Each tree-sitter grammar ships a node-types.json alongside its parser,
generated by tree-sitter generate. It is the authoritative description
of the grammar's node types: which fields they have, which child types
they accept, which supertypes they participate in.
Diffract carries that data into the binary at build time. Nothing reads it right now — it is groundwork, kept because it costs only build plumbing and would otherwise have to be reintroduced by hand.
grammars/build-grammars.sh copies each package's node-types.json into
grammars/metadata/<language>.node-types.json after npm install. The
files are committed so they version with the grammar binaries. Five
languages: typescript, tsx, kotlin, php (php_only), scala.
lib/dune then inlines each file's bytes as a raw OCaml string constant
in a generated embedded_metadata.ml (generated, not committed), which
exposes one string per language plus
by_language : string -> string option. Only data is embedded; the JSON
in grammars/metadata/ stays the single source of truth, and the single
binary needs nothing from disk at runtime.
Regenerating is automatic: run build-grammars.sh after a grammar bump.
unwrap_root. A list_shape_wrappers projection over
node-types.json (named nodes with no field-named children) supplied
candidate wrapper types, and Tree.unwrap_root peeled them at the root
of a parsed pattern fragment so a single-node pattern matched the
corresponding source node rather than the whole file. The universal
tokenizer made this unnecessary — a pattern becomes a leaf stream rather
than a parsed fragment, so there is no root to peel — and both were
removed.
The DEL definition. The pipeline also parsed each grammar's
grammar.json to derive bracket pairs, string-literal shapes and comment
markers per language, behind Grammar_metadata.del_definition, with a
per-language extension table for languages whose strings or comments live
in external scanners and are therefore opaque in the grammar JSON. It
served the DEL-based hybrid matcher, which needed hand-declared lexical
tables to lex patterns without a parser; that design was abandoned for
the universal tokenizer, which reuses the source language's own parser in
error-tolerant mode (docs/universal-tokenizer.md §1–2).
The accessor then survived unused, its only callers its own tests — a
green test over data nothing consumes, which is worse than no test: it
reports on a derivation whose output cannot affect the product, so it
fails (or passes) for reasons no reader can act on. Grammar_metadata,
its extension table, the grammar.json embedding, the committed
*.grammar.json files and the yojson dependency were all removed.
If lexical metadata is wanted again, note the limit that shaped it:
derivation from grammar.json only works for markers appearing as
literals in the grammar, and external scanners hide exactly the
interesting cases (Kotlin's multiline_comment, PHP heredocs, Scala
interpolated strings). A revived version needs a declared fallback per
language and, more importantly, a real consumer.
- Adding a language stays mechanical: register the grammar binding, drop
in
node-types.json, the pipeline picks it up. - Metadata versions with the grammar; no manual curation drift.
- The committed JSON is a few hundred KB and must be regenerated on a
grammar bump (
build-grammars.shdoes it). - Upstream schema drift would need whatever future parser consumes it to adapt. The schema is stable in practice but not formally versioned.