feat: add nested table syntax for uniform nested objects#293
feat: add nested table syntax for uniform nested objects#293Turtle-dev3 wants to merge 7 commits intotoon-format:mainfrom
Conversation
Add opt-in nestedTables option that flattens uniform nested objects
into tabular format: {id,customer{name,country},total}
Rows are inlined instead of falling back to list items, saving tokens
when data has consistent nested structure. Non-uniform objects fall
back to the existing list syntax automatically.
Parse nested field syntax in headers: {id,customer{name,country},total}
and reconstruct nested objects from flattened row values.
Also strips type hint suffixes (e.g., field:int) for forward
compatibility with future encoders that may add them.
9 tests covering encode, decode, round-trip, non-uniform fallback, multiple nested fields, and forward-compatible type hint stripping.
Compare TOON vs TOON+nested vs JSON across 7 datasets including a new uniform-nested dataset (500 shipments with sender/receiver/dimensions). Key result: 53.8% token savings vs baseline TOON and 41.9% fewer tokens than JSON on uniform nested data.
|
Hi @Turtle-dev3, They are looking into another aproach for handling this. Please refer #290 |
Hi @asriva404 , thanks for the pointer! I have looked into #290 and I believe the two proposals solve different problems. My PR extends the existing array table syntax ([N]{fields}:) to support nested sub-objects within rows, so uniform nested fields get flattened inline instead of falling back to list items. #290 introduces an entirely new container type for keyed object collections ({fields}*:), where the parent is an object with string keys rather than an array. So this PR targets nested objects inside arrays, while #290 targets table-collapse for dictionaries/maps. They could actually complement each other quite nicely if both were to land. Happy to hear if the maintainers see it differently, though! (Or I have misunderstood something ^^) |
2 participants
Summary
Adds opt-in nested table encoding that flattens uniform nested objects into TOON's tabular format, significantly reducing token count for data with consistent nested structure.
Before (falls back to list items):
After (
nestedTables: true):How it works
field{sub1,sub2}header syntax with values inlined in rows.field{sub1,sub2}in headers and reconstructs nested objects from flattened row values. Also strips type hint suffixes (e.g.,field:int) for forward compatibility.Benchmark results
Tested across 7 datasets including a new
uniform-nesteddataset (500 shipment records with sender/receiver/dimensions objects):For data without nested objects, output is identical to standard TOON — zero overhead.
Breaking changes
None. The feature is fully opt-in via
encode(data, { nestedTables: true }). Default encoder output is unchanged. The decoder handles nested table syntax automatically with no option needed.Test plan