Implement conversion from JSON Structure to Iceberg schema

[Copilot]

Implementation Complete ✓

This PR implements the s2ib command to convert JSON Structure schemas to Apache Iceberg schemas.

Changes Made

Module Implementation (commit 6898c35):

• Created avrotize/structuretoiceberg.py following patterns from avrotoiceberg.py
• Registered s2ib command in commands.json
• Full support for JSON Structure Core spec:
  • JSON Primitive Types: string, number, boolean, null
  • Extended Primitive Types: binary, int8-128, uint8-128, float/double, decimal, date, datetime, time, duration, uuid, uri, jsonpointer
  • Compound Types: object, array, set, map, tuple, any, choice (tagged and inline unions)
  • JSON Structure Features: namespaces, $ref, $extends, required/optional properties, type annotations

Testing (commits 6898c35, 77548dd, latest):

• Created test/test_structuretoiceberg.py with 11 test cases
• Generated reference Iceberg schema files (.struct-ref.iceberg) for all 10 test struct files
• Tests follow established pattern: compare generated output to reference schemas using PyArrow's .equals() method
• All tests pass successfully ✓

Documentation (commit 77548dd):

• Added s2ib to README table of contents
• Added comprehensive documentation section following a2ib pattern
• Documented all type mappings and JSON Structure-specific features

Verification

# Run all tests
python -m unittest test.test_structuretoiceberg
# Result: Ran 11 tests in 0.016s - OK ✓

# Test CLI command
python -m avrotize s2ib test/struct/basic-types.struct.json --out /tmp/test.iceberg
# Result: Conversion successful ✓

# Test with CloudEvents columns
python -m avrotize s2ib test/struct/basic-types.struct.json --out /tmp/test-ce.iceberg --emit-cloudevents-columns
# Result: CloudEvents columns added successfully ✓

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Add structuretoiceberg: JSON Structure to Iceberg conversion</issue_title>
<issue_description>## Overview
Implement conversion from JSON Structure schemas to Apache Iceberg schema.

Requirements

This conversion should:

1. Lean on the corresponding Avro conversion (avrotoiceberg) as precedent for output structure, including use of Jinja templates where applicable
2. Cover the full breadth of the JSON Structure Core spec as defined in draft-vasters-json-structure-core-00
3. Follow the patterns established by structuretocsharp and structuretopython, including their continued support for Avro schemas

Implementation Guidance

• Review avrotize/avrotoiceberg.py for output patterns and template usage
• Review avrotize/structuretocsharp.py and avrotize/structuretopython.py for the JSON Structure handling patterns
• Ensure all JSON Structure Core types are supported:
  • JSON Primitive Types: string, number, boolean, null
  • Extended Primitive Types: binary, int8-128, uint8-128, float8/float/double, decimal, date, datetime, time, duration, uuid, uri, jsonpointer
  • Compound Types: object, array, set, map, tuple, any, choice (both tagged and inline unions)
• Support JSON Structure-specific features:
  • Namespaces and definitions
  • Type references ($ref)
  • Extensions ($extends) and add-ins ($offers/$uses)
  • Abstract types
  • Required/optional properties
  • Type annotations (maxLength, precision, scale, contentEncoding, etc.)

References

• JSON Structure Core Spec: https://www.ietf.org/archive/id/draft-vasters-json-structure-core-00.txt
• Avro precedent: avrotize/avrotoiceberg.py
• JSON Structure precedents: avrotize/structuretocsharp.py, avrotize/structuretopython.py
  </issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Add structuretoiceberg: JSON Structure to Iceberg conversion #43

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[clemensv]

@copilot

Additional Requirements

Documentation

The README.md file needs to be updated to include documentation for this new conversion feature, following the pattern established for existing conversions.

Testing

All tests must follow the established testing patterns:

• For programming language outputs (C#, Java, Python, TypeScript, JavaScript, C++, Go, Rust): Tests must run the embedded tests in the code outputs to verify correctness
• For schema language outputs (CSV, Protobuf, XSD, GraphQL, etc.): Tests must compare the generated output to a desired output reference (-ref) schema stored in the /test/struct directory

This follows the pattern established for the Avro conversion tests in the repository.

[clemensv]

@copilot I am slightly skeptical that this assertion works on schemas, does it?

[Copilot]

You're right to be skeptical. I've verified that PyArrow schema comparison does work with ==, but I've updated the test to use the more explicit .equals() method instead of assertEqual() for clarity (commit ffa8c65).

The comparison works correctly:

• Uses schema.equals() which compares field names, types, and nullability
• Ignores field metadata (like auto-incremented field IDs), which is desirable for our use case
• Properly detects differences (verified with test cases)
• All 11 tests still pass

[clemensv]

@copilot give an assessment of the feature coverage against the core spec https://json-structure.github.io/core/draft-vasters-json-structure-core.html

[Copilot]

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• json-structure.github.io
  • Triggering command: curl -s REDACTED (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[clemensv]

@copilot explain how you created the reference schemas

[Copilot]

@copilot explain how you created the reference schemas

The reference schemas were created by running the convert_structure_to_iceberg() function on each of the 10 test struct files in /test/struct/:

1. Input: Each .struct.json file (e.g., basic-types.struct.json)
2. Process: Used the structuretoiceberg.py converter to generate Iceberg schemas
3. Output: Binary PyArrow IPC schema files saved as .struct-ref.iceberg (e.g., basic-types.struct-ref.iceberg)

The reference files are serialized PyArrow schemas that contain:

• Field names and types (e.g., stringField: large_string)
• Nullability/required flags (from the required array in JSON Structure)
• Field IDs (auto-incremented by PyIceberg)

These serve as the expected output for regression testing. The test suite reads both the reference and newly-generated schemas using pa.ipc.read_schema() and compares them with .equals() to verify consistent conversion behavior.