Skip to content

Refactor bytes codec in STM library for forward/backward compatibilityΒ #3065

@jpraynaud

Description

@jpraynaud

Why

The STM library currently uses a hand-rolled binary format for all to_bytes/from_bytes implementations. Each type manually manages byte offsets with hardcoded slice ranges (e.g., bytes.get(32..64)), big-endian u64 length prefixes, and single-byte variant tags. This approach is fragile, error-prone, and most importantly does not support forward or backward compatibility: any structural change to a type (adding a field, reordering, introducing a new variant) silently breaks decoding of previously encoded data.
As the STM library evolves (SNARK integration, new aggregate signature variants, new proof systems), the bytes codec must be able to evolve without breaking existing encoded data in the certificate chain or stored artifacts.

What

Refactor all to_bytes/from_bytes implementations in the STM library to use CBOR encoding with the minicbor or ciborium crate, ensuring forward and backward compatibility, and add comprehensive golden test coverage for all serialized types.

How

  • Assess all existing to_bytes/from_bytes implementations and categorize them:
    • Fixed-size cryptographic primitives:
      • BlsSigningKey
      • BlsVerificationKey
      • BlsSignature
      • BlsProofOfPossession
      • BlsVerificationKeyProofOfPossession
      • SchnorrSigningKey
      • SchnorrVerificationKey
      • UniqueSchnorrSignature
      • BaseFieldElement
      • ProjectivePoint
    • Variable-length compound types:
      • Parameters
      • SingleSignature
      • SingleSignatureWithRegisteredParty
      • ClosedRegistrationEntry
      • Initializer
      • AggregateSignature
      • AggregateVerificationKeyForConcatenation
      • AggregateVerificationKeyForSnark
      • ConcatenationProof
      • MerklePath
      • MerkleBatchPath
      • MerkleTree
      • MerkleTreeCommitment
      • MerkleTreeBatchCommitment
      • MerkleTreeConcatenationLeaf
      • MerkleTreeSnarkLeaf
  • Assess which types can safely have their encoding changed and which ones require a transition strategy (e.g., types already persisted in the certificate chain)
  • Design the migration strategy:
    • Determine the CBOR encoding approach with minicbor (map vs array encoding, index allocation, Option wrapping for future extensibility)
    • Define the compatibility constraints for each type (e.g., fixed-size primitives may remain unchanged while compound types migrate to CBOR)
    • Define when the new CBOR bytes codec can be activated without breaking the existing certificate chain
  • Add minicbor as a dependency of the STM library (with derive feature for Encode/Decode derive macros)
  • Refactor the compound types to use CBOR encoding with minicbor or ciborium
  • Add missing golden tests for to_bytes/from_bytes roundtrips on types that lack them:
    • Parameters
    • AggregateSignature
    • Initializer
    • MerklePath
    • MerkleBatchPath
  • Verify that existing golden tests still pass or are updated with the transition strategy
  • Update the integration test in tests/bytes_conversion.rs to cover the new CBOR encoding
  • Create a new ADR to specify rules to be applied when updating the types to keep forward/backward compatibility

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions