-
Notifications
You must be signed in to change notification settings - Fork 53
Refactor bytes codec in STM library for forward/backward compatibilityΒ #3065
Description
Why
The STM library currently uses a hand-rolled binary format for all to_bytes/from_bytes implementations. Each type manually manages byte offsets with hardcoded slice ranges (e.g., bytes.get(32..64)), big-endian u64 length prefixes, and single-byte variant tags. This approach is fragile, error-prone, and most importantly does not support forward or backward compatibility: any structural change to a type (adding a field, reordering, introducing a new variant) silently breaks decoding of previously encoded data.
As the STM library evolves (SNARK integration, new aggregate signature variants, new proof systems), the bytes codec must be able to evolve without breaking existing encoded data in the certificate chain or stored artifacts.
What
Refactor all to_bytes/from_bytes implementations in the STM library to use CBOR encoding with the minicbor or ciborium crate, ensuring forward and backward compatibility, and add comprehensive golden test coverage for all serialized types.
How
- Assess all existing
to_bytes/from_bytesimplementations and categorize them:- Fixed-size cryptographic primitives:
-
BlsSigningKey -
BlsVerificationKey -
BlsSignature -
BlsProofOfPossession -
BlsVerificationKeyProofOfPossession -
SchnorrSigningKey -
SchnorrVerificationKey -
UniqueSchnorrSignature -
BaseFieldElement -
ProjectivePoint
-
- Variable-length compound types:
-
Parameters -
SingleSignature -
SingleSignatureWithRegisteredParty -
ClosedRegistrationEntry -
Initializer -
AggregateSignature -
AggregateVerificationKeyForConcatenation -
AggregateVerificationKeyForSnark -
ConcatenationProof -
MerklePath -
MerkleBatchPath -
MerkleTree -
MerkleTreeCommitment -
MerkleTreeBatchCommitment -
MerkleTreeConcatenationLeaf -
MerkleTreeSnarkLeaf
-
- Fixed-size cryptographic primitives:
- Assess which types can safely have their encoding changed and which ones require a transition strategy (e.g., types already persisted in the certificate chain)
- Design the migration strategy:
- Determine the CBOR encoding approach with
minicbor(map vs array encoding, index allocation,Optionwrapping for future extensibility) - Define the compatibility constraints for each type (e.g., fixed-size primitives may remain unchanged while compound types migrate to CBOR)
- Define when the new CBOR bytes codec can be activated without breaking the existing certificate chain
- Determine the CBOR encoding approach with
- Add
minicboras a dependency of the STM library (withderivefeature forEncode/Decodederive macros) - Refactor the compound types to use CBOR encoding with
minicbororciborium - Add missing golden tests for
to_bytes/from_bytesroundtrips on types that lack them:-
Parameters -
AggregateSignature -
Initializer -
MerklePath -
MerkleBatchPath
-
- Verify that existing golden tests still pass or are updated with the transition strategy
- Update the integration test in
tests/bytes_conversion.rsto cover the new CBOR encoding - Create a new ADR to specify rules to be applied when updating the types to keep forward/backward compatibility