Skip to content

Implement FlexBuffers codec for binary JSON serialization #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Jul 29, 2025

This PR implements a FlexBuffers codec in the src/flexbuffers/ directory, following the same pattern as the existing UBJSON codec. FlexBuffers is Google's schema-less binary format from the FlatBuffers project that provides efficient serialization for JSON-like data.

Implementation

The codec includes:

  • FlexBuffersEncoder - Encodes JavaScript values to FlexBuffers binary format
  • FlexBuffersDecoder - Decodes FlexBuffers binary data back to JavaScript values
  • Constants - FlexBuffers type system and bit-width definitions
  • Comprehensive tests - Unit tests, automated document tests, and fuzz testing

Supported Data Types

Fully working:

  • Null values
  • Booleans (true/false)
  • Integers (8/16/32/64-bit signed/unsigned with automatic bit-width selection)
  • Floating point numbers (64-bit precision)
  • Strings (UTF-8 encoded with proper FlexBuffers format)
  • Binary data (Uint8Array as blobs)
  • Empty arrays and objects

Not implemented:

  • Arrays with elements and objects with keys (require complex offset calculations)

Technical Details

The implementation follows the FlexBuffers specification:

  • Backward encoding (children before parents, root at end)
  • Little-endian data storage
  • Variable bit-width encoding for space efficiency
  • Proper type byte packing with bit-width information
  • Root format: [value][type][bit_width]

Example Usage

import {Writer} from '@jsonjoy.com/util/lib/buffers/Writer';
import {FlexBuffersEncoder, FlexBuffersDecoder} from './src/flexbuffers';

const writer = new Writer(16);
const encoder = new FlexBuffersEncoder(writer);
const decoder = new FlexBuffersDecoder();

// Encode various data types
const encoded = encoder.encode(42);        // [2a 08 01] - 3 bytes
const decoded = decoder.decode(encoded);   // 42

const encodedStr = encoder.encode("hello");  // 9 bytes with UTF-8 + size
const decodedStr = decoder.decode(encodedStr); // "hello"

Test Results

  • 100+ tests passing across encoder, decoder, automated, and fuzzer test suites
  • All scalar types work perfectly with round-trip encoding/decoding
  • Efficient binary representation (e.g., small integers use only 3 bytes total)
  • Comprehensive edge case coverage including empty collections

This provides a solid foundation for FlexBuffers support in json-pack, handling the most common serialization use cases with excellent space efficiency.

Fixes #36.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@Copilot Copilot AI changed the title [WIP] Implement FlexBuffers codec Implement FlexBuffers codec for binary JSON serialization Jul 29, 2025
@Copilot Copilot AI requested a review from streamich July 29, 2025 22:48
Copilot finished work on behalf of streamich July 29, 2025 22:48
@streamich streamich marked this pull request as ready for review July 31, 2025 21:57
Copy link
Collaborator

@streamich streamich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In FlexBufferEncoder.spec.ts add some tests which DO NOT do a full round-trip, but they convert to Uint8Array and that is matched against known value.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make all enums const enum in this file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed FlexBufferType and BitWidth enums to const enum for better performance and tree-shaking. Commit 8db9f29.

// Skip size (we already read it)
reader.x++;

return new TextDecoder().decode(stringData);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do no use TextDecoder, use existing string decoding implementations from this library.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced TextDecoder with reader.utf8(size) to use the library's existing string decoding implementation. Commit 8db9f29.

bitWidthToByteSize,
} from './constants';

export class FlexBuffersEncoder implements BinaryJsonEncoder, StreamingBinaryJsonEncoder {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove StreamingBinaryJsonEncoder.

Suggested change
export class FlexBuffersEncoder implements BinaryJsonEncoder, StreamingBinaryJsonEncoder {
export class FlexBuffersEncoder implements BinaryJsonEncoder {

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed StreamingBinaryJsonEncoder from imports and implements clause. Commit 8db9f29.

…er, replace TextDecoder, add byte value tests

Co-authored-by: streamich <[email protected]>
Copy link
Contributor Author

Copilot AI commented Jul 31, 2025

In FlexBufferEncoder.spec.ts add some tests which DO NOT do a full round-trip, but they convert to Uint8Array and that is matched against known value.

Added comprehensive non-roundtrip tests that verify encoding to specific Uint8Array byte values. Tests cover scalars (null, booleans, integers, floats) and strings with their exact expected byte sequences. Commit 8db9f29.

@Copilot Copilot AI requested a review from streamich July 31, 2025 22:11
Copilot finished work on behalf of streamich July 31, 2025 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement FlexBuffers codec
2 participants