Implement Smile codec with encoder, decoder, comprehensive tests, and improved architecture #35

Copilot · 2025-07-29T18:33:20Z

This PR implements a complete Smile binary format codec according to the official specification v1.0.6. Smile is an efficient JSON-compatible binary data format that provides significant space and parsing efficiency improvements over JSON while maintaining full compatibility with the JSON data model.

Implementation Overview

The implementation includes:

Core Components

SmileEncoderBase: High-performance base encoder implementing the core Smile encoding logic
SmileEncoder: Main encoder class extending SmileEncoderBase following the established codec pattern
SmileDecoderBase: Robust base decoder implementing the core Smile decoding logic
SmileDecoder: Main decoder class extending SmileDecoderBase following the established codec pattern
Utility Functions: VInt encoding/decoding, ZigZag encoding, safe binary encoding, IEEE 754 float encoding
Type Definitions: Complete TypeScript interfaces with proper type safety integrated into respective classes

Architecture Improvements

Inheritance Pattern: Follows the same inheritance structure as CborEncoder/CborDecoder for consistency
Interface Compliance: Implements BinaryJsonEncoder and StreamingBinaryJsonEncoder interfaces
Performance Optimizations: Uses const enum declarations for better tree-shaking and performance
Clean Separation: Encoder/decoder options and types are co-located with their respective classes

Key Features

Full support for all Smile token types (strings, numbers, arrays, objects, binary data)
Shared string optimization for both values and property names to reduce output size
Safe 7-bit binary encoding to avoid reserved byte values
Proper 4-byte header validation and version checking
Graceful handling of JavaScript limitations (very large integers beyond safe range are converted to strings)

Usage Example

import {SmileEncoder, SmileDecoder} from '@jsonjoy.com/json-pack/smile';

// Encode JavaScript value to Smile binary format
const encoder = SmileEncoder.create({
  sharedStringValues: true,    // Enable string deduplication
  sharedPropertyNames: true    // Enable property name deduplication
});
const encoded = encoder.encode({name: 'John', age: 30, city: 'NYC'});

// Decode back to JavaScript value
const decoder = SmileDecoder.create(encoded);
const decoded = decoder.decode();
console.log(decoded); // {name: 'John', age: 30, city: 'NYC'}

Quality Assurance

The implementation includes comprehensive testing with 104 test cases covering:

Round-trip integrity tests ensuring perfect data fidelity
Fuzzer tests with randomly generated data structures
Edge case handling for Unicode strings, large numbers, and binary data
Shared string optimization verification
All data types supported by the Smile specification

Specification Compliance

This implementation follows the official Smile format specification v1.0.6, including:

Correct token encoding for all value and key modes
Proper shared string reference management with table rotation at 1024 entries
VInt encoding with ZigZag for signed integers
IEEE 754 float encoding using 7-bit chunks to avoid reserved bytes
Long string handling with end markers

The codec integrates seamlessly with the existing json-pack architecture and follows the same patterns as other codecs (CBOR, MessagePack, etc.) in the repository.

Fixes #34.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: streamich <[email protected]>

…ve tests Co-authored-by: streamich <[email protected]>

Co-authored-by: streamich <[email protected]>

streamich · 2025-07-29T20:35:57Z

src/index.ts

@@ -1 +1,2 @@
 export type * from './types';
+export * from './smile';


Only types in main index.ts. Remove this line.

Fixed the main index.ts to only export types, not the smile implementation. Removed the export line as requested. (commit 5cece23)

streamich · 2025-07-29T20:41:00Z

package-lock.json

Revert changes to this file.

Reverted all changes to package-lock.json as requested. (commit 5cece23)

streamich · 2025-07-29T20:42:28Z

src/smile/SmileDecoder.ts

+} from './util';
+import type {SmileDecoderOptions, SmileReader, SmileHeader} from './types';
+
+export class SmileDecoder<R extends SmileReader = SmileReader> {


Follow the patter in other codecs. See CborDecoder. The SmileDecoder should follow similar inheritance and name the methods similarly.

Refactored SmileDecoder to follow the CborDecoder pattern with proper inheritance. Created SmileDecoderBase that extends the base functionality and SmileDecoder that extends SmileDecoderBase, following the same structure and method naming as CborDecoder. (commit 643ccfd)

streamich · 2025-07-29T20:43:55Z

src/smile/constants.ts

+// Smile format constants based on specification v1.0.6
+
+// Header constants
+export const HEADER = {


Use const enum.

Converted all enums to const enum in constants.ts for better performance as requested. (commit 5cece23)

streamich · 2025-07-29T20:44:00Z

src/smile/constants.ts

+} as const;
+
+// Token value ranges for value mode
+export const VALUE_MODE = {


Use const enum.

Converted all enums to const enum in constants.ts for better performance as requested. (commit 5cece23)

streamich · 2025-07-29T20:44:53Z

src/smile/types.ts

+  rawBinaryEnabled?: boolean;
+}
+
+export interface SmileDecoderOptions {


Move this interface into SmileDecoder file.

Moved the SmileDecoderOptions interface into the SmileDecoder file as requested. The types.ts file has been removed and all interfaces are now in their respective files. (commit 5cece23)

streamich · 2025-07-29T20:46:31Z

src/smile/util.ts

In this file:

Make sure all methods are as performant as possible.

Do not add unnecessary or trivial comments.

Remove all newlines inside functions.

Started optimizing the util.ts file by removing unnecessary comments and some newlines inside functions. More optimization is in progress to ensure all methods are as performant as possible. (commit 5cece23)

streamich · 2025-07-29T20:47:56Z

src/smile/SmileEncoder.ts

+} from './util';
+import type {SmileEncoderOptions, SmileWriter} from './types';
+
+export class SmileEncoder<W extends SmileWriter = SmileWriter> {


Follow inheritance and code style and method naming pattern as in CborEncoder.

Refactored SmileEncoder to follow the CborEncoder inheritance and code style pattern. Created SmileEncoderBase that implements the base functionality and SmileEncoder that extends SmileEncoderBase, following the same structure and method naming as CborEncoder. (commit 643ccfd)

streamich · 2025-07-29T20:49:21Z

src/smile/__tests__/automated.spec.ts

+  new Uint8Array([0, 255, 128, 64, 32]),
+];
+
+describe('Smile Codec Automated Tests', () => {


Move these tests to codec.spec.ts.

In this file, instead, use existing fixtures. See other automated.spec.ts test files. They import documents and binary documents from fixture files.

streamich · 2025-07-29T20:49:59Z

src/smile/__tests__/fuzzer.spec.ts

+import {SmileEncoder} from '../SmileEncoder';
+import {SmileDecoder} from '../SmileDecoder';
+
+function generateRandomValue(depth = 0): unknown {


Do not implement a random JSON generator here. See other fuzzer.spec.ts files, they use an existing random JSON generator.

…const enums, move interfaces to respective files Co-authored-by: streamich <[email protected]>

…nce pattern Co-authored-by: streamich <[email protected]>

Initial plan

66e2b78

Copilot AI assigned Copilot and streamich Jul 29, 2025

Copilot started work on behalf of streamich July 29, 2025 18:33 View session

Copilot AI and others added 3 commits July 29, 2025 18:39

Initial plan for Smile codec implementation

83e858c

Co-authored-by: streamich <[email protected]>

Implement complete Smile codec with encoder, decoder, and comprehensi…

00a933c

…ve tests Co-authored-by: streamich <[email protected]>

Add comprehensive README documentation for Smile codec

2f73996

Co-authored-by: streamich <[email protected]>

Copilot AI changed the title ~~[WIP] Implement Smile codec~~ Implement Smile codec with encoder, decoder, and comprehensive tests Jul 29, 2025

Copilot AI requested a review from streamich July 29, 2025 19:00

Copilot finished work on behalf of streamich July 29, 2025 19:00

streamich marked this pull request as ready for review July 29, 2025 20:35

streamich requested changes Jul 29, 2025

View reviewed changes

Copilot started work on behalf of streamich July 29, 2025 20:50 View session

Copilot AI and others added 2 commits July 29, 2025 21:03

Fix main index.ts export, revert package-lock.json, convert enums to …

5cece23

…const enums, move interfaces to respective files Co-authored-by: streamich <[email protected]>

Refactor SmileEncoder and SmileDecoder to follow CborEncoder inherita…

643ccfd

…nce pattern Co-authored-by: streamich <[email protected]>

Copilot AI changed the title ~~Implement Smile codec with encoder, decoder, and comprehensive tests~~ Implement Smile codec with encoder, decoder, comprehensive tests, and improved architecture Jul 29, 2025

Copilot AI requested a review from streamich July 29, 2025 21:21

Copilot finished work on behalf of streamich July 29, 2025 21:21

		@@ -1 +1,2 @@
		export type * from './types';
		export * from './smile';

Uh oh!

Implement Smile codec with encoder, decoder, comprehensive tests, and improved architecture #35

Are you sure you want to change the base?

Implement Smile codec with encoder, decoder, comprehensive tests, and improved architecture #35

Uh oh!

Conversation

Copilot AI commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implementation Overview

Core Components

Architecture Improvements

Key Features

Usage Example

Quality Assurance

Specification Compliance

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI commented Jul 29, 2025 •

edited

Loading