Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
471 lines (381 loc) · 18 KB

ARCHITECTURE.md

File metadata and controls

471 lines (381 loc) · 18 KB
Raw

Sesi Runtime Architecture

Overview

Sesi is a concise and highly legible programming language. It uses a high-performance bytecode compilation model built in TypeScript. The architecture features a compiler that lowers the AST to a flat bytecode chunk, executed by a register-based virtual machine, with the original tree-walking interpreter retained as a fallback path. The architecture is optimized for simplicity and eliminating the need for async/await syntax or heavy module imports. By acting as a minimal orchestration layer, Sesi allows developers to write clean, straightforward logic that natively handles background processes and API or Reasoning calls without SDK overhead.

Component Stack

┌─────────────────────────────────────────────┐
│         Sesi Program (.sesi file)           │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│      Lexer (src/lexer.ts)                   │
│  Converts source text → Tokens              │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│      Parser (src/parser.ts)                 │
│  Converts Tokens → Abstract Syntax Tree     │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│      Compiler (src/compiler.ts)             │
│  Lowers AST → Bytecode Chunk                │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│      VM (src/vm.ts)                         │
│  Bytecode virtual machine                   │
│  - Executes OpCode instructions             │
│  - Manages call frames & value stack        │
│  - Supports closures, try/catch, imports    │
└──────┬───────────────────────┬──────────────┘
       │                       │
       │                       │
┌──────▼──────────────┐  ┌────▼────────────────┐
│ Builtins            │  │ AI Runtime          │
│ (src/builtins.ts)   │  │ (src/ai-runtime.ts) │
│                     │  │                     │
│ - print()           │  │ - Gemini API calls  │
│ - len()             │  │ - Memory mgmt       │
│ - read_file()       │  │ - Structured output │
│ - write_file()      │  │ - Tool calling      │
│ - spawn()           │  │                     │
│ - exec()            │  │                     │
│ - sesi()            │  │                     │
│ - time()            │  │                     │
│ - random()          │  │                     │
│ - etc.              │  │                     │
└─────────────────────┘  └────┬────────────────┘
                              │
                    ┌─────────▼──────────┐
                    │  Gemini API        │
                    │  (@google/genai)   │
                    └────────────────────┘

Execution Flow

1. Lexical Analysis (Lexer)

  • Input: Source code string
  • Process: Character-by-character tokenization
    • Identifier and keyword recognition
    • Number and string literal parsing
    • Operator and delimiter recognition
    • Comment stripping
  • Output: Token stream

Example:

Source: let x = 10
Tokens: [LET, IDENTIFIER("x"), EQUAL, NUMBER(10), EOF]

2. Parsing (Parser)

  • Input: Token stream
  • Process: Recursive descent parsing
    • Statement parsing (declarations, control flow)
    • Expression parsing (operators, function calls, Reasoning constructs)
    • AST construction
    • Error recovery
  • Output: Abstract Syntax Tree (AST)

Example AST Node:

{
  type: 'LetStatement',
  name: 'x',
  value: {
    type: 'Literal',
    value: 10,
    rawType: 'number'
  }
}

3. Compilation & Execution (Compiler → VM)

  • Input: AST
  • Compiler process: Single-pass AST lowering
    • Emits flat OpCode instructions into a Chunk
    • Resolves local variable slots at compile time
    • Compiles closures, loops, conditionals, and try/catch
  • VM process: Stack-based execution
    • Dispatches OpCode instructions in a tight loop
    • Maintains a value stack and call frame stack
    • Calls into builtins and AI runtime as needed
  • Output: Program side effects (print, Reasoning calls, etc.)

Note: The original tree-walking interpreter (src/interpreter.ts) is retained as a fallback execution path for any constructs not yet supported by the compiler.

Scope and Environment Management

Sesi uses lexical scoping with an environment chain:

┌─────────────────────────┐
│  Global Environment     │
│  - Built-in functions   │
│  - Global variables     │
└────────┬────────────────┘
         │
    ┌────▼────────────────┐
    │ Function Environment│
    │ - Parameters        │
    │ - Local variables   │
    └────┬────────────────┘
         │
    ┌────▼────────────────┐
    │  Block Environment  │
    │  - Block variables  │
    └─────────────────────┘

Environment Lookup

  1. Check current environment
  2. Walk up parent chain until found
  3. If not found anywhere, error

Variable Assignment

  1. Try to update in current scope
  2. If not in current scope, try parent scopes
  3. Update in the scope where variable was found

Type System

Sesi has a dynamic type system with runtime type checking:

RuntimeValue =
  | number
  | string
  | boolean
  | null
  | Array<RuntimeValue>
  | Object<string, RuntimeValue>
  | RuntimeFunction
  | RuntimeModule

Type Coercion Rules

String Concatenation (operator +):

"Hello" + 5        → "Hello5"
"Age:" 30          → "Age: 30"
any + string       → toString(any) + string
string + any       → string + toString(any)

Numeric Operations:

10 + 20            → 30
"10" + 20          → "1020" (not numeric!)
10 20              → "10 20" (not numeric!)

Truthiness:

null   → false
false  → false
0      → false
""     → false
[]     → true
{}     → true

Reasoning Integration

Reasoning Runtime Lifecycle

  1. Initialization:
    • Load @google/genai SDK
    • Validate GEMINI_API_KEY environment variable
  2. During Execution:
    • Parse model calls from AST
    • Construct prompt from string concatenation
    • Make async API call to Gemini
  • Wait for response (blocking in v1.x)
  • Validate finish reason and non-empty text
  • Return text response to program or throw
  1. Memory Management:
    • Simple string buffers per memory ID
    • Append/update operations
    • Passed to next model call

Model Call Flow

ModelCallExpression (AST)
    │
    ├─ Evaluate prompt expression
    ├─ Extract configuration (`thinkingLevel`, `max_tokens`, `cache`, etc.)
    ├─ Call AIRuntime.callModel()
    │   │
    │   ├─ Create Gemini interaction request
    │   ├─ Send to Gemini API
    │   ├─ Wait for response
    │   ├─ Validate finish reason == STOP
    │   └─ Extract text from response
    │
    └─ Return text to program or throw

Error Handling (V1)

Sesi now has basic exception-style error handling in v1:

  1. Parse Errors: Parser logs the error and synchronizes to continue parsing later statements.
  2. Runtime Errors: Interpreter errors throw and can be caught with try/catch.
  3. Built-in I/O Errors: read_file(), write_file(), and list_dir() throw on filesystem failure.
  4. Reasoning Errors: model() throws when the SDK fails, when no text is returned. (Note: MAX_TOKENS finish reasons are now handled natively via an automatic async polling loop that prompts the model to continue where it left off, and therefore does not throw errors).
  5. Structured Output: structured_output() attempts recovery, but currently logs parsing failures and returns {} if coercion still fails.

Memory Model

Stack

  • Local variables
  • Function parameters
  • Scope frames

Heap

  • Arrays (dynamic)
  • Objects (key-value maps)
  • Strings (immutable)

Reasoning Context

  • Conversation memory per memory ID
  • String buffers, not structured storage

Performance Characteristics

Operation Time Notes
Variable lookup O(n) n = scope depth
Array access O(1) Direct indexing
Object access O(1) Map lookup
Function call O(1) + body execution
Model call ~2-5s Depends on Gemini latency
Array iteration O(n) Foreach loop

Limitations (V1)

  • VM Single-threaded: Each individual Sesi process is single-threaded.
  • Process-level Concurrency: Sesi uses a multi-process model via spawn() for concurrent task execution.
  • Bytecode coverage: The compiler covers expressions, loops, functions, closures, try/catch, and imports; edge-case constructs fall back to the tree-walking interpreter.
  • No JIT: The VM executes bytecode directly; JIT compilation is a V4+ goal.
  • Simple type system: Runtime checking only.
  • No macro system: No compile-time code generation
  • No introspection: Can't inspect function bodies
  • Limited error info: Basic error messages

Future Architecture (V2+)

V2 Delivered ✅

  • Bytecode compiler (src/compiler.ts) that lowers the AST to a flat Chunk
  • Stack-based virtual machine (src/vm.ts) with OpCode dispatch
  • Closures, loops, try/catch, and imports handled natively by the VM
  • Legacy interpreter flag (-t) to enable the tree-walking execution path
  • Streaming response support
  • Advanced error handling with stack traces

V3+ Vision

  • Agent framework with state machines
  • Knowledge base integration
  • Advanced memory with embedding search
  • Multi-agent orchestration
  • Custom type definitions
  • Macro system

Code Organization

SKILLS.md                 # AI-agent workspace context and repo guardrails
eslint.config.mjs         # ESLint configuration
dist/                     # Compiled TypeScript output
example.js                # Helper script to run basic examples
example-ai.js             # Helper script to run Reasoning examples
examples.sesi             # Central execution suite for examples
package.json              # Dependencies & scripts
tsconfig.json             # TypeScript configuration
QUICKSTART.md             # Quick start guide
IMPLEMENTATION_SUMMARY.md # Progress and tracking
README.md                 # Project overview

src/
├── types.ts              # Type definitions and AST
├── lexer.ts              # Tokenization
├── parser.ts             # AST generation
├── compiler.ts           # AST → Bytecode compiler
├── chunk.ts              # Bytecode Chunk & OpCode definitions
├── vm.ts                 # Bytecode virtual machine
├── interpreter.ts        # Tree-walking interpreter (fallback)
├── ai-runtime.ts         # Gemini integration
├── builtins.ts           # Built-in functions
└── index.ts              # Main entry point

bin/
└── sesi.js               # CLI executable

main/                     # Main user scripts and debugging
├── build_website.sesi    # Sesi-generated landing page builder
└── tests/                # Additional syntax validation scripts

examples/
├── main/                        # Core language & systems feature examples
│   ├── 01_hello.sesi            # Hello World
│   ├── 02_variables.sesi        # Variables & operations
│   ├── 03_functions.sesi        # Functions with parameters
│   ├── 04_conditionals.sesi     # If/else control flow
│   ├── 05_loops.sesi            # While, for, for-in loops
│   ├── 06_arrays_objects.sesi   # Collections
│   ├── 07_prompts.sesi          # Prompt blocks composition
│   ├── 09_structured_output.sesi # Type-safe reasoning responses
│   ├── 11_memory_storage.sesi   # Multi-turn stateful reasoning
│   ├── 12_classification.sesi   # Systems classification loop
│   ├── 13_data_pipeline.sesi    # Complete systems pipeline
│   ├── 16_modules.sesi          # Modules & standard library imports
│   ├── 17_http_client.sesi      # Native HTTP client (web_get/web_send)
│   ├── 18_parallel_requests.sesi # Parallel request concurrency (multi_req)
│   ├── 19_search_web.sesi       # Built-in search_web integration
│   ├── 21_custom_tools.sesi     # Declaring and using custom tools
│   ├── 23_file_conversion.sesi  # File and format conversions
│   ├── 24_http_handler.sesi     # Native HTTP server request routing
│   ├── 24_http_server.sesi      # Non-blocking native HTTP server
│   ├── 25_webpage_server.sesi   # Native webpage hosting
│   ├── 26_database.sesi         # Embedded document database operations
│   ├── 27_robust_web_db.sesi    # Dynamic database-backed web analytics
│   ├── 29_tool_piping.sesi      # Functional piping operator (|)
│   ├── 30_error_recovery.sesi   # Error retry backoff handling
│   ├── 31_synthesizer.sesi      # Sound/music synthesis & SVG drawing
│   ├── 32_browser_automation.sesi # Headless browser automation with Playwright
│   ├── 33_base64.sesi           # Base64
│   └── 34_sesi_api.sesi         # SwaggerUI API setup
│
└── optional/                    # Optional reasoning & advanced AI examples
│   ├── 08_model_call.sesi       # Basic reasoning model calls
│   ├── 10_code_generation.sesi  # Code generation logic
│   ├── 14_folder_explainer.sesi # Workspace directory analyzer
│   ├── 15_image_generation.sesi # Image generation API
│   ├── 20_model_aliases.sesi    # Custom model naming aliases
│   ├── 22_reasoning_plus_custom_tools.sesi # Reasoning composed with custom tools
│   └── 28_streaming.sesi        # Real-time response streaming

docs/
├── SPECIFICATION.md      # Language spec
├── ARCHITECTURE.md       # This file
├── BUILTINS.md           # Built-in reference
├── CLI.md                # Comprehensive CLI & Parametric Eval guide
├── IMAGE_GENERATION.md   # Image generation guide
├── COMPARISON.md         # Language comparison showcase
├── REASONING.md          # Reasoning and simple logic guide
└── ROADMAP.md            # Future plans

tests/
├── basic.test.ts         # Core parsing & evaluation tests
├── cache.test.ts         # Execution caching tests
├── http.test.ts          # Web request builtins testing
├── module.test.ts        # Imports & module loading tests
├── parallel.test.ts      # Concurrent execution tests
├── security.test.ts      # Sandbox & guardrail tests
├── test-gemini.ts        # Base model integration test
├── test-gemini2.ts       # Extended model integration test
└── workflow.test.ts      # Complex sequence workflows tests

Workspace Context File

The root-level SKILLS.md file is part of the practical repo architecture. It is not consumed by the Sesi runtime, but it is intended to guide AI-assisted development in this workspace.

Testing Strategy

Unit Tests (per component):

// Lexer: tokenization correctness
// Parser: AST structure correctness
// Interpreter: evaluation correctness
// AI Runtime: Gemini integration

Integration Tests:

// Full program execution
// Complex Reasoning workflows
// Error handling

Example Programs:

  • Basic arithmetic and control flow
  • Functions and scoping
  • Arrays and objects
  • Reasoning model calls
  • Structured output parsing
  • Memory-based conversations

Debugging

Debug Output

Enable debug statement tracing:

SESI_DEBUG=1 sesi program.sesi

AST Visualization

Print the pretty-printed AST tree:

sesi --ast program.sesi

Token Stream

Print the scanned token stream table:

sesi --tokens program.sesi

Conclusion

Sesi's architecture has evolved from a simple tree-walking interpreter to a dual-mode engine. The bytecode compiler + VM is now the primary execution path, delivering faster startup and lower per-instruction overhead. The tree-walking interpreter is retained as a well-understood fallback. Both paths share the same Lexer, Parser, builtins, and AI Runtime.

  • High-throughput execution via bytecode VM
  • Easy extensibility via the shared AST and builtin layer
  • Smooth Reasoning integration regardless of execution path

As the language matures, JIT compilation and further VM optimizations can be added without changing the Sesi API.