🚀 JetPascal™ Architecture Deep Dive - Post v0.1.0

[jarroddavis68]

Compiler Architecture Built from the Ground Up

Hey JetPascal community! 👋

I'm excited to share the technical details of our redesigned compiler architecture that will be powering JetPascal v0.2.0+. This is a enhancement of the architecture in v0.1.0 and represents careful design and refinement.

🎯 The Big Picture: What Changed

Out with the Old

• ❌ Removed DelphiAST dependency - We've completely ditched the third-party AST library
• ❌ No more external parsing dependencies
• ❌ No more rigid, inflexible tree structures

In with the New

• ✅ Custom tokenizer built specifically for JetPascal's needs
• ✅ Two-layer architecture separating concerns cleanly
• ✅ SmartToken system - our innovative approach to transpilation
• ✅ Complete control over every aspect of compilation

🏗️ The Two-Layer Architecture

We've separated the compiler into two distinct layers, each with specific responsibilities:

Layer 1: JetPascal.Transpiler (Core Engine)

The transpiler is the heart of the compilation system. It provides:

Tokenizer Services:

• Custom-built lexical analyzer that understands Pascal syntax
• Sophisticated token stream management with peek/push/pop state
• Support for Pascal-specific constructs (nested comments, string literals, hex numbers)
• Precise line/column tracking for excellent error messages

Error Management:

• Four severity levels: Hints, Warnings, Errors, Fatal
• Rich error messages with file, line, and column information
• Non-blocking error accumulation (continue parsing to find multiple errors)
• Beautiful console output formatting

Code Generation:

• StringBuilder-based output with indentation management
• Support for C++23 features
• Smart header management (automatic deduplication and sorting)
• #line directive emission for debugging (C++ errors map back to Pascal source!)
• Type mapping (Pascal Integer → C++ int, etc.)

Build Settings:

• Optimization levels (Debug, ReleaseSafe, ReleaseFast, ReleaseSmall)
• Target platforms (Native, x86_64-Windows, x86_64-Linux, aarch64-macOS, wasm32-wasi)
• ABI modes (C vs C++)
• Path management (units, includes, libraries)

Layer 2: JetPascal.Compiler (Project Manager)

The compiler sits on top of the transpiler and handles project-level concerns:

Project Management:

• Project identity (name, directory structure)
• Source and generated file tracking
• Template system (Program, Library, Unit)

Build System Integration:

• Dynamic build.zig generation based on project settings
• Source path and file registration
• Library and include path management
• Zig execution with real-time console output capture

High-Level API:

• Init() - Create new projects from templates
• Build() - Full compile-and-build pipeline
• Clean() - Remove generated artifacts
• Run() - Execute compiled programs
• Zig() - Direct Zig command passthrough

💡 The SmartToken Innovation

This is where JetPascal really shines. Instead of building a traditional Abstract Syntax Tree (AST), we use SmartTokens - intelligent, self-contained compilation units.

What is a SmartToken?

Every Pascal language construct is represented by a SmartToken class that knows how to:

1. Parse itself - Each token type understands its own syntax
2. Register its dependencies - Declares what C++ headers it needs
3. Emit C++ code - Generates its corresponding C++ implementation
4. Provide expressions - Returns C++ expressions when used in larger constructs

The SmartToken Base Class

TSmartToken = class(TBaseObject)
public
  procedure Parse(const ACompiler: TTranspiler); virtual; abstract;
  procedure RegisterHeaders(const ACompiler: TTranspiler); virtual;
  function EmitCppExpression(const ACompiler: TTranspiler): string; virtual;
  procedure EmitCpp(const ACompiler: TTranspiler); virtual;
  procedure SetLocation(const ALine: Integer; const AColumn: Integer);
end;

Real-World Example: The WriteLn Token

Let's see how a WriteLn statement works:

Pascal Source:

WriteLn('Hello, ', name, '!');

What TWriteLnToken Does:

1. Parse Phase:

   • Parses the opening parenthesis
   • Identifies arguments: string literal, identifier, string literal
   • Stores them internally with type tags
   • Validates syntax

2. RegisterHeaders Phase:

   • Calls ACompiler.RegisterHeader('<print>') to ensure std::print is available

3. EmitCpp Phase:

   • Builds format string: "Hello, {}!"
   • Builds argument list: name
   • Emits: std::println("Hello, {}!", name);

Why SmartTokens Are Brilliant

Encapsulation: Each language feature is completely self-contained in its own unit file

• JetPascal.Token.WriteStmt.pas - Write/WriteLn statements
• JetPascal.Token.ForLoopStmt.pas - For loops
• JetPascal.Token.VarDecl.pas - Variable declarations
• JetPascal.Token.ProgramDecl.pas - Program declarations
• And so on...

Extensibility: Adding new language features is straightforward:

1. Create a new JetPascal.Token.XXX.pas unit
2. Implement the Parse and EmitCpp methods
3. Register the keyword with the transpiler
4. Done! The feature is now part of the language

Maintainability: When a bug is found or enhancement is needed:

• You know exactly which file to modify
• Changes are isolated and don't affect other features
• Testing is simpler (test the token in isolation)

🔄 The Compilation Pipeline

Here's how everything flows together:

Step 1: Registration

RegisterKeyword('program', TProgramToken);
RegisterKeyword('begin', TBeginEndToken);
RegisterKeyword('writeln', TWriteLnToken);
RegisterKeyword('for', TForLoopToken);
// ... etc

Step 2: Source Loading

Transpiler.SetSource(PascalCode, 'myprogram.pas');

Step 3: Root Token Detection

Token := PeekToken();  // Look at first token
RootToken := CreateTokenForKeyword(Token.Text);  // Create TProgramToken, TUnitToken, etc.

Step 4: Parse Tree Construction

RootToken.Parse(Transpiler);  
// Recursively creates child tokens:
// - Uses clause → loads and parses imported units
// - Var section → parses variable declarations  
// - Begin/end block → parses statements

Step 5: Header Registration

RootToken.RegisterHeaders(Transpiler);
// Each token in the tree registers its required C++ headers

Step 6: Code Generation

RootToken.EmitCpp(Transpiler);
// Recursively generates C++ code
// Headers are automatically injected at the top

Step 7: File Output

RootToken.WriteToDirectory(Transpiler, OutputDir);
// Writes .cpp and .h files as appropriate

Step 8: Zig Build

Compiler.GenerateBuildZig();  // Create build.zig
Compiler.ExecuteBuild(ExitCode);  // Invoke Zig to compile C++

🎨 Design Patterns We Use

1. Factory Pattern

Keywords are mapped to token classes, allowing dynamic instantiation:

FKeywordMap.AddOrSetValue('writeln', TWriteLnToken);

2. Visitor Pattern (Sort Of)

The transpiler "visits" tokens by calling their Parse and EmitCpp methods.

3. Strategy Pattern

Different token types provide different parsing and emission strategies.

4. Template Method Pattern

TCompilationUnit provides a base template that TProgramToken, TUnitToken, and TLibraryToken customize.

📊 Benefits of This Architecture

For Users:

✅ Better error messages - Precise line/column information with context
✅ Faster compilation - No intermediate AST serialization
✅ Smaller binaries - Direct Pascal→C++ translation without overhead
✅ Debugging support - C++ errors map back to Pascal source via #line directives

For Contributors:

✅ Easy to understand - Each token is self-documenting
✅ Easy to extend - Add features without touching existing code
✅ Easy to test - Unit test individual token types
✅ Easy to debug - Clear separation of concerns

For the Project:

✅ No external dependencies for parsing (only relies on Delphi RTL)
✅ Complete control over compilation process
✅ Future-proof - Easy to add new features as Pascal or C++ evolves

🔮 What's Next?

With this architecture in place, we're now positioned to rapidly add features:

Coming in a future release:

• Records and classes
• Generic types
• Interfaces
• Procedural types
• Inline assembly
• And much more!

Each feature will be a new SmartToken, plugged into the system without disrupting what already works.

🤔 Technical Q&A

Q: Why not use a traditional AST?
A: Traditional ASTs require building a complete tree in memory, then traversing it multiple times. Our SmartToken approach processes the source in a single pass (with recursive sub-passes for units), reducing memory overhead and simplifying the code.

Q: How does unit dependency resolution work?
A: When a uses clause is parsed, TUsesDeclToken locates each unit file, pushes the current parser state, processes the unit recursively, then pops back to continue with the main file. Circular dependencies are detected and prevented.

Q: What about compile times?
A: By skipping the AST construction phase, we save significant time. The bottleneck is actually the Zig C++ compilation phase, not our Pascal→C++ transpilation.

Q: How are you handling Pascal's case-insensitivity?
A: The tokenizer handles this transparently. Keywords are always matched case-insensitively, and we maintain the original casing for identifiers while doing case-insensitive lookups in symbol tables.

🙏 Acknowledgments

This architecture wouldn't exist without:

• The Zig project for an excellent build system
• The C++23 standard for modern features like std::print
• This community for feedback and feature requests!

📝 Closing Thoughts

We didn't just build a compiler - we built a platform for language innovation. The SmartToken architecture gives us the flexibility to experiment, iterate, and deliver features quickly while maintaining code quality.

Want to contribute? Check out our GitHub repo! Adding a new language feature is as simple as creating a new JetPascal.Token.XXX.pas file. The compiler handles the rest.

Let's accelerate Pascal together! 🚀

Have questions about the architecture? Drop them in the comments and I'll do my deep dive explanations!