An Isolating Multi-Language Frontend approach for inline Cpp blocks.

[OmniVertX]

Hi everyone 👋,

I've been following the fascinating design dilemma regarding embedded C++ snippets within Carbon (e.g., using a raw string literal model vs a block syntax model).

The concern about avoiding grammar pollution and complex tokenization inside the Carbon lexer is absolutely valid; forcing Carbon's frontend to handle brace-matching or contextual rules for C++ is a recipe for a fragile, slow lexer. However, relying purely on regular expression matching or basic string parsing in IDEs often breaks down in complex, nested files.

I would like to propose a radical, foundational compromise that bridges language frontend design and modern tooling: An Isolating Multi-Language Frontend approach.

The Core Idea:

Instead of choosing between a generic string literal or a completely integrated syntax block, Carbon could implement a dedicated sub-lexer or an external token-stream handoff protocol.

1. The Lexer Handoff Protocol: When the Carbon lexer encounters a contextual phrase like inline Cpp {, it shouldn't attempt to parse standard C++ tokens. Instead, it temporarily yields control to a lightweight sub-lexer (or a shared LLVM/Clang lexer instance). This sub-lexer only tracks balance for the outermost curly braces {} of that block, bypassing Carbon's token rules entirely and treating the interior payload as an isolated token stream.
2. First-Class AST Nodes via External Attributes: Instead of dropping this into the AST as a generic string literal, it enters the Carbon Abstract Syntax Tree as a dedicated InlineCppBlock node.

Why this benefits both sides:

• For Compiler Engineers: The core backend gets a clean, bounded block that prevents C++ tokens from bleeding into Carbon’s primary grammar, keeping parsing fast and simple.
• For Tooling & IDEs: The Language Server Protocol (LSP) receives a native, semantic node to attach to. The LSP can instantly map a full C++ parser instance onto that specific node range, providing robust syntax highlighting, auto-complete, and proper macro expansions without guessing via text pattern matching.

This path allows Carbon to support genuine curly-brace syntax (inline Cpp { ... }) that treats C++ as a first-class citizen of the ecosystem, without the lexical complexity of standard language nesting.

Would love to hear the community's thoughts on this architectural direction!