fix: lex a template substitution inline instead of pre-scanning it - #52
Merged
marevol merged 1 commit intoAug 28, 2026
Merged
Conversation
Any slash inside a substitution broke the template, division included:
var a = 6, b = 3;
print(`${a / b}`); Expected an operand but found template_tail
print(`${1 / 2}`); Expected an operand but found template_tail
print(`${/c+/.source}`); Expected an operand but found /
print(`${"aa".replace(/a/g, "b")}`); Expected an operand but found /
scanTemplate pre-scanned the substitution by counting braces, then handed the
region to a nested lexer and called lexify() on it exactly once. But lexify()
deliberately returns as soon as it emits an ambiguous token so the parser can
decide whether a "/" starts a regular expression - so one call stopped at the
first slash and the rest of the expression was never tokenized. The parser then
met TEMPLATE_TAIL where it wanted an operand.
Looping lexify() would have fixed division and not the regular expression: the
literal is only recoverable while its token is the last one in the stream.
So the substitution is no longer pre-scanned. scanTemplate now emits just the
part that opens the template and returns; the ordinary lexer loop tokenizes the
expression, and the brace matching the opening ${ brings it back to emit
TEMPLATE_MIDDLE or TEMPLATE_TAIL. Which brace that is comes from a small stack
of brace depths, one entry per open substitution, so templates still nest.
That stack is lexer state, so the speculative lookahead added for arrow
functions has to rewind it along with the position. Leaving it behind puts the
lexer back inside a substitution while the depth says otherwise, and the brace
that ends the substitution is then read as an ordinary one.
Letting the real loop do the work fixes three more things the brace pre-scan got
wrong, because it knew about strings but not about comments or regular
expressions:
print(`x${ 1 // }
}y`); printed "x1\n}y", now "x1y" - silently wrong before
print(`x${ /* } */ 1 }y`); was a SyntaxError
print(`x${ 1 /* don't */ }y`); the apostrophe ate the rest of the file
print(`${ /}/.source }`); the brace in the regex ended the substitution
skipStringInSubstitution() existed only for that pre-scan and is removed.
One separate off-by-one goes with it: the part scanner consumed an escaped line
terminator without counting the line, because it used "else if (isEOL(ch0))"
where scanString uses a nested if. Every line number after such a template was
one too low. A template and a string with the same escaped newline now report
the same line.
./gradlew build testOptimistic testPessimistic:
suite before after
test 665, 0 fail 665, 0 fail
testOptimistic 1715, 0 fail 1715, 0 fail
testPessimistic 1715, 0 fail 1715, 0 fail
marevol
force-pushed
the
es6/fix-template-substitution
branch
from
August 28, 2026 05:11
4067b64 to
bb0c0ea
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #51.
Any slash inside a substitution broke the template, division included:
Cause
scanTemplatepre-scanned the substitution by counting braces, then handed the region to a nested lexer and calledlexify()on it exactly once. Butlexify()deliberately returns as soon as it emits an ambiguous token so the parser can decide whether a/starts a regular expression — so one call stopped at the first slash and the rest of the expression was never tokenized. The parser then metTEMPLATE_TAILwhere it wanted an operand.Looping
lexify()would have fixed division and not the regular expression: the literal is only recoverable while its token is the last one in the stream.Fix
The substitution is no longer pre-scanned.
scanTemplatenow emits just the part that opens the template and returns; the ordinary lexer loop tokenizes the expression, and the brace matching the opening${brings it back to emitTEMPLATE_MIDDLEorTEMPLATE_TAIL. Which brace that is comes from a small stack of brace depths, one entry per open substitution, so templates still nest.That stack is lexer state, so the speculative lookahead added in #51 has to rewind it along with the position. Leaving it behind puts the lexer back inside a substitution while the depth says otherwise, and the brace that ends the substitution is then read as an ordinary one.
Things the brace pre-scan also got wrong
It knew about strings but not about comments or regular expressions:
skipStringInSubstitution()existed only for that pre-scan and is removed.One separate off-by-one goes with it: the part scanner consumed an escaped line terminator without counting the line, because it used
else if (isEOL(ch0))wherescanStringuses a nestedif. Every line number after such a template was one too low. A template and a string with the same escaped newline now report the same line.Verification
./gradlew build testOptimistic testPessimistic