fix: keep a regular expression literal readable across the ES6 lookahead - #51
Merged
Merged
Conversation
Under --language=es6 a regular expression literal stopped parsing wherever the
arrow or destructuring-assignment lookahead ran first:
var x = (/a/); Expected an operand but found /
var x = [/a/]; Expected an operand but found /
var o = { re: /a+/ }; Expected an operand but found /
var a = [1, /a/]; Expected an operand but found /
var x = (c ? /a/ : 1); Expected an operand but found /
and, under -scripting, a here string in the same positions. es5 was unaffected.
Bisected to the arrow-function change for the parenthesised form and to the
destructuring-assignment change for the bracket and brace forms.
A "/" is ambiguous, so lexify() adds the token and breaks out of its loop right
there, leaving the parser to call scanLiteral() and reinterpret it as a literal
if the grammar wants one. scanLiteral() refuses once the stream has moved on -
"we break on ambiguous tokens so if we already moved on it can't be a literal".
isArrowFunction() and isDestructuringAssignment() scan forward with T(i) to find
a closing bracket, and getToken() satisfies that by calling lexify() again, which
resumes past the slash. From then on the literal is unrecoverable.
So the lookahead now runs through AbstractParser.lookahead(), which snapshots the
lexer and the token stream, runs the probe, and rewinds both. The ambiguity is
left exactly where it was, for the real parse to resolve with the grammar context
it actually has. TokenStream.removeLast() and Lexer.saveState()/restoreState()
already existed; only pauseOnNextLeftBrace needed accessors, since it is lexer
state the State record does not carry.
Rewinding rather than bailing out at the slash matters: an arrow parameter
default may legitimately contain one. (a = 1 / 2) => a kept working throughout
because the lookahead and the real parse agreed on division; (a = /h+/) => a was
broken before this change and works now.
basic/es6/regex-literals.js covers each position, both arrow-parameter forms,
and plain division. No new es6 test used a regular expression literal at all,
which is why 1714 passing tests said nothing about this.
./gradlew build testOptimistic testPessimistic:
suite before after
test 665, 0 fail 665, 0 fail
testOptimistic 1714, 0 fail 1715, 0 fail
testPessimistic 1714, 0 fail 1715, 0 fail
This was referenced Aug 28, 2026
Fix typo in comment regarding ambiguous tokens in lexer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #50.
Under
--language=es6a regular expression literal stopped parsing wherever the arrow or destructuring-assignment lookahead ran first:and, under
-scripting, a here string in the same positions. es5 was unaffected. Bisected to the arrow-function change (#33) for the parenthesised form and to the destructuring-assignment change (#44) for the bracket and brace forms.Cause
A
/is ambiguous, solexify()adds the token and breaks out of its loop right there, leaving the parser to callscanLiteral()and reinterpret it as a literal if the grammar wants one.scanLiteral()refuses once the stream has moved on — "we break on ambiguous tokens so if we already moved on it can't be a literal".isArrowFunction()andisDestructuringAssignment()scan forward withT(i)to find a closing bracket, andgetToken()satisfies that by callinglexify()again, which resumes past the slash. From then on the literal is unrecoverable.Fix
The lookahead now runs through
AbstractParser.lookahead(), which snapshots the lexer and the token stream, runs the probe, and rewinds both. The ambiguity is left exactly where it was, for the real parse to resolve with the grammar context it actually has.TokenStream.removeLast()andLexer.saveState()/restoreState()already existed; onlypauseOnNextLeftBraceneeded accessors, since it is lexer state theStaterecord does not carry.Rewinding rather than bailing out at the slash matters: an arrow parameter default may legitimately contain one.
(a = 1 / 2) => akept working throughout because the lookahead and the real parse agreed on division;(a = /h+/) => awas broken before this change and works now.basic/es6/regex-literals.jscovers each position, both arrow-parameter forms, and plain division. No new es6 test used a regular expression literal at all, which is why 1714 passing tests said nothing about this.Verification
./gradlew build testOptimistic testPessimistic