Skip to content

fix: keep a regular expression literal readable across the ES6 lookahead - #51

Merged
marevol merged 2 commits into
es6/fix-arrow-this-lazyfrom
es6/fix-regex-after-lookahead
Aug 28, 2026
Merged

fix: keep a regular expression literal readable across the ES6 lookahead#51
marevol merged 2 commits into
es6/fix-arrow-this-lazyfrom
es6/fix-regex-after-lookahead

Conversation

@marevol

@marevol marevol commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Stacked on #50.

Under --language=es6 a regular expression literal stopped parsing wherever the arrow or destructuring-assignment lookahead ran first:

var x = (/a/);          // Expected an operand but found /
var x = [/a/];          // Expected an operand but found /
var o = { re: /a+/ };   // Expected an operand but found /
var a = [1, /a/];       // Expected an operand but found /
var x = (c ? /a/ : 1);  // Expected an operand but found /

and, under -scripting, a here string in the same positions. es5 was unaffected. Bisected to the arrow-function change (#33) for the parenthesised form and to the destructuring-assignment change (#44) for the bracket and brace forms.

Cause

A / is ambiguous, so lexify() adds the token and breaks out of its loop right there, leaving the parser to call scanLiteral() and reinterpret it as a literal if the grammar wants one. scanLiteral() refuses once the stream has moved on — "we break on ambiguous tokens so if we already moved on it can't be a literal".

isArrowFunction() and isDestructuringAssignment() scan forward with T(i) to find a closing bracket, and getToken() satisfies that by calling lexify() again, which resumes past the slash. From then on the literal is unrecoverable.

Fix

The lookahead now runs through AbstractParser.lookahead(), which snapshots the lexer and the token stream, runs the probe, and rewinds both. The ambiguity is left exactly where it was, for the real parse to resolve with the grammar context it actually has. TokenStream.removeLast() and Lexer.saveState()/restoreState() already existed; only pauseOnNextLeftBrace needed accessors, since it is lexer state the State record does not carry.

Rewinding rather than bailing out at the slash matters: an arrow parameter default may legitimately contain one. (a = 1 / 2) => a kept working throughout because the lookahead and the real parse agreed on division; (a = /h+/) => a was broken before this change and works now.

basic/es6/regex-literals.js covers each position, both arrow-parameter forms, and plain division. No new es6 test used a regular expression literal at all, which is why 1714 passing tests said nothing about this.

Verification

./gradlew build testOptimistic testPessimistic

suite before after
test 665, 0 fail 665, 0 fail
testOptimistic 1714, 0 fail 1715, 0 fail
testPessimistic 1714, 0 fail 1715, 0 fail

Under --language=es6 a regular expression literal stopped parsing wherever the
arrow or destructuring-assignment lookahead ran first:

    var x = (/a/);            Expected an operand but found /
    var x = [/a/];            Expected an operand but found /
    var o = { re: /a+/ };     Expected an operand but found /
    var a = [1, /a/];         Expected an operand but found /
    var x = (c ? /a/ : 1);    Expected an operand but found /

and, under -scripting, a here string in the same positions. es5 was unaffected.
Bisected to the arrow-function change for the parenthesised form and to the
destructuring-assignment change for the bracket and brace forms.

A "/" is ambiguous, so lexify() adds the token and breaks out of its loop right
there, leaving the parser to call scanLiteral() and reinterpret it as a literal
if the grammar wants one. scanLiteral() refuses once the stream has moved on -
"we break on ambiguous tokens so if we already moved on it can't be a literal".
isArrowFunction() and isDestructuringAssignment() scan forward with T(i) to find
a closing bracket, and getToken() satisfies that by calling lexify() again, which
resumes past the slash. From then on the literal is unrecoverable.

So the lookahead now runs through AbstractParser.lookahead(), which snapshots the
lexer and the token stream, runs the probe, and rewinds both. The ambiguity is
left exactly where it was, for the real parse to resolve with the grammar context
it actually has. TokenStream.removeLast() and Lexer.saveState()/restoreState()
already existed; only pauseOnNextLeftBrace needed accessors, since it is lexer
state the State record does not carry.

Rewinding rather than bailing out at the slash matters: an arrow parameter
default may legitimately contain one. (a = 1 / 2) => a kept working throughout
because the lookahead and the real parse agreed on division; (a = /h+/) => a was
broken before this change and works now.

basic/es6/regex-literals.js covers each position, both arrow-parameter forms,
and plain division. No new es6 test used a regular expression literal at all,
which is why 1714 passing tests said nothing about this.

./gradlew build testOptimistic testPessimistic:

  suite            before         after
  test             665, 0 fail    665, 0 fail
  testOptimistic   1714, 0 fail   1715, 0 fail
  testPessimistic  1714, 0 fail   1715, 0 fail
Fix typo in comment regarding ambiguous tokens in lexer.
@marevol marevol self-assigned this Aug 28, 2026
@marevol
marevol merged commit e7227d3 into master Aug 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant