tls: defer re-entrant calls to SSL state machine from JS

[pimterry]

This is a step towards #65035: making sure that we never mess with the OpenSSL/BoringSSL state machine from inside it.

Rather than hooking each of our callbacks/events inside the SSL library scope, I've wrapped each call that goes into OpenSSL at the top level, and then I'm checking against this in DoWrite, DoShutdown and Cycle, which can be triggered from JS. In each case, if we're inside the SSL library scope (which means we're being called from a JS callback that was triggered by the SSL library) then we defer.

This covers some cases that currently work by luck but will break soon in BoringSSL and some cases that already fail today, like the ALPNCallback write example (see #65035 for details).

This just covers SSL_read/SSL_write/SSL_shutdown which are impacted by the pending BoringSSL patch, we may need to review other SSL_set... calls as well later.

[nodejs-github-bot]

Review requested:

• @nodejs/crypto

[codecov]

Codecov Report

❌ Patch coverage is 79.74684% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.15%. Comparing base (fe4a42b) to head (c03259b).
⚠️ Report is 83 commits behind head on main.

Files with missing lines	Patch %	Lines
src/crypto/crypto_tls.cc	78.08%	6 Missing and 10 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #65105      +/-   ##
==========================================
- Coverage   90.18%   90.15%   -0.03%     
==========================================
  Files         771      771              
  Lines      264911   265513     +602     
  Branches    50309    50484     +175     
==========================================
+ Hits       238919   239385     +466     
- Misses      16966    17054      +88     
- Partials     9026     9074      +48     

Files with missing lines	Coverage Δ
src/crypto/crypto_tls.h	90.00% <100.00%> (+2.50%)	⬆️
src/crypto/crypto_tls.cc	78.44% <78.08%> (-0.33%)	⬇️

... and 67 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pimterry]

@nodejs/crypto some eyes on this would be helpful. This fixes various cases (see tests) where using a TLS socket from inside our own callbacks breaks Node today, due to bad behaviour that BoringSSL is planning to soon directly guard against: https://boringssl-review.googlesource.com/c/boringssl/+/97087.

#65035 has the full context.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/77343/

[nodejs-github-bot]

Commit Queue failed

This pull request has multiple commits, but no landing policy was selected.

Add commit-queue-squash PRs the Commit Queue should land as one squashed commit. to land it as one commit, or commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. to land the commits separately.

The pull request was removed from the Commit Queue and labeled commit-queue-failed PRs whose Commit Queue landing failed and need manual intervention before retrying. . After resolving the failure, remove that label and add commit-queue PRs queued for automated landing through the Commit Queue. to retry.

Full Commit Queue output

- Loading data for nodejs/node/pull/65105
✔  Done loading data for nodejs/node/pull/65105
----------------------------------- PR info ------------------------------------
Title      tls: defer re-entrant calls to SSL state machine from JS (#65105)
   ⚠  Could not retrieve the email or name of the PR author's from user's GitHub profile!
Branch     pimterry:fix-alpn-cb-write -> nodejs:main
Labels     crypto, c++, author ready, needs-ci, commit-queue
Commits    3
 - tls: defer re-entrant calls to SSL state machine from JS
 - Defer shutdown until after newSession resolves
 - Fix newSession test to work consistently for both OpenSSL & BoringSSL
Committers 1
 - Tim Perry <pimterry@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/65105
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/65105
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
--------------------------------------------------------------------------------
   ℹ  This PR was created on Fri, 07 Aug 2026 14:41:00 GMT
   ✔  Approvals: 1
   ✔  - Filip Skokan (@panva) (TSC): https://github.com/nodejs/node/pull/65105#pullrequestreview-5156570615
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2026-09-11T08:14:32Z: https://ci.nodejs.org/job/node-test-pull-request/77343/
- Querying data for job/node-test-pull-request/77343/
✔  Build data downloaded
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  No git cherry-pick in progress
   ✔  No git am in progress
   ✔  No git rebase in progress
--------------------------------------------------------------------------------
- Bringing origin/main up to date...
From https://github.com/nodejs/node
 * branch                  main       -> FETCH_HEAD
✔  origin/main is now up-to-date
- Downloading patch for 65105
From https://github.com/nodejs/node
 * branch                  refs/pull/65105/merge -> FETCH_HEAD
✔  Fetched commits as 7709fca530a8..c03259b7d05a
--------------------------------------------------------------------------------
[main 85d5b5d90c] tls: defer re-entrant calls to SSL state machine from JS
 Author: Tim Perry <pimterry@gmail.com>
 Date: Fri Aug 7 16:24:15 2026 +0200
 5 files changed, 294 insertions(+), 6 deletions(-)
 create mode 100644 test/parallel/test-tls-alpn-callback-sync-end.js
 create mode 100644 test/parallel/test-tls-alpn-callback-sync-write.js
 create mode 100644 test/parallel/test-tls-keylog-sync-write.js
[main 68983fbf86] Defer shutdown until after newSession resolves
 Author: Tim Perry <pimterry@gmail.com>
 Date: Wed Sep 9 12:40:30 2026 +0200
 2 files changed, 61 insertions(+), 2 deletions(-)
 create mode 100644 test/parallel/test-tls-alpn-callback-sync-end-newsession.js
[main 8bda43a426] Fix newSession test to work consistently for both OpenSSL & BoringSSL
 Author: Tim Perry <pimterry@gmail.com>
 Date: Wed Sep 9 14:40:50 2026 +0200
 1 file changed, 8 insertions(+), 9 deletions(-)
   ✔  Patches applied
There are 3 commits in the PR. Attempting autorebase.
(node:393) [DEP0190] DeprecationWarning: Passing args to a child process with shell option true can lead to security vulnerabilities, as the arguments are not escaped, only concatenated.
(Use `node --trace-deprecation ...` to show where the warning was created)
Rebasing (2/6)
Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
tls: defer re-entrant calls to SSL state machine from JS

Signed-off-by: Tim Perry <pimterry@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/65105
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
--------------------------------------------------------------------------------
[detached HEAD feeacbf21f] tls: defer re-entrant calls to SSL state machine from JS
 Author: Tim Perry <pimterry@gmail.com>
 Date: Fri Aug 7 16:24:15 2026 +0200
 5 files changed, 294 insertions(+), 6 deletions(-)
 create mode 100644 test/parallel/test-tls-alpn-callback-sync-end.js
 create mode 100644 test/parallel/test-tls-alpn-callback-sync-write.js
 create mode 100644 test/parallel/test-tls-keylog-sync-write.js
Rebasing (3/6)
Rebasing (4/6)
Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
Defer shutdown until after newSession resolves

PR-URL: https://github.com/nodejs/node/pull/65105
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
--------------------------------------------------------------------------------
[detached HEAD 8a58c14b26] Defer shutdown until after newSession resolves
 Author: Tim Perry <pimterry@gmail.com>
 Date: Wed Sep 9 12:40:30 2026 +0200
 2 files changed, 61 insertions(+), 2 deletions(-)
 create mode 100644 test/parallel/test-tls-alpn-callback-sync-end-newsession.js
Rebasing (5/6)
Rebasing (6/6)
Executing: git node land --amend --yes
--------------------------------- New Message ----------------------------------
Fix newSession test to work consistently for both OpenSSL & BoringSSL

PR-URL: https://github.com/nodejs/node/pull/65105
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
--------------------------------------------------------------------------------
[detached HEAD a5f22a336b] Fix newSession test to work consistently for both OpenSSL & BoringSSL
 Author: Tim Perry <pimterry@gmail.com>
 Date: Wed Sep 9 14:40:50 2026 +0200
 1 file changed, 8 insertions(+), 9 deletions(-)
Successfully rebased and updated refs/heads/main.
--------------------------------------------------------------------------------
   ℹ  Add `commit-queue-squash` label to land the PR as one commit, or `commit-queue-rebase` to land as separate commits.

View workflow run

[nodejs-github-bot]

Landed in 46bbfc4