EPIPE write, ECONNRESET issues when switch to nodejs 20

[roman-kupriyanov]

Hello,

After the migration to nodejs 20.16.0 from 18.20.4 some of our tests that use Mockttp server started to fail with EPIPE write or ECONNRESET read or ECONNRESET socket hang up errors. We have around 200 tests in general and only some of them are affected. A weird part is that if to run failed tests individually they are passing, which makes me think that it may be related to some parallel running issues or race conditions. We use Jest as a test runner.

Any help would be appreciated. Thanks!

[pimterry]

Oooh, that sounds very interesting! I haven't seen this before myself, and Mockttp's own tests seem to run fine in 20.16.0 so I'm not sure exactly what would cause this.

Is there any pattern to which tests fail? Is there anything notable about those tests?

If you try using other Node versions (older Node v20 versions, or Node v22) do those fail in the same way? If not, it would be really interesting to try a few different versions and see if you can work out exactly which version is breaking this for you.

[roman-kupriyanov]

Hi @pimterry,

Indeed a very interesting issue it is 🙂 Regarding your questions:

If you try using other Node versions (older Node v20 versions, or Node v22) do those fail in the same way? If not, it would be really interesting to try a few different versions and see if you can work out exactly which version is breaking this for you.

I tried version 20.0.0 and it is not working right away, so seems that the issue is appearing from very beginning of the node 20 version. I also tried 22.6.0 and it fails the tests in the same way. And again the latest 18.x version fixes everything immediately.

I also use the latest to date mockttp version: 3.15.1.

Is there any pattern to which tests fail? Is there anything notable about those tests?

There are two patterns usually:

• When we expect some specific response from API, but instead getting the socket errors:

    expect(received).toEqual(expected) // deep equality

    - Expected  - 3
    + Received  + 7

      Object {
        "details": Object {
    -     "messages": Array [],
    -     "returnCode": 0,
    +     "connectionCode": "ECONNRESET",
    +     "httpStatusCode": undefined,
    +     "messages": Array [
    +       "read ECONNRESET",
    +     ],
        },
    -   "status": "OK",
    +   "status": "ERROR",
    +   "type": "CONNECTION_ERROR",
      }

      10244 |       )(packageName)(packageOptions);
      10245 |       // assert
    > 10246 |       expect(createPackageResult).toEqual({
            |                                   ^
      10247 |         status: ResponseStatus.OK,
      10248 |         details: {
      10249 |           returnCode: 0,

These tests usually use forAnyRequest().thenJson() mock:

      await mockServer.forAnyRequest().thenJson(
        200,
        {
          data: [],
          messages,
          reports: {},
          returnCode,
        },
        {
          version: '2.5',
          'content-type': 'application/json',
        }
      );

• And the second case is during the check if API was even called:

    expect(received).toBe(expected) // Object.is equality

    Expected: true
    Received: false

      6726 |         const seenRequests = await endevorEndpoint.getSeenRequests();
      6727 |         const calledOnce = seenRequests.length === 1;
    > 6728 |         expect(calledOnce).toBe(true);
           |                            ^
      6729 |
      6730 |         expect(isErrorEndevorResponse(generateResult)).toBe(false);
      6731 |       });

This one happens usually with the forPut() or forGet() mock.

[pimterry]

Ok, that's very useful thanks. I think it's likely that this is a Node bug (or intentional breaking change that's causing issues), but unfortunately I'm also a Node maintainer, so I can't really pass the buck here 😆, and whether the fix is in Node or Mockttp it'd be useful to know exactly what's going on.

Can you share more detail about how exactly are you're sending the requests, and any kinds of client configuration there? One thing that has changed a few times in Node during major version bumps is a slow tightening of which kinds of subtly invalid connections or unusual requests are accepted, so it might be that there's something unusual there triggering this.

If you test with some v19 versions, does this fail the same way there too? If we can pin this down to a specific version where it breaks then that would reduce the set of possibly related changes in Node significantly, which would be very helpful to understand the issue.

Is there any chance you can put together a reproduction of the issue, or share enough code that I can run the tests myself? That would make it very dramatically easier to investigate and fix.

[jaramir]

I think I managed to replicate the same problem on a simpler codebase. Pushed the code here: https://github.com/codurance/mockttp-test

This project has two tests, reusing the same server which will start on the same port. The first one passes and the second one fails.

I think this also depends on the specific version of Jest used but still looking into that. [update: nope, upgraded everything to latest and it still fails]

[pimterry]

Thanks @jaramir, that's definitely an interesting example. It looks like node-fetch (via cross-fetch) is the real culprit here. If you comment out that import entirely and just use Node's fetch built-in instead then the test passes fine with Node v20 or v22. Meanwhile if you use fetch from node-fetch or cross-fetch, the 2nd test fails.

Any idea why that might be? I know Jest has some weird & wonderful hooks into various internals, I wonder if that might interact with fetch in some unexpected ways somewhere. Adding a bunch of debug logging, it looks like Mockttp doesn't even receive a TCP connection at all in the 2nd test. If you try using a simpler test runner (Node's built-in test runner, or mocha or something) does that work?

[vokrik]

I am not sure if it helps, but to me it seems to be happening on every other test. My setup is node-fetch, jest as a runner. I have a completely wild theory without any proof, but I wonder if the node-fetch is not using some persistent tcp connection. I remember that node in certain versions was not sending the closing message and just ended the connection on it's end. If the node-fetch somehow tried to re-use the connection, it would make sense for it to fail. (and it would explain why it would fail every other test)

Don't want to add a red herring here, cause I am really just a user of both of the libs, just wanted to add a theory to test if you run out of ideas

EDIT:
Found this mention in the node-fetch docs:
https://github.com/node-fetch/node-fetch/blob/8b3320d2a7c07bce4afc6b2bf6c3bbddda85b01f/README.md?plain=1#L560

I was using node 20, when I downgrade to 16 (just picked random installed prior v19), all tests pass. So now I am quite confident this might be the problem.

Not sure how to fix it though :D

[jaramir]

I've removed cross-fetch. It still fails if the first test does two requests: https://github.com/codurance/mockttp-test

It works if the port is randomised with .start(0) which is what I've done in the production code I tend to.

I've followed the traffic with Wireshark: what I see is the node build-in fetch reusing the connection used by the first request after sending FIN and before receiving FIN from the server.

Edit: this also fails on node 18 so we might have stumbled on an entirely different problem.

[pimterry]

I've followed the traffic with Wireshark: what I see is the node build-in fetch reusing the connection used by the first request after sending FIN and before receiving FIN from the server.

Yep, I can see this too.

I think this is a known Node bug via Undici's fetch implementation: nodejs/undici#3492. As you can see from discussion there, it seems you can hit it in other cases, but the fetch case is more common.

In short: the client has already received a FIN packet, and so it definitely shouldn't be trying to continue using the connection. It only comes up when there's a very short delay between connection close (triggered by server.stop()) and the subsequent request, which results in the request is sent before the connection FIN is fully processed by the client. This only applies if the port is the same (if the port is different, the client doesn't reuse the connection) - if you use start(0) that will mostly work, but could still fail if you get the same random port twice in a row.

In general this is Mockttp working correctly - the server closes and sends a FIN before server.stop() resolves, it's the client that's doing the wrong thing. That said, obviously it would be good for Mockttp to make this case easy to manage correctly. And also, I'm a Node maintainer, so the client doing the wrong thing is kind of my problem as well anyway 😆

It looks like you can fix this by adding a tiny delay after server shutdown (5ms works reliably on my machine) so that the close packets are definitely processed. That's obviously suboptimal of course, since it's a bit arbitrary/flaky and it's an annoying extra step, but it might be useful as another workaround. I'll see if there's anything we can do to improve this here so that waiting for server.stop() handles this correctly automatically. Not sure how difficult it would be to resolve this in Node itself, it looks like the existing discussion didn't find any easy solutions unfortunately.

[roman-kupriyanov]

It works if the port is randomised with .start(0) which is what I've done in the production code I tend to.

Thank you @jaramir! This workaround fixes the problem for us too with nodejs v20.18.3 and mockttp v3.16.

[pimterry]

@roman-kupriyanov @jaramir I think I now have a proper fix for this. The internal dependency destroyable-server has a new v1.1.1 version just published, which ensures that server.stop() waits until the connections have fully closed before it resolves, which should stop this happening.

I've just tested and with this change I can run the https://github.com/codurance/mockttp-test repo tests in a constant loop for 5 minutes straight without any failures, so this does seem to be fairly reliable.

If you update the destroyable-server dependency in your projects, does that resolve the issue?

[roman-kupriyanov]

@pimterry thank you for the suggestion! Updating of the destroyable-server@1.1.1 helped to overcome the issue with the sockets on nodejs 20+ that I was initially reporting.

However, all the tests fail with the same error after switching back to nodejs 18.20.6 and using the updated destroyable-server@1.1.1 package:

    - Expected  - 3
    + Received  + 4

      Object {
        "details": Object {
    +     "connectionCode": undefined,
    +     "httpStatusCode": 503,
          "messages": Array [
    -       "API0034S INVALID USERID OR PASSWORD DETECTED",
    +       "Rest API failure with HTTP(S) status 503",
          ],
    -     "returnCode": 20,
        },
        "status": "ERROR",
    -   "type": "WRONG_CREDENTIALS_ENDEVOR_ERROR",
    +   "type": "GENERIC_ERROR",
      }

We do not plan to use nodejs 18.x runtimes anymore after migration, but I thought you may want to know that some latest version of the mockttp's dependency can cause issues for the users that may still use nodejs 18.

Thanks a lot everyone who provided their reports and the test project to play with 🙏

[pimterry]

Oooh, thanks for reporting that @roman-kupriyanov, that's interesting. Mockttp's own tests do run with Node 18 in CI and they're passing, but it's quite possible there's a specific case they're not covering.

It's not very clear what's going on from the error shown there though, most of the specifics seem related to your test code. I think this is saying there's some kind of 503 being returned, but it's hard to know what's actually going on without more context.

Do you have a way I can reproduce this locally? Or any more details on exactly what's going wrong?

[roman-kupriyanov]

@pimterry oh yeah, sure 👍 I will try to reproduce the same error on the test project to be sure..