Shared Memory "Already Free" Bug?

[lancedockins]

One of our servers ran into this error after the shared memory zone ran out of free space.
2026/03/16 10:53:18 [alert] 3931255#3931255: ngx_slab_free(): chunk is already free in js shared zone "zone_name"

It's on NJS 0.9.5 and I know that the message that preceded it has been silenced in 0.9.6 if timeout + eviction are enabled in the zone.
2026/03/16 10:53:18 [crit] 3931255#3931255: ngx_slab_alloc() failed: no memory in js shared zone "zone_name"

Grant that the message is only an alert but it sounded like an attempt to double free memory behind the scenes and that would imply some sort of memory management bug in NJS for this so I wanted to relay this and ask.

We've only seen this occur one time now so I can't specifically name a way to reproduce this but I'd assume that you'd just have to overfill the memory zone. When this did happen, Nginx CPU use spiked to 100% and we had to restart the Nginx process to fix it.

Our memory zone definition is:
js_shared_dict_zone zone=zone_name:10m type=string timeout=30m evict;

This is an Nginx 1.29.5 build with NJS 0.9.5 on AlmaLinux 9.

We aren't using any memory clear/delete NJS functions. The code that uses this zone strictly uses get/set functions and it does sets only if the memory zone doesn't have the key/value pair set already.

If there's anything else that you'd need to know to look into this or confirm that it is resolved in 0.9.6 or otherwise, please let me know. If there's something about this setup that you'd advise doing differently, I'm happy to hear that as well.

[xeioex]

hi @lancedockins,

Grant that the message is only an alert but it sounded like an attempt to double free memory behind the scenes

The change you are referring to (8767aea),
silences the alloc errors because in evict mode they are expected and are handled by the njs code.

all the allocations after the change look like this:

static void *
ngx_js_dict_alloc(ngx_js_dict_t *dict, size_t n)
{
    void  *p;

    dict->shpool->log_nomem = !dict->evict;
    p = ngx_slab_alloc_locked(dict->shpool, n);
    dict->shpool->log_nomem = 1;

    if (p == NULL && dict->evict) {
        ngx_js_dict_evict(dict, 16);
        p = ngx_slab_alloc_locked(dict->shpool, n);
    }

    return p;
}

we are trying to allocate memory (suppressing error report for evict mode),
if allocations did not succeed we try to evict some elements and try again (this time we do not suppress error reporting).

We've only seen this occur one time now so I can't specifically name a way to reproduce this but I'd assume that you'd just have to overfill the memory zone.

we have tests for the overfilling (in nginx/t/js_shared_dict.t: /overflow and evict_stress tests).

Given that you saw it only once, and simple overfilling works fine, it seems we need a bit more leads before looking into it.
Feel free to report them as well.

Please, note that you can get 100% CPU due to erroneous JS code too.

[lancedockins]

Thanks, Dmitry. We'll watch for this to occur again. We don't have endless loops or anything in our JS code. Honestly there aren't a ton of loops in what we're doing in general. We've only ever seen Nginx spike to that sort of CPU use three times. Once was a previous NJS bug, once was related to Brotli compression, and this instance (which seems to concur with an overfull NJS memory zone).

Is there anything that you would recommend that would be particularly helpful to try to track this down assuming that it happens again? I won't be able to enable debug mode as these are production environments. The only thing that I could think to do would be something like using the memory zone size checks to determine free space/zone full situations on set and log that if we see it. But I don't know that that would really tell us more than what we know from the error log as it is.

[lancedockins]

@xeioex this is definitely proving to be a recurring issue. Since I reached out, we've upgraded our entire fleet to the latest NJS to silence messages about the zone being full/out of memory but we've seen several instances now where the shared dict zone fills up and then we see immense Nginx related CPU spikes. The spikes only subside if and when we restart the Nginx process. The commonality in every instance where this occurs is this error shows in the logs right before the CPU storm occurs:

2026/03/25 12:57:29 [alert] 2110902#2110902: ngx_slab_free(): chunk is already free in js shared zone "zone_name"
2026/03/25 12:57:29 [alert] 3820900#3820900: worker process 2110902 exited on signal 11 (core dumped)

There is definitely some sort of double memory free problem occurring with these zones. I included the details of the shared dict zone in my original post. It's the same zone that's running into issues.

For context, prior to a recent change in our timeout length, we had never seen this occur. Previously we were using 60s timeouts. When we extended the timeout to 30m, this started happening periodically on servers even with evict enabled. Just to try to reduce the propensity for this to happen, we've already increased the size of the memory zone. But it occurred again on another server (now like the 3rd+ time that this has happened) today.

How can we get to the bottom of this?

[xeioex]

@lancedockins

I will investigate
can you please confirm the previous version of njs and nginx with which everything worked fine?

[lancedockins]

All that I can say for sure is that we weren't having issues with this until we changed the longevity of the timeout. We've run the stack on both Nginx 1.29.5 w/ NJS 0.9.5 and 1.29.6 w/ NJS 0.9.6. The problem started on the same day that we updated the length of the zone's timeout. Prior to that we had had no issues on 1.29.5 w/ NJS 0.9.5 for weeks. So I don't know that I can specifically attribute this to a particular Nginx+NJS version. From all evidence that I have, the issue is tied to the timeout length.

[xeioex]

@lancedockins

OK, I will try to stress test the shared dict more.

[lancedockins]

If it's of any help, all that we're doing is caching data in the zone to avoid heavier lookups. JS use for this particular zone is limited to "has", "get", and "set". The flow is basically to check whether the zone has the key/value that we're looking for. If it doesn't, we grab it from the slower source and set it into the key/value in the zone. Otherwise if the key/value exists in the zone, we get the data from the zone. There are no clear/delete calls.

It's pretty vanilla use. But I thought that that might help if it's some sort of use-case specific bug that only occurs with that sort of flow + prolonged use of the zone.

[lancedockins]

Last detail that might be relevant. We seem to have only seen this on our higher traffic servers as well (both more total traffic and more variable traffic). Often these same servers have long running bot crawls (e.g. Googlebot) due to the volume of content on them as well. So it does at least seem like the issue is provoked by higher load.

[xeioex]

Hi @lancedockins,

I was able to reproduce the issue with added info (100% matches your description). The bug triggers in the set() update path when set() is called on a key that already exists while the zone is full.

In some rare cases, when evicting and having high traffic we may evict the node we are currently updating - which leads to double-free and worker crash. when a worker dies under a lock the lock remains forever locked explaining 100% for all nginx processes.

Mitigation:

1. reducing timeout, or enlarging the zone should help.
2. alternatively, if it works for your case

replace set() with add().

  // Instead of:
  if (!zone.has(key)) {
      zone.set(key, fetchValue(key));
  }
  var val = zone.get(key);

  // Use:
  var val = zone.get(key);
  if (val === undefined) {
      val = fetchValue(key);
      zone.add(key, val);  // no-op if another worker added it first
  }

add() returns false when the key already exists and never enters the update path, so it sidesteps the bug entirely.

[lancedockins]

Thank you, @xeioex. I'm glad that you were able to find and fix this. Really appreciate the help. I'll look into the workaround pending the bug fix.

That said, I think there is still an issue with the other error message in this thread. Maybe it's connected to the same bug so I'll defer to you on that. But on the same server where we saw the core dump associated with the shared memory zone filling yesterday, we're seeing tons of the below error messages today. That's despite the fact that we're on 1.29.6 and NJS 0.9.6 on that server right now.

Isn't this error supposed to be suppressed with the latest NJS updates if the zone uses evict?

2026/03/26 08:26:18 [crit] 1167100#1167100: ngx_slab_alloc() failed: no memory in js shared zone "zone_name"
2026/03/26 08:26:18 [error] 1167100#1167100: *60757 js exception: SharedMemoryError

[lancedockins]

Update. I did switch to .add for our use case and the same server that I mentioned earlier is still showing the nix_slab_alloc() failed and SharedMemoryError exception logs. The backtrace just refers to the add method now instead of the set method.

So it appears that there is still some sort of bug with at least error emits. But the SharedMemoryError exception at least strongly implies that the shared memory zone's code flow is being derailed by a full zone despite the use of evict.

We aren't currently using try/catch for this particular code flow but regardless of that, we wouldn't want the code to fail when adding to the shared dict if that is what is happening. The expected behavior would be that the add would succeed but the oldest values would be evicted.