ERROR: Failed to retrieve the next key.

[nitriques]

I had the 6.1Gb file and tried to shrink it with this tools.
I was able to process 1250 records and failed with:

modsec-sdbm-util.c:332:dump_database(): Failed to retrieve the next key.

I then move the 6.1Gb file and let mod_sec recreate the file. But I still get the same error when running either -x or -k, on the newly created file.

I tried this on the lastest master branch.

[bostrt]

Interesting, I've had some large modsec db files before but haven't run into this before unless the db files are corrupted. Are you able to perform any other operations on the database or does it result in the same error?

[zimmerle]

@nitriques this is an exception or that is happening consistently with different databases?

[nitriques]

@bostrt Can you tell me how can I perform other operation on the database ?

@zimmerle consistenly, even with fresh/new databases that are a couple of minutes old.

I had a 6.1Gb file this morning also. so it grows at 1.2Gb per day right now.

I do run OWASP ruleset BTW.

[bostrt]

@nitriques I recommend creating a copy of the db that you can reproduce issues with. Try running the shrink operation (-k) or extract (-D). Basically, anything else besides dumping; just curious if the issue is isolated to dump.

I tried this on the lastest master branch.

Did you test with latest ModSecurity master branch?

[nitriques]

@bostrt Thanks for the help

I just copied the existing ip.pag file. Then:

# /root/modsec-sdbm-util/modsec-sdbm-util -k ip.pag.bak
Checking target directory: /tmp
Target directory exists.
Opening file: ip.pag.bak
Failed to open sdbm: ip.pag.bak

# /root/modsec-sdbm-util/modsec-sdbm-util -D ip.pag.bak

-D yeilds no output.

Did you test with latest ModSecurity master branch?

No, I can't. I have

Server version: Apache/2.2.34 (Unix)
Server built:   Aug 28 2017 14:35:15
Cpanel::Easy::Apache v3.34.16 rev9999

And from what I can tell from this page it is running 2.9

[bostrt]

@nitriques I think you are running into an issue with how modsec-sdbm-util picks up files by name:

Failed to open sdbm: ip.pag.bak
Try dropping the .bak extension like this:

# mkdir /tmp/sdbm
# cp ip.pag.bak /tmp/sdbm/ip.pag
# cp ip.dir /tmp/sdbm/ip.dir

Note that I copied in the ip.dir file too. The database is split into two the .pag and .dir so both are required to get some output.

Basically, I'm curious if you can shrink (-k) or extract (-n) the database and then run the dump operation.

Testing with the latest ModSecurity would probably be best next step, but sounds like that might not be an option. FYI, latest version is ModSecurity 2.9.2: https://github.com/SpiderLabs/ModSecurity/releases/tag/v2.9.2

[nitriques]

FYI, latest version is ModSecurity 2.9.2:

I still can't find how to get the version I am running :(

# /root/modsec-sdbm-util/modsec-sdbm-util -D /tmp/ip.pag

Still nothing

# /root/modsec-sdbm-util/modsec-sdbm-util -k ip.pag
Checking target directory: /tmp
Target directory exists.
Opening file: ip.pag
Database ready to be used.
 [/] 700 records so far.
Total of 700 elements processed.
700 elements removed.
Expired elements: 700, inconsistent items: 0
Fragmentation rate: 100.00% of the database is/was dirty data.

And then,

# /root/modsec-sdbm-util/modsec-sdbm-util -x ip.pag
Checking target directory: /tmp
Target directory exists.
Opening file: ip.pag
Database ready to be used.

Should I run /root/modsec-sdbm-util/modsec-sdbm-util -k ip.pag via cron ?

[nitriques]

With -v I still see

modsec-sdbm-util.c:163:modsec_unpack(): Expired: 1508449508, 1508449128 delta: -380
modsec-sdbm-util.c:332:dump_database(): Failed to retrieve the next key.

[bostrt]

I still can't find how to get the version I am running :(

You should see a notice in your main Apache log stating the version. Something like this:

[Fri Oct 20 18:43:49.298214 2017] [:notice] [pid 25472:tid 140137344624832] ModSecurity for Apache/2.9.1 (http://www.modsecurity.org/) configured.
[Fri Oct 20 18:43:49.298220 2017] [:notice] [pid 25472:tid 140137344624832] ModSecurity: APR compiled version="1.5.2"; loaded version="1.6.2"
[Fri Oct 20 18:43:49.298222 2017] [:warn] [pid 25472:tid 140137344624832] ModSecurity: Loaded APR do not match with compiled!
[Fri Oct 20 18:43:49.298242 2017] [:notice] [pid 25472:tid 140137344624832] ModSecurity: PCRE compiled version="8.40 "; loaded version="8.41 2017-07-05"

Yours will look a bit different b/c above is from Apache 2.4 and you are using 2.2.

So I'm not sure of why exactly is causing the Failed to retrieve the next key, has got to be some sort of database corruption/badness.

700 elements removed.
Expired elements: 700, inconsistent items: 0
Fragmentation rate: 100.00% of the database is/was dirty data.

This is promising though and potentially a workaround. You can try shrinking the database (-k) or extracting it to a new file (-n) occasionally using a cron job. However, some funny race conditions might come up doing a extract (-n) while Apache is up-and-running trying to access the ModSecurity db. I advise a cron job that stops Apache, extracts, starts Apache back up.

[nitriques]

Thanks for your help @bostrt

You should see a notice in your main Apache log stating the version. Something like this:

I did, and could not find anything :(

Thanks for the idea about the cron job.