Resource Efficiency & Performance

[KevinW1998]

Hi! I am interested in this platform, but I have some questions about its overall performance.

How well does the MWDB perform when storing large numbers of samples (up to millions)? Has it been used in this way before?

Is there any reason why hashes are stored as hex-text rather than binary? https://github.com/CERT-Polska/mwdb-core/blob/master/mwdb/model/file.py#L32-L37
Wouldn't storing it in binary be more resource efficient?

The same question applies to blobs: https://github.com/CERT-Polska/mwdb-core/blob/master/mwdb/model/blob.py#L49.

_content=content.encode("unicode_escape").decode("utf-8")

Escaping a byte would take 4–5 bytes.

[psrok1]

Hello!

How well does the MWDB perform when storing large numbers of samples (up to millions)? Has it been used in this way before?

Of course, it's a backend for our https://mwdb.cert.pl instance that is storing milions of samples. The most important factor is the performance of the database - PostgreSQL in our case.

Is there any reason why hashes are stored as hex-text rather than binary? https://github.com/CERT-Polska/mwdb-core/blob/master/mwdb/model/file.py#L32-L37. Wouldn't storing it in binary be more resource efficient?

I don't know and I've never benchmarked that. Hex-text was always good enough in our case, so if there's no visible bottleneck, it would be a premature optimization to change that.

The same question applies to blobs: https://github.com/CERT-Polska/mwdb-core/blob/master/mwdb/model/blob.py#L49.
_content=content.encode("unicode_escape").decode("utf-8")

That one is a dirty workaround for PostgreSQL limitations that doesn't allow to store arbitrary bytes or codepoints in text fields. Blob is meant to be a text file, but world isn't ideal so we assume that it may contain NULL bytes and other weird bytes as well. PostgreSQL doesn't allow to store such bytes in text/varchar fields. We could use bytea for that, but that would be a very significant change that would change the hashes of existing objects.

[ya-solnyshko]

Hi,

Hex-text was always good enough in our case, so if there's no visible bottleneck, it would be a premature optimization to change that.

The use of strings doubles the amount of storage used for hashes, so just sha256 for 100 million samples adds additional 3GB of used storage, and it must have a similar effect on the index size.

I don't know and I've never benchmarked that.

That is a good point, I think that providing some kind of a benchmark would be beneficial for the community. If you have a running database with millions of samples, it should be easy for you to check an average insertion/lookup time, and give some average storage requirements for the database. That also can help to evaluate the applicability of mwdb for use-cases different from pure malware, as it provides a convenient platform for storing, processing and indexing any type of files with their metadata.

[yankovs]

We also have an instance that is storing millions of samples. Deployed on AWS using a pretty beefy RDS instance. Please note that what I'm saying is based on MWDB v12.2.0. Things may be different in the newest release.

The vast majority of sample queries (viewing a sample in MWDB) are near instant on the postgres side in our case. The only places where I've seen noticeable slowdowns in the general user expereicen are: (1) samples with a lot (10,000+) of relations, or (2) searching in general, specifically searching on attributes. We replicate the data in elasticsearch for this reason

If you have specific questions, I can try to answer

[psrok1]

I did some benchmarks with BYTEA hashes and I don't think it's worth it.

• our database has 26 million of objects (mostly samples)
• before conversion: 17GB on table data, 23GB on index.
• after conversion: 12GB on table data, 18GB on index.

But it's not for free. API still operates on hexlified hashes so we need to have conversions everywhere which adds some complexity to the code. But the worst thing are searches: we lose the direct possibility to make a LIKE query when someone queries for a sample, knowing only the partial hash value.

A very nice feature of MWDB is that even if I have a screenshot of an IoC that looks like this:

I can still search for the sample using query like that: sha256:"25e2c799475058c07*f9fd73f11f9c". Of course you can still find a way to make such query even for BYTEA fields, but as I said before - cost of added complexity is just not worth the benefits.

The thing that I would still experiment with is keeping the blob content in BYTEA field instead of doing our weird escaping. We would still need to do it during hashing (to keep the ids unchanged), but at least the representation will be more simple.