[Performance] S3Boto3StorageWrapper caches file per-analyzer instead of sharing a single cached copy

[mannubaveja007]

What happened

In intel_owl/settings/storage.py, the S3Boto3StorageWrapper.retrieve() method caches
downloaded S3 files under MEDIA_ROOT/<analyzer>/<filename>. This means the file
The file from S3 is downloaded and is stored separately for each process.

If 10 analyzers are set to run on the same sample, the file will be retrieved from the disk, written to the HDD or sent over the network 10 times.

Environment

1. OS: Any
2. IntelOwl version: latest develop

What did you expect to happen

The file should be cached once at MEDIA_ROOT/ and then reused by all the analyzers.
avoiding redundant S3 HTTP requests and duplicate disk writes.

How to reproduce your issue

1. Set LOCAL_STORAGE=False (S3 mode in env_file_app)
2. Upload a file and run multiple analyzers on it
3. Inspect MEDIA_ROOT/ — a separate subdirectory per analyzer is created with identical file content.

Error messages and logs

No error. This is a known FIXME left by maintainers:
https://github.com/intelowlproject/IntelOwl/blob/develop/intel_owl/settings/storage.py#L37

[mlodic]

considering you know how to reproduce the issue, do it and show proof that this is indeed a problem

[mannubaveja007]

considering you know how to reproduce the issue, do it and show proof that this is indeed a problem

working on it just allow me this sunday!

[mannubaveja007]

ran the benchmark test and this is the results PR is ready

1. Before

[BENCHMARK] N=20 events → 61 SQL queries
FAIL: decay() ran 61 queries for N=20; expected O(1), got O(N).

2. After

[BENCHMARK] N=20 events → 4 SQL queries
ok

[mlodic]

@mannubaveja007 are you still working on this? if yes please open a new draft PR within a week

[mannubaveja007]

@mannubaveja007 are you still working on this? if yes please open a new draft PR within a week

It's ready I will just push the things and create a proper PR to merge

[mannubaveja007]

sorry for late

thank you for giving me a chance!