S3 cleanup needs a full redesign

[ivukotic]

Minio cleanup did not work as minio API always return only first 1000 objects in the list.
I have created a new cleanup code that relies on AWS boto3 API.
It now lists all the objects in the bucket but listing buckets returns only first 1000.
This is obviously a problem...
I have made a code that iteratively deletes empty buckets and buckets over the time limit.
Then it deletes oldest 500 buckets in the list of 1000, and repeats the process until we are under 1000 basket limit.

Once we have tens of users and some of them create hundreds or requests this all becomes a huge problem.

We have three solutions:

• use a single bucket and have requestId as a prefix to the object name
• use a single bucket but put object in "subdirectories"
• we use multiple buckets as is the case now, but cleanup connects to postgresql DB to find what are all the buckets.

[ponyisi]

Is this addressed in #811 ?

[ivukotic]

yes