Delegate deletion of Finished tasks to workers

[rakovskij-stanislav]

Hello!

I noticed that karton-system became irresponsible on 500k+ keys. I know, we should avoid such situations, but I was careless to send several millions tasks to Low priority for retro-scan :)

While troubleshooting how to deal with a situation of 27M tasks without deleting them from a queue, I noticed that 67% of tasks are in Finished state but karton-system's GC is too slow to both process new tasks and delete old ones.

I see that previously services asked karton-system to change status (https://github.com/CERT-Polska/karton/blob/master/karton/system/system.py#L284). By moving task state change to a task worker, you lifted extra signals to karton-system, and that's good.

I suggest to delete Finished task immediately as a part of worker routine.

Advantages:

• Speed up karton-system's GC stage -- both with deleting Finished tasks and searching for deletion candidates of S3 resources.

Disadvantages:

• We are not deleting tasks in bulk. BUT otherwise we need to change task status to Finished in a worker anyway and it's also not a bult operation.
• We lose a possibility to do something with Finished tasks in karton-system before gc-ing them.

Share with me your thoughts -- maybe you know severe issues with such approach?

[psrok1]

We recently had to deal with GC issues that completely stopped routing and the solution was to run separate instances of karton-system: one with enable_gc=False and another one with enable_router=False

The bottleneck was S3 listing, that's pretty slow operation. Especially if you have lots of objects there because of GC failure 🤦

I suggest to delete Finished task immediately as a part of worker routine.

I think we should analyze if there are any consequences, finished tasks and deleted/missing tasks are considered to be same thing in various places.

As you noticed, initial versions of Karton were serializing task status changes by delegating them to karton.system. That was considered to be the safest option that limited the risks of a race condition, so we don't bother ourselves with such bugs during early development. Then we analyzed it more thoroughly and made it more distributed. Maybe finished tasks are another case that can be optimized!

[rakovskij-stanislav]

Hi, @psrok1!

While handling the situation with GC, I also separated karton-system to two instances -- enable_router=False (called karton-system-gc in my docker stack) and enable_gc=False (karton-system-router), and it solved the situation with unrouted tasks.

I can suggest a trick that we can apply additionally with moving deletion of Finished task to worker -- we can also delete unpersistent resources at worker side.

Probably we also need to add a list of task uid's at redis karton:resource:{res_id} to track services that use a certain resource, this way we can speed up deals with s3 while avoiding frequent listing of s3 directory and iteration all task bodies. It can greatly help at GC stage, but we also need to implement it in a proper way to avoid race conditions.

[mak]

Hi folks,

From our experience, biggest problem when you have more than few hundred thousand tasks is collecting them from redis, this can be improved by using SCAN rather than GET, SCAN is fast but somehow inaccurate, works for us when we need to dump everything and examine the state tho

other approach might to change how gc work and i just submitted a poc for that ;]