bodhi composer runs out of open files after some time

[nirik]

 Active: active (running) since Sat 2025-08-30 03:37:42 UTC; 4 days ago

: ERROR/ForkPoolWorker-16] Task clean_old_composes[937e35d9-e594-45d1-b90c-6e041231955d] raised unexpected: OSError(24, 'Too many open files')
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]: Traceback (most recent call last):
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:   File "/usr/lib/python3.13/site-packages/celery/app/trace.py", line 477, in trace_task
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:     R = retval = fun(*args, **kwargs)
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:                  ~~~^^^^^^^^^^^^^^^^^
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:   File "/usr/lib/python3.13/site-packages/celery/app/trace.py", line 760, in __protected_call__
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:     return self.run(*args, **kwargs)
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:            ~~~~~~~~^^^^^^^^^^^^^^^^^
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:   File "/usr/lib/python3.13/site-packages/bodhi/server/tasks/__init__.py", line 123, in clean_old_composes_task
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:     main(num_to_keep)
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:     ~~~~^^^^^^^^^^^^^
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:   File "/usr/lib/python3.13/site-packages/bodhi/server/tasks/clean_old_composes.py", line 68, in main
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:     shutil.rmtree(d)
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:     ~~~~~~~~~~~~~^^^
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:   File "/usr/lib64/python3.13/shutil.py", line 763, in rmtree
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:     _rmtree_safe_fd(stack, onexc)
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:     ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:   File "/usr/lib64/python3.13/shutil.py", line 707, in _rmtree_safe_fd
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:     onexc(func, path, err)
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:     ~~~~~^^^^^^^^^^^^^^^^^
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:   File "/usr/lib64/python3.13/shutil.py", line 682, in _rmtree_safe_fd
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:     with os.scandir(topfd) as scandir_it:
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]:          ~~~~~~~~~~^^^^^^^
Sep 03 03:03:00 bodhi-backend01.rdu3.fedoraproject.org celery-3[1169191]: OSError: [Errno 24] Too many open files: '/mnt/koji/compose/updates/Fedora-41-updates-20250824.0/compose/Everything/x86_64'

wc -l /tmp/apache-open-files
18654 /tmp/apache-open-files

seems to be a lot of pipes?
celery-3 1169175 apache 17w FIFO 0,15 0t0 48983819 pipe