✨🎨Computational backend: Automatically stop a running job if no logs are detected for 1h

[sanderegg]

What do these changes do?

As it happens more and more this is a measure to protect the platform from long running jobs that are probably hanging and it protects the user from paying for jobs that are hanged.

If a job does not produce any logs during 1h a timeout triggers and will stop the job.

Bonus:

• fixes [Feature]: Ensure manually aborted task also upload the logs if any #8569
• Show better logs when out of memory issues arise

Related issue/s

• fixes

How to test

Dev-ops

[codecov]

Codecov Report

❌ Patch coverage is 87.96296% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.47%. Comparing base (de246e2) to head (69968f7).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8549      +/-   ##
==========================================
+ Coverage   87.55%   89.47%   +1.92%     
==========================================
  Files        2012     1382     -630     
  Lines       78982    58411   -20571     
  Branches     1369      187    -1182     
==========================================
- Hits        69151    52264   -16887     
+ Misses       9425     6088    -3337     
+ Partials      406       59     -347     

Flag	Coverage Δ
integrationtests	63.87% <ø> (-0.04%)	⬇️
unittests	87.75% <87.96%> (+1.46%)	⬆️

Components	Coverage Δ
pkg_aws_library	∅ <ø> (∅)
pkg_celery_library	∅ <ø> (∅)
pkg_dask_task_models_library	79.37% <100.00%> (+0.36%)	⬆️
pkg_models_library	∅ <ø> (∅)
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	∅ <ø> (∅)
pkg_service_library	∅ <ø> (∅)
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	84.95% <ø> (ø)
agent	93.10% <ø> (ø)
api_server	91.35% <ø> (ø)
autoscaling	95.83% <ø> (ø)
catalog	92.06% <ø> (ø)
clusters_keeper	99.14% <ø> (ø)
dask_sidecar	91.51% <86.73%> (-0.32%)	⬇️
datcore_adapter	97.95% <ø> (ø)
director	75.72% <ø> (ø)
director_v2	90.90% <ø> (ø)
dynamic_scheduler	96.66% <ø> (ø)
dynamic_sidecar	90.44% <ø> (ø)
efs_guardian	89.83% <ø> (ø)
invitations	90.90% <ø> (ø)
payments	92.80% <ø> (ø)
resource_usage_tracker	92.16% <ø> (+0.05%)	⬆️
storage	86.84% <ø> (-0.09%)	⬇️
webclient	∅ <ø> (∅)
webserver	87.06% <ø> (+0.06%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update de246e2...69968f7. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for 69968f7.

✅ Passed Jobs With Interesting Signals

Pipeline	Job	Signal	Health on master	Retries	🔍 CI Insights	📄 Logs
CI	system-tests	You had a 25% chance of failing… lucky you! 🎲		Configure an automatic retry	View	View
	unit-tests	You had a 50% chance of failing… lucky you! 🎲		Configure an automatic retry	View	View

[Copilot]

Pull Request Overview

This PR implements automatic termination of computational jobs that produce no logs for 1 hour, protecting the platform from hanging jobs and preventing unnecessary charges. It also improves error handling for out-of-memory scenarios with better log visibility.

Key changes:

• Added configurable timeout (DASK_SIDECAR_MAX_LOG_SILENCE_TIMEOUT) for monitoring service log activity
• Introduced new exception types (ServiceTimeoutLoggingError, ServiceOutOfMemoryError) with detailed error messages
• Enhanced error handling and logging throughout the computational sidecar execution flow

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
services/dask-sidecar/src/simcore_service_dask_sidecar/settings.py	Added configuration for maximum log silence timeout (default: 1 hour)
packages/dask-task-models-library/src/dask_task_models_library/container_tasks/errors.py	Introduced new exception hierarchy with ServiceTimeoutLoggingError and ServiceOutOfMemoryError
services/dask-sidecar/src/simcore_service_dask_sidecar/computational_sidecar/docker_utils.py	Implemented timeout detection in log monitoring and improved error handling
services/dask-sidecar/src/simcore_service_dask_sidecar/computational_sidecar/core.py	Added OOMKilled detection and enhanced error reporting for service execution
services/dask-sidecar/src/simcore_service_dask_sidecar/worker.py	Added exception handler to avoid dask serialization issues
services/dask-sidecar/tests/unit/test_computational_sidecar_tasks.py	Added comprehensive tests for timeout and OOM scenarios
services/dask-sidecar/tests/unit/conftest.py	Added RAM resources to test cluster configuration

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[matusdrobuliak66]

Thanks a lot

[pcrespov]

thx!

[GitHK]

thanks

[sanderegg]

@mergify queue

[mergify]

queue

🛑 Configuration not compatible with a branch protection setting

The branch protection setting Require branches to be up to date before merging is not compatible with draft PR checks. To keep this branch protection enabled, update your Mergify configuration to enable in-place checks: set merge_queue.max_parallel_checks: 1, set every queue rule batch_size: 1, and avoid two-step CI (make merge_conditions identical to queue_conditions). Otherwise, disable this branch protection.

[matusdrobuliak66]

@sanderegg, maybe we should consider this an issue on our side and not bill the user? This can be done automatically if you send an ERROR state in the Rabbit message that is sent to RUT.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud