⚠️🎨✨ Execute the api-server's run and map function api endpoints in celery worker

[bisgaard-itis]

This is a mega PR which introduces a celery worker into the api-server based on our wonderful celery_library.

What do these changes do?

• Introduce a celery worker associated with the api-server
• Modify the tasks section of the api-server to use the celery task_manager directly.
• Modify the following two endpoints so that they are handled by the new celery worker:

• As discussed with @wvangeit the implemention "hides" the celery_worker from the user in the sense that the user gets a function_job/function_collection back immediately when running the run/map endpoints respectively, and when the user requests the status of the job, the api-server keeps track of whether or not celery task is still running or the backend project has already been created.

• @wvangeit note the following points:

  • that the job_ids of the job_collection which is returned by the map endpoint don't have the same ordering as the job_inputs (because of the way the caching is done. Since the job_inputs are stored with the job itself I figured that is not really required, but please let me know if that conclusion is wrong.
  • In the map endpoint there is still a python loop over the job inputs which is executed before the response is returned in the endpoint. This is because of the caching. We could try to avoid this if you feel it is important. Let me know

Related issue/s

• closes Use Celery workers in the run/map function calls (Voyager) osparc-issues#1973

How to test

From inside /services/api-server run

make install-dev
python -m pytest tests/unit/api_functions

Dev-ops

• We need to setup an api-worker service on the ops-side before deploying this, see Setup api-worker service on deployments osparc-ops-environments#1192

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for 9172a39.

✅ Passed Jobs With Interesting Signals

Pipeline	Job	Signal	Health on master	Retries	🔍 CI Insights	📄 Logs
CI	integration-tests	Base branch is healthy, but retries were needed. Could be early signs of flakiness 👀		1	View	View
	system-tests	Base branch is healthy, but retries were needed. Could be early signs of flakiness 👀		1	View	View

[codecov]

Codecov Report

❌ Patch coverage is 88.02589% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.86%. Comparing base (8768457) to head (9172a39).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8233      +/-   ##
==========================================
- Coverage   87.89%   87.86%   -0.03%     
==========================================
  Files        1933     1939       +6     
  Lines       74941    75186     +245     
  Branches     1309     1309              
==========================================
+ Hits        65867    66062     +195     
- Misses       8681     8731      +50     
  Partials      393      393              

Flag	Coverage Δ
integrationtests	64.13% <ø> (+0.02%)	⬆️
unittests	86.53% <88.02%> (-0.02%)	⬇️

Components	Coverage Δ
pkg_aws_library	93.59% <ø> (ø)
pkg_celery_library	87.37% <ø> (ø)
pkg_dask_task_models_library	79.62% <ø> (ø)
pkg_models_library	93.11% <100.00%> (+0.02%)	⬆️
pkg_notifications_library	85.20% <ø> (ø)
pkg_postgres_database	88.02% <ø> (ø)
pkg_service_integration	70.19% <ø> (ø)
pkg_service_library	71.09% <0.00%> (-0.10%)	⬇️
pkg_settings_library	90.17% <71.42%> (ø)
pkg_simcore_sdk	85.03% <ø> (ø)
agent	93.53% <ø> (ø)
api_server	91.93% <91.33%> (-0.81%)	⬇️
autoscaling	95.77% <ø> (ø)
catalog	92.34% <ø> (ø)
clusters_keeper	99.13% <ø> (ø)
dask_sidecar	92.37% <ø> (+0.22%)	⬆️
datcore_adapter	97.94% <ø> (ø)
director	75.81% <ø> (ø)
director_v2	90.93% <ø> (-0.07%)	⬇️
dynamic_scheduler	96.27% <ø> (ø)
dynamic_sidecar	90.46% <ø> (ø)
efs_guardian	89.62% <ø> (ø)
invitations	91.44% <ø> (ø)
payments	92.61% <ø> (ø)
resource_usage_tracker	92.08% <ø> (+0.53%)	⬆️
storage	86.57% <ø> (+0.08%)	⬆️
webclient	∅ <ø> (∅)
webserver	88.08% <ø> (+<0.01%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8768457...9172a39. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sanderegg]

Do not forget to create the MRs for OPS (also for the ENVs) and please check my last comments.

[pcrespov]

thx

[GitHK]

Thanks, OK with me

[sonarqubecloud]

Quality Gate failed

Failed conditions
5.2% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

[bisgaard-itis]

@Mergifyio queue

[mergify]

queue

🟠 Waiting for conditions to match

• -closed [📌 queue requirement]
• any of: [🔀 queue conditions]
  • all of: [📌 queue conditions of queue default]
    • branch-protection-review-decision = APPROVED [🛡 GitHub branch protection]
    • any of: [🛡 GitHub branch protection]
      • check-neutral = SonarCloud Code Analysis
      • check-skipped = SonarCloud Code Analysis
      • check-success = SonarCloud Code Analysis
    • #approved-reviews-by >= 2 [🛡 GitHub branch protection]
    • #approved-reviews-by>=2
    • #changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
    • #changes-requested-reviews-by=0
    • #review-threads-unresolved = 0 [🛡 GitHub branch protection]
    • #review-threads-unresolved=0
    • -conflict
    • -draft
    • base=master
    • label!=🤖-do-not-merge
    • label=🤖-automerge
    • any of: [🛡 GitHub branch protection]
      • check-skipped = deploy to dockerhub
      • check-neutral = deploy to dockerhub
      • check-success = deploy to dockerhub
    • any of: [🛡 GitHub branch protection]
      • check-success = system-tests
      • check-neutral = system-tests
      • check-skipped = system-tests
    • any of: [🛡 GitHub branch protection]
      • check-success = unit-tests
      • check-neutral = unit-tests
      • check-skipped = unit-tests
    • any of: [🛡 GitHub branch protection]
      • check-success = check OAS' are up to date
      • check-neutral = check OAS' are up to date
      • check-skipped = check OAS' are up to date
    • any of: [🛡 GitHub branch protection]
      • check-success = integration-tests
      • check-neutral = integration-tests
      • check-skipped = integration-tests
    • any of: [🛡 GitHub branch protection]
      • check-success = build-test-images (frontend) / build-test-images
      • check-neutral = build-test-images (frontend) / build-test-images
      • check-skipped = build-test-images (frontend) / build-test-images
• -conflict [📌 queue requirement]
• -draft [📌 queue requirement]
• any of: [📌 queue -> configuration change requirements]
  • -mergify-configuration-changed
  • check-success = Configuration changed