feat(worker): make qgis jobs defined in a single file #1367

suricactus · 2025-09-21T12:01:52Z

This PR addressed multiple problems:

job definition requires changes in multiple files - entrypoint.py, custom file etc
extending QFieldCloud with custom jobs is impossible unless entrypoint.py is overwritten
testability of the qgis worker as a standalone module without going through the worker wrapper is cumbersome
some files (e.g. apply_deltas.py) defined two CLI interfaces to be called - one via entrypoint.py, and one by running the file itself.

duke-nyuki · 2025-09-21T12:01:56Z

Task linked: QF-6714 Bump to the latest libqfieldsync version in the workers
Task linked: QF-6803 Make the job parser in QFieldCloud qgis workers extendible so new jobs can be added with a single file change

The utils.py is way too complex with all the functionality in there.

Make sure the steps in a workflow are always unique.

Used by projects based on QFieldCloud but need to modify the existing jobs.

Instead of having `QfcBaseWorkflow.handle` to take care of building the workflow and making sure the `feedback.json` is generated, we make the `handle` still prepare the `feedback.json`, but now in the base class and the new method `get_workflow` must be implemented in the children. This also allow for easier extension of already existing jobs.

…ting all of them in `_run_docker`

…that returns job mapping

…ing the job type

it should always been `upload_project_directory` as there is no export

suricactus · 2025-10-02T00:19:51Z

docker-app/qfieldcloud/core/management/commands/dequeue.py

+        return {
            Job.Type.PACKAGE: PackageJobRun,
-            Job.Type.DELTA_APPLY: DeltaApplyJobRun,
+            Job.Type.DELTA_APPLY: ApplyDeltaJobRun,


I promise one day we also change the DELTA_APPLY name with APPLY_DELTA. I am sorry for the dislexic name I introduced back then.

Noted that down for the follow up task to make the Job.Type extensible (so that custom jobs are correctly displayed in the admin)

Do you already have CU task? Can you post a link here?

docker-qgis/qfc_worker/commands_base.py

Copilot

Pull Request Overview

This PR restructures the QGIS worker to consolidate job definitions into a single file architecture. It addresses the problem where job definitions required changes across multiple files and makes extending QFieldCloud with custom jobs easier.

Consolidates QGIS worker job definitions from multiple scattered files into a unified command-based architecture
Introduces a new workflow system that enables better testability and extensibility
Restructures entrypoint.py to use the new command system instead of hardcoded functions

Reviewed Changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
scripts/check_envvars.sh	Updates environment variable exclusion list for new debug variable
docker-qgis/qfc_worker/workflow.py	New workflow execution framework with step-based processing
docker-qgis/qfc_worker/utils.py	Removes workflow classes moved to separate workflow.py file
docker-qgis/qfc_worker/exceptions.py	New centralized exception definitions extracted from utils.py
docker-qgis/qfc_worker/commands_base.py	New command registration system and base class for QGIS commands
docker-qgis/qfc_worker/commands/*.py	Refactored job implementations using the new command architecture
docker-qgis/entrypoint.py	Simplified to delegate to the new command system
docker-compose.override.local.yml	Updates debug environment variables for new development setup
docker-app/worker_wrapper/wrapper.py	Refactored container execution and debug mounting logic
docker-app/qfieldcloud/settings.py	Consolidated debug path settings
docker-app/qfieldcloud/core/tests/test_delta.py	Minor test improvements for better error reporting
docker-app/qfieldcloud/core/models.py	Added explicit project_id type annotation
docker-app/qfieldcloud/core/management/commands/dequeue.py	Updated job class references and improved structure
.env.example	Updated environment variable documentation

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

suricactus · 2025-10-08T23:15:51Z

@boardend can you have a look on this PR this week or the next week latest?

boardend · 2025-10-09T18:14:40Z

Sure, will review and test over the weekend

boardend

LGTM 😍

boardend · 2025-10-13T09:26:30Z

docker-app/qfieldcloud/core/management/commands/dequeue.py

+        return {
            Job.Type.PACKAGE: PackageJobRun,
-            Job.Type.DELTA_APPLY: DeltaApplyJobRun,
+            Job.Type.DELTA_APPLY: ApplyDeltaJobRun,


Noted that down for the follow up task to make the Job.Type extensible (so that custom jobs are correctly displayed in the admin)

docker-qgis/qfc_worker/commands_base.py

docker-qgis/qfc_worker/exceptions.py

suricactus · 2025-10-15T08:14:15Z

@boardend thanks for the review!

Base automatically changed from QF-6714-libqfieldsync-followup to master September 23, 2025 22:50

suricactus force-pushed the QF-6803-extendible-jobs branch 2 times, most recently from a41db4a to 04bf260 Compare September 24, 2025 00:44

suricactus changed the base branch from master to bump_dependencies September 24, 2025 00:45

suricactus changed the title ~~feat(worker): make qgis jobs defined in a single file~~ [WIP] feat(worker): make qgis jobs defined in a single file Sep 24, 2025

suricactus force-pushed the QF-6803-extendible-jobs branch from e9c77f0 to 08d6500 Compare September 30, 2025 00:15

Base automatically changed from bump_dependencies to master October 1, 2025 11:07

suricactus force-pushed the QF-6803-extendible-jobs branch 3 times, most recently from 8741d20 to f10e6a9 Compare October 1, 2025 19:24

suricactus added 13 commits October 1, 2025 22:24

feat(worker): make qgis jobs defined in a single file

06f3508

refactor(worker): move workflow stuff away from utils.py

a8ca196

The utils.py is way too complex with all the functionality in there.

feat: validate for workflow step id uniqueness

7024fdb

Make sure the steps in a workflow are always unique.

feat(worker): add the ability to insert steps in the Workflow

168de52

Used by projects based on QFieldCloud but need to modify the existing jobs.

typing: add typing that Job.project_id exists

455af2b

feat(worker): add get_volumes/environment/ports instead of crea…

0635f54

…ting all of them in `_run_docker`

feat(worker): make the jobs easily extendible by making a new method …

c17dabf

…that returns job mapping

refactor(test): make the test_delta.QfcTestCase.fail typed and show…

14c2b9d

…ing the job type

refactor(worker): rename wrongly named step in apply_deltas

f14fa74

it should always been `upload_project_directory` as there is no export

refactor: simplify volume loading and DEBUG variables

7a8ce36

refactor(worker): move the command registry to a metaclass

e2c6afd

refactor(worker): be more explicit about the docker types

30c5e52

suricactus force-pushed the QF-6803-extendible-jobs branch from f10e6a9 to 30c5e52 Compare October 1, 2025 19:24

suricactus commented Oct 2, 2025

View reviewed changes

docker-qgis/qfc_worker/commands_base.py Show resolved Hide resolved

suricactus changed the title ~~[WIP] feat(worker): make qgis jobs defined in a single file~~ feat(worker): make qgis jobs defined in a single file Oct 2, 2025

suricactus requested review from boardend and Copilot October 2, 2025 00:23

Copilot AI reviewed Oct 2, 2025

View reviewed changes

suricactus marked this pull request as ready for review October 2, 2025 00:24

boardend approved these changes Oct 13, 2025

View reviewed changes

suricactus merged commit 74dabaa into master Oct 15, 2025
21 checks passed

suricactus deleted the QF-6803-extendible-jobs branch October 15, 2025 08:14

Uh oh!

feat(worker): make qgis jobs defined in a single file #1367

feat(worker): make qgis jobs defined in a single file #1367

Uh oh!

Conversation

suricactus commented Sep 21, 2025

Uh oh!

duke-nyuki commented Sep 21, 2025

Uh oh!

suricactus Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

boardend Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

suricactus Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

suricactus commented Oct 8, 2025

Uh oh!

boardend commented Oct 9, 2025

Uh oh!

boardend left a comment

Choose a reason for hiding this comment

Uh oh!

boardend Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

suricactus commented Oct 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants