Set up docker with dispatcherd for local development #18

abikouo · 2025-07-04T15:35:10Z

Add Docker compose configuration with PostgreSQL service, Worker service, and API/Ui service

Note: #25 has to be merged first

GomathiselviS · 2025-07-10T15:34:47Z

Hi @abikouo, I couldn't get the migrate command to work with manage.py inside src/pattern_service/. I tested moving manage.py to the src/ directory and adjusting its references, which fixed all the builds (docker build and docker compose) and migration.

Is there a step I'm overlooking, or should manage.py be located in the src/ directory?

hakbailey

In addition to the inline comments, I would suggest we remove the following from this PR:

Movement of code to the src directory. There are ongoing conversations org-wide about how we want to standardize platform service repository structure; let's hold off on making big changes in this regard until we know what those decisions are.
Dependency management with Poetry. This isn't standard for platform services either; let's stick with pyproject.toml for now.
Bash files to run tests and mypy, this shouldn't be necessary.
Addition of copyright statements to individual files. We may want to do this; however I would suggest we do it in a single PR to all existing files because adding it in this PR to only some files is introducing inconsistency across the repo. I would also like to verify what the org-wide guideline is for this.

Makefile

dispatcher_dev.yml

src/pattern_service/core/tasks/worker.py

src/pattern_service/core/utils/logging.py

src/pattern_service/core/views.py

hakbailey · 2025-07-18T00:08:36Z

tools/docker/README.md

@@ -0,0 +1,51 @@
+# Docker Compose for Development


There is a push to move towards podman over docker within Red Hat generally; let's try to refer either to podman or just containers generally instead of specifying docker.

abikouo · 2025-07-21T13:21:32Z

@hakbailey I have updated the PR based on what you did on #25. #25 has to be merged first.

tools/docker/Dockerfile

alinabuzachis · 2025-07-21T14:48:08Z

tools/docker/Dockerfile

+
+ENV PATTERN_SERVICE_MODE=development
+
+RUN python3.11 manage.py migrate


I suggest you run the migration when the container starts because this adds a read-only layer.

I was able to create a read-write file for the db, what is the error on your side?

I wasn't able to add anything else to the db because the migration was run in a read only layer. When creating a pattern I was getting something like "read-only database".

django.db.utils.OperationalError: attempt to write a readonly database

sonarqubecloud · 2025-07-24T23:58:07Z

Quality Gate failed

Failed conditions
1 Security Hotspot

See analysis details on SonarQube Cloud

hakbailey

It would be good to have the test endpoint submit a task to the dispatcher worker so we can verify this functionality; something simple like this would be fine.

I think it would also be valuable to update the logging section of the development settings to include dispatcherd DEBUG logs.

.dockerignore

Makefile

tools/docker/docker-compose.yaml

tools/docker/README.md

hakbailey · 2025-07-29T17:43:40Z

pattern_service/settings/defaults.py

+DISPATCHER_CONFIG = {
+    "version": 2,
+    "service": {
+        "pool_kwargs": {"min_workers": 2, "max_workers": 12, "scaledown_wait": 15},


I'm not sure we know yet what reasonable values will be for any of this. I would vote that we leave the default values for everything that we don't have to explicitly define here and then update them as needed. The settings used for EDA may not be relevant to our service, they're doing different kinds of async tasks.

I have updated the configuration accordingly

hakbailey · 2025-07-29T17:45:09Z

pattern_service/settings/dispatcher.py

+
+
+def override_dispatcher_settings(loaded_settings: Dynaconf) -> None:
+    feature_dispatcherd = convert_to_bool(loaded_settings.get("FEATURE_DISPATCHERD"))


I'm not sure why this conditional is here either, FEATURE_DISPATCHERD isn't a setting in our app which means this function doesn't currently do anything. The way our settings are written write now, I'm not sure this override function is even needed. We will need to add it when we start pulling some of these values from env variables or files, but at the moment it doesn't seem necessary.

The override here is to switch between SQLite DB and PostgreSQL DB while running with the dispatcher. We can remove that if we assume that the default behavior is with the dispatcher.
This will also break the current deployment on aap-dev.

pyproject.toml

hakbailey · 2025-07-29T20:20:10Z

tools/docker/docker-compose.yaml

@@ -0,0 +1,84 @@
+x-environment: &common-env
+  PATTERN_SERVICE_DB_HOST: postgres


Are these env variables used for anything?

These environment variables will be part of the settings without the PATTERN_SERVICE prefix. They are used to configure the DB.

hakbailey · 2025-07-29T20:21:16Z

tools/docker/docker-compose.yaml

+    image: localhost/ansible-pattern-service-worker
+    environment:
+      << : *common-env
+      PATTERN_SERVICE_DISPATCHER_NODE_ID: pattern-service-worker


Does this env variable get used for anything?

This is the node ID that will be set in the dispatcher config.

abikouo force-pushed the dispatcherd_1 branch 7 times, most recently from 19a6fc5 to ad9b029 Compare July 9, 2025 09:00

hakbailey requested changes Jul 18, 2025

View reviewed changes

abikouo force-pushed the dispatcherd_1 branch from 6622288 to d03de2c Compare July 21, 2025 13:13

abikouo force-pushed the dispatcherd_1 branch from 083193d to 85585e2 Compare July 21, 2025 13:26

alinabuzachis reviewed Jul 21, 2025

View reviewed changes

abikouo force-pushed the dispatcherd_1 branch from 306796d to b14a2c7 Compare July 29, 2025 11:20

abikouo requested review from hakbailey and alinabuzachis July 29, 2025 11:23

abikouo force-pushed the dispatcherd_1 branch 2 times, most recently from e14894d to b45ca24 Compare July 29, 2025 11:28

Set up docker for local development

4be3cad

abikouo force-pushed the dispatcherd_1 branch from d3a5a50 to 4be3cad Compare July 29, 2025 11:41

hakbailey requested changes Jul 29, 2025

View reviewed changes

abikouo added 2 commits July 30, 2025 11:26

Code review updates

34a85f1

Fix check

c1b8be6


		ENV PATTERN_SERVICE_MODE=development

		RUN python3.11 manage.py migrate



		def override_dispatcher_settings(loaded_settings: Dynaconf) -> None:
		feature_dispatcherd = convert_to_bool(loaded_settings.get("FEATURE_DISPATCHERD"))

		@@ -0,0 +1,84 @@
		x-environment: &common-env
		PATTERN_SERVICE_DB_HOST: postgres

Set up docker with dispatcherd for local development #18

Are you sure you want to change the base?

Set up docker with dispatcherd for local development #18

Uh oh!

Conversation

abikouo commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GomathiselviS commented Jul 10, 2025

Uh oh!

hakbailey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abikouo commented Jul 21, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Quality Gate failed

Uh oh!

hakbailey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

abikouo commented Jul 4, 2025 •

edited

Loading

sonarqubecloud bot commented Jul 24, 2025 •

edited

Loading