Merge pull request #49 from dimagi/nh/refactor

Refactor Ansible for CommCare Data Pipeline

Author: kaapstorm

.ansible-lint

@@ -0,0 +1,15 @@
+---
+profile: production
+
+exclude_paths:
+  - .venv/
+
+warn_list:
+  - experimental
+
+# skip_list entries require a justification comment
+skip_list:
+  # These variables are intentionally shared across multiple roles as project-level
+  # variables — renaming them to per-role prefixes would be a backwards-incompatible
+  # change affecting all inventories.
+  - var-naming[no-role-prefix]

.github/workflows/lint.yml

@@ -0,0 +1,29 @@
+---
+name: Lint
+
+on:
+  push:
+    branches: ["**"]
+  pull_request:
+    branches: ["**"]
+jobs:
+  ansible-lint:
+    name: ansible-lint
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install dependencies
+        run: |
+          pip install ansible-dev-tools
+          ansible-galaxy collection install -r requirements.yml
+
+      - name: Run ansible-lint
+        run: ansible-lint

.github/workflows/molecule-test.yml

@@ -0,0 +1,37 @@
+---
+name: Molecule Test
+
+on:
+  push:
+    branches: [master, nh/refactor]
+  pull_request:
+    branches: [master, nh/refactor]
+jobs:
+  test:
+    name: Molecule Test
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        role: [postgres, redis, nginx]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.11"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install ansible molecule molecule-docker docker
+          ansible-galaxy collection install -r requirements.yml
+
+      - name: Run Molecule test
+        run: |
+          cd roles/${{ matrix.role }}
+          molecule test
+        env:
+          PY_COLORS: "1"
+          ANSIBLE_FORCE_COLOR: "1"

.readthedocs.yml

@@ -1,3 +1,4 @@
+---
 # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
 
 version: 2

CLAUDE.md

@@ -0,0 +1,34 @@
+# CommCare Sync Ansible
+
+Ansible playbooks and roles for deploying [CommCare Data Pipeline](https://github.com/dimagi/commcare-sync).
+
+## Setup
+
+```bash
+source .venv/bin/activate
+```
+
+## Commands
+
+```bash
+# Lint
+ansible-lint
+
+# Run Molecule tests for a role (requires Docker)
+cd roles/<role>
+molecule test
+
+# Run playbook
+ansible-playbook -i inventories/<env> commcare_sync.yml
+```
+
+## Structure
+
+- `roles/` — Ansible roles (`commcare_sync`, `django`, `nginx`, `postgres`, `redis`, `superset`)
+- `inventories/` — Per-environment inventory files
+- `requirements.yml` — Ansible Galaxy role/collection dependencies
+- `requirements-dev.txt` — Python dev tools (`ansible-dev-tools`, `molecule-docker`)
+
+## Molecule tests
+
+Each role has tests in `roles/<role>/molecule/default/`. Tests use Docker (Ubuntu 24.04). Run from the role directory.

Vagrantfile

@@ -15,7 +15,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
   config.vm.define "commcare_sync", primary: true do |obj|
     # Every Vagrant development environment requires a box. You can search for
     # boxes at https://atlas.hashicorp.com/search.
-    obj.vm.box = "bento/ubuntu-18.04"
+    obj.vm.box = "bento/ubuntu-24.04"
     obj.vm.hostname = "dev-commcare-sync"
     # obj.vm.boot_timeout = 600
     obj.vm.network "private_network", ip: "192.168.11.10"

commcare_sync.yml

@@ -1,16 +1,20 @@
 ---
 # Issue: https://github.com/ansible/ansible/issues/57529#issuecomment-513143430
-- hosts: all
+- name: Gather facts
+  hosts: all
 
 - name: Create CommCare Data Pipeline server.
   hosts: all
-  become: yes
+  become: true
   become_user: root
 
-  gather_facts: no
-
+  gather_facts: false
 
   roles:
+    - postgres
+    - redis
+    - nginx
+    - role: superset
+      when: superset_enabled
+    - django
     - commcare_sync
-
-...

docs/development.md

@@ -31,3 +31,44 @@ This should download and deploy the latest CommCare Data Pipeline code to a loca
 
 If everything is successful you can load [http://192.168.11.10/](http://192.168.11.10/) in a browser
 and get going!
+
+## Running tests with Molecule
+
+Individual roles can be tested in isolation using
+[Molecule](https://ansible.readthedocs.io/projects/molecule/), which
+provisions a Docker container and runs the role against it. Tests
+currently exist for the `postgres`, `redis`, and `nginx` roles.
+
+### Install Molecule and dependencies
+
+```bash
+uv venv
+source .venv/bin/activate
+uv pip install -r requirements-dev.txt
+```
+
+You will also need [Docker](https://docs.docker.com/get-docker/)
+installed and running.
+
+### Run tests
+
+To test a single role:
+
+```bash
+cd roles/postgres
+molecule test
+```
+
+This will create an Ubuntu 24.04 Docker container, run the role, execute
+the verification checks, and tear down the container.
+
+To test all roles that have Molecule scenarios:
+
+```bash
+for role in postgres redis nginx; do
+  (cd roles/$role && molecule test)
+done
+```
+
+Tests also run automatically on push and pull request via the GitHub
+Actions workflow in `.github/workflows/molecule-test.yml`.

docs/migration.md

@@ -47,7 +47,7 @@ createdb -U commcare_sync -h localhost -p 5432 superset
 ```
 
 If you get errors about active connections when dropping databases try stopping all processes 
-with `supervisorctl stop all` and killing any other active connections in a `psql` shell with something like the below:
+by stopping all services with `sudo systemctl stop gunicorn celery superset` and killing any other active connections in a `psql` shell with something like the below:
 
 ```
 SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'superset' AND pid <> pg_backend_pid();

docs/system-administration.md

@@ -15,8 +15,8 @@ Services
 --------
 
 The complete list of services is available in the
-[roles/commcare_sync/tasks/main.yml](https://github.com/dimagi/commcare-sync-ansible/blob/master/roles/commcare_sync/tasks/main.yml)
-file.
+[commcare_sync.yml](https://github.com/dimagi/commcare-sync-ansible/blob/master/commcare_sync.yml)
+playbook and its individual roles under `roles/`.
 
 The most important ones are summarized on the
 [What's Installed](whats-installed.md) page.
@@ -115,12 +115,12 @@ environment is in `~ansible/www/.virtualenvs/superset`.
 
 ### Other Useful Commands
 
-| Task                             | Command                                     |
-|----------------------------------|---------------------------------------------|
-| See running Supervisor processes | `sudo supervisorctl status`                 |
-| Restart a Supervisor process     | `sudo supervisorctl restart [process name]` |
-| Connect to the local database    | `sudo -u postgres psql`                     |
-| Restart nginx                    | `sudo service nginx restart`                |
+| Task                           | Command                                       |
+|--------------------------------|-----------------------------------------------|
+| See running services           | `sudo systemctl status gunicorn celery superset` |
+| Restart a service              | `sudo systemctl restart [service name]`       |
+| Connect to the local database  | `sudo -u postgres psql`                       |
+| Restart nginx                  | `sudo systemctl restart nginx`                |
 
 
 ### Checking for Stuck Exports