feat(loaders): migrate Connect exports to v2 paginated JSON

[jjackson]

Summary

CommCare Connect deprecated its streaming CSV export endpoints in favor of keyset-paginated JSON (Accept: application/json; version=2.0) to eliminate the 5-minute gunicorn worker timeout on large UserVisit exports. This PR migrates scout's seven Connect list-export loaders to follow the v2 contract: GET, then walk the next URL until null.

The downstream _write_connect_* writers in materializer.py are unchanged — scalars are stringified to preserve the v1 CSV-shaped TEXT-column contract, and form_json / images pass through as native dict/list for the JSONB columns. No DB migrations, no consumer-facing changes.

What changed

mcp_server/loaders/connect_base.py

• New _paginate_export_pages(suffix, params) — yields page lists, follows the server's next URL until null, sends the versioned Accept header per call so ConnectMetadataLoader's non-versioned endpoints (/export/opp_org_program_list/, /export/opportunity/<id>/) stay untouched.
• New stringify / stringify_record helpers and a ConnectExportError exception type.
• Removed _get_csv and the csv / io imports.

mcp_server/loaders/connect_visits.py

• Rewrote _normalize_visit for v2 dict input. Dropped the ast.literal_eval Python-repr fallback — form_json is now a real dict from the wire.
• Preserved the id → visit_id rename so the existing writer schema is unchanged.
• Falls back to the loader's opportunity_id when the serializer omits it (matches connect-labs behavior).

6 simple loaders (users, completed_works, payments, invoices, assessments, completed_modules)

• Thin paginate-then-stringify loops. Now stream page-by-page rather than buffering the full export — free memory win for large opportunities.

Regression pin for dimagi/commcare-connect#1109

While running this work end-to-end against connect.dimagi.com we hit a server bug: the production gunicorn config defaults --forwarded-allow-ips to 127.0.0.1, so it strips X-Forwarded-Proto: https from Traefik. Django's request.build_absolute_uri() then falls back to http, and IdKeysetPagination.get_next_link() returns next URLs with the http:// scheme even on HTTPS requests. Clients following those URLs receive a 301 redirect.

#1109 is the upstream fix; until it lands, scout has to tolerate the bug. The mitigation here:

• requests.Session.get defaults to allow_redirects=True, so we follow the edge's 301 http→https automatically.
• requests.Session.should_strip_auth has a special case for same-host HTTP→HTTPS upgrades on default ports that preserves the Authorization header — verified empirically before adding the test.
• An inline comment in _paginate_export_pages documents the dependency and links to #1109.
• A new test test_follows_http_to_https_redirect_on_next_url simulates the bug end-to-end (page 1 returns next: http://..., mock 301-redirects to https, page 2 returns final results) and asserts both row aggregation and that the bearer token survives the redirect. This pins both behaviors so a future change (e.g., switching to httpx, or someone passing allow_redirects=False) cannot silently regress.

Reference implementation: connect-labs PR #55 — same migration on the labs side, where the mitigation is httpx's follow_redirects=True (httpx default is False).

Test plan

• uv run pytest tests/test_connect_base_loader.py tests/test_connect_data_loaders.py — 56 passed (12 base + 13 visit + 31 simple-loader matrix)
• uv run pytest tests/ --ignore=tests/qa — 797 passed, 15 skipped, 0 regressions
• uv run ruff check + ruff format --check — clean
• Manual verification that ConnectMetadataLoader is unaffected (the versioned Accept header is per-call, not session-global, and a base-loader test pins this)
• After merge: end-to-end materialization run against a real opportunity to confirm the writer contract holds with native v2 JSON types

🤖 Generated with Claude Code

[snopoke]

Why is everything being converted to strings? We'll loose all the type information by doing this.

The root issue is that the connect tables all have TEXT columns. We should update these tables to use the correct data types.

[jjackson]

Fair point, you're right. I was treating this as a behavior-preserving refactor from CSV to JSON and preserved the TEXT-column contract via stringification, but that punted on the actual data-modeling fix you're describing.

Addressed in 234d043. Summary:

Loader side: deleted stringify and stringify_record from connect_base.py. The 6 simple loaders now collapse to thin pass-throughs that yield pages directly from _paginate_export_pages. connect_visits.py::_normalize_visit stops wrapping scalars in stringify() — it still does the id → visit_id rename, the opportunity_id fallback, and the defensive dict/list coercion for form_json/images, but everything else flows through untouched.

Writer side: every raw_* table's CREATE TABLE DDL in materializer.py updated to use proper types. Summary of what changed:

• IDs → BIGINT (visit_id as PK, opportunity_id across all tables)
• Booleans → BOOLEAN (flagged, confirmed, suspended, passed)
• Money → NUMERIC(14, 2) (amounts, payment_accrued, saved_payment_accrued*, saved_org_payment_accrued*)
• exchange_rate → NUMERIC(14, 6) for sub-cent precision
• Dates/datetimes → TIMESTAMPTZ (visit_date, date_created, status_modified_date, last_active, created_at, etc.)
• raw_invoices.date → DATE (invoices are day-level)
• Counts → INTEGER (saved_completed_count, saved_approved_count, score, passing_score, duration)

Business identifiers with ambiguous upstream shape stay TEXT (deliver_unit_id, entity_id, payment_unit_id, completed_work_id, invoice_id, invoice_number), since we don't control whether those are ints, UUIDs, or slugs. String columns stay TEXT.

Insert tuples changed from r.get("field", "") to r.get("field") on typed columns so missing keys bind to SQL NULL via psycopg instead of passing an empty string that'd fail the cast. Text columns keep the "" default for backwards compat.

No migration needed: the writers already DROP TABLE IF EXISTS ... CASCADE + CREATE TABLE on every materialization, so the next sync for any tenant recreates their schema with the new types inside the same transaction. No backfill script, no per-tenant DDL loop.

Downstream audit: grepped the whole repo for SQL that reads raw_* assuming TEXT values. Clean — no DBT assets reference these tables, no recipes, no agent prompt templates hard-code types, no Python result-processing code. The chat agent's LLM-generated SQL will actually improve since it no longer needs ::numeric casts or flagged = 'True' string comparisons.

Tests: test_flagged_bool_stringified → test_flagged_bool_passes_through (asserts is True). test_native_types_stringified → test_native_types_preserved with a per-field type round-trip check. Added two visit tests to pin specific risks: test_flagged_false_not_coerced_to_none (guards against the False or default falsy trap on nullable booleans), and test_missing_datetime_field_is_none (verifies null datetime → Python None → SQL NULL). 58 loader tests pass, full repo suite 800/800.

Thanks for the push — the query ergonomics are going to be much better this way.

[jjackson]

Follow-up: production validation and a bug fix worth naming

After 234d043 (the typed-columns change per your review, @snopoke) I shipped it to our fork's deployment and did a smoke test on a 70k+ visit opp — it crashed immediately with psycopg.ProgrammingError: cannot adapt type 'dict' using placeholder '%s' in _write_connect_visits.

Root cause: when I did the typed-columns rewrite I only read the UserVisit model far enough to see flagged and visit_date, and I assumed the rest of the scalar fields were plain CharFields. Wrong. A full walkthrough of commcare_connect/data_export/serializer.py against the underlying Django models revealed 15 column type mismatches across all 7 raw_ tables*. Every one was either a hidden JSONField, a ForeignKey that DRF's default ModelSerializer renders as the related PK int, a DurationField that serializes to a string like "0:30:00", or PaymentInvoice.service_delivery which I had misread as a label when it's actually a BooleanField. Fixed in 811a625. The specific corrections:

• raw_visits: flag_reason TEXT → JSONB (it's a JSONField, not a CharField); deliver_unit, completed_work, deliver_unit_id, completed_work_id TEXT → BIGINT
• raw_users: claim_limits TEXT → JSONB (it's a SerializerMethodField that returns [dict(row) for row in data])
• raw_completed_works: payment_unit_id TEXT → BIGINT
• raw_payments: payment_unit, invoice_id TEXT → BIGINT
• raw_invoices: service_delivery TEXT → BOOLEAN; exchange_rate NUMERIC(14,6) → BIGINT (it's a ForeignKey(ExchangeRate), not the numeric rate — my previous NUMERIC guess was semantically wrong too)
• raw_assessments: app TEXT → BIGINT
• raw_completed_modules: module TEXT → BIGINT; duration INTEGER → TEXT (DRF's DurationField.to_representation returns strings like "0:30:00")

Added a _json_or_none helper for the nullable JSONB columns because json.dumps(None) produces the literal string "null" which inserts as JSONB null — not the same as SQL NULL. Worth being explicit about that distinction in the writer so nullable JSONFields round-trip correctly.

Production validation

After deploying 811a625, two back-to-back materializations on the same tenant 765 (76,512 visits, the one that originally crashed) completed cleanly:

Run	Elapsed	HTTP response	Backend errors
1	3m 42s	POST /scout/api/chat/ 200 5735	0
2	3m 50s	POST /scout/api/chat/ 200 5196	0

Zero ERROR / Traceback / Pipeline failed / psycopg events in CloudWatch across both windows. Chat agent confirmed row counts populated and query-able with native types on the second run.

~77 paginated v2 JSON requests per run under the 600s nginx proxy_read_timeout I also raised in 9e06ee3 — the same bug class that made the sync-blocking-gunicorn problem go away for our Labs deployment (details on that one in the 9e06ee3 commit message; it's fork-local infrastructure config so not strictly upstream-relevant, but useful context).

Gap I want to own for the reviewer

The loader tests use requests_mock and never exercise the writer against a real Postgres. That's why 234d043 shipped with 15 wrong column types and my initial smoke test on a smaller opp didn't notice — the smaller opp had no flagged visits, so flag_reason was always None and never hit psycopg's adapt check. Adding a writer integration test suite that runs the full _write_connect_* pipeline against a throwaway schema with realistic fixtures is the right follow-up and is on my list. Flagged in the 811a625 commit message too.

Current PR head is 811a625 and fully validated against production data. Happy to take another round of review if you have cycles.