Make content_ids migration incremental and restartable - #8021
Conversation
The 0152 data fill recomputed each repository version from scratch in Python inside one transaction, which DBaaS statement timeouts killed on large upgrades. Fill caches in Postgres from the previous version and commit per repository. Assisted By: Cursor Grok 4.6 Co-authored-by: Cursor <cursoragent@cursor.com>
|
|
||
| class Migration(migrations.Migration): | ||
| # Per-repository commits inside RunPython; AlterField runs afterwards. | ||
| atomic = False |
There was a problem hiding this comment.
This opens up a window, right?
But as long as nothing runs at the same time, or (in the case of ZDU) the currently running code is sufficiantly new there is nothing to introduce new repository versions with NULL. I think this is actually fine.
There was a problem hiding this comment.
Maybe, if the user is doing a big jump and trying to do ZDU they might run into a problem if an old worker gets a task to create a new repo-vesion. Theoretically if that were to happen and then the ALTER TABLE call fails, they could just rerun the migration again and we would only need to update that one new repo-version that slipped through.
|
|
||
| import django.contrib.postgres.fields | ||
| from django.db import migrations, models | ||
| from django.db import migrations, models, transaction |
There was a problem hiding this comment.
I was thinking about something along the lines of
UPDATE core.repositoryversion AS rv SET content_ids=(SELECT ARRAY_AGG(content_id) FROM core.repositoryversion_content WHERE version_added <= rv.version and version_removed > rv.version) WHERE rv.content_ids is NULL;
Maybe needs a few more joins and some care for version_removed = NULL.
Am oversimplifying the challenge here?
Maybe one can even batch that by adding a LIMIT 1000 clause.
2 participants
Summary
content_idsdata fill to compute each repository version in Postgres from the previous version (plus added, minus removed) instead of an N+1 ORM loop that ships UUID arrays through Python.atomic = False) so a DBaaS statement/session timeout does not roll back already-filled repos. Re-running migrate continues from remaining NULLs, thenAlterFieldmakes the column required.bulk_updatebatches does not fix the per-version recompute or the single wrapping transaction.Test plan
Made with Cursor