Skip to content

Merge V7.12.0.5 into main#8043

Open
acwhite211 wants to merge 41 commits into
mainfrom
v7_12_0_5
Open

Merge V7.12.0.5 into main#8043
acwhite211 wants to merge 41 commits into
mainfrom
v7_12_0_5

Conversation

@acwhite211
Copy link
Copy Markdown
Member

@acwhite211 acwhite211 commented Apr 28, 2026

Merge the PRs added in the v7_12_0_5 branch into main.

Checklist

  • Self-review the PR after opening it to make sure the changes look good and
    self-explanatory (or properly documented)
  • Add relevant issue to release milestone
  • Add pr to documentation list

Testing instructions

Summary by CodeRabbit

  • Bug Fixes

    • Improved delete blocker handling for many-to-many relationships with normalized output display.
    • Enhanced consistency in column definition selection with deterministic OS preference.
  • Chores

    • Optimized schema configuration migrations and database initialization processes.
    • Improved permissions policy JSON generation for deterministic output.
    • Refined migration utilities for more reliable and efficient database operations.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 28, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 777b9ca4-24bc-4dee-9fa2-9cdf1bc54ac7

📥 Commits

Reviewing files that changed from the base of the PR and between 888aca8 and 534e726.

📒 Files selected for processing (4)
  • specifyweb/frontend/js_src/lib/components/FormParse/index.ts
  • specifyweb/specify/management/commands/run_key_migration_functions.py
  • specifyweb/specify/migration_utils/tectonic_ranks.py
  • specifyweb/specify/tests/test_delete_blockers.py
✅ Files skipped from review due to trivial changes (1)
  • specifyweb/specify/tests/test_delete_blockers.py
🚧 Files skipped from review as they are similar to previous changes (3)
  • specifyweb/frontend/js_src/lib/components/FormParse/index.ts
  • specifyweb/specify/migration_utils/tectonic_ranks.py
  • specifyweb/specify/management/commands/run_key_migration_functions.py

📝 Walkthrough

Walkthrough

This PR introduces determinism and idempotence across multiple systems: FormParse column selection now prioritizes Linux OS definitions, tectonic-rank creation prevents duplicate subtree generation via root tracking, the migration startup pipeline is refined to exclude unreliable transformations, and delete-blocker test coverage validates many-to-many normalization behavior.

Changes

Determinism and Blocker Normalization

Layer / File(s) Summary
FormParse OS-based column selection
specifyweb/frontend/js_src/lib/components/FormParse/index.ts, specifyweb/frontend/js_src/lib/components/FormParse/__tests__/index.test.ts
getColumnDefinition now prefers columnDef entries with os="lnx", then those without os attribute, and finally the first available entry when os is undefined, replacing non-deterministic broad matching.
Tectonic ranks root creation idempotence
specifyweb/specify/migration_utils/tectonic_ranks.py
Root TectonicUnitTreeDefItem creation is tracked via get_or_create return value to skip duplicate rank subtree creation; root and tree-definition lookups now use structural constraints (parent, rankid) instead of names for deterministic resolution.
Migration function startup pipeline refinement
specifyweb/specify/management/commands/run_key_migration_functions.py
Schema-config startup pipeline is narrowed to schema defaults, picklist, and field updates while commenting out destructive/unreliable startup transformations (CO children, quantities, paleo, accession schemas, hidden-prop migrations) with documentation of startup-incompatible steps.
Delete-blocker normalization test validation
specifyweb/specify/tests/test_delete_blockers.py
Test helpers enforce strict HTTP 200 responses and normalize blocker ids ordering for deterministic assertions; new tests verify that many-to-many join-table blockers are normalized to related-model form and that cascaded collection excludes underlying join-table entries.

Possibly Related PRs

  • specify/specify7#6308: Introduced the run_key_migration_functions command and startup/runner wiring that this PR refines by narrowing the fix_schema_config pipeline.
  • specify/specify7#8039: Applies overlapping migration and pipeline refactors including typed defaults for schema-config helpers, deterministic locale container handling, and removal of MIGRATION_0038_UPDATE_FIELDS.
  • specify/specify7#8028: Modifies the same FormParse getColumnDefinition(s) logic to enforce deterministic OS (os="lnx") precedence.

Suggested Reviewers

  • grantfitzsimmons
  • emenslin
🚥 Pre-merge checks | ✅ 3 | ❌ 3

❌ Failed checks (2 warnings, 1 inconclusive)

Check name Status Explanation Resolution
Automatic Tests ⚠️ Warning Delete-blocker and FormParse logic have unit tests, but Permissions.sortPolicies, load_datamodel._all_fields, and migration utilities lack test coverage. Add unit tests for Permissions.sortPolicies, load_datamodel._all_fields, and key migration utility changes (tectonic_ranks, update_schema_config).
Testing Instructions ⚠️ Warning Testing instructions lack clarity. "Test each PR individually" provides no actionable guidance on HOW (test commands), WHAT (specific scenarios), or scope (unit vs integration). Add: (1) specific test commands (pytest for backend, jest for frontend); (2) scenarios per PR (delete-blocker M2M normalization, FormParse OS precedence, schema migration validation); (3) clear pass/fail criteria.
Title check ❓ Inconclusive The title describes a version merge operation that encompasses multiple PRs with substantial changes across backend and frontend systems, but it doesn't specify what those changes accomplish. Consider adding a brief summary of the main technical changes (e.g., 'Merge V7.12.0.5: fix delete-blocker serialization, improve schema config migrations, and enhance form parsing determinism').
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch v7_12_0_5

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Also, define a helper on Table to fetch/filter fields
…alecontaineritem

Duplicate items and strings would be created everytime the function is run.
The cause of this is the same as #7988.
There's no easy way to differentiate schema config items for
which these should be applied.
I suppose the BEST method might be to be exact: only updating
Containers/Items/Strings that exactly match the previous values
expected with the migrations they aim to fix/resolve
@melton-jason
Copy link
Copy Markdown
Contributor

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 29, 2026

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
specifyweb/specify/tests/test_delete_blockers.py (1)

165-329: ⚠️ Potential issue | 🟠 Major

Please remove the duplicated module block appended at the end.

The entire test module is repeated from Line 165 onward, including imports and class redefinitions. This shadows earlier class bindings and makes the test file brittle/confusing. Keep a single copy of the module content.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@specifyweb/specify/tests/test_delete_blockers.py` around lines 165 - 329, The
file contains a duplicated module block (duplicate imports and repeated
definitions of TestDeleteBlockers, TestDeleteBlockersCascade, helper _url and
tests) appended after the original content; remove the entire second copy so
only one definition of helpers (_url, _get_blockers, _assertSame,
_assertContains) and the test classes TestDeleteBlockers and
TestDeleteBlockersCascade remain, ensuring references to
_collect_delete_blockers, GeographyTree, and model usages (e.g. Spexportschema,
Spexportschemamapping) are preserved from the first occurrence.
specifyweb/specify/migration_utils/default_cots.py (1)

71-86: ⚠️ Potential issue | 🟠 Major

Backfill the default COG items on reruns.

Picklistitem.get_or_create(...) was already idempotent, so moving the item loop under picklist_created means fix_cots can no longer heal an existing SystemCOGTypes picklist that is missing one or more default rows. This turns a safe rerun path into a partial no-op.

Suggested fix
-        if picklist_created:
-            for cog_type in DEFAULT_COG_TYPES:
-                Picklistitem.objects.using(using).get_or_create(
-                    title=cog_type,
-                    value=cog_type,
-                    picklist=cog_type_picklist
-                )
+        for cog_type in DEFAULT_COG_TYPES:
+            Picklistitem.objects.using(using).get_or_create(
+                title=cog_type,
+                value=cog_type,
+                picklist=cog_type_picklist
+            )
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@specifyweb/specify/migration_utils/default_cots.py` around lines 71 - 86, The
code currently only seeds DEFAULT_COG_TYPES into the Picklist when
picklist_created is True, which prevents healing missing Picklistitem rows on
reruns; update the logic in the function (e.g., fix_cots in default_cots.py) so
that after obtaining cog_type_picklist via
Picklist.objects.using(using).get_or_create(...) you always iterate
DEFAULT_COG_TYPES and call Picklistitem.objects.using(using).get_or_create(...)
for each cog_type (using cog_type_picklist as the picklist argument) rather than
restricting that loop to the picklist_created branch, ensuring idempotent
backfill of missing items.
🧹 Nitpick comments (2)
specifyweb/backend/delete_blockers/views.py (1)

45-49: Optional cleanup: remove the temporary nested list + flatten() roundtrip.

You can return the serialized list directly for simpler and clearer code.

Proposed simplification
-    return flatten([
-        [
-            _serialize_delete_blocker(field, sub_objs)
-        ] for field, sub_objs in collector.delete_blockers
-    ])
+    return [
+        _serialize_delete_blocker(field, sub_objs)
+        for field, sub_objs in collector.delete_blockers
+    ]
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@specifyweb/backend/delete_blockers/views.py` around lines 45 - 49, The
current return builds a temporary nested list and then flattens it; simplify by
returning a flat list comprehension directly instead of wrapping each item in
its own list and calling flatten — replace the expression that returns
flatten([... for field, sub_objs in collector.delete_blockers]) with a direct
comprehension that calls _serialize_delete_blocker(field, sub_objs) for each
(referencing collector.delete_blockers and _serialize_delete_blocker) so the
function returns the serialized list without the unnecessary nested-list +
flatten roundtrip.
specifyweb/specify/models_utils/load_datamodel.py (1)

152-174: Consider making this iterator public.

update_schema_config.py now depends on table._all_fields(...) directly, so this is effectively part of the external Table API now. Renaming it to a public method like iter_all_fields(...) would make that contract explicit and reduce future refactor breakage.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@specifyweb/specify/models_utils/load_datamodel.py` around lines 152 - 174,
The iterator method _all_fields is being used externally so make it an explicit
public API by adding a new method iter_all_fields(self, exclude_fields=False,
exclude_relationships=False, exclude_id_field=False,
exclude_virtual_fields=True) that implements the same yield logic and update the
all_fields property to call iter_all_fields(); then keep _all_fields as a thin
compatibility wrapper that calls iter_all_fields (or vice versa) and update any
internal call sites to use iter_all_fields to avoid future breakage while
preserving backward compatibility.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@specifyweb/specify/migration_utils/tectonic_ranks.py`:
- Around line 115-126: The current lookup for tectonic_tree_def_item only
filters by treedef and parent=None and can pick a wrong parentless row; update
the initial query on TectonicUnitTreeDefItem (the tectonic_tree_def_item
assignment) to include rankid=0 (i.e., filter by treedef=tectonic_tree_def,
parent=None, rankid=0) so it matches the same root invariant used in the
get_or_create path; ensure the get_or_create call for TectonicUnitTreeDefItem
also uses rankid=0 as a lookup key (not just a value) so both retrieval and
creation use the same unique identifying fields.

In `@specifyweb/specify/migration_utils/update_schema_config.py`:
- Around line 1963-1969: The loop currently calls
update_table_field_schema_config_with_defaults which only applies ishidden when
creating new Splocalecontaineritem rows; change the call (or the function) to
ensure existing Splocalecontaineritem rows are updated to set ishidden=True for
the targeted fields. Concretely, modify
update_table_field_schema_config_with_defaults (or add a small wrapper) so that
for each (table, field_name) it queries Splocalecontaineritem for the matching
table/field/discipline and if a row exists performs a direct update (e.g., set
ishidden=True and save or use queryset.update), otherwise create with the
default; keep references to MIGRATION_0038_FIELDS, Discipline.objects.all(),
update_table_field_schema_config_with_defaults and Splocalecontaineritem when
implementing the change.

---

Outside diff comments:
In `@specifyweb/specify/migration_utils/default_cots.py`:
- Around line 71-86: The code currently only seeds DEFAULT_COG_TYPES into the
Picklist when picklist_created is True, which prevents healing missing
Picklistitem rows on reruns; update the logic in the function (e.g., fix_cots in
default_cots.py) so that after obtaining cog_type_picklist via
Picklist.objects.using(using).get_or_create(...) you always iterate
DEFAULT_COG_TYPES and call Picklistitem.objects.using(using).get_or_create(...)
for each cog_type (using cog_type_picklist as the picklist argument) rather than
restricting that loop to the picklist_created branch, ensuring idempotent
backfill of missing items.

In `@specifyweb/specify/tests/test_delete_blockers.py`:
- Around line 165-329: The file contains a duplicated module block (duplicate
imports and repeated definitions of TestDeleteBlockers,
TestDeleteBlockersCascade, helper _url and tests) appended after the original
content; remove the entire second copy so only one definition of helpers (_url,
_get_blockers, _assertSame, _assertContains) and the test classes
TestDeleteBlockers and TestDeleteBlockersCascade remain, ensuring references to
_collect_delete_blockers, GeographyTree, and model usages (e.g. Spexportschema,
Spexportschemamapping) are preserved from the first occurrence.

---

Nitpick comments:
In `@specifyweb/backend/delete_blockers/views.py`:
- Around line 45-49: The current return builds a temporary nested list and then
flattens it; simplify by returning a flat list comprehension directly instead of
wrapping each item in its own list and calling flatten — replace the expression
that returns flatten([... for field, sub_objs in collector.delete_blockers])
with a direct comprehension that calls _serialize_delete_blocker(field,
sub_objs) for each (referencing collector.delete_blockers and
_serialize_delete_blocker) so the function returns the serialized list without
the unnecessary nested-list + flatten roundtrip.

In `@specifyweb/specify/models_utils/load_datamodel.py`:
- Around line 152-174: The iterator method _all_fields is being used externally
so make it an explicit public API by adding a new method iter_all_fields(self,
exclude_fields=False, exclude_relationships=False, exclude_id_field=False,
exclude_virtual_fields=True) that implements the same yield logic and update the
all_fields property to call iter_all_fields(); then keep _all_fields as a thin
compatibility wrapper that calls iter_all_fields (or vice versa) and update any
internal call sites to use iter_all_fields to avoid future breakage while
preserving backward compatibility.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0711e959-159f-42a0-9232-05dbdd5b23fa

📥 Commits

Reviewing files that changed from the base of the PR and between b4b1602 and 888aca8.

📒 Files selected for processing (13)
  • specifyweb/backend/delete_blockers/views.py
  • specifyweb/frontend/js_src/lib/components/FormParse/__tests__/index.test.ts
  • specifyweb/frontend/js_src/lib/components/FormParse/index.ts
  • specifyweb/frontend/js_src/lib/components/Permissions/index.ts
  • specifyweb/specify/api/crud.py
  • specifyweb/specify/management/commands/run_key_migration_functions.py
  • specifyweb/specify/migration_utils/default_cots.py
  • specifyweb/specify/migration_utils/sp7_schemaconfig.py
  • specifyweb/specify/migration_utils/tectonic_ranks.py
  • specifyweb/specify/migration_utils/update_schema_config.py
  • specifyweb/specify/migrations/0039_agent_fields_for_loan_and_gift.py
  • specifyweb/specify/models_utils/load_datamodel.py
  • specifyweb/specify/tests/test_delete_blockers.py
💤 Files with no reviewable changes (2)
  • specifyweb/specify/migrations/0039_agent_fields_for_loan_and_gift.py
  • specifyweb/specify/migration_utils/sp7_schemaconfig.py

Comment thread specifyweb/specify/migration_utils/tectonic_ranks.py Outdated
Comment thread specifyweb/specify/migration_utils/update_schema_config.py
@github-project-automation github-project-automation Bot moved this from 📋Back Log to Dev Attention Needed in General Tester Board Apr 29, 2026
@acwhite211 acwhite211 marked this pull request as ready for review May 19, 2026 15:19
Copy link
Copy Markdown
Collaborator

@emenslin emenslin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only tested the first two as the last one was only dev tested before, Looks good, I didn't run into any issues!

@emenslin emenslin requested a review from a team May 19, 2026 15:34
Copy link
Copy Markdown
Collaborator

@bhumikaguptaa bhumikaguptaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two work PRs work as expected! Unable to test the third one.

@bhumikaguptaa bhumikaguptaa requested a review from a team May 19, 2026 15:52
Copy link
Copy Markdown
Contributor

@Iwantexpresso Iwantexpresso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

last pr was a bit of doozy but I think I didn't spot any duplicate fields manually. and the query didn't show any duplicate fields either. nice work, on all three!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Dev Attention Needed

Development

Successfully merging this pull request may close these issues.

6 participants