feat(#537)!: error when form internalId does not match file name

[jkuester]

This saga with the internalId has been bothering me for awhile. It is yet another unnecessary foot-gun and I figured we could continue the process of fixing it as a part of this major release in cht-conf. See #537 and medic/cht-core#3342 for more context.

The main change in this PR is that when there is a miss-match between the form_id in the xlsx, the file name for the xml, and the internalId from the properties file cht-conf will now raise an error when you run the convert-*-forms action. (Previously, the internalId value on the form doc in the DB was set according to this order of precedence: properties file > form_id in xlsx (instance id in xml) > form xlsx/xml file name.)

Now if the ids all match, the form will be successfully converted (though a warning will still be printed if the internalId is set in the properties file). If no form_id value is provided on the settings tab in the xlsx, then new logic will make sure to set the proper id value into the form xml so everything remains consistent. So, you will no longer be able to specify an id value that is miss-matched from your form name, but now you never actually have to specify an id value at all. cht-conf/pyxform will automatically assign one based on your file name.

The internalId property is still populated as before on the form doc. Now it is just guaranteed to match the form file name and form doc id.

I spent a lot of time deliberating and tinkering in various error scenarios until I was confident that there should be a migration path available for any viable existing error case. So, an app-dev should always have a way to resolve the error without breaking their whole instance (or doing big migrations):

• app forms:
  • The internalId of the form gets set on report docs as the form property.
  • So, when there is a miss-match, the file name of the form xml/xlsx should be updated match the existing internalId. You should not update the internalId because that would start writing reports with a different form value.
  • The renamed form will be seen as a "new" form with a different form doc, etc. So, the form doc for the old form should be removed to prevent duplication.
• contact forms:
  • As far as I can tell, the internalId value on contact forms is not used anywhere (and not recorded in on contact docs in the DB).
  • Since the format for contact form file names is already very specific (with the contact type and action), it seems reasonable that if you hit an error with the contact form id, you can just remove the form_id column from the xlsx and everything should be fine.
• training forms:
  • Training forms are an interesting case. The internalId value does get set on the training "reports" as the form. So, we don't want to change the internalId value or we could lose the mapping of which trainings the user has completed.
  • So, when there is a miss-match in the training form, the name of the form should be updated to match the internalId.
  • One edge case I considered here was what happens if the existing internalId does not even have the training: prefix. (In that case where would be no way to update the file names to match the internalId.)
    • However, I confirmed that the webapp logic for rendering the training forms relies on the internalId having the training: prefix. Without it, the form cannot be rendered. So, I think we do not need to worry about this case since the form would never have worked.

---

FYI, I am targeting the 6_0-staging branch with this PR so that I can group these changes with #709 and release both these breaking changes in the same 6.0.0 major release. When this PR is approved, I will "Squash and merge" it into 6_0-staging branch. Then I raise a final PR from 6_0-staging branch into main which can be added into main with the "Rebase and merge" action so that (hopefully) semantic-release will just cut one major release for two breaking changes... 🤞

AI Disclosure: some parts of this PR were suggested/discussed with Claude Code, but nothing was generated solely by AI.

Code review items

• Readable: Concise, well named, follows the style guide, documented if necessary.
• Documented: Configuration and user documentation on cht-docs
• Tested: Unit and/or integration tests where appropriate
• Backwards compatible: Works with existing data and configuration. Any breaking changes documented in the release notes.

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

[jkuester]

Now (with the new pyxform) the name of the primary instance node is data by default. So we do not have to do anything special for contact forms.

[jkuester]

This form an an invalid form_id value of "district". It was causing an error. I updated the form_id value to be district_hospital to cover the case of a contact form with the default "file name" id.

[jkuester]

This form used to have the invalid form_id of person. I actually just totally removed the form_id column from this form to test the case where pyxform auto-assigns the id based on the file name.

[jkuester]

I updated this test case to have a valid (matching) internalId to avoid the new error. Then I added the invalid-internal-id case to test the error.

[dianabarsan]

Looking good! I added one comment about the intermediary save logic.

[dianabarsan]

VERY VERY NICE!