chore(#10700): re-write Enketo form save workflow

[jkuester]

Description

This PR basically re-writes the doc-saving workflow when an Enekto form (either contact/app/training form) is completed. The existing legacy code suffered from several major problems that made it very difficult to continue developing features on top of it:

• The doc-processing pipeline for contact docs vs reports was almost completely disjointed. Aside from the core xml-to-object functions, none of the other logic/workflows was coherently shared between contacts and reports. This meant that we had a lot of duplication and inconsistency between how things were being handled. And, any new features (e.g. attachments) needed to be essentially implemented twice.
• Historical changes to these workflows have been haphazard. The current structure is now incredibly complex with very poor abstraction/encapsulation. It is difficult to understand the data flow and/or which parts of the code are responsible for what.
• The code does a poor job of communicating the baked-in assumptions (instead nesting the handling of them deep in the workflows).

The main impetus behind is PR is to provide a better structured base for solving #10700. The hope is that the new functionality in #10923 can be ported on top of the changes in this PR in a way that is way more simple and maintainable.

What is in this PR

The goal was to have this PR be a complete drop-in replacement for the existing functionality. I have done my best to ensure the new code accurately matches the behavior of the old code and there are no new features that I intended to directly support by these changes.

The only behavior difference that I am aware of currently is a new error I added to the save workflow that will fail with a helpful error message when a contact form does not have the contact data in a group that is named for the actual contact type. (So, a person-create form needs to have the contact data in the person group.) This requirement is listed explicitly in our docs and it is implicitly expected by the pre-population logic for setting the initial data onto the contact. Previously it was technically possible to save a contact form with data in a different group. Because our documentation is explicit about how the forms are supposed to be structured, I do not see this as a breaking change. Also, if some does have this issue, they will hit the error and be unable to save those contact forms. It will not just cause weird data issues by failing silently. The fix would just be to update the form config to name the group properly.

Added FormConfig

A new FormConfig class has been added in webapp/src/ts/services/form/form-config.ts that serves as a wrapper for the form doc and its relevant attachments. This FormConfig is retrieved via the XmlFormsService when rendering the form and then can be re-used when saving the form. This wrapper basically centralizes the handling of the form doc and its attachments and avoids the need to retrieve these things multiple times during the render/save process.

Added new EnketoFormData classes

The new EnketoFormData classes added in webapp/src/ts/services/form/form-data.ts are wrappers to encapsulate the XML data model produced when saving an Enekto form. These data classes own the logic for interacting/querying the XML data as well as de-serializing that data to JS objects.

Being able to operate against the XML data is useful for queries (either via css selectors or XPaths) which do not have a simple alternative when using normal JS objects. These wrappers help define the available data queries as well as hold on to the _id references for the data to allow for easy interlinking.

The xml de-serialization logic from the now-deleted EnketoTranslationService was moved into these EnketoFormData classes. (The JS -> XML serialization logic was just moved to where it was being used in the EnketoPrepoulationDataService.)

Move Contact Save functionality into EnketoService

Previously, we had the logic for saving contact forms separated out into the ContactSaveService. I have move this into the EnektoService so now there are only two entry points for saving a form: EnketoService.saveContact and EnketoService.saveReport. Keeping the code in the same service allows for better alignment between the workflows.

As before, there is still probably "too much" code in the EnektoService, but I have balanced things out a bit by deleting duplicate logic and/or moving it into the EnektoFormData classes. The end result is the actual LOC in the enketo.service file has only grown around 30 lines...

Clean up form/enketo service unit tests

A large percentage of the changed lines in this PR are a result of me overhauling the unit tests for the FormService and EnketoService. Previously there was a TON of duplication/leaking between those unit tests (because of how I previously handled separating the FormService from enketo... 😓). The time has come to pay the piper and I have refactored the FormService tests especially to just mock the EnektoService.

AI Disclosure

Basically every single line of the implementation code was hand-written by me. I leveraged Claude a lot for the investigation and minor code structure advice, but ultimately, I really wanted full control/responsibility for porting all the logic line-by-line. Any bugs will be ones that I personally wrote. 👍 😅

I did leverage Claude heavily for the updates to the unit tests. Even there, I have gone through everything line-by-line and made many editorial changes and requested additional tests, etc. There is a lot of lines changed, so it is possible we missed some stuff, but I am highly confident the functionality is well tested and the tests are very maintainable going forwards.

Code review checklist

• UI/UX backwards compatible: Test it works for the new design (enabled by default). And test it works in the old design, enable can_view_old_navigation permission to see the old design. Test it has appropriate design for RTL languages.
• Readable: Concise, well named, follows the style guide
• Tested: Unit and/or e2e where appropriate
• Backwards compatible: Works with existing data and configuration or includes a migration. Any breaking changes documented in the release notes.
• AI disclosure: Please disclose use of AI per the guidelines.

Compose URLs

If Build CI hasn't passed, these may 404:

• Core
• CouchDB Single
• CouchDB Cluster

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

[jkuester]

The changes here are actually due to the updates I made in the cht-form/src/app.component.ts. When we write a doc to Couch, properties set to undefined are not included in the JSON data. cht-form returns the JS objects directly and was not doing anything to account for undefined properties. Unfortunately, Chai treats an object with a property set to undefined as not deep equal to an object that just does not have the property at all. So, these values were previously needed here to account for these props getting set in the Enekto service, etc.

[jkuester]

FTR, this property was no longer being used prior to my changes in this PR.

[jkuester]

This is now done in the EnketoService.saveContact.

[jkuester]

The most important thing is that this runs before the dupchecking logic in formService.saveContact. I do not think it is an issue that this now runs before the Enketo validation for the form. The Enketo validation is still going to run before the dup checking.

[mrjones-plip]

I came here to see what this was all about and saw this:

I do not see this as a breaking change

Amazing work Josh!!