Missing Details Design Decisions

Key design constraints, architectural rationale, tradeoffs, and extensibility notes for the Missing Details feature across livehealth-frontend, crelio-app, and livehealthapp.

Design Intent

Missing Details was not built as a "nice-to-have data entry shortcut". It was built as a controlled failure mode for registration and billing.

The core problem is operational, not cosmetic:

• labs cannot afford to hard-stop high-volume registration or billing every time one field is unavailable,
• but they also cannot silently accept incomplete data and lose the fact that something was missing,
• and downstream teams still need visibility so those gaps can be corrected before they become reporting, billing, or compliance issues.

The design premise is simple:

let the workflow continue, but make incompleteness explicit, queryable, reversible, and eventually resolvable.

That single premise explains most of the implementation.

Decision Stack

Key Design Constraints

Architectural Rationale

1. Workflow continuity was preferred over hard blocking

The first design fork was straightforward but high stakes:

Should the system reject registration/billing when a field is missing, or should it let the workflow continue under controlled conditions?

We chose controlled continuation.

Why this was preferred

Hard blocking looks neat in theory and behaves badly at a live counter. In practice it causes:

• queue buildup,
• billing delays,
• front-desk workarounds,
• junk values being typed just to satisfy validation,
• and a general loss of trust in the product.

That is the wrong optimization target. The system should protect data quality without forcing operational deadlock.

So the chosen tradeoff was:

• allow continuation,
• but only for a known set of fields,
• only with lab-approved defaults,
• and only while creating an explicit unresolved record.

Missing Details is therefore not a generic "ignore validation" switch. It is a controlled bypass with traceability attached.

Alternative we did not choose

Alternative: make the fields mandatory and reject the save

Why it was not chosen:

• operationally brittle,
• encourages fake-value workarounds,
• gives no structured way to revisit deferred data,
• and converts a data-quality problem into a throughput problem.

2. Config-time and runtime concerns were deliberately separated

One of the strongest design choices in this feature is the separation between setup state and business incidents.

Why this split was preferred

If config and runtime were fused into one table:

• config edits could distort the meaning of old incidents,
• disabled configs would become awkward lifecycle flags,
• historical analysis would become muddier,
• and runtime queries would drag around setup metadata they do not actually own.

Separating the layers keeps the model sane:

• templates define vocabulary,
• config defines lab behavior,
• runtime rows capture actual incompleteness,
• trace rows capture config-side side effects.

3. A global template catalog was preferred over lab-defined free-form fields

It would have been possible to let every lab invent its own missing-field definitions from scratch and skip the master template layer.

That sounds flexible. It is also how you end up with a support nightmare.

Why the template layer exists

The global MissingDetailsTemplates catalog gives the system a stable vocabulary:

• field names stay canonical,
• categories stay broadly aligned,
• modules stay explicit,
• resolution mapping can be written against known names,
• frontend enums and backend logic can target the same identifiers.

Without that layer:

• each lab would effectively invent its own DSL,
• resolution would degrade into string-matching chaos,
• analytics would become inconsistent,
• and cross-lab rollout of new supported fields would become manual and error-prone.

Tradeoff we accepted

The template layer reduces free-form flexibility. That is intentional. Missing Details is meant to be configurable, not anarchic.

4. Placeholder values are stored on the runtime row, not only in config

At first glance, storing LabMissingDetails.field_value can look redundant because the config row already has a default value.

It is not redundant. It is necessary.

Why this was preferred

The runtime row must capture the value that was actually used when the incident occurred.

That matters because:

• lab config defaults can change later,
• unresolved old rows still need to compare against the original placeholder,
• and resolution logic needs a deterministic baseline for actual_value != stored_missing_value.

If runtime rows only referenced the current config value:

• a config edit could silently rewrite the meaning of history,
• old unresolved rows could begin comparing against the wrong placeholder,
• and auto-resolution would become nondeterministic over time.

This is a classic configuration-vs-event boundary: runtime incidents must preserve the effective value that existed at the time they were created.

5. Patient-module and order-module missing details were modeled separately

Missing data in this feature is not one homogeneous thing. Some gaps belong to the patient profile, others belong to a specific order/bill.

That distinction is reflected in:

• template module,
• config field_module,
• runtime field_module,
• frontend tabs,
• resolution scope,
• and list filtering.

Why this split was preferred

If everything were modeled as patient-scoped:

• order fields like test, ICDs, pre-auth number, and order number would have ambiguous ownership,
• resolution would be too broad,
• and UI semantics would get muddy.

If everything were modeled as bill-scoped:

• patient-level incompleteness would be duplicated per bill,
• longitudinal patient gaps would fragment,
• and downstream badge logic would get noisy.

So the model uses a clean split:

• patient module = persistent profile-like gaps,
• order module = bill-specific incompleteness.

6. Configs are soft-disabled instead of deleted

Disabling a missing field in setup sets is_disabled = 1. It does not remove the config row.

Why this was preferred

Soft-disable preserves:

• historical interpretation,
• traceability from MissingDetailsEntityCreation.config_id,
• audit continuity,
• and rollback friendliness.

Hard delete would be cheaper in the short term and more expensive in every debugging, support, and audit conversation after that. For a feature whose purpose is to track incomplete business data, historical continuity is more valuable than storage neatness.

7. Resolution is hybrid: implicit where possible, manual where necessary

The feature supports two resolution paths:

1. implicit resolution through normal business flows,
2. manual resolution through the Missing Data UI.

Why implicit resolution was preferred

If every resolved field required a second explicit user step:

• many rows would stay unresolved forever,
• the Missing Data list would become stale,
• badge accuracy would drift,
• and the system would accumulate long-dead missing incidents.

The backend already has the updated patient/order values in hand during save flows. Not using that context to resolve existing rows would be wasted signal.

Why manual resolution was retained

Implicit resolution is not enough on its own. There are plenty of real-world cases where:

• the correction happened through a side workflow,
• the backend does not have a rich enough values map to compare safely,
• or the operator knows the issue is operationally resolved even if the system cannot infer it cleanly.

That is why the manual PATCH /api-v3/account/missing-details/lab-fields path remains necessary. The design is intentionally hybrid:

• let the system self-heal when it can,
• let humans close the loop when it cannot.

8. A mapper bridge was preferred over field-specific resolution branches

MISSING_FIELDS_MODEL_FIELD_MAPPER is one of the key architecture choices in the feature.

What problem it solves

The product thinks in business labels:

• Patient Name
• Order Number
• Bill ICD codes

The backend objects think in model fields:

• fullName
• order_number
• bill_icd

The mapper is the translation layer between those two worlds.

Why this was preferred

Without a mapper, the resolution engine would degenerate into scattered special-case conditionals:

• if field_name == "Patient Name": use obj.fullName
• if field_name == "Address": use obj.area
• and so on.

That would spread field knowledge across multiple save flows and make extension ugly.

With a mapper:

• the engine stays generic,
• adding supported fields becomes largely a mapping exercise,
• and the business vocabulary remains stable even when object shapes differ.

Tradeoff we accepted

The mapper can drift. That is real. But centralized drift is still better than distributed branch logic drift.

9. Downstream systems get a badge flag, not full missing-detail payloads

Most downstream screens do not need the full LabMissingDetails row set. They only need to answer:

should I show a Missing Data badge here?

Why the flag approach was preferred

Pushing full missing-detail payloads into every waiting list, dashboard, accession, or report screen would:

• inflate payload size,
• duplicate data-fetch work,
• complicate rendering logic,
• and couple simple badge consumers to the full missing-detail model.

The chosen design is intentionally cheap:

• compute unresolved existence once,
• attach has_missing_fields,
• let the screen render a red badge.

The detailed list still exists separately because the Missing Data module genuinely needs row-level detail. That separation keeps edge consumers light without taking away investigative depth.

10. Runtime rows are lab-owned first, org-scoped second

The feature is modeled as lab-owned first. org_id exists where needed, but lab_id remains the primary partition.

Why this was preferred

Labs are the operational owners of:

• field configuration,
• registration infrastructure,
• billing workflows,
• and activity traceability.

Organization visibility matters, but it is not the root tenancy model. That means:

• lab ownership stays clear,
• org filtering can be layered in where needed,
• and org/CC login behavior can remain access semantics rather than becoming the primary data model.

This is subtle, but it is the correct bias for how the product actually operates.

11. Entity-backed defaults are bootstrapped at config time, not at registration/billing time

Some missing fields are not plain scalars. They correspond to real entities such as referral, organization, test, ICD, insurance, or group.

The chosen design is:

• when config is created, optionally create or queue creation of the backing entity,
• then let runtime workflows reference something that already exists.

Why this was preferred

Doing entity creation inside registration or billing would have been the wrong coupling:

• save paths would become slower and more failure-prone,
• hot operational workflows would inherit catalog-creation side effects,
• and the feature would blur the line between "this data is missing" and "also create a new master record right now."

By moving entity bootstrap closer to config-time:

• the lab defines allowed missing-field behavior once,
• expensive or legacy entity creation happens outside the hot path,
• and runtime behavior becomes simpler and more predictable.

This is also why the feature tolerates a py3/py2 split. Some entities already have usable py3 creation paths. Others still rely on mature py2 controllers. The design chose interoperability over waiting for a perfect migration boundary.

12. Activity logging is separate from business-state storage

The feature writes to both business tables and activity logs, but it does not confuse those concerns.

Why this was preferred

Each store answers a different question:

• LabMissingDetails = business state,
• ActivityLog = operator/system trace,
• MissingDetailsEntityCreation = config-side side-effect trace.

Overloading one into the other would make both querying and debugging noisier. Keeping them separate makes it easier to answer:

• "is this row unresolved?",
• "who queued the entity create?",
• "which config created this domain entity?"

Those are different questions and deserve different storage shapes.

Decision Boundaries

Alternatives We Deliberately Did Not Choose

Robustness, Reusability, and Tradeoffs

The implementation is not tiny, but the tradeoffs are coherent.

Robustness highlights

Reusability and extensibility

The feature is reasonably extensible because it is built out of generic primitives:

• a master field catalog,
• a per-lab config layer,
• a mapper bridge,
• a generic runtime incident table,
• a list/resolve API,
• and reusable frontend checkbox + auto-fill mechanics.

That makes these changes relatively straightforward:

Negative consequences we accepted

• The implementation spans more files than a single-module feature.
• Engineers need both py3 and py2 mental models for full-stack debugging.
• Mapper drift becomes a maintenance concern, especially for entity-backed fields.
• Some fields will always be awkward because domain object shapes do not line up neatly.

That is the cost of choosing operational fit and staged migration over architectural minimalism.

Final Take

The Missing Details design is opinionated in the right places.

It does not try to make incomplete data disappear.
It does not overreact by blocking the workflow completely.
It does not treat setup and runtime as the same thing.
It does not assume users will remember to clean everything up manually.
It does not pretend the stack migration is more important than feature value.

Instead, it chooses a very product-engineering middle path:

• accept messy operational reality,
• constrain it with configuration,
• record it explicitly,
• surface it downstream,
• and let the system heal itself when real data catches up.

That is the real design philosophy behind Missing Details.