# Welcome to CrelioHealth Docs Welcome to the CrelioHealth Internal Documentation portal. This resource is designed to help new joiners and existing team members navigate our development processes and setup procedures. Getting Started [#getting-started] Engineering Workflow [#engineering-workflow] Services & Architecture [#services--architecture] Start here to understand the core systems. Quick Links [#quick-links] | Resource | Description | | -------------------------------------------------- | ---------------------------------------------------------------- | | [Local Environment Setup](/docs/local-environment) | Complete guide for setting up your local development environment | | [LLM Context API](/llms-full.txt) | Full documentation dump in plain text for LLM context injection | *** **Last Updated:** January 2026 # Application Flow Application Flow [#application-flow] This diagram illustrates the complete flow of requests through the CrelioHealth infrastructure, from the internet through the CDN, load balancer, and application tier to various backend services. Architecture Diagram [#architecture-diagram] Component Breakdown [#component-breakdown] Request Flow [#request-flow] 1. **Internet → CDN** - Incoming requests first hit the CDN for cached content 2. **CDN → Load Balancer** - Non-cached requests route to the load balancer 3. **Load Balancer → Target Group** - Traffic distributed across healthy application nodes 4. **Target Group → Application Tier** - Requests reach auto-scaling application servers Application Tier [#application-tier] * **Django/Flask** - Main web frameworks handling HTTP requests * **API Services** - Python 2 and Python 3 API endpoints * **Static Assets** - Served directly from application nodes * **Async Services** - Producer/Consumer pattern for background jobs Data Layer [#data-layer] | Service | Purpose | | ------------- | ---------------------------------------------- | | MySQL | Primary relational database | | MongoDB | Document storage | | Redis Queue | Async job queue | | Elasticsearch | Search functionality (Density & Thor clusters) | | S3 | Object storage | External Services [#external-services] * **SNS/SES** - AWS notification and email services * **Lambda** - Serverless functions * **AI Services** - Machine learning integrations Integration Engine [#integration-engine] * **Mirth** - Integration platform for healthcare data exchange * **HL7/JSON** - Standard healthcare data formats * **External Apps** - Third-party system integrations # Infrastructure Infrastructure Overview [#infrastructure-overview] This section contains documentation about the CrelioHealth infrastructure architecture, including: * **[Application Flow](/docs/infra/application-flow)** - Complete service flow diagram showing how requests flow through the system * Service deployment architecture * Cloud infrastructure components Key Components [#key-components] | Component | Description | | ------------------ | ------------------------------------------------- | | CDN | Content delivery for static assets | | Load Balancer | Traffic distribution across application nodes | | Application Tier | Auto-scaling Django/Flask services | | Elasticsearch | High-performance search clusters (Density & Thor) | | Message Queue | Redis-based async job processing | | Integration Engine | Mirth-based HL7/JSON integrations | # Bitbucket Branch-Based Workflow Bitbucket Branch-Based Workflow (2026 Update) [#bitbucket-branch-based-workflow-2026-update] Removal and Backup of Current Setup (if applicable) [#removal-and-backup-of-current-setup-if-applicable] If you are migrating from an older fork-based workflow, follow these steps to back up and clean up your current Git configuration before switching to the new branch-based workflow: 1. **Backup your current git remote configuration:** ```bash git remote -v > ~/git-remotes-backup.txt ``` 2. **Remove old fork and upstream remotes (if present):** ```bash git remote remove origin git remote remove upstream ``` 3. **(Optional) Delete your old fork on Bitbucket after migration is complete.** Once this is done, proceed with the steps below for the new branch-based workflow. > **Note:** Bitbucket Cloud is deprecating cross-workspace fork support. All new joiners must use the branch-based workflow described below. Forking is no longer required or supported for CrelioHealth repositories. Overview [#overview] All development now happens via feature branches in the main repository. Forks are not used. You will push your feature branches directly to the main repository and create pull requests from there. **Protected branches:** `main`, `develop`, `hotfix` (no direct pushes allowed) **Feature branches:** Must follow the naming convention below and are created from `develop` or `hotfix`. *** Initial Setup [#initial-setup] 1. **Your Bitbucket access will be provided by the onboarding team.** 2. **Clone the repository:** ```bash git clone git@bitbucket.org:creliohealth-repo/livehealthapp.git cd livehealthapp ``` 3. **Sync your local base branches:** ```bash git fetch origin git checkout main && git reset --hard origin/main git checkout develop && git reset --hard origin/develop git checkout hotfix && git reset --hard origin/hotfix # if applicable ``` *** Creating and Working on Feature Branches [#creating-and-working-on-feature-branches] 1. **Start from the correct base branch:** ```bash git checkout develop # or hotfix git pull origin develop ``` 2. **Create your feature branch (use the naming convention):** ```bash git checkout -b develop/yourname/EN-1234-brief-description ``` * Example: `develop/ritu/EN-1123-aoe-in-crm` 3. **Work on your changes and commit:** ```bash git add . git commit -m "EN-1234: Add MRN duplicate validation" ``` 4. **Push your branch to the main repository:** ```bash git push origin develop/yourname/EN-1234-brief-description ``` 5. **Create a Pull Request (PR) on Bitbucket:** * Source: your feature branch * Target: `develop` or `hotfix` (as appropriate) *** Keeping Your Feature Branch Updated [#keeping-your-feature-branch-updated] ```bash git checkout develop/yourname/EN-1234-brief-description git pull --rebase origin develop # Resolve conflicts if any, then: git rebase --continue git push origin develop/yourname/EN-1234-brief-description ``` *** Branch Naming Convention [#branch-naming-convention] Format: `base-branch/username/TICKET-ID-brief-description` * `base-branch`: develop, hotfix, or main (source branch) * `username`: your identifier (e.g., ritu, rahul, sai) * `TICKET-ID`: Jira ticket number (e.g., EN-1123) * `brief-description`: short summary in kebab-case **Examples:** * `develop/ritu/EN-1123-aoe-in-crm` * `hotfix/rahul/EN-1200-fix-login-bug` *** Branch Cleanup Policy [#branch-cleanup-policy] Branches for completed (Done) Jira tickets are deleted from the repository every weekend. You are encouraged to delete your local and remote feature branches after your PR is merged. ```bash # Delete local branch git branch -d develop/yourname/EN-1234-brief-description # Delete remote branch git push origin --delete develop/yourname/EN-1234-brief-description ``` *** FAQ [#faq] * **Do I need a fork?** No, all work is done in the main repository. * **Can I push to main/develop/hotfix?** No, these are protected. Use feature branches and PRs. * **What if my branch name is wrong?** Rename it: `git branch -m old-name develop/yourname/EN-1234-new-name` * **Who merges PRs?** Only authorized reviewers/engineers. * **What if I have issues?** Contact a senior developer or Bitbucket admin. *** **This branch-based workflow is now the standard for all new joiners.** # Database Configuration Database Configuration [#database-configuration] 10. Run Database Migrations [#10-run-database-migrations] Step 1: Execute Migrations [#step-1-execute-migrations] 1. Enter the chapp Docker container 2. Run the migration command: ```bash make migrate ``` Step 2: Handle Failed Migrations (if any) [#step-2-handle-failed-migrations-if-any] If any migration fails, you can fake the migration and manually create the required tables: ```bash ./develop.py migrate app_name migration_name --fake ``` Then manually create the required tables using SQL queries. Step 3: Access MySQL Shell (Optional) [#step-3-access-mysql-shell-optional] If you need to perform SQL operations without DataGrip: ```bash docker exec -it mysql -u -p ``` *** # Docker Build Docker Image Setup [#docker-image-setup] 9. Build Docker Images [#9-build-docker-images] ```bash make build ``` > \[!WARNING] > **Fusion Requirements Fix:** If the build breaks for `netifaces` and `ssh-import-id` packages: > > * Remove version numbers from these packages in the requirements file > * This will install the latest versions instead Optional Optimization [#optional-optimization] * Change Elasticsearch templates to use shard 1 for better local performance *** # Elasticsearch Setup Elasticsearch Configuration [#elasticsearch-configuration] 11. Run Elasticsearch Migration [#11-run-elasticsearch-migration] Step 1: Create Python Virtual Environment [#step-1-create-python-virtual-environment] ```bash python3 -m venv ~/.venv ``` Step 2: Activate Virtual Environment [#step-2-activate-virtual-environment] ```bash source ~/.venv/bin/activate ``` Step 3: Install Required Package [#step-3-install-required-package] ```bash pip install requests ``` Step 4: Navigate to ES Migration Scripts [#step-4-navigate-to-es-migration-scripts] ```bash cd ./scripts/es_migration ``` Step 5: Create Templates [#step-5-create-templates] ```bash python create_templates.py ``` Step 6: Create Indexes [#step-6-create-indexes] ```bash python create_indexes.py ``` *** 12. Run Database Fixtures [#12-run-database-fixtures] Set the deployment zone to **IN** (India) and load the initial data: ```bash make load-fixtures ``` *** # Git & SSH Setup import Image from 'next/image'; import sshSetup from '@/images/ssh-setup.png'; Git Configuration & SSH Setup [#git-configuration--ssh-setup] 4. Install Git Driver Files and Create SSH Key [#4-install-git-driver-files-and-create-ssh-key] Step 1: Generate and Copy SSH Key [#step-1-generate-and-copy-ssh-key] ```bash cat /Users/creliohealth/.ssh/id_ed25519.pub | pbcopy ``` This command copies your SSH public key to your clipboard. > \[!NOTE] > If you don't have an SSH key yet, generate one first using: > > ```bash > ssh-keygen -t ed25519 -C "your_email@creliohealth.com" > ``` Step 2: Add SSH Key to Bitbucket [#step-2-add-ssh-key-to-bitbucket] 1. Go to [Bitbucket SSH Keys settings](https://bitbucket.org/account/settings/ssh-keys/) 2. Click "Add key" 3. Paste the copied key and save SSH Key Setup *** # Local Environment Setup Guide Local Environment Setup Guide [#local-environment-setup-guide] **For New Joiners** This guide will help you set up the CrelioHealth development environment on your local machine. Follow each section carefully and in order. Overview [#overview] The setup process involves the following steps: 1. **[Prerequisites](/docs/local-environment/prerequisites)** - Install essential development tools 2. **[Git & SSH Setup](/docs/local-environment/git-ssh-setup)** - Configure Git and SSH keys 3. **[Project Repository](/docs/local-environment/project-repository)** - Clone repositories and install Docker 4. **[Docker Build](/docs/local-environment/docker-build)** - Build Docker images 5. **[Database Configuration](/docs/local-environment/database-config)** - Run database migrations 6. **[Elasticsearch](/docs/local-environment/elasticsearch)** - Set up Elasticsearch 7. **[Running the Application](/docs/local-environment/running-application)** - Start services 8. **[Verification](/docs/local-environment/verification)** - Final checks *** **Document Version:** 1.0\ **Last Updated:** January 2026 # Prerequisites Prerequisites - Development Tools Installation [#prerequisites---development-tools-installation] Before starting with the project setup, you need to install essential development tools. 1. Install Visual Studio Code (VSCode) [#1-install-visual-studio-code-vscode] Download and install VSCode from the [official website](https://code.visualstudio.com/). This will be your primary code editor. 2. Install a Terminal Application (Choose One) [#2-install-a-terminal-application-choose-one] Option A: iTerm2 (Recommended) [#option-a-iterm2-recommended] * iTerm2 is an enhanced terminal application for macOS that provides better functionality than the default Terminal app * Download from: [https://iterm2.com/downloads.html](https://iterm2.com/downloads.html) Option B: Warp (Alternative) [#option-b-warp-alternative] * Warp is a modern, AI-powered terminal with excellent features * Download from: [https://www.warp.dev/](https://www.warp.dev/) Choose either iTerm2 or Warp based on your preference. Both work well for development. 3. Install Oh My Zsh [#3-install-oh-my-zsh] Oh My Zsh makes your command line interface more efficient, customizable, and visually appealing. * Installation link: [https://ohmyz.sh/#install](https://ohmyz.sh/#install) * Follow the installation instructions on the website 4. Bitbucket Access [#4-bitbucket-access] Your Bitbucket account and access to the CrelioHealth workspace and repositories will be set up by the onboarding team. You will receive an invitation or direct access as part of your onboarding process. Once you have access, proceed with Git & SSH setup and start working directly in the main repository using feature branches. > \[!IMPORTANT] > If you have not received Bitbucket access, contact your onboarding coordinator or a team admin. *** # Project Repository Setup Project Repository Setup [#project-repository-setup] 5. Clone the CrelioHealth Local Repository [#5-clone-the-creliohealth-local-repository] Clone the main repository to your local machine using Git: [https://bitbucket.org/creliohealth-repo/creliohealth-local/src/main/](https://bitbucket.org/creliohealth-repo/creliohealth-local/src/main/) ```bash git clone git@bitbucket.org:creliohealth-repo/creliohealth-local.git ``` 6. Install Docker [#6-install-docker] Docker is required to run the application services in containers. * Download from: [https://www.docker.com/products/docker-desktop](https://www.docker.com/products/docker-desktop) * Install Docker Desktop and ensure it's running 7. Clone All Required Repositories [#7-clone-all-required-repositories] Follow the steps in the `readme.md` file of the cloned repository. Run the clone script: ```bash ./scripts/clone.sh ``` This script clones all necessary project repositories. 8. Copy Secret Configuration Files [#8-copy-secret-configuration-files] ```bash make copy-secrets ``` This command copies all required secret configuration files to their proper locations. *** # Running the Application Running the Application [#running-the-application] 13. Start Database Services [#13-start-database-services] ```bash make run-db ``` This starts all database-related services (MySQL, Elasticsearch, Redis, etc.) 14. Start All Application Services [#14-start-all-application-services] ```bash make run-all ``` This command starts all backend services at once. *** 15. Build Frontend Assets [#15-build-frontend-assets] For building frontend assets, refer to the detailed instructions: * Slack Link: [Frontend Build Instructions](https://crelio.slack.com/archives/C04BL2BM0A2/p1771226234848669) *** # Verification Verification [#verification] After completing all steps, verify that: * [ ] All Docker containers are running without errors * [ ] Database migrations completed successfully * [ ] Elasticsearch indexes are created * [ ] You can access the application locally Troubleshooting [#troubleshooting] If you encounter any issues during setup, please reach out to the development team on Slack. Common Issues [#common-issues] | Issue | Solution | | ---------------------------------- | -------------------------------------------- | | Docker build fails for `netifaces` | Remove version number from requirements file | | Migration fails | Use `--fake` flag and manually create tables | | Elasticsearch connection error | Ensure Docker containers are running | *** # Migration Migration [#migration] This section contains documentation for ongoing and planned migrations across our infrastructure and services. Current Migrations [#current-migrations] * [GitHub Migration](/docs/migration/github-migration) — Transitioning from Bitbucket to GitHub # Onboarding Guide Onboarding Guide [#onboarding-guide] Welcome to CrelioHealth! This guide will help you get set up and integrated with the team. Getting Started [#getting-started] Checklist [#checklist] Use this checklist to track your onboarding progress: * [ ] Join essential Slack channels * [ ] Set up local development environment * [ ] Complete Git & SSH configuration * [ ] Run the application locally *** **Last Updated:** January 2026 # Slack Channels Slack Channels to Join [#slack-channels-to-join] Join the following Slack channels to stay connected with the team and receive important updates. Engineering Channels [#engineering-channels] | Channel | Purpose | | ------------------ | -------------------------------------- | | `#engineering` | Main engineering team channel | | `#eng-discussions` | Technical discussions and decisions | | `#eng-pr-reviews` | Pull request reviews and notifications | Release & QA [#release--qa] | Channel | Purpose | | ------------------------ | ------------------------------------ | | `#release-p0-approvals` | Critical release approvals | | `#release-notifications` | Release announcements and updates | | `#qa-verification` | QA verification requests and updates | Support & Requests [#support--requests] | Channel | Purpose | | ------------------- | ----------------------------------------------- | | `#technical-issues` | Technical issue discussions and troubleshooting | | `#db-requests` | Database-related requests and queries | *** > \[!TIP] > You can join all channels at once by searching for them in Slack and clicking "Join Channel". # GitHub PR Guidelines PR Review Guidelines — CrelioHealth [#pr-review-guidelines--creliohealth] Branch Protection Rules [#branch-protection-rules] All repositories follow the **standard-web** ruleset preset. Direct pushes to protected branches are blocked — all changes go through pull requests. develop [#develop] * **2 approvals** required * Stale reviews dismissed on new push * All review threads must be resolved * Code owner approval required (CODEOWNERS file) * PR title must follow conventional commits (web & ai repos) * No force push, no deletion hotfix [#hotfix] * **2 approvals** required * Stale reviews dismissed on new push * All review threads must be resolved * Code owner approval required * PR title must follow conventional commits (web & ai repos) * No force push, no deletion main [#main] * **2 approvals** required * Stale reviews dismissed on new push * All review threads must be resolved * Code owner approval required * PR title must follow conventional commits (web & ai repos) * No force push, no deletion production / prod-* / moh-production [#production--prod---moh-production] * **No direct push allowed** — fully locked * No deletion, no force push, no updates * Changes reach production only through branch merges (hotfix → main → production) develop-moh / develop-nrl (MOH/NRL staging) [#develop-moh--develop-nrl-mohnrl-staging] * **1 approval** required * Stale reviews dismissed on new push * All review threads must be resolved * Code owner approval required * No force push, no deletion *** PR Title Format (Conventional Commits) [#pr-title-format-conventional-commits] All PRs to web and ai repos must have titles following the conventional commits format. This is enforced by the **Validate PR Title** status check. Format [#format] ``` (): ``` Allowed types [#allowed-types] `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`, `revert` Examples [#examples] ``` feat(billing): add invoice PDF export fix(auth): handle expired session redirect chore: update dependencies refactor(api): simplify error handling middleware docs(readme): add deployment instructions ``` Rules [#rules] * Type is required, scope is optional * Description must be lowercase and not end with a period * Breaking changes: add `!` after type/scope — `feat(api)!: change response format` *** Automated Merges [#automated-merges] Two GitHub Actions workflows handle branch synchronization automatically using the **CrelioHealth Bot** GitHub App: hotfix → develop [#hotfix--develop] When a PR is merged to `hotfix`, the workflow auto-merges hotfix into `develop`. If there's a conflict, it aborts and sends a Slack notification. main → hotfix & develop [#main--hotfix--develop] When a PR is merged to `main`, the workflow auto-merges main into both `hotfix` and `develop`. Conflicts trigger Slack notifications per branch. These workflows skip manual intervention for routine merges. If a conflict is reported on Slack, a developer must resolve it manually. *** Code Owner Review [#code-owner-review] Every repo has a `.github/CODEOWNERS` file. When `require_code_owner_review` is enabled in rulesets, at least one code owner must approve the PR before merge. Code owners are assigned by repo category: | Category | Owners | | ---------------------------------- | ----------------- | | web / pacs / interfacing / landing | engineering-leads | | qa | qa-devs | | infra | devops | | android | android-leads | | ios | ios-leads | | ai | ai-leads | | docs | docs-maintainers | Some repos have additional reviewers (e.g. backend-reviewers for crelio-app, livehealthapp, fusion, fusion\_worker; frontend-reviewers for livehealth-frontend). *** Review Checklist [#review-checklist] Before approving a PR, reviewers should verify: 1. Code compiles and passes existing tests 2. No unresolved review threads 3. PR title follows conventional commit format 4. No secrets, credentials, or PII in the diff 5. Changes are scoped — one concern per PR 6. Breaking changes are clearly documented in the PR description # PR Review Process PR Review Process [#pr-review-process] Comprehensive code review guidelines, standards, and best practices to ensure code quality, maintainability, and consistency across all projects. Contents [#contents] * **[General Guidelines](/docs/pr-review/general)** - Universal code review rules applicable across all teams and tech stacks * **[Backend Review (Django)](/docs/pr-review/backend-review)** - Backend-specific code review process and guidelines for Django/Python Overview [#overview] Code reviews are a critical part of our development workflow. They help us: 1. **Maintain Code Quality** - Catch bugs, ensure standards compliance, and improve code readability 2. **Share Knowledge** - Learn from each other's approaches and techniques 3. **Ensure Consistency** - Keep the codebase uniform and maintainable 4. **Improve Security** - Identify potential security vulnerabilities before they reach production Review Philosophy [#review-philosophy] * **Constructive Feedback** - Reviews should be helpful, not critical * **Learn and Teach** - Both reviewers and authors should learn from the process * **Timely Reviews** - Respond to PRs within established SLAs * **Thorough but Efficient** - Balance thoroughness with speed General Principles [#general-principles] All code reviews, regardless of the tech stack, should verify: * **Functionality** - Does the code do what it's supposed to do? * **Tests** - Are there adequate tests covering the changes? * **Security** - Are there any security concerns? * **Performance** - Are there any performance implications? * **Documentation** - Is the code well-documented where necessary? Select the appropriate review guide for your project from the menu. # Product Engineering This section outlines our end-to-end product engineering workflow and documents what we build and how we build it—from feature behavior to reusable utilities and implementation guidelines. It is designed to help new team members ramp up quickly, while also serving as a shared reference for Product, Sales, Customer Support, and other cross-functional teams who need clear visibility into system behavior and engineering decisions. # GitHub Release Process GitHub Release Process [#github-release-process] Our release process is fully automated via GitHub Actions dispatch workflows. Release engineers trigger workflows from the GitHub Actions UI — no manual git merges or CLI commands needed. Who Can Trigger Releases [#who-can-trigger-releases] Only members of these teams can trigger release workflows: * `release-devs` — release engineers * `devops` — infrastructure team Unauthorized users will be denied at the `authorize` job. *** Release Flow Overview [#release-flow-overview] *** Step 1: Merge to Main [#step-1-merge-to-main] **Workflow:** `Release: Merge to Main` **Location:** Actions tab → Release: Merge to Main → Run workflow This merges either `develop` or `hotfix` into `main`. Inputs [#inputs] | Input | Options | Description | | ------------ | ------------------------------------ | --------------------------------------------------- | | Release type | `feature-release` / `hotfix-release` | Merges `develop` or `hotfix` into `main` | | Dry run | true / false | Preview what would be merged without making changes | What happens [#what-happens] 1. Validates the triggering user is in `release-devs` or `devops` 2. Fetches `main` and the source branch 3. Merges source → main with `--no-ff` 4. Pushes to `main` 5. On conflict: sends Slack notification with conflicting files, aborts merge After merge [#after-merge] The `Merge main → develop & hotfix` workflow triggers automatically on push to `main`. This keeps `develop` and `hotfix` in sync with `main`. If there's a conflict during realignment, a Slack notification is sent. *** Step 2: Pre-Production (Release PR + Notes) [#step-2-pre-production-release-pr--notes] **Workflow:** `Release: Pre-Production` **Location:** Actions tab → Release: Pre-Production → Run workflow **Restriction:** Can only be triggered from the `main` branch This creates (or updates) a release PR from `main → production` with auto-generated release notes. Inputs [#inputs-1] | Input | Options | Description | | ------------- | ------------ | ----------------------------------------------------- | | Target branch | `production` | Target deployment branch | | Dry run | true / false | Show diff without creating PR | | Notes only | true / false | Update existing PR body with latest notes (no new PR) | | Skip AI | true / false | Skip Gemini AI-enhanced notes for faster runs | What happens [#what-happens-1] 1. Pre-flight checks: verifies `main` is ahead of `production`, warns if `develop` has unmerged commits 2. Walks all commits in `production...main` diff 3. For each commit, finds associated PRs, extracts Jira ticket IDs from PR titles, branch names, and commit messages 4. Links Jira tickets as `[EN-XXXXX](https://crelio.atlassian.net/browse/EN-XXXXX)` 5. Optionally generates AI-enhanced release notes via Gemini (with raw fallback) 6. Creates or updates a PR from `main → production` with labels: `release`, `production`, `ignore-semantic-pull-request` 7. Requests review from `release-devs` team Notes-only mode [#notes-only-mode] If a release PR already exists, use `notes_only: true` to refresh the PR body with the latest notes without creating a new PR. Useful after cherry-picks or last-minute merges. *** Step 3: Merge Release PR [#step-3-merge-release-pr] Once QA approves the release PR on the sanity environment: 1. Go to the release PR (`main → production`) 2. Review the release notes 3. Merge the PR (squash or merge commit) After merge [#after-merge-1] The branch realignment workflows handle syncing `production` changes back to `develop` and `hotfix` automatically. *** Step 4: Release Summary (Cross-Repo) [#step-4-release-summary-cross-repo] **Workflow:** `Release: Summary` **Repo:** [CrelioHealth/release-notes](https://github.com/CrelioHealth/release-notes) **Location:** Actions tab → Release: Summary → Run workflow This generates a combined release summary across all three main repos and posts it to Slack. Inputs [#inputs-2] | Input | Options | Description | | ------- | ------------ | ---------------------------------------- | | Dry run | true / false | Preview summary without posting to Slack | What happens [#what-happens-2] 1. Compares `production...main` diff in each repo: * `livehealth-frontend` * `crelio-app` * `livehealthapp` 2. Extracts Jira ticket IDs from commits, PR titles, and branch names 3. Fetches Jira issue titles from the Jira REST API 4. Deduplicates tickets across repos (same EN-XXXXX in multiple repos shows once) 5. Removes reverted tickets 6. Posts Slack Block Kit message with: * Numbered Jira ticket list with links * Repo tags per ticket * Buttons to release PRs * User and group tags *** Conflict Handling [#conflict-handling] All merge workflows send Slack Block Kit notifications on conflict. Each notification includes: * Header with merge direction (e.g., `hotfix → develop`) * Repo link * Source PR (last merged PR that triggered the merge) * Target PR (last PR on the target branch) * Who triggered it * Conflicting files in a code block * View Diff and Workflow Run buttons Resolving conflicts [#resolving-conflicts] 1. Check the Slack notification for conflicting files 2. Locally: ```bash git checkout git merge origin/ # Resolve conflicts git commit git push origin ``` 3. Or create a PR from `source → target` to resolve in the GitHub UI *** Automated Branch Realignment [#automated-branch-realignment] These workflows run automatically — you don't trigger them manually: | Trigger | Workflow | What it does | | ---------------- | ----------------------------- | ------------------------------------------ | | Push to `main` | Merge main → develop & hotfix | Keeps develop and hotfix in sync with main | | Push to `hotfix` | Merge hotfix → develop | Keeps develop in sync with hotfix changes | Both have concurrency groups to prevent race conditions when multiple PRs merge quickly. *** Quick Reference [#quick-reference] Release day checklist [#release-day-checklist] 1. **Merge to Main** — trigger `Release: Merge to Main` (feature-release or hotfix-release) 2. **Wait** for branch realignment to complete 3. **Pre-Production** — trigger `Release: Pre-Production` (target: production) 4. **QA verifies** on sanity environment 5. **Merge** the release PR 6. **Release Summary** — trigger `Release: Summary` in the release-notes repo 7. **Verify** production deployment Workflow locations [#workflow-locations] | Workflow | Repo | Trigger | | ----------------------------- | ------------- | ---------------------------------- | | Release: Merge to Main | Each repo | Manual dispatch | | Release: Pre-Production | Each repo | Manual dispatch (main branch only) | | Merge main → develop & hotfix | Each repo | Auto on push to main | | Merge hotfix → develop | Each repo | Auto on push to hotfix | | Release: Summary | release-notes | Manual dispatch | Required secrets [#required-secrets] | Secret | Scope | Used by | | ---------------------------- | ------------------ | ------------------------------- | | `APP_ID` | Org | All workflows (GitHub App auth) | | `APP_PRIVATE_KEY` | Org | All workflows (GitHub App auth) | | `SLACK_CONFLICT_WEBHOOK_URL` | Org | Conflict notifications | | `GEMINI_API_KEY` | Org | AI release notes | | `JIRA_EMAIL` | release-notes repo | Release summary | | `JIRA_API_TOKEN` | release-notes repo | Release summary | | `SLACK_RELEASE_WEBHOOK_URL` | release-notes repo | Release summary Slack post | Required variables [#required-variables] | Variable | Value | Used by | | -------------- | ------------------------ | ---------------- | | `GEMINI_MODEL` | `gemini-3.1-pro-preview` | AI release notes | *** **Last Updated:** March 2026 # Release Workflow Release Workflow [#release-workflow] Looking for the step-by-step guide to trigger releases? See [GitHub Release Process](/docs/release-workflow/github-release-process) for the workflow-based release steps. Overview [#overview] Our release process is designed to ensure stable, predictable deployments while maintaining flexibility for urgent fixes. We follow a structured branching strategy with automated testing and staged deployments across multiple environments. Release workflows are automated via GitHub Actions dispatch workflows. Release engineers (`release-devs` and `devops` teams) trigger workflows from the GitHub Actions UI. Key Principles [#key-principles] * **Regular Release Cadence**: Code releases every Tuesday * **Dual-Track Development**: Separate branches for features and hotfixes * **Progressive Testing**: E2E → Sanity → Production * **Regional Deployments**: Phased rollout across production regions *** Branching Strategy [#branching-strategy] We maintain two primary development branches: Develop Branch [#develop-branch] * **Purpose**: Features and changes with large impact areas * **Release Cycle**: Every two weeks * **Use Case**: New features, major refactoring, significant improvements Hotfix Branch [#hotfix-branch] * **Purpose**: Bug fixes and changes with small impact areas * **Release Cycle**: Every week * **Use Case**: Critical bugs, minor fixes, low-risk updates *** Release Schedule [#release-schedule] Weekly Timeline [#weekly-timeline] | Day | Activity | Description | | --------------------------------------------------- | ------------------ | --------------------------------- | | **Tuesday** | Release Day | Code deployed to production | | **Wednesday - Thursday** | Active Development | PRs created and reviewed | | **Friday (Second Half)** or **Monday (First Half)** | Code Freeze | Release PR created against `main` | Release Cycles [#release-cycles] * **Hotfix**: Weekly releases (every Tuesday) * **Develop**: Bi-weekly releases (every two weeks on Tuesday) *** Development Workflow [#development-workflow] 1. Create Pull Request [#1-create-pull-request] Create your PR against the appropriate branch: ```bash # For features or large changes git checkout -b feature/your-feature-name # Create PR against 'develop' branch # For bug fixes or small changes git checkout -b fix/your-bug-fix # Create PR against 'hotfix' branch ``` 2. Merge to Development Branch [#2-merge-to-development-branch] Once your PR is approved and merged: * **Develop branch** → Automatically deployed to `e2e-lhapp.crelop.solutions` * **Hotfix branch** → Automatically deployed to `e2e-lhapp.crelop.solutions` 3. Testing on E2E Environment [#3-testing-on-e2e-environment] After merge, your changes are available on the E2E environment: * **URL**: [https://e2e-lhapp.crelop.solutions](https://e2e-lhapp.crelop.solutions) * **Purpose**: Developer testing and verification * **Action**: Verify your changes work as expected 4. QA Verification [#4-qa-verification] Once you've verified your changes: 1. Update the ticket status 2. Request QA team to verify 3. QA performs functional testing *** Release Process [#release-process] Release workflows are triggered from the GitHub Actions UI. See the [GitHub Release Process](/docs/release-workflow/github-release-process) page for the complete step-by-step guide. Summary [#summary] 1. **Merge to Main** — `Release: Merge to Main` workflow merges `develop` or `hotfix` into `main` 2. **Branch realignment** — `main → develop & hotfix` runs automatically 3. **Pre-Production** — `Release: Pre-Production` creates a release PR with AI-generated notes 4. **QA verification** — QA tests on sanity environment 5. **Merge release PR** — `main → production` 6. **Release Summary** — Cross-repo Slack notification from `release-notes` repo Phase 1: Code Freeze [#phase-1-code-freeze] **Timeline**: Friday (second half) or Monday (first half) * Development teams stop merging new changes * Release PR created against `main` branch * Final review of included changes Phase 2: Sanity Environment [#phase-2-sanity-environment] **Environment**: `main` branch * **URL**: [https://sanity-lhapp.crelio.solutions](https://sanity-lhapp.crelio.solutions) * **Purpose**: QA automation suite execution * QA team runs comprehensive sanity tests * Regression testing performed Phase 3: Production Deployment [#phase-3-production-deployment] **Triggered**: After QA approval 1. `main` branch merged to `prod-` branches 2. Serves production servers per region 3. Deployment pipeline triggered manually ```bash # Regional production branches prod-us prod-eu prod-asia # ... other regions ``` *** Environment Hierarchy [#environment-hierarchy] *** Quick Reference [#quick-reference] When to Use Each Branch [#when-to-use-each-branch] **Use `develop` when:** * Adding new features * Making architectural changes * Refactoring with wide impact * Changes affecting multiple services **Use `hotfix` when:** * Fixing production bugs * Making small UI tweaks * Updating copy or content * Changes with isolated impact Important URLs [#important-urls] | Environment | URL | Purpose | | ----------- | ------------------------------------------------------------------------------ | -------------------- | | E2E | [https://e2e-lhapp.crelop.solutions](https://e2e-lhapp.crelop.solutions) | Developer testing | | Sanity | [https://sanity-lhapp.crelio.solutions](https://sanity-lhapp.crelio.solutions) | QA automation | | Production | Various regional URLs | Live customer-facing | *** Best Practices [#best-practices] ✅ Do's [#-dos] * Test your changes thoroughly on E2E before requesting QA * Create PRs well before code freeze to allow time for review * Communicate with team if your change might affect others * Monitor deployments after production release ❌ Don'ts [#-donts] * Don't merge to `develop`/`hotfix` during code freeze * Don't skip E2E verification before involving QA * Don't create large PRs against `hotfix` branch * Don't trigger production deployment without QA approval *** **Last Updated:** March 2026 # Services Services [#services] Available Services [#available-services] * **LiveHealth Frontend** - TypeScript frontend application * **LiveHealthApp** - Python application service * **ClickStack HyperDX Infra** - Self-hosted observability service for OpenTelemetry traces, logs, metrics, and debugging * **Phoenix Search** - FastAPI patient and user search service * **Fusion** - Python backend service * **Fusion Worker** - Background worker service * **Crelio App** - Crelio App backend service * **Crelio AI** - AI service for healthcare automation # Webhook Actions Webhook Actions [#webhook-actions] This document provides a comprehensive list of webhook actions configured in the system. Each action is identified by a unique **Webhook ID** and serves a specific purpose in the integration workflows. These IDs are critical for ensuring accurate event handling and seamless communication between systems. List of Webhook Actions [#list-of-webhook-actions] | **Webhook ID** | **Action Name** | **Action Description** | | -------------- | ------------------------------------------- | ----------------------------------------------------------------- | | 1 | Bill Generation | Triggered when a new bill is generated. | | 2 | Bill Update | Triggered when an existing bill is updated. | | 3 | Bill Cancel | Triggered when a bill is canceled. | | 4 | Bill Reset | Triggered when a bill is reset to its initial state. | | 5 | Bill Refund | Triggered when a refund is processed for a bill. | | 6 | Sample Receive | Triggered when a sample is received in the lab. | | 7 | Sample Reject | Triggered when a sample is rejected. | | 8 | Sample Dismiss | Triggered when a sample is dismissed. | | 9 | Sample Redraw | Triggered when a sample needs to be redrawn. | | 10 | Report Save (With Values) | Triggered when a report is saved with test values. | | 11 | Report Save (Without Values) | Triggered when a report is saved without test values. | | 12 | Report Signed | Triggered when a report is signed by an authorized person. | | 13 | Report Submit | Triggered when a report is submitted. | | 14 | Test Dismiss | Triggered when a test is dismissed. | | 15 | Report Submit (With Values) | Triggered when a report with values is submitted. | | 16 | With Attachment | Triggered when a report is submitted with an attachment. | | 17 | Bill Settlement | Triggered when a bill is settled. | | 18 | Add Test To Bill | Triggered when a test is added to an existing bill. | | 19 | Report PDF (Webhook) | Triggered when a report is submitted. | | 20 | With PDF | Triggered when a report is submitted. | | 21 | With Structured | Triggered when a report is submitted in a structured format. | | 22 | Create & Update Debtor/Patient | Triggered to create or update debtor/patient information. | | 23 | Post Invoice | Triggered to post an invoice. | | 24 | Consolidated Report Email | Triggered to send a consolidated report via email. | | 25 | Consolidated Report Submit (Profile) | Triggered to submit a consolidated report for a profile. | | 26 | Consolidated All Report Submit | Triggered to submit all consolidated reports. | | 27 | Portea Integration | Triggered for Portea integration workflows. | | 28 | HL7 Integration | Triggered for HL7 integration workflows. | | 29 | Quest 4 Health | Triggered for Quest 4 Health integration workflows. | | 30 | Sample Collect | Triggered when a sample is collected. | | 31 | Sample Uncollect | Triggered when a sample collection is undone. | | 32 | Appointment Status | Triggered to update the status of an appointment. | | 33 | Report Submit Consolidate PDF FTP | Triggered to submit a consolidated report as a PDF via FTP. | | 34 | Report Submit Consolidate Base 64 | Triggered to submit a consolidated report in Base64 format. | | 35 | Report Submit Consolidate Structured Format | Triggered to submit a consolidated report in a structured format. | | 36 | Report Submit (With Values) COVID-19 | Triggered to submit a report with values for COVID-19 tests. | | 37 | Home Collection Assigned/Reassigned | Triggered when a home collection is assigned or reassigned. | | 38 | Home Collection Reschedule/Cancel | Triggered when a home collection is rescheduled or canceled. | | 39 | Bill Outsource | Triggered when a bill is outsourced. | | 40 | Tests Outsourced as Quest Order | Triggered when tests are outsourced as a Quest order. | | 41 | Bill Generation For Promotions | Triggered when a bill is generated for promotions. | | 42 | Report Submit (Critical Values) | Triggered to submit a report with critical values. | | 43 | Home Collection Initiated | Triggered when a home collection is initiated. | | 44 | Home Collection Billing | Triggered for billing related to home collections. | | 45 | Appointment Booking Initiated | Triggered when an appointment booking is initiated. | | 46 | Appointment Booking Billing | Triggered for billing related to appointment bookings. | | 47 | Billing Webhook - HIPAA | Triggered for HIPAA-compliant billing workflows. | | 48 | Test Cancel through HL7 Integration | Triggered to cancel a test via HL7 integration. | | 49 | Bill Generation HL7 for SFTP (Outbound) | Triggered after generating a bill for Integrations. | | 50 | Report Submit with SFTP | Triggered to submit a report via SFTP. | | 51 | Order Generation using SFTP (Inbound) | Triggered to generate an order via SFTP (inbound). | | 52 | Organisation Update | Triggered to update organization details. | | 53 | Attach Report through JSON Integration | Triggered to attach a report via JSON integration. | | 54 | Smart Report Generation | Triggered for smart report generation workflows. | | 55 | Create Batch | Triggered to create a batch. | | 56 | Send Reports via Fax | Triggered to send reports via fax. | | 57 | New Patient Registered | Triggered when a new patient is registered. | | 58 | Update Patient Info | Triggered to update patient information. | | 59 | Accession-wise Report Submit | Triggered to submit reports accession-wise. | | 60 | Claim Submission Webhook | Triggered for claim submission workflows. | | 61 | Claim Approval Webhook (Remittance) | Triggered for claim approval workflows (remittance). | | 62 | Create/Update/Delete Invoice | Triggered to create, update, or delete an invoice. | | 63 | Sample ID Update | Triggered to update a sample ID. | Additional Details [#additional-details] * These webhook actions are configured in the database and are essential for various integration workflows. * Each action is associated with a specific event or process, ensuring seamless communication between systems. * Proper configuration and testing of these webhook actions are crucial for the smooth functioning of integrations. # Backend Backend [#backend] Architecture Overview [#architecture-overview] The SPA backend is entirely within `livehealthapp` (Django/Py2). It handles three responsibilities: controlling which labs get SPA vs. legacy builds via a Redis feature flag, serving the SPA HTML template, and routing login redirects to the correct URL. System Design Diagram [#system-design-diagram] The is_spa_enabled() Function [#the-is_spa_enabled-function] Source: [`labs/cacheFunction.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/labs/cacheFunction.py) This is the central feature flag that controls SPA vs. legacy mode. It's a simple Redis cache lookup: ```python def is_spa_enabled(lab_id): """ Flag will be used to get/set SPA (Single Page App). """ if not lab_id: return False cache_key = "is_spa_enabled_{lab_id}".format(lab_id=lab_id) return cache.get(cache_key) or False ``` Key Characteristics [#key-characteristics] | Property | Value | | -------------- | ------------------------------------------------------------------------------------- | | **Storage** | Redis (Django cache backend) | | **Key format** | `is_spa_enabled_{lab_id}` | | **Default** | `False` (absent key = SPA disabled) | | **TTL** | No expiry — persists until explicitly deleted | | **Set by** | Account creation (`selfserved/account_creation/account.py`) or manually via Redis CLI | CrelioDashboardView [#creliodashboardview] Source: [`livehealth_4/views.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/livehealth_4/views.py) This is the Django view that serves the SPA HTML: ```python class CrelioDashboardView(View): def get(self, request, *args, **kwargs): lab_id = request.session.get('labId') is_support_login = request.session.get('isSupportLogin') if not lab_id or not (is_spa_enabled(lab_id) or is_support_login): return HttpResponseRedirect('/') rendered_html = render( request, 'build_assets/dashboard/build/crelio-dashboard.html' ) return prepare_favicon_change_response(request, rendered_html) ``` Access Control [#access-control] | Check | Purpose | | ---------------------------------------------- | -------------------------------------------------------------------- | | `lab_id` must exist | Rejects unauthenticated requests | | `is_spa_enabled(lab_id)` OR `is_support_login` | Only SPA-enabled labs or support users can access | | Falls back to `HttpResponseRedirect('/')` | Non-SPA labs hitting `/crelio-dashboard/` are sent to the login page | Support users (`isSupportLogin = True`) always get access to the SPA, regardless of whether the lab has `is_spa_enabled` set. This is intentional — support needs the unified view for faster debugging. Login Flow — URL Resolution [#login-flow--url-resolution] The login flow determines the user's `loginURL` based on their role and SPA status. This happens in multiple places across the codebase. Session Helpers [#session-helpers] Source: [`logins/utils/session_helpers.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/logins/utils/session_helpers.py) The session builder determines where a user lands after login: ```python # For operations staff login_url = "/crelio-dashboard/#/operation" if is_spa_enabled(user.labId) else '/waitingList/' # For accession staff login_url = "/crelio-dashboard/#/accession/" if is_spa_enabled(user.labId) else "/sample-accession/" # For finance staff login_url = "/crelio-dashboard/#/finance/dashboard" if is_spa_enabled(user.labId) else "/finance-new/#/dashboard" # For registration staff login_url = "/crelio-dashboard/#/registration/directRegistration" if is_spa_enabled(user.labId) else "/billing/#directRegistration" # For inventory staff login_url = "/crelio-dashboard/#/inventory/inventory-dashboard" if is_spa_enabled(user.labId) else "/inventory-v5/" ``` The `is_spa_enabled` flag is also stored in the session itself for frontend and template access: ```python # In session_helpers.py - get_lab_user_session() "is_spa_enabled": is_spa_enabled(getattr(user, "labId_id", False)) or False, ``` SPA URL Mapper [#spa-url-mapper] Source: [`livehealth_4/decorators/utils.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/livehealth_4/decorators/utils.py) ```python SPA_URL_MAPPER = { "/accountOverview/": "/crelio-dashboard/#/admin/account-overview", "/billing/#directRegistration": "/crelio-dashboard/#/registration/", "/waitingList/": "/crelio-dashboard/#/operation", "/finance-new/#/dashboard": "/crelio-dashboard/#/finance/", "/sample-accession/": "/crelio-dashboard/#/accession/", "/inventory-v5/": "/crelio-dashboard/#/inventory", "/crm/dashboard/": "/crelio-dashboard/#/crm", "/referral-management/": "/crelio-dashboard/#/finance/referral-management/referral-list", "/adminTestList-v5/#/admin/account-overview": "/crelio-dashboard/#/admin/account-overview" } ``` This dictionary maps every legacy module URL to its SPA equivalent. It's used by the `mapped_spa_url()` function during login and lab-switching. Version Detection — v4 vs v5 [#version-detection--v4-vs-v5] Before any SPA logic runs, the system first determines whether the lab is on **v4** (legacy app) or **v5** (React builds). This is controlled by two session values: | Condition | Version | UI | | -------------------------- | ------- | --------------------------------------- | | `reactVersionAccess == 1` | **v5** | React single-build modules | | `newRegistrationPage == 3` | **v5** | React registration and accession builds | | Neither of the above | **v4** | Legacy app | **v5 is the prerequisite for SPA.** A lab must be on v5 before it can be SPA-enabled. Labs on v4 always receive the legacy app regardless of the `is_spa_enabled` flag. Within v5, the SPA is an additional opt-in controlled by the `is_spa_enabled` Redis flag: | Version | `is_spa_enabled` | What the user gets | | ------- | ------------------- | ----------------------------------------------------- | | v4 | N/A | Legacy app | | v5 | `False` (or absent) | React single-build modules (one HTML per module) | | v5 | `True` | SPA — unified `crelio-dashboard.html` for all modules | The mapped_spa_url() Function [#the-mapped_spa_url-function] Source: [`livehealth_4/decorators/version_decorator.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/livehealth_4/decorators/version_decorator.py) This function translates legacy `loginURL` values to SPA URLs during lab switching. It checks version first, then SPA eligibility: ```python def mapped_spa_url(session, lab_id, use_spa_routing=False): if not session or not lab_id: return "/" login_url = session.get('loginURL') react_version_access = session.get("reactVersionAccess") new_registration_page = session.get("newRegistrationPage") is_support_login = session.get('isSupportLogin') # Step 1: Version check — v5 non-SPA builds return URL as-is # reactVersionAccess == 1 OR newRegistrationPage == 2 → v5 but not SPA-eligible if any([react_version_access == 1, new_registration_page == 2]): return login_url # Step 2: Lab switch with SPA enabled if use_spa_routing and is_spa_enabled(lab_id): return SPA_URL_MAPPER.get(login_url) or login_url # Step 3: Normal login — check if v5 SPA-eligible # reactVersionAccess == 2 OR newRegistrationPage == 3 → v5 SPA-eligible if is_spa_enabled(lab_id) and ( react_version_access == 2 or new_registration_page == 3 ) or is_support_login: return SPA_URL_MAPPER.get(login_url) or login_url return login_url ``` Decision Tree [#decision-tree] The asset_provider Decorator [#the-asset_provider-decorator] Source: [`livehealth_4/decorators/version_decorator.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/livehealth_4/decorators/version_decorator.py) For legacy module views that support multiple build versions (v4 Jinja templates vs. v5 React builds), the `asset_provider` decorator determines which template to render: ```python @access_labUser() @asset_provider("accession") def sample_accession_v4_view(request, template): rendered_html = render(request, template, {...}) return prepare_favicon_change_response(request, rendered_html) ``` | Module | Version Key | Version 0-2 | Version 3-4 | | ------------ | ------------------------- | --------------- | ------------------------------------------------ | | Registration | `defaultRegistrationPage` | Jinja templates | `build_assets/dashboard/build/registration.html` | | Accession | `defaultRegistrationPage` | Jinja templates | `build_assets/dashboard/build/accession.html` | | Operations | `reactVersionAccess` | Jinja templates | `build_assets/dashboard/build/operations.html` | | Finance | `reactVersionAccess` | Jinja templates | `build_assets/dashboard/build/finance.html` | The `asset_provider` handles v4 → v5 template resolution for the legacy single-build mode. It does **not** handle SPA routing — that's done via the login flow and `CrelioDashboardView`. Template Serving — SPA vs. Legacy [#template-serving--spa-vs-legacy] | Aspect | SPA Mode | Legacy Single-Build Mode | | -------------------- | --------------------------------------------- | ---------------------------------------------------- | | **Template served** | `crelio-dashboard.html` | `registration.html`, `operations.html`, etc. | | **Django views hit** | Single `CrelioDashboardView` | Module-specific views (`registration_view_v5`, etc.) | | **Module switching** | Client-side hash route change | Full browser navigation → Django → new template | | **Static assets** | Single JS bundle (with code-split chunks) | Per-module JS bundles | | **Public path** | `/static/build_assets/CrelioDashboard/build/` | `/static/build_assets/dashboard/build/` | Key Backend Locations [#key-backend-locations] | File | Purpose | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | | [`labs/cacheFunction.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/labs/cacheFunction.py) | `is_spa_enabled()` — Redis feature flag lookup | | [`livehealth_4/views.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/livehealth_4/views.py) | `CrelioDashboardView` — serves `crelio-dashboard.html` | | [`livehealth_4/decorators/version_decorator.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/livehealth_4/decorators/version_decorator.py) | `mapped_spa_url()`, `asset_provider` decorator | | [`livehealth_4/decorators/utils.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/livehealth_4/decorators/utils.py) | `SPA_URL_MAPPER`, `ASSET_MAPPER` dictionaries | | [`logins/utils/session_helpers.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/logins/utils/session_helpers.py) | Login URL resolution with `is_spa_enabled` checks | | [`logins/views.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/logins/views.py) | Login view with SPA session setup | | [`selfserved/account_creation/account.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/selfserved/account_creation/account.py) | Sets `is_spa_enabled` for newly created labs | # Design Decisions Design Decisions & Architecture [#design-decisions--architecture] *** Key Design Decisions & Constraints [#key-design-decisions--constraints] SPA via CrelioDashboard vs. Micro-Frontends [#spa-via-creliodashboard-vs-micro-frontends] **Decision:** Unify all staff-facing modules into a single `CrelioDashboard` React application rather than using Module Federation or a micro-frontend architecture. **Why:** The modules share \~80% of their dependencies (React, Redux, AG Grid, i18n, reactstrap, axios) and all operate within the same staff session context. Module Federation would introduce: * Runtime dependency resolution overhead * Version skew risks across independently deployed modules * Complex shared state management (each micro-frontend would need its own Redux store or a shared event bus) * Additional CI/CD complexity (each module deployed independently) Since all SPA-eligible modules (Registration, Accession, Operations, Finance, Admin, LabAdmin, Inventory, CRM Dashboard) are in the same repository (`livehealth-frontend`) and share the same release cadence, a monolithic SPA with code-splitting per module gives the same lazy-loading benefits without federation overhead. **Trade-off:** All modules must be built and deployed together. A change to the Finance module requires rebuilding the entire SPA. This is acceptable because the team already follows a single-branch, unified release model. Hash-Based Routing (/#/) vs. HTML5 History API [#hash-based-routing--vs-html5-history-api] **Decision:** Use hash-based routing (`/crelio-dashboard/#/finance/dashboard`) instead of clean URLs (`/crelio-dashboard/finance/dashboard`). **Why:** The frontend is served by Django, which owns URL routing. With HTML5 History API: * Every frontend route would need a corresponding Django URL pattern pointing to `CrelioDashboardView` * Deep-linking would fail if Django didn't recognize the path (404 before React could handle it) * Every new frontend route addition would require a backend URL config change With hash routing, Django only needs a single URL pattern (`/crelio-dashboard/`) — everything after `#` is handled entirely client-side. This decouples frontend route additions from backend deployments. **Trade-off:** URLs are less "clean" and hash fragments don't participate in server-side redirects. This is acceptable because the SPA is an internal tool (not SEO-sensitive) and all navigations happen client-side after initial load. is_spa_enabled as a Redis Flag vs. Database Column [#is_spa_enabled-as-a-redis-flag-vs-database-column] **Decision:** Store the SPA enablement flag in Redis (`is_spa_enabled_{lab_id}`) rather than as a column on the `labs` or `labFeatures` table. **Why:** * **Performance** — the flag is checked on every login and every module-level URL redirect. A Redis lookup is sub-millisecond; a DB query adds latency. * **Rollback speed** — if the SPA causes issues for a lab, support can flip the flag instantly via Redis without touching the DB or deploying code. * **Gradual rollout** — new labs can have the flag set automatically at account creation time without requiring a migration. **Trade-off:** The flag is not queryable via SQL (can't do `SELECT COUNT(*) FROM labs WHERE is_spa_enabled = 1`). For admin reporting, support must use the Redis CLI or a dedicated admin endpoint. The flag also lacks audit trail — there's no record of when the flag was changed or by whom. **Implementation:** ```python # labs/cacheFunction.py def is_spa_enabled(lab_id): """ Flag will be used to get/set SPA (Single Page App). """ if not lab_id: return False cache_key = "is_spa_enabled_{lab_id}".format(lab_id=lab_id) return cache.get(cache_key) or False ``` Eager Session Initialization vs. Lazy Per-Module Loading [#eager-session-initialization-vs-lazy-per-module-loading] **Decision:** Fetch all module settings (operations settings, report settings, department list, doctor list, CRM permissions, registration config, bill settings, inventory settings) during the SPA bootstrap rather than lazily when the user navigates to a module. **Why:** In the legacy single-build architecture, each module independently fetched its required settings on mount. If the SPA deferred these fetches too, the first navigation to each module would incur the same delay as the old full-page reload. By front-loading all settings during the initial SPA bootstrap: * Module switches become **instant** — no loading spinners, no network delays * Settings are available in Redux for cross-module features (e.g., a Registration form that checks Operations settings) **Trade-off:** The initial SPA load time is longer (more API calls during bootstrap). This is mitigated by: 1. Using `Promise.all` / `axiosInstance.all` for parallel fetching 2. The bootstrap happens once per session, not on every module switch 3. The total initialization time is still less than two legacy module loads combined SPA_URL_MAPPER: Static Mapping vs. Dynamic URL Rewriting [#spa_url_mapper-static-mapping-vs-dynamic-url-rewriting] **Decision:** Use a hardcoded dictionary (`SPA_URL_MAPPER`) to map legacy module URLs to SPA hash routes. ```python SPA_URL_MAPPER = { "/accountOverview/": "/crelio-dashboard/#/admin/account-overview", "/billing/#directRegistration": "/crelio-dashboard/#/registration/", "/waitingList/": "/crelio-dashboard/#/operation", "/finance-new/#/dashboard": "/crelio-dashboard/#/finance/", "/sample-accession/": "/crelio-dashboard/#/accession/", "/inventory-v5/": "/crelio-dashboard/#/inventory", "/crm/dashboard/": "/crelio-dashboard/#/crm", "/referral-management/": "/crelio-dashboard/#/finance/referral-management/referral-list", "/adminTestList-v5/#/admin/account-overview": "/crelio-dashboard/#/admin/account-overview" } ``` **Why:** The mapping is finite and well-known. There are exactly 9 legacy module URLs that need mapping. A regex-based or algorithmic approach would be over-engineered and harder to debug. The static dictionary: * Is self-documenting — you can see every mapping at a glance * Is easy to extend — adding a new mapping is a one-line change * Has zero runtime parsing cost **Constraint:** If a new module URL is added without updating `SPA_URL_MAPPER`, it will fall through to the legacy URL, which will cause a full-page reload instead of SPA navigation. Module-Aware Sidebar via URL Path Matching [#module-aware-sidebar-via-url-path-matching] **Decision:** Determine the active module from the URL path rather than maintaining explicit module-selection state in Redux. ```typescript const matchingModule = useMemo(() => { const pathParts = location.pathname.split("/").filter(Boolean); const moduleName = pathParts[0]?.toLowerCase(); const moduleByRoute = modules.find((module) => module.menu.some((item) => { const itemRouteBase = item.route?.split("/").filter(Boolean)[0]?.toLowerCase(); return location.pathname === item.route || pathParts[0]?.toLowerCase() === itemRouteBase; }) ); const moduleByName = modules.find( (module) => module.selectedModule?.toLowerCase() === moduleName ); return moduleByRoute || moduleByName || defaultModule; }, [location.pathname, modules]); ``` **Why:** This makes the sidebar purely reactive to the URL. If a user deep-links to `/finance/referral-management/referral-list`, the sidebar correctly shows Finance menus without any explicit "select Finance" action. This also means browser back/forward navigation correctly updates the sidebar. **Trade-off:** The resolution logic runs on every route change. With \~7 modules × \~10 menu items each, this is \~70 string comparisons per navigation — negligible. *** Architectural Rationale [#architectural-rationale] Why Keep Both SPA and Single-Build? [#why-keep-both-spa-and-single-build] The dual-mode architecture exists for a critical reason: **backward compatibility and safe migration**. | Concern | Why Both Modes | | ------------------- | ---------------------------------------------------------------------------------------------------------------- | | **Gradual rollout** | Labs have different configurations, customizations, and user workflows. A big-bang migration is risky. | | **Rollback safety** | If the SPA has a bug affecting a specific lab's workflow, support can disable it instantly without a deployment. | | **CI/CD** | Jenkins builds both modes in the same pipeline. If a single-build regression is found, the fix works for both. | | **Testing** | QA can validate both modes in parallel before enabling SPA for a customer. | Why Hash Routing Instead of Migrating to React Router v6 + BrowserRouter? [#why-hash-routing-instead-of-migrating-to-react-router-v6--browserrouter] The codebase uses `react-router-dom@5.3.4` with `HashRouter` semantics (via hash fragments in `useLocation`). Migrating to `BrowserRouter` in React Router v6 would require: 1. Django URL patterns for every frontend route (or a catch-all `re_path(r'^crelio-dashboard/.*$', ...)`) 2. React Router v5 → v6 migration (breaking API changes: `Switch` → `Routes`, `component` → `element`, removed `exact`) 3. Testing every deep-link, bookmark, and shared URL This migration is planned but deferred to avoid coupling it with the SPA rollout. *** Known Constraints & Gotchas [#known-constraints--gotchas] | Constraint | Impact | Mitigation | | ----------------------------------------- | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | **Hash-based routing** | URLs contain `/#/` which is unusual for users sharing links | Planned migration to HTML5 History API in a future phase | | **All modules loaded eagerly** | SPA bootstrap fetches settings for all modules, even if the user only uses Registration | `React.lazy()` + code splitting ensures only the active module's components are loaded; settings APIs are lightweight | | **Redis flag has no audit trail** | Cannot track when `is_spa_enabled` was changed or by whom | Wrap flag changes in a support dashboard action that logs to `crelio_data_migrations` | | **SPA\_URL\_MAPPER must be kept in sync** | New legacy URLs won't redirect to SPA if not added to the mapper | The mapper is in `decorators/utils.py` — easy to grep and update | | **Support users always get SPA** | Support login bypasses the `is_spa_enabled` check | This is intentional for faster debugging, but means support may see different behavior than the lab they're supporting if that lab has SPA disabled | | **Standalone modules don't benefit** | DoctorLogin, ReferralLogin, OrganisationLogin remain separate full-page apps | These serve different user roles with different sessions — unifying them would require a multi-session architecture | | **No server-side rendering** | Initial load depends on client-side React bootstrap | Acceptable for internal tool; mitigated by loading spinner in `index.html` | *** Extensibility Guide [#extensibility-guide] | What you want to do | How to do it | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Add a new module to the SPA** | 1. Create a `useXxxRoutes()` and `useXxxSidebarMenu()` hook in `CrelioDashboard/RoutesAndMenus/`
2. Register them in `routes.tsx`'s `modules` array
3. Add the legacy URL → SPA URL mapping in `SPA_URL_MAPPER`
4. Update the Django login flow to redirect to the SPA URL when `is_spa_enabled` | | **Enable SPA for a lab** | Set `is_spa_enabled_{lab_id}` to `True` in Redis | | **Disable SPA for a lab** | Delete the `is_spa_enabled_{lab_id}` key from Redis (or set to `False`) | | **Add a new route to an existing module** | Add a `CrelioRoute` entry in the module's `*RoutesAndMenu` file. No backend changes needed. | | **Add a new sidebar item** | Add a `CrelioSidebarItem` entry in the module's `useXxxSidebarMenu()` hook | | **Migrate to HTML5 History API** | 1. Add a Django catch-all for `/crelio-dashboard/*`
2. Switch from hash-based `useLocation` to `BrowserRouter`
3. Update all internal links to remove `/#/`
4. Update `SPA_URL_MAPPER` to remove hash fragments | # Developer Handbook Developer Handbook [#developer-handbook] A practical, step-by-step guide for developers working with the `CrelioDashboard` SPA. Use this as a quick reference when adding features, debugging issues, or onboarding. Adding a New Module to the SPA [#adding-a-new-module-to-the-spa] Step 1: Create the Routes and Menu File [#step-1-create-the-routes-and-menu-file] Create a new file in `RoutesAndMenus/`: ``` src/modules/CrelioDashboard/RoutesAndMenus/myModuleRoutesAndMenu.tsx ``` Export two hooks — one for routes and one for sidebar: ```typescript import { CrelioRouteType, CrelioSidebarItem } from "../routeTypes"; import { useTranslation } from "react-i18next"; import React from "react"; // Lazy-load your components const MyDashboard = React.lazy(() => import("src/components/MyModule/Dashboard")); const MySettings = React.lazy(() => import("src/components/MyModule/Settings")); export const useMyModuleRoutes = (): CrelioRouteType[] => { return [ { path: "/my-module", Component: MyDashboard }, { path: "/my-module/settings", Component: MySettings }, ]; }; export const useMyModuleSidebarMenu = (): CrelioSidebarItem[] => { const { t } = useTranslation(); return [ { name: t("Dashboard"), icon: "fa-chart-line", route: "/my-module" }, { name: t("Settings"), icon: "fa-cog", route: "/my-module/settings" }, ]; }; ``` Step 2: Register in routes.tsx [#step-2-register-in-routestsx] Open `RoutesAndMenus/routes.tsx` and: 1. **Import** your hooks: ```typescript import { useMyModuleRoutes, useMyModuleSidebarMenu } from "./myModuleRoutesAndMenu"; ``` 2. **Call** the hooks inside the `Routes` component: ```typescript const myModuleRoutes = useMyModuleRoutes(); const myModuleSidebarMenu = useMyModuleSidebarMenu(); ``` 3. **Add** to the `modules` array: ```typescript const modules: Module[] = useMemo(() => [ // ... existing modules { menu: myModuleSidebarMenu, routes: myModuleRoutes, selectedModule: t("MyModule") }, ], [/* dependencies */]); ``` Step 3: Register in navigate.ts [#step-3-register-in-navigatets] Add your module's URL prefix to the `moduleNames` array: ```typescript export const moduleNames: string[] = [ "registration", "accession", "operation", "finance", "crm", "admin", "inventory", "my-module", // ← add here ]; ``` This ensures the `navigate()` utility correctly prefixes intra-module URLs. Step 4: Add Backend URL Mapping [#step-4-add-backend-url-mapping] In `livehealth_4/decorators/utils.py`, add the legacy-to-SPA URL mapping: ```python SPA_URL_MAPPER = { # ... existing mappings "/my-module-legacy/": "/crelio-dashboard/#/my-module", } ``` Step 5: Update Session Helpers [#step-5-update-session-helpers] In `logins/utils/session_helpers.py`, add the SPA login URL for your module's user role: ```python # For my-module staff login_url = "/crelio-dashboard/#/my-module" if is_spa_enabled(user.labId) else "/my-module-legacy/" ``` Checklist [#checklist] * [ ] Created `RoutesAndMenus/myModuleRoutesAndMenu.tsx` with routes + sidebar hooks * [ ] Registered hooks in `routes.tsx` `modules` array * [ ] Added module name to `moduleNames` in `navigate.ts` * [ ] Added URL mapping in `SPA_URL_MAPPER` (backend) * [ ] Updated session helpers for login redirect (backend) * [ ] All route paths start with `/my-module/` prefix * [ ] Components are lazy-loaded with `React.lazy()` *** Adding Routes to an Existing Module [#adding-routes-to-an-existing-module] Adding a Simple Route [#adding-a-simple-route] Add a new entry to your module's `useXxxRoutes()` hook: ```typescript export const useOperationsRoutes = (): CrelioRouteType[] => { return [ // ... existing routes { path: "/operation/my-new-page", Component: MyNewPage }, ]; }; ``` Adding a Route with Permission Check [#adding-a-route-with-permission-check] Use `routeProps` to restrict access: ```typescript { path: "/operation/admin-only-page", Component: AdminOnlyPage, routeProps: { requiredPermissions: ["canAccessAdminPage"], }, } ``` The `AuthRoute` component checks `routeProps.requiredPermissions` against the user's session permissions before rendering. Adding a Redirect [#adding-a-redirect] Use the redirect route type: ```typescript { from: "/operation/old-path", to: "/operation/new-path", isRedirect: true, } ``` *** Adding Sidebar Items [#adding-sidebar-items] Simple Menu Item [#simple-menu-item] ```typescript { name: t("My Page"), icon: "fa-file", route: "/operation/my-page" } ``` Menu Item with Submenu [#menu-item-with-submenu] ```typescript { name: t("Reports"), icon: "fa-chart-bar", submenuArr: [ { name: t("Daily Report"), route: "/operation/reports/daily" }, { name: t("Monthly Report"), route: "/operation/reports/monthly" }, ], } ``` Conditional Menu Item (Permission-Based) [#conditional-menu-item-permission-based] ```typescript { name: t("Admin Settings"), icon: "fa-cog", route: "/operation/admin-settings", hasPermission: () => sessionData?.isAdmin === true, } ``` Hidden Menu Item [#hidden-menu-item] Use `hidden: true` for items that need a route but shouldn't appear in the sidebar: ```typescript { name: t("Hidden Page"), route: "/operation/hidden-page", hidden: true, } ``` *** Navigation Patterns [#navigation-patterns] Within the Same Module [#within-the-same-module] Use `navigate()` with relative-style paths — the utility auto-prefixes: ```typescript import { navigate } from "src/utils/navigate"; // From within Operations module navigate("/testwise-waiting-list/all-tests"); // → resolves to /operation/testwise-waiting-list/all-tests ``` Cross-Module Navigation [#cross-module-navigation] Explicitly include the target module name: ```typescript navigate("/finance/dashboard"); // → no prefix added, navigates directly to Finance ``` With Route State [#with-route-state] ```typescript navigate({ pathname: "/patient-details", state: { patientId: 123, fromModule: "registration" }, }); ``` Replace History (No Back Button) [#replace-history-no-back-button] ```typescript navigate("/settings", { replace: true }); ``` *** Local Development [#local-development] Starting the SPA Dev Server [#starting-the-spa-dev-server] ```bash cd apps/livehealth-frontend yarn start local.crelioDashboard ``` This starts the SPA on port 3000 with hot-reload. Starting a Single Module (Non-SPA) [#starting-a-single-module-non-spa] ```bash yarn start local.Operations # or Registration, Finance, etc. ``` Full Stack Setup [#full-stack-setup] ```bash # Terminal 1: SPA frontend (port 3000) cd apps/livehealth-frontend yarn start local.crelioDashboard # Terminal 2: Py2 backend (port 8000) cd livehealthapp python manage.py runserver 0.0.0.0:8000 # Terminal 3: Py3 backend (port 9000) cd crelioapp python manage.py runserver 0.0.0.0:9000 ``` Enabling SPA Locally [#enabling-spa-locally] Set the Redis flag for your local lab: ```bash redis-cli SET is_spa_enabled_YOUR_LAB_ID True ``` Or via Django shell: ```python from django.core.cache import cache cache.set("is_spa_enabled_YOUR_LAB_ID", True) ``` *** Debugging [#debugging] Module Not Resolving [#module-not-resolving] **Symptom:** Navigating to your module shows Registration (the fallback) instead of your module. **Checklist:** 1. Is the module name in the `modules` array in `routes.tsx`? 2. Does the sidebar route's first path segment match your URL? (e.g., `/my-module/dashboard` should match a sidebar item starting with `/my-module/`) 3. Is the `selectedModule` value in the modules array matching the expected name? Route Not Rendering [#route-not-rendering] **Symptom:** URL changes but the page shows `PageNotFound`. **Checklist:** 1. Does the route `path` exactly match the URL? Routes use `exact={true}`. 2. Is the route in the correct module's `useXxxRoutes()` hook? 3. Is the component exported correctly and lazy-loaded properly? Navigate Not Prefixing Module Name [#navigate-not-prefixing-module-name] **Symptom:** `navigate("/dashboard")` goes to `/dashboard` instead of `/operation/dashboard`. **Checklist:** 1. Is the current URL within a recognized module? Check `moduleNames` in `navigate.ts`. 2. Is `isLabLogin()` returning `true`? 3. Is the target path accidentally matching a module name? (e.g., `/admin` won't be prefixed because it IS a module name) SPA Not Loading [#spa-not-loading] **Symptom:** Hitting `/crelio-dashboard/` redirects to `/` (login page). **Checklist:** 1. Is `is_spa_enabled_{lab_id}` set to `True` in Redis? 2. Is the user authenticated (valid `labId` in session)? 3. Is `crelio-dashboard.html` deployed to `build_assets/CrelioDashboard/build/`? 4. Try logging in as support user — they bypass the flag check. *** Common Pitfalls [#common-pitfalls] ❌ Using window.location.href for Navigation [#-using-windowlocationhref-for-navigation] ```typescript // DON'T — this triggers a full page reload and destroys SPA state window.location.href = "/operation/dashboard"; // DO — use the navigate utility for in-app navigation navigate("/operation/dashboard"); ``` ❌ Forgetting to Prefix Routes with Module Name [#-forgetting-to-prefix-routes-with-module-name] ```typescript // DON'T — this will conflict with other modules using "/dashboard" { path: "/dashboard", Component: MyDashboard } // DO — prefix all routes with your module name { path: "/my-module/dashboard", Component: MyDashboard } ``` ❌ Module-Specific State in Global Redux [#-module-specific-state-in-global-redux] ```typescript // DON'T — all modules share the same Redux store in SPA mode // Avoid generic keys that other modules might also use dispatch({ type: "SET_DASHBOARD_DATA", payload: data }); // DO — namespace your actions with the module name dispatch({ type: "OPERATIONS/SET_DASHBOARD_DATA", payload: data }); ``` ❌ Initializing Settings in Component useEffect [#-initializing-settings-in-component-useeffect] ```typescript // DON'T — this re-fetches on every module switch useEffect(() => { fetchSettings(); }, []); // DO — initialize in the SPA bootstrap (routes.tsx) so it runs once // Or check if data already exists before fetching useEffect(() => { if (!settings) fetchSettings(); }, []); ``` ❌ Using history.push Directly [#-using-historypush-directly] ```typescript // DON'T — bypasses module-aware URL prefixing import history from "src/utils/history"; history.push("/dashboard"); // DO — use navigate which handles module prefixing import { navigate } from "src/utils/navigate"; navigate("/dashboard"); ``` *** Testing [#testing] Testing SPA Mode vs v5 Mode [#testing-spa-mode-vs-v5-mode] To compare behavior between SPA and v5 single-build: | Test | SPA Mode | v5 Single-Build | | ----------------- | ----------------------------------- | ------------------------------ | | **Start server** | `yarn start local.crelioDashboard` | `yarn start local.Operations` | | **URL pattern** | `/crelio-dashboard/#/operation/...` | `/waitingList/...` or `/#/...` | | **Redis flag** | `is_spa_enabled_{lab_id} = True` | Flag absent or `False` | | **Module switch** | Instant (no reload) | Full page reload | Quick Smoke Test [#quick-smoke-test] After making changes: 1. Start the SPA dev server 2. Log in and verify you land on `/crelio-dashboard/#/{module}` 3. Switch between at least 2 modules via the sidebar — verify no page reload 4. Use browser back/forward buttons — verify navigation works 5. Use `Cmd+K` spotlight search — verify cross-module results 6. Hard-refresh the page — verify the SPA reloads correctly at the same route Testing a New Route [#testing-a-new-route] 1. Navigate to the new route directly via URL 2. Navigate to it via sidebar click 3. Navigate to it programmatically via `navigate()` 4. Refresh the page on the new route — verify it loads correctly 5. Use browser back button from the new route *** Quick Reference [#quick-reference] File Locations [#file-locations] | What | Where | | ----------------------- | ----------------------------------------------------------------- | | Module routes + sidebar | `src/modules/CrelioDashboard/RoutesAndMenus/xxxRoutesAndMenu.tsx` | | Module assembly | `src/modules/CrelioDashboard/RoutesAndMenus/routes.tsx` | | Route types | `src/modules/CrelioDashboard/routeTypes.ts` | | Navigate utility | `src/utils/navigate.ts` | | History instance | `src/utils/history.ts` | | Sidebar component | `src/modules/CrelioDashboard/Sidebar/sidebar.tsx` | | Module switcher | `src/modules/CrelioDashboard/SwitchModuleDropdown/` | | SPA entry point | `src/modules/CrelioDashboard/index.tsx` | | Build config | `app-config.js`, `config-overrides.js` | Key Imports [#key-imports] ```typescript // Navigation import { navigate } from "src/utils/navigate"; // Route types (for module hooks) import { CrelioRouteType, CrelioSidebarItem, Module } from "../routeTypes"; // Redux state access import { getReduxState } from "src/utils/helpers"; // History (only if you need location info, NOT for navigation) import history from "src/utils/history"; ``` # Frontend Frontend [#frontend] What Frontend Owns [#what-frontend-owns] | Concern | Frontend responsibility | | :------------------ | :----------------------------------------------------------------------------------------------------------------------- | | Provider stack | Initialize and share Redux, i18n, LaunchDarkly, Sentry, AG Grid, ErrorBoundary across all modules — once per SPA session | | Module registration | Each module exports `useXxxRoutes()` and `useXxxSidebarMenu()` hooks consumed by the unified Routes component | | Module resolution | Determine the active module from the URL path via `matchingModule` logic | | Route rendering | Render only the active module's routes in the `Switch`, with `AuthRoute` permission checks | | Sidebar | Render a unified sidebar with the active module's menu items and a module switcher dropdown | | SPA bootstrap | Eager-load all module settings (session, operations, finance, registration, CRM, inventory) during initial mount | | Code splitting | Use `React.lazy()` + `Suspense` for per-route code splitting | | Spotlight search | `Cmd+K` cross-module search integration | | Build pipeline | Webpack configuration for SPA, single-build, and all-modules build modes | Component Tree [#component-tree] Entry Point: CrelioDashboard/index.tsx [#entry-point-creliodashboardindextsx] Source: [`src/modules/CrelioDashboard/index.tsx`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/modules/CrelioDashboard/index.tsx) The SPA entry point renders `
` with its ``: ```typescript import ReactDOM from "react-dom"; import { Main } from "../../index"; import { Routes } from "./RoutesAndMenus/routes"; import "src/stylesheets/viewport.scss"; const Container = () => { return
} />; }; ReactDOM.render(, document.getElementById("root")); ``` The difference from standalone modules is that `` here contains **all module routes**, not just a single module's routes. Main — The Provider Stack [#main--the-provider-stack] Source: [`src/index.tsx`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/index.tsx) `Main` is shared across all modules (SPA and standalone). It initializes: | Provider | Purpose | | ------------------ | ------------------------------------------------------ | | `Suspense` | Fallback loading spinner for `React.lazy()` components | | `LDProvider` | LaunchDarkly feature flag context | | `Provider` (Redux) | Global state store | | `I18nextProvider` | Internationalization | | `ErrorBoundary` | Sentry error boundary for crash reporting | | `Router` | React Router with shared `history` object | | `AppAlert` | Global toast notification system | | `DetectNetwork` | Offline/online detection banner | All of these are initialized **once** when the SPA loads and remain alive across module switches. Module Registration Pattern [#module-registration-pattern] Each module contributes to the SPA via two custom hooks: | Hook | Returns | Purpose | | --------------------- | --------------------- | ----------------------------------------------------- | | `useXxxRoutes()` | `CrelioRouteType[]` | Array of route definitions (path + component + props) | | `useXxxSidebarMenu()` | `CrelioSidebarItem[]` | Array of sidebar menu items for the module | Example: Operations Module [#example-operations-module] Source: [`RoutesAndMenus/operationsRoutesAndMenu.tsx`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/modules/CrelioDashboard/RoutesAndMenus/operationsRoutesAndMenu.tsx) ```typescript // Routes hook export const useOperationsRoutes = (): CrelioRouteType[] => { return [ { path: "/operation", Component: OperationsDashboard }, { path: "/operation/operation-dashboard/:route", Component: OperationsDashboard }, { path: "/operation/testwise-waiting-list/:route", Component: WaitingList }, // ... more routes ]; }; // Sidebar hook export const useOperationsSidebarMenu = (): CrelioSidebarItem[] => { return [ { name: t("Operations Dashboard"), icon: "fa-chart-line", route: "/operation" }, { name: t("Waiting List"), icon: "fa-list", route: "/operation/testwise-waiting-list/all-tests" }, // ... more menu items ]; }; ``` All Registered Modules [#all-registered-modules] These hooks are consumed in `routes.tsx` and assembled into a `modules` array: ```typescript const modules: Module[] = useMemo(() => [ { menu: financeSidebarMenu, routes: financeRoutes, selectedModule: t("Finance") }, { menu: labAdminSidebarMenu, routes: labAdminRoutes, selectedModule: t("Admin") }, { menu: accessionSidebarMenu, routes: accessionRoutes, selectedModule: t("Accession") }, { menu: inventorySidebarMenu, routes: inventoryRoutes, selectedModule: t("Inventory") }, { menu: operationSidebarMenu, routes: operationsRoutes, selectedModule: t("Operation") }, { menu: registrationSidebarMenu, routes: registrationRoutes, selectedModule: t("Registration") }, { menu: crmDashboardSidebarMenu, routes: crmDashboardRoutes, selectedModule: t("CRM") }, ], [/* dependencies */]); ``` Module Resolution — matchingModule [#module-resolution--matchingmodule] Source: [`RoutesAndMenus/routes.tsx`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/modules/CrelioDashboard/RoutesAndMenus/routes.tsx) The SPA determines which module is active purely from the URL: ```typescript const matchingModule: Module = useMemo(() => { const pathParts: string[] = location.pathname.split("/").filter(Boolean); const moduleName: string = pathParts[0]?.toLowerCase(); // Strategy 1: Match by sidebar route const moduleByRoute: Module | undefined = modules.find((module: Module) => module.menu.some((item: CrelioSidebarItem) => { if (item.route) { const itemRouteBase = item.route.split("/").filter(Boolean)[0]?.toLowerCase(); const locationPathBase = pathParts[0]?.toLowerCase(); const isExactMatch = location.pathname === item.route; const isBaseMatch = locationPathBase === itemRouteBase; return isExactMatch || isBaseMatch; } return false; }) ); // Strategy 2: Match by module name const moduleByName: Module | undefined = modules?.find( (module: Module) => module.selectedModule?.toLowerCase() === moduleName ); // Fallback: Registration return moduleByRoute || moduleByName || { menu: registrationSidebarMenu, routes: registrationRoutes, selectedModule: t("Registration"), }; }, [location.pathname, modules, t]); ``` Resolution Priority [#resolution-priority] | Priority | Strategy | Example | | -------- | ------------------------------------------------------------------------ | ------------------------------------- | | 1 | **Route match** — first path segment matches a sidebar item's route base | `/finance/dashboard` → Finance module | | 2 | **Name match** — first path segment matches a module's `selectedModule` | `/operation` → Operation module | | 3 | **Fallback** — if nothing matches, default to Registration | `/unknown-path` → Registration module | Route Type System [#route-type-system] Source: [`CrelioDashboard/routeTypes.ts`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/modules/CrelioDashboard/routeTypes.ts) Core Types [#core-types] ```typescript // A regular route with a component type CrelioRoute = { path: string; Component: React.ElementType; routeProps?: RouteProps; }; // A redirect route type CrelioRedirectRoute = { to: string; from: string; isRedirect: true | false; }; // Union type used in route arrays type CrelioRouteType = CrelioRoute | CrelioRedirectRoute; // Sidebar item type CrelioSidebarItem = { name: string | JSX.Element; icon?: string; route?: string; onClick?: Function; isCustom?: true | false; hasPermission?: Function; submenuArr?: CrelioSidebarItem[]; showPartition?: boolean; requiredPermissions?: string[]; hidden?: boolean; count?: number | string; redirection?: boolean; }; // Module aggregate interface Module { menu: CrelioSidebarItem[]; routes: CrelioRouteType[]; selectedModule: string; } ``` Routes support permission-based access control via `routeProps`, consumed by the `AuthRoute` component which checks the user's session permissions before rendering. Route Rendering [#route-rendering] The `Switch` block dynamically renders only the matching module's routes: ```typescript {matchingModule?.routes?.map((route: any) => { const { to = "", from = "", path = "", Component = null, isRedirect = false, routeProps = {}, } = route; const key = `crelio-dashboard-route-${path || to}`; if (isRedirect) { return ; } return ( ); })} ``` Only the `matchingModule.routes` are rendered at any time. Route paths across modules don't need to be globally unique — `/dashboard` in Finance and `/dashboard` in Operations won't conflict because only one module's routes are active. The navigate Utility — v5 ↔ SPA Redirection [#the-navigate-utility--v5--spa-redirection] Source: [`src/utils/navigate.ts`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/utils/navigate.ts) The `navigate` utility is the primary programmatic navigation function used across all modules. It wraps React Router's `history.push` / `history.replace` and adds module-aware URL prefixing that makes the same navigation calls work correctly in both **v5 single-build mode** and **SPA mode**. How It Works [#how-it-works] ```typescript import history from "src/utils/history"; import { getReduxState, isLabLogin } from "./helpers"; export type NavigateInput = string | { pathname: string; state?: JsonObject }; export const moduleNames: string[] = [ "registration", "accession", "operation", "finance", "crm", "admin", "inventory", ]; const getUpdatedUrl = (pathname: string, locationPathname: string): string => { const sessionData = getReduxState("SESSION"); const [firstPart] = locationPathname.split("/").filter(Boolean); const [path] = pathname.split("/").filter(Boolean); const urlBeforeHash = window.location.href.split("#")[0]; const updatedUrl = isLabLogin() && !urlBeforeHash.includes("superAdminUserManagement") && moduleNames.includes(firstPart) && !moduleNames.includes(path) && !urlBeforeHash.includes("inventory") ? `/${firstPart}/${pathname}` : pathname; return removeDoubleSlashes(updatedUrl); }; export const navigate: Navigate = (urlOrObject, options?) => { const method = options?.replace ? history.replace : history.push; const updatedUrl = getUpdatedUrl( typeof urlOrObject === "string" ? urlOrObject : urlOrObject.pathname, history?.location?.pathname ); method(typeof urlOrObject === "string" ? updatedUrl : { pathname: updatedUrl, state: urlOrObject.state }); }; ``` URL Prefixing Logic — getUpdatedUrl [#url-prefixing-logic--getupdatedurl] The key function is `getUpdatedUrl`, which decides whether to prefix the target URL with the current module name: | Condition | Result | Example | | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | -------------------------------------------------------------------- | | User is a lab login AND current URL is in a known module AND target path is NOT a known module name | **Prefix** the target with the current module | In `/operation`, navigating to `/dashboard` → `/operation/dashboard` | | Target path IS a known module name | **No prefix** — it's a cross-module navigation | Navigating to `/finance/dashboard` → `/finance/dashboard` | | URL contains `superAdminUserManagement` | **No prefix** — admin context bypass | Admin pages skip module prefixing | | URL contains `inventory` | **No prefix** — inventory has its own routing structure | Inventory module navigation is passed through directly | Why This Matters [#why-this-matters] In **v5 single-build mode**, each module runs in its own React app instance with its own ``. Routes like `/dashboard` or `/settings` are relative to the module and don't need prefixing because the module context is implicit in the HTML entry point. In **SPA mode**, all modules share a single ``. A component calling `navigate("/dashboard")` from within the Operations module needs the URL to resolve as `/operation/dashboard`, not just `/dashboard` (which would lose the module context). The `navigate` utility handles this transparently — components don't need to know if they're running in SPA or v5 mode. Usage [#usage] ```typescript import { navigate } from "src/utils/navigate"; // Simple string navigation navigate("/testwise-waiting-list/all-tests"); // With state navigate({ pathname: "/patient-details", state: { patientId: 123 } }); // Replace instead of push (no back button entry) navigate("/settings", { replace: true }); // Cross-module navigation (explicitly includes module name) navigate("/finance/dashboard"); ``` SPA Bootstrap — Initialization Sequence [#spa-bootstrap--initialization-sequence] When the SPA loads, the `Routes` component runs a comprehensive initialization sequence: ```typescript useEffect(() => { const initialize = async () => { // 1. Session + bill settings await getAppLicationLevelSessionAndBillSetting(); // 2. Registration-specific await getCustomRegistrationPage(); await getBillSettingData(dispatch); // 3. Operations-specific dispatch(getSettingsData("/getOperationSetting/", false)); dispatch(getReportSettings("/getAllReportSettings/", true)); // 4. Shared data dispatch(getDepartmentList(true)); await getDepartmentWiseDoctor(dispatch); await getDepartmentAndSignDocRel(dispatch); await getDoctorList(dispatch); // 5. Inventory (conditional) inventoryManagement && userInventoryManagement && await fetchInventorySettings(); // 6. CRM (parallel batch) axiosInstance.all([ "/crm/onboarding/user/permissions/", "/crm/store/store-enable-status/", "/crm/onboarding/walkthrough/configurations/", "/crm/onboarding/walkthrough/", "/crm/home-collections/get-crm-settings/", ].map(endpoint => axiosInstance.get(endpoint))) .then(/* dispatch all CRM state */) .finally(() => setLoading(false)); }; initialize(); }, []); ``` This runs **once** on SPA mount. All subsequent module switches use the data already in Redux. Spotlight Search [#spotlight-search] The SPA includes a cross-module search feature (`Cmd+K` / `Ctrl+K`): ```typescript useEffect(() => { const handleKeyDown = (e: KeyboardEvent): void => { if ((e.metaKey || e.ctrlKey) && e.key.toLowerCase() === "k") { e.preventDefault(); onSpotlightSearch(); } }; document.addEventListener("keydown", handleKeyDown); return () => document.removeEventListener("keydown", handleKeyDown); }, []); ``` The `SpotlightSearch` component receives the `selectedModule` to scope search results to the current module context while still enabling cross-module navigation. Build Pipeline [#build-pipeline] app-config.js — Entry Point Registry [#app-configjs--entry-point-registry] Source: [`app-config.js`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/app-config.js) ```javascript const allApps = [ "all", // Clubbable modules (SPA-eligible) "Admin", "Finance", "inventory", "LabAdmin", "Accession", "Operations", "Registration", "crmDashboard", "CrelioDashboard", // Standalone modules (separate builds) "crm", "DoctorLogin", "ReferralLogin", "MarketingLogin", "centerdashboard", "OrganisationLogin", ]; const allBuilds = { Admin: "src/modules/Admin/index.tsx", Finance: "src/modules/Finance/index.tsx", CrelioDashboard: "src/modules/CrelioDashboard/index.tsx", // ... each module maps to its entry point }; ``` Build Path Resolution [#build-path-resolution] | Build | Output Directory | Public URL Path | | ----------------- | ------------------------- | -------------------------------------------- | | `CrelioDashboard` | `./build/CrelioDashboard` | `/static/build_assets/CrelioDashboard/build` | | All other modules | `./build/dashboard` | `/static/build_assets/dashboard/build` | config-overrides.js — Webpack Customization [#config-overridesjs--webpack-customization] Source: [`config-overrides.js`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/config-overrides.js) Key customizations using `react-app-rewired` + `customize-cra`: **Multi-entry support:** ```javascript const build = process?.env?.REACT_APP_BUILD_NAME || "DoctorLogin"; const entries = appConfig.getBuild(build); const multipleEntry = require("react-app-rewire-multiple-entry")(entries); ``` **esbuild-loader (replaces babel-loader):** ```javascript rule.oneOf[babelLoaderIndex] = { test: /\.(js|jsx|ts|tsx)$/, exclude: /node_modules/, use: [{ loader: "esbuild-loader", options: { loader: "tsx", target: "es2015" }, }], }; ``` **Chunk splitting strategy:** ```javascript splitChunks: { chunks: "all", maxInitialRequests: 20, minSize: 20000, cacheGroups: { vendor: { test: /[\\/]node_modules[\\/]/, name(module) { const packageName = module.context.match(/[\\/]node_modules[\\/](.*?)([\\/]|$)/)?.[1]; return packageName ? `vendor.${packageName.replace("@", "")}` : "vendor"; }, priority: 20, }, commons: { test: /[\\/]src[\\/]/, name: "commons", minChunks: 2, priority: 10, }, }, }, ``` This creates per-package vendor chunks (`vendor.react.js`, `vendor.ag-grid-community.js`), a commons chunk for shared source code, and a single runtime chunk. Production Optimizations [#production-optimizations] | Plugin | Purpose | | ------------------------------- | ------------------------------------------------------- | | `TerserPlugin` | JavaScript minification | | `EsbuildPlugin` | Additional minification + CSS minification | | `CompressionPlugin` | Gzip pre-compression for `.js`, `.css`, `.html`, `.svg` | | `MiniCssExtractPlugin` | Extract CSS into separate files with content hashes | | `ModuleConcatenationPlugin` | Scope hoisting for smaller bundles | | `BundleAnalyzerPlugin` | Static HTML report of bundle composition | | `ContextReplacementPlugin` | Strip unused Moment.js locales (keep only English) | | `NormalModuleReplacementPlugin` | Replace `lodash` with `lodash-es` for tree-shaking | Build Output Comparison [#build-output-comparison] | Aspect | SPA Build | All Build (Legacy) | Single Module Build | | ---------------- | ------------------------------- | ---------------------------------- | ---------------------------- | | **Command** | `nps build.CrelioDashboardProd` | `nps build.allProd` | `nps build.RegistrationProd` | | **Entry points** | 1 (`CrelioDashboard/index.tsx`) | 15 (all modules) | 1 (specific module) | | **HTML files** | 1 (`index.html`) | 15 (one per module) | 1 (`index.html`) | | **JS bundles** | Unified + code-split per module | Shared chunks + per-module bundles | Module-specific bundle | | **Output dir** | `build/CrelioDashboard/` | `build/dashboard/` | `build/dashboard/` | | **Build time** | \~3-5 min | \~15-25 min | \~2-3 min | Asset Deployment — move-assets.sh [#asset-deployment--move-assetssh] After building, assets are copied into the Django project's static directory: ```bash prepare_and_move_assets() { cd ../livehealthapp/static rm -vrf build_assets/$1 mkdir -p build_assets/$1/build cd ../../livehealth-frontend cp -vrf build/$1/* ../livehealthapp/static/build_assets/$1/build } ``` CI/CD Pipeline [#cicd-pipeline] Bitbucket Pipelines [#bitbucket-pipelines] Source: [`bitbucket-pipelines.yml`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/bitbucket-pipelines.yml) Each module has staging and production build steps: | Build Step | Build Command | | ------------------------- | ------------------------ | | `registration-prod-build` | `build.RegistrationProd` | | `finance-prod-build` | `build.FinanceProd` | | `crm-prod-build` | `build.crmProd` | Jenkins Integration [#jenkins-integration] For the `all` build (including SPA), Jenkins is triggered via the Bitbucket pipeline: 1. Checks out the branch. 2. Runs `yarn install` and `yarn start build.allProd`. 3. Uploads bundled assets to S3 or deploys to EC2. **Post-build steps (Jenkins only):** 1. `sentry:production` — Uploads source maps to Sentry for error deobfuscation. 2. `delete-sourcemaps` — Removes `.map` files from the build output to prevent client-side source exposure. Comparison: SPA Module vs. Standalone Module [#comparison-spa-module-vs-standalone-module] | Aspect | SPA Module (CrelioDashboard) | Standalone Module (e.g., DoctorLogin) | | -------------------- | ------------------------------------- | ------------------------------------- | | **Entry point** | `CrelioDashboard/index.tsx` | `DoctorLogin/index.tsx` | | **Routes** | All modules combined in one `Switch` | Module-specific routes only | | **Sidebar** | Shared `SideBar` with module switcher | Module-specific `SideBar` | | **Initialization** | All settings fetched once | Module-specific settings only | | **Module switching** | In-memory route change | Full page reload | | **Session** | Shared staff session | Role-specific session | | **Build output** | `crelio-dashboard.html` | `doctor-login.html` | Key Frontend Locations [#key-frontend-locations] | File | Purpose | | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------- | | [`CrelioDashboard/index.tsx`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/modules/CrelioDashboard/index.tsx) | SPA entry point — renders `Main` with unified `Routes` | | [`RoutesAndMenus/routes.tsx`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/modules/CrelioDashboard/RoutesAndMenus/routes.tsx) | SPA-level routing: module resolution, initialization, sidebar binding | | [`routeTypes.ts`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/modules/CrelioDashboard/routeTypes.ts) | TypeScript type definitions for routes, sidebar items, and modules | | [`Sidebar/sidebar.tsx`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/modules/CrelioDashboard/Sidebar/sidebar.tsx) | Unified sidebar with module-aware menu rendering | | [`SwitchModuleDropdown/`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/modules/CrelioDashboard/SwitchModuleDropdown/) | Module switcher dropdown component | | [`src/utils/navigate.ts`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/utils/navigate.ts) | Module-aware navigation utility — handles URL prefixing for v5 ↔ SPA compatibility | | [`app-config.js`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/app-config.js) | Module registry: maps build names to entry points | | [`config-overrides.js`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/config-overrides.js) | Webpack customization: esbuild-loader, chunk splitting | | [`package-scripts.js`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/package-scripts.js) | `nps` build commands for local dev and production builds | | [`bitbucket-pipelines.yml`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/bitbucket-pipelines.yml) | CI/CD pipeline: per-module build steps | | [`move-assets.sh`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/move-assets.sh) | Asset deployment: copies build output into Django static directory | | [`src/setupProxy.js`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/setupProxy.js) | Dev server proxy: routes `/api-v3/` to Py3 backend | # Overview SPA Architecture [#spa-architecture] Unify eight previously independent React module builds into a single React application served from `/crelio-dashboard/`, eliminating full-page reloads between modules while maintaining backward-compatible single-build deployments. SPA Architecture [#spa-architecture-1] SPA Architecture is the transition of `livehealth-frontend` from a multi-HTML-entry-point model to a single unified React application called `CrelioDashboard`. In the legacy model, navigating between staff modules (Registration → Operations → Finance) triggered a complete browser reload: new HTML document, fresh React bootstrap, re-initialization of Redux, i18n, AG Grid, LaunchDarkly, Sentry, and re-fetch of session, settings, department lists, and doctor lists. This created 3-5 second delays on every module switch, destroyed transient state, and caused redundant API calls. The SPA keeps a single React tree alive across module switches. Module navigation becomes an in-memory hash route change — instant, with no network overhead. Session data, Redux store, sidebar navigation, and all initialized providers are shared across modules. Both architectures coexist in production. The `is_spa_enabled` Redis flag (`is_spa_enabled_{lab_id}`) controls which mode a lab receives. New labs default to SPA; existing labs migrate via the support dashboard. Support users always receive the SPA regardless of the lab flag. Related Jira Tickets [#related-jira-tickets] | Ticket | Title | Notes | | :----------------------------------------------------- | :--------------- | :------------------------------------------------------------------------------------- | | [EN-5793](https://crelio.atlassian.net/browse/EN-5793) | SPA Architecture | Internal engineering initiative to unify staff modules into a single React application | Prerequisites [#prerequisites] | Requirement | Why it matters | Where it is enforced | | :----------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `is_spa_enabled_{lab_id}` must be `True` in Redis | The feature flag determines whether the lab receives SPA or legacy builds | [`labs/cacheFunction.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/labs/cacheFunction.py) | | Lab must be on React v5 builds (`reactVersionAccess == 1` or `newRegistrationPage == 3`) | The SPA only applies to v5 React module builds, not legacy v4 Jinja templates. `reactVersionAccess == 1` or `newRegistrationPage == 3` indicates v5; anything else is v4. | [`livehealth_4/decorators/version_decorator.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/livehealth_4/decorators/version_decorator.py) — `mapped_spa_url()` | | `crelio-dashboard.html` build artifact must be deployed to `build_assets/CrelioDashboard/build/` | The Django view serves this HTML file | [`livehealth_4/views.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/livehealth_4/views.py) — `CrelioDashboardView` | | Login flow must populate `is_spa_enabled` in the Django session | Frontend and Django templates can check SPA status without additional Redis lookups | [`logins/utils/session_helpers.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/logins/utils/session_helpers.py) | > **Note:** If a user logs in via **Support Login** (`isSupportLogin = True`), the system **always opens the SPA** — regardless of whether the lab has `is_spa_enabled` set to `True` or not. This allows support users to debug any lab using the unified SPA view without needing to toggle the feature flag. What Is It For [#what-is-it-for] Frontend perspective [#frontend-perspective] * Eliminate full-page reloads between staff module switches (Registration, Operations, Finance, etc.). * Keep a single React tree alive across all module navigations — Redux store, providers, and sidebar remain initialized. * Enable instant module switching via in-memory hash route changes instead of browser navigation. * Share session state, settings, department lists, and doctor lists across all modules — fetched once during bootstrap. * Provide a unified sidebar with a module switcher dropdown and cross-module spotlight search (`Cmd+K`). * Maintain code-splitting via `React.lazy()` so only the active module's components are loaded. Backend perspective [#backend-perspective] * Serve a single HTML template (`crelio-dashboard.html`) for all SPA-eligible modules instead of per-module templates. * Control SPA rollout per-lab via the `is_spa_enabled_{lab_id}` Redis flag with instant rollback capability. * Map legacy module URLs to SPA hash routes using `SPA_URL_MAPPER` so existing bookmarks and redirects continue working. * Maintain backward compatibility — labs with SPA disabled continue receiving per-module HTML builds without any change. * Support login bypass — support users always receive the SPA for faster debugging regardless of lab flag state. Module Classification [#module-classification] SPA-Eligible Modules (Clubbable) [#spa-eligible-modules-clubbable] These modules share the same staff session context and are unified under the `CrelioDashboard` SPA build: | Module | SPA Route | Build Entry (Legacy) | | ------------- | ----------------- | ------------------------------------ | | Registration | `/registration/` | `src/modules/Registration/index.tsx` | | Accession | `/accession/` | `src/modules/Accession/index.tsx` | | Operations | `/operation/` | `src/modules/Operations/index.tsx` | | Finance | `/finance/` | `src/modules/Finance/index.tsx` | | Admin | `/admin/` | `src/modules/Admin/index.tsx` | | LabAdmin | `/lab-admin/` | `src/modules/LabAdmin/index.tsx` | | Inventory | `/inventory/` | `src/modules/inventory/index.tsx` | | CRM Dashboard | `/crm-dashboard/` | `src/modules/crmDashboard/index.tsx` | Standalone Modules (Separate Builds) [#standalone-modules-separate-builds] These modules serve different user roles and require their own isolated sessions: | Module | URL | Reason for Isolation | | ----------------- | -------------------- | -------------------------------------------------------- | | DoctorLogin | `/v4/doctor/` | Doctor-specific session, different settings API | | ReferralLogin | `/v4/referral/` | Referral doctor session, subset of features | | OrganisationLogin | `/org-login/` | Organization-specific session and billing context | | MarketingLogin | `/marketing/` | Marketing-only access, no lab operations | | centerdashboard | `/center-dashboard/` | Multi-centre aggregation view, separate data model | | CRM | `/crm/` | Standalone CRM module with its own entry point and build | Key Features [#key-features] * **Instant module switching** — in-memory hash route changes instead of full browser reloads, eliminating 3-5 second delays. * **Shared session and state** — session data, Redux store, settings, department lists, and doctor lists initialized once and shared across all modules. * **Unified sidebar and module switcher** — single sidebar with a dropdown to switch between modules, dynamically resolving the active module from the URL. * **Spotlight search** — `Cmd+K` / `Ctrl+K` cross-module search integrated into the SPA. * **Code-splitting per module** — `React.lazy()` + `Suspense` ensures only the active module's components are loaded, keeping the initial bundle size manageable. * **Dual-mode coexistence** — SPA and legacy single-build deployments coexist in production, controlled per-lab via Redis flag with instant rollback. * **Backward-compatible URL mapping** — `SPA_URL_MAPPER` translates legacy module URLs to SPA hash routes so existing bookmarks, redirects, and login flows continue working. * **Support user override** — support users always receive the SPA regardless of lab flag, enabling faster debugging with the unified view. # Workflow Guide import Image from 'next/image' import enableSpa from '@/images/enable-spa-support-dashboard.jpg' Workflow Guide [#workflow-guide] This section provides a practical walkthrough for enabling, configuring, and working with the SPA — covering both operator-facing behavior and developer-facing setup. How To Enable SPA For A Lab [#how-to-enable-spa-for-a-lab] Via Redis CLI (Direct) [#via-redis-cli-direct] The simplest way to enable SPA for a lab: ```bash # Enable SPA for lab 12345 redis-cli SET is_spa_enabled_12345 True # Verify the flag redis-cli GET is_spa_enabled_12345 ``` The change takes effect on the next login for that lab. No deployment or restart required. Via Account Creation (Automatic) [#via-account-creation-automatic] New labs created through the self-serve account creation flow have the SPA flag set automatically: Source: [`selfserved/account_creation/account.py`](https://bitbucket.org/creliohealth-repo/livehealthapp/src/develop/selfserved/account_creation/account.py) Via Support Dashboard [#via-support-dashboard] Support staff can toggle the flag through the support admin interface: 1. Open the Support Dashboard → navigate to the lab's **Centre Details**. 2. Go to the **Configurations** tab → **Workflow Configurations**. 3. Under **Enable Workflows/Features** → **Features**, find the **Enable SPA** toggle. 4. Toggle it on to enable, off to disable. This wraps the Redis operation in a logged action for traceability. Enable SPA toggle in Support Dashboard What Happens After Enabling [#what-happens-after-enabling] 1. User logs in → Django checks `is_spa_enabled(lab_id)` via Redis. 2. If `True`, the session's `loginURL` is set to `/crelio-dashboard/#/{module-path}`. 3. Browser is redirected to `/crelio-dashboard/`. 4. `CrelioDashboardView` serves `crelio-dashboard.html`. 5. React bootstraps once — all providers, session, and settings initialized. 6. User sees the unified sidebar with module switcher. Bulk Enable For Multiple Labs (Django Shell) [#bulk-enable-for-multiple-labs-django-shell] To enable SPA for multiple labs at once, use the Django shell: ```python from django.core.cache import cache lab_ids = [1134, 3042, 3333, 6284, 7493, 7726, 8440, 8774, 8968, 9354, 9448, 9573, 10446, 11802] [cache.set("is_spa_enabled_{lab_id}".format(lab_id=lab_id), True) for lab_id in lab_ids] ``` This sets the `is_spa_enabled_{lab_id}` Redis key to `True` for each lab in the list. The change takes effect on the next login for each lab. To bulk disable: ```python from django.core.cache import cache lab_ids = [1134, 3042, 3333] [cache.delete("is_spa_enabled_{lab_id}".format(lab_id=lab_id)) for lab_id in lab_ids] ``` How To Disable SPA For A Lab [#how-to-disable-spa-for-a-lab] Via Redis CLI (Direct) [#via-redis-cli-direct-1] ```bash # Disable SPA for lab 12345 — delete the key redis-cli DEL is_spa_enabled_12345 # Or set to False explicitly redis-cli SET is_spa_enabled_12345 False ``` The change takes effect on the next login. Active sessions continue using the SPA until the user logs out and back in. Via Support Dashboard [#via-support-dashboard-1] Use the same toggle shown in the enable section above: 1. Open the Support Dashboard → navigate to the lab's **Centre Details**. 2. Go to the **Configurations** tab → **Workflow Configurations**. 3. Under **Enable Workflows/Features** → **Features**, find the **Enable SPA** toggle. 4. Toggle it **off** to disable SPA for the lab. Bulk Disable For Multiple Labs (Django Shell) [#bulk-disable-for-multiple-labs-django-shell] ```python from django.core.cache import cache lab_ids = [1134, 3042, 3333] [cache.delete("is_spa_enabled_{lab_id}".format(lab_id=lab_id)) for lab_id in lab_ids] ``` Rollback scenario [#rollback-scenario] If the SPA causes issues for a specific lab: 1. Disable the Redis flag immediately (no deployment needed). 2. Ask the user to log out and log back in. 3. They will now receive the legacy per-module builds. This provides an instant rollback path without code changes or deployments. How To Configure [#how-to-configure] Adding A New Module To The SPA [#adding-a-new-module-to-the-spa] | Step | What To Do | Where | | :--- | :-------------------------------------------------------------------- | :---------------------------------------------------------------- | | 1 | Create `useXxxRoutes()` hook returning `CrelioRouteType[]` | `src/modules/CrelioDashboard/RoutesAndMenus/xxxRoutesAndMenu.tsx` | | 2 | Create `useXxxSidebarMenu()` hook returning `CrelioSidebarItem[]` | Same file | | 3 | Register both hooks in the `modules` array | `RoutesAndMenus/routes.tsx` | | 4 | Add legacy URL → SPA URL mapping | `SPA_URL_MAPPER` in `livehealth_4/decorators/utils.py` | | 5 | Update Django login flow to redirect to SPA URL when `is_spa_enabled` | `logins/utils/session_helpers.py` | Adding A New Route To An Existing Module [#adding-a-new-route-to-an-existing-module] Add a `CrelioRoute` entry in the module's `*RoutesAndMenu` file. No backend changes needed — hash routing means Django doesn't need to know about new frontend routes. ```typescript // Example: adding a new route to Operations { path: "/operation/new-feature", Component: NewFeatureComponent } ``` Adding A New Sidebar Item [#adding-a-new-sidebar-item] Add a `CrelioSidebarItem` entry in the module's `useXxxSidebarMenu()` hook: ```typescript { name: t("New Feature"), icon: "fa-star", route: "/operation/new-feature" } ``` Navigating Through The SPA Workflow [#navigating-through-the-spa-workflow] Login → SPA Bootstrap → Module Navigation [#login--spa-bootstrap--module-navigation] Module Switching [#module-switching] Once the SPA is bootstrapped: 1. User clicks a module in the sidebar module switcher dropdown. 2. The URL hash changes (e.g., `/crelio-dashboard/#/finance/dashboard`). 3. `matchingModule` resolves the active module from the URL. 4. The sidebar renders the new module's menu items. 5. The `Switch` renders the new module's routes. 6. **No network request, no re-initialization, no state loss.** Spotlight Search [#spotlight-search] Press `Cmd+K` (Mac) or `Ctrl+K` (Windows) to open the cross-module spotlight search. Results are scoped to the current module context but enable cross-module navigation. Running Locally [#running-locally] Dev Server Setup [#dev-server-setup] ```bash # 1. Start the SPA dev server (port 3000) cd apps/livehealth-frontend yarn start local.crelioDashboard # 2. Start the Django backend (port 8000) — in a separate terminal cd livehealthapp python manage.py runserver 0.0.0.0:8000 # 3. Start the Py3 backend (port 9000) — in a separate terminal cd crelioapp python manage.py runserver 0.0.0.0:9000 ``` Proxy Configuration [#proxy-configuration] The dev server automatically proxies API calls: | API Prefix | Proxy Target | Source | | :------------------ | :-------------------------- | :--------------------------------------------------------------------------------------------------------------- | | Default (no prefix) | `http://0.0.0.0:8000` (Py2) | `package.json` proxy field | | `/api-v3/` | `http://0.0.0.0:9000` (Py3) | [`src/setupProxy.js`](https://bitbucket.org/creliohealth-repo/livehealth-frontend/src/develop/src/setupProxy.js) | Building For Production [#building-for-production] ```bash # SPA production build nps build.CrelioDashboardProd # All modules (legacy) production build nps build.allProd # Move assets to Django static directory bash move-assets.sh CrelioDashboard # SPA bash move-assets.sh dashboard # Legacy ``` Where SPA Behavior Is Visible To The User [#where-spa-behavior-is-visible-to-the-user] | Screen | What shows up | | -------------------- | -------------------------------------------------------------------------------------- | | Login redirect | User lands on `/crelio-dashboard/#/{module}` instead of `/waitingList/` or `/billing/` | | Module switching | Clicking a different module in the sidebar is instant — no page reload | | Sidebar | Unified sidebar with module switcher dropdown at the top | | URL pattern | URLs contain `/#/` hash fragment (e.g., `/crelio-dashboard/#/finance/dashboard`) | | Browser back/forward | Navigates between previously visited modules without reload | | Spotlight search | `Cmd+K` opens cross-module search | # GitHub Migration GitHub Migration [#github-migration] We are migrating our source code repositories from **Bitbucket** to **GitHub**. This section covers everything you need for a smooth transition. # Post Migration Post Migration [#post-migration] Once the migration to GitHub is complete, we'll be rolling out several new capabilities and improvements to our development workflow. *** AI-Powered Code Review [#ai-powered-code-review] GitHub's ecosystem enables us to integrate AI-based code review tools directly into our pull request workflow. This means: * Automated code suggestions and improvements on every PR * AI-assisted detection of bugs, security issues, and code smells * Faster review cycles with intelligent pre-review before human reviewers *** CODEOWNERS [#codeowners] We'll be implementing [GitHub CODEOWNERS](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) to automatically tag the right reviewers based on which modules are changed. * Each module/directory will have designated owners * PRs touching those paths will automatically request reviews from the relevant team members * No more guessing who should review what — ownership is defined in the repo itself Example `CODEOWNERS` file: ``` # Frontend /src/components/ @frontend-team /src/pages/ @frontend-team # Backend /apps/core/ @backend-core-team /apps/finance/ @finance-team /apps/report/ @report-team # Infrastructure /infra/ @devops-team /k8s/ @devops-team ``` *** Agent-Based Bugfixes [#agent-based-bugfixes] With GitHub's integration ecosystem, we can leverage AI agents that: * Automatically investigate bug reports * Propose fixes as pull requests * Run tests to validate the fix before human review This reduces the time from bug report to fix significantly. *** Enhanced PR Review Experience [#enhanced-pr-review-experience] GitHub provides a much improved PR review UI compared to Bitbucket: * Inline comments with threaded discussions * Suggested changes that can be committed directly from the review * Review summaries and approval workflows * Better diff visualization and file tree navigation *** Custom Pipelines & Workflows [#custom-pipelines--workflows] Our CI/CD pipelines will be migrated to **GitHub Actions**, giving us: * Native integration with the repository * Custom workflows triggered by PRs, pushes, releases, and more * Reusable workflow templates across repos * Matrix builds, caching, and parallel execution out of the box * Marketplace actions for common tasks (linting, testing, deployment) *** What's Coming [#whats-coming] | Feature | Status | Expected | | ------------------------------- | ------- | -------------- | | AI Code Review | Planned | Post migration | | CODEOWNERS setup | Planned | Post migration | | Agent-based bugfixes | Planned | Post migration | | GitHub Actions pipelines | Planned | Post migration | | PR review workflow improvements | Planned | Post migration | These features will be rolled out incrementally after the migration is complete. Stay tuned for updates on each. # Prerequisites import Image from 'next/image'; import githubSsh from '@/images/github-ssh.png'; Prerequisites [#prerequisites] Before the migration cutoff, make sure you have the following set up. 1. Create / Verify Your GitHub Account [#1-create--verify-your-github-account] If you don't already have a GitHub account, create one at [github.com/signup](https://github.com/signup). Make sure you use your work email or link it to your account. 2. Set Up SSH Keys [#2-set-up-ssh-keys] SSH keys allow you to authenticate with GitHub without entering your password every time. Generate a New SSH Key [#generate-a-new-ssh-key] Open your terminal and run: ```bash ssh-keygen -t ed25519 -C "your_email@example.com" ``` When prompted, press **Enter** to accept the default file location. Optionally set a passphrase. Start the SSH Agent [#start-the-ssh-agent] ```bash eval "$(ssh-agent -s)" ``` Add Your Key to the Agent [#add-your-key-to-the-agent] ```bash ssh-add ~/.ssh/id_ed25519 ``` Copy the Public Key [#copy-the-public-key] ```bash pbcopy < ~/.ssh/id_ed25519.pub ``` > On Linux, use `xclip -selection clipboard < ~/.ssh/id_ed25519.pub` instead. Add the Key to GitHub [#add-the-key-to-github] 1. Go to [GitHub SSH Key Settings](https://github.com/settings/ssh/new) 2. Click **New SSH key** 3. Give it a title (e.g., "Work Laptop") 4. Paste your public key 5. Click **Add SSH key** GitHub SSH Key Settings Verify the Connection [#verify-the-connection] ```bash ssh -T git@github.com ``` You should see: ``` Hi ! You've successfully authenticated, but GitHub does not provide shell access. ``` # Repositories Repositories [#repositories] Live sync status of all repositories being migrated from Bitbucket to GitHub. Data is fetched from the GitHub API and refreshed every 5 minutes. # Timeline Timeline [#timeline] The migration must be completed **before March 26, 2026** — this is when the Bitbucket plan expires. Key Dates [#key-dates] | Date | Event | | ------------------------ | ------------------------------------------------------------------------------------------------------------- | | **Now → March 12, 2026** | Continue pushing all code to Bitbucket upstream as usual | | **March 9, 2026** | Rohit Gaikwad begins migrating **Fusion** and **Fusion Worker** to GitHub, along with DevOps pipeline changes | | **March 12, 2026**\* | **Bitbucket hard stop** — merge access revoked, start mirroring changes to GitHub | | **March 26, 2026** | **Bitbucket plan expiration** — all repos must be on GitHub by this date | | **Post March 26, 2026** | All new development moves exclusively to GitHub | > \* *March 12 is the tentative date for the Bitbucket hard stop. This is still pending final confirmation.* *** Responsibilities [#responsibilities] | Person | Responsibility | | ----------------- | ----------------------------------------------------------------------------- | | **Rohit Gaikwad** | All infrastructure changes — code mirroring, pipeline setup, DevOps workflows | | **Sai Tharun** | GitHub migration execution — repo transfers, configuration, coordination | *** Migration Order [#migration-order] | Priority | Service | Notes | | -------- | -------------- | ------------------------------------------------------------- | | 1 | Fusion | Starting week of March 9 — along with pipeline/DevOps changes | | 2 | Fusion Worker | Starting week of March 9 — along with pipeline/DevOps changes | | TBD | Other services | To be announced | *** Important Notes [#important-notes] * **Bitbucket repos will still be accessible** after migration — there is no loss of backup. The repos won't disappear, but all new development must happen on GitHub. * Fusion and Fusion Worker are first because they are low-hanging, service-based repos that are straightforward to migrate with pipeline changes. *** Questions? [#questions] For any queries regarding the migration, reach out on Slack: [#github-migration](https://crelio.slack.com/archives/D027ZCTJE8Y) # Transition Steps Transition Steps [#transition-steps] For All Developers [#for-all-developers] 1. **Before March 12** — Push all your branches and WIP to Bitbucket upstream 2. **Set up SSH keys** for GitHub (see [Prerequisites](/docs/migration/github-migration/prerequisites)) 3. **After cutoff** — Clone repos from GitHub and update your local remotes Updating Your Local Remote [#updating-your-local-remote] Once a repo is migrated, update your remote URL: ```bash git remote set-url origin git@github.com:/.git ``` Verify with: ```bash git remote -v ``` FAQ [#faq] **Q: Can I still push to Bitbucket after March 12?** A: No. After the cutoff, all new commits should go to GitHub. **Q: What if I have open PRs on Bitbucket?** A: Merge or close them before the cutoff. Any unmerged work should be rebased onto the GitHub repo after migration. **Q: Do I need a new SSH key for GitHub?** A: If you already have an SSH key, you can add the same public key to GitHub. You don't need to generate a new one unless you prefer to. # Backend Review Guidelines (Django) Code Review Guidelines - Backend (Django) [#code-review-guidelines---backend-django] This document contains **backend-specific code review rules** for Django/Python development. Refer to [General Guidelines](/docs/pr-review/general) for general review process, SLAs, and reviewer assignment rules. *** Part 1: Developer Self-Review (Before Creating PR) [#part-1-developer-self-review-before-creating-pr] Step 1: Run Pre-commit Hooks on Changed Files [#step-1-run-pre-commit-hooks-on-changed-files] ```bash # Run on staged files only pre-commit run # Or run on specific changed files pre-commit run --files $(git diff --name-only HEAD) # Or run on files changed vs main/develop branch pre-commit run --files $(git diff --name-only origin/develop) ``` Step 2: AI-Assisted Self-Review [#step-2-ai-assisted-self-review] In VS Code/Cursor, select your changed files and use one of these prompts: **Quick Review Prompt:** ``` Review this code against our backend_development.mdc standards. Check for: BaseModel inheritance, ValidationError usage, mapper patterns, QuerySet optimization, naming conventions, and response format compliance. Flag any violations as CRITICAL, MAJOR, or MINOR. ``` **Comprehensive Review Prompt:** ``` Perform a thorough code review of my changes using @backend_development.mdc rules. Check these categories: 1. CODE QUALITY: snake_case naming, no nested ternaries, f-strings, blank line before return 2. ARCHITECTURE: BaseModel/GenericView inheritance, prepare_urls usage, serializer location 3. SECURITY: @allow_guest usage, permission checks, no hardcoded secrets 4. PERFORMANCE: select_related/prefetch_related, no N+1, cache-aware methods 5. PATTERNS: Model.get() not objects.get(), mapper dicts not if/else, ValidationError raised For each issue found, provide: - Severity (CRITICAL/MAJOR/MINOR) - Line reference - What's wrong - How to fix it ``` Step 3: Fix All Issues Before PR [#step-3-fix-all-issues-before-pr] | AI Severity | Action Required | | ----------- | ---------------------------------- | | CRITICAL | Must fix - PR will be rejected | | MAJOR | Must fix - Blocks approval | | MINOR | Should fix - Reviewer will request | *** Part 2: Backend Pre-Submission Checklist [#part-2-backend-pre-submission-checklist] | # | Check | Verification | | - | --------------------- | ------------------------------------------------------------ | | 1 | Pre-commit hooks pass | Run `pre-commit run` - must show all green | | 2 | Self-review completed | AI review + manual diff review | | 3 | No debug code | Remove `print()`, `breakpoint()`, commented code | | 4 | Migrations reviewed | Check generated migration matches intent, no data migrations | | 5 | Documentation updated | Docstrings for new classes/complex methods | | 6 | Fixtures updated | Update fixture files if adding new master data (see below) | Fixture File Updates (Mandatory) [#fixture-file-updates-mandatory] If your changes involve adding new entries to tables with fixtures, you **MUST** update the corresponding fixture file in `fixtures/` directory: | Change Type | Fixture File to Update | | -------------------------------- | ----------------------------- | | New activity log master category | `master_log_category.json` | | New activity log sub-category | `sub_log_category.json` | | New communication trigger | `communication_triggers.json` | | New communication variable | `communication_variable.json` | | New center feature | `center_feature_list.json` | | New assistant prompt | `assistant_prompt.json` | > ⚠️ **Failure to update fixtures will cause issues in fresh deployments and new environments.** *** Part 3: Reviewer Checklist - Code Quality (Blocking) [#part-3-reviewer-checklist---code-quality-blocking] | Check | Standard Reference | | ------------------------------------------------------------------------ | --------------------- | | ✅ Follows `snake_case` for variables/functions, `PascalCase` for classes | Naming Conventions | | ✅ No nested ternaries | Core Principles | | ✅ Uses `Model.get()` instead of `objects.get()` | BaseModel Pattern | | ✅ Uses mapper dictionaries instead of if/else chains | Configuration Mappers | | ✅ `ValidationError` raised (not returned) with proper status codes | Error Handling | | ✅ `JsonResponse` with consistent structure (`status`, `message`, `data`) | Response Format | | ✅ F-strings used for string formatting | Code Style | | ✅ Blank line before `return` statements | Formatting Rules | *** Part 4: Reviewer Checklist - Architecture Compliance (Blocking) [#part-4-reviewer-checklist---architecture-compliance-blocking] | Check | Standard Reference | | --------------------------------------------------------- | ------------------ | | ✅ Models inherit from `core.models.BaseModel` | BaseModel Pattern | | ✅ Views inherit from `core.view.GenericView` | View Structure | | ✅ Serializers inherit from `DynamicFieldsModelSerializer` | Serializers | | ✅ URL patterns use `prepare_urls()` utility | URL Patterns | | ✅ Proxy models in `{app}/proxies/` directory | Proxy Models | | ✅ Serializers in `{app}/serializers/` directory | Folder Structure | | ✅ No custom managers unless explicitly required | Custom Managers | *** Part 5: Reviewer Checklist - Security (Blocking) [#part-5-reviewer-checklist---security-blocking] | Check | Verification | | --------------------------------------------------- | ---------------------- | | ✅ `@allow_guest()` only on public endpoints | No accidental exposure | | ✅ `@validate_feature()` for feature-gated endpoints | Feature validation | | ✅ Permission checks before sensitive operations | Authorization | | ✅ No hardcoded secrets/credentials | Security | | ✅ User input validated before use | Input validation | | ✅ SQL injection prevention (ORM usage, no raw SQL) | Database security | *** Part 6: Reviewer Checklist - Performance (Blocking for High-Impact) [#part-6-reviewer-checklist---performance-blocking-for-high-impact] | Check | Standard Reference | | -------------------------------------------------------------------------- | ---------------------- | | ✅ `select_related()` for ForeignKey traversal | QuerySet Optimization | | ✅ `prefetch_related()` for reverse FK/M2M | QuerySet Optimization | | ✅ `.values()` / `.values_list()` when full objects not needed | QuerySet Optimization | | ✅ No N+1 query patterns | QuerySet Optimization | | ✅ Cache-aware methods used for cached models (`LabFeature.get_features()`) | Cached Model Retrieval | | ✅ `@transaction.atomic` for multi-step DB operations | Defensive Programming | *** Part 7: Reviewer Checklist - Documentation (Non-Blocking) [#part-7-reviewer-checklist---documentation-non-blocking] | Check | Verification | | --------------------------------------------------- | ---------------- | | ✅ Docstrings for classes and complex methods | Documentation | | ✅ `category_id_mapper` defined for activity logging | Activity Logging | | ✅ Complex logic explained (why, not how) | Documentation | *** Part 8: Module Owners (Backend) [#part-8-module-owners-backend] Developers MUST add the relevant **module owner** to their PR (Some of the module owners are): | Module/Area | Module Owner | When to Add | | ----------------- | ------------------------------------------------- | --------------------------------------------------------- | | AI/Assistant | Sai Tharun | Any changes in `assistant/`, AI prompts, LLM integrations | | Payments/Finance | Rahul Bhangale | Changes in `payments/`, `finance/` | | Interfacing | Sumit Rajenimbalkar | Changes in `interfacing/`, devices, reports | | Integrations | Abhijeet Mane | Changes in `integration/` | | Communication | Sai Tharun | Changes in `communication/`, SMS, email | | Lab Reports | Sumit Rajenimbalkar / Rahul Bhangale / Sai Tharun | Changes in `lab_reports/` | | Accession | Rahul Bhangale | Changes in `accession/` | | Billing | Sai Tharun | Changes in `billing/` | | Insurance | Milind Naik / Sidhharth Chakraborty | Changes in `insurance/` | | Bulk Registration | Sidhharth Chakraborty | Changes in `patient/` | | Inventory | Subham Kumar Mal | Changes in `inventory/` | | CRM | Sai Tharun / Akshay Goregankar / Ritu Kataria | Changes in `crm/` | | Lab Forms | Ritu Kataria | Changes in `lab_forms/` | Adding the module owner is the **developer's sole responsibility**. *** Part 9: Backend-Specific Severity Examples [#part-9-backend-specific-severity-examples] | Severity | Backend Examples | | ----------------- | ------------------------------------------------------------------------------------------------------------------------- | | **🔴 Critical** | Missing `@allow_guest` on auth endpoint, raw SQL with user input, exposed secrets in settings | | **🟠 Major** | Missing `BaseModel` inheritance, N+1 queries in loops, no tests for new feature, `objects.get()` instead of `Model.get()` | | **🟡 Minor** | Missing docstring, camelCase variable name, hardcoded string instead of constant | | **🔵 Suggestion** | Could use `values_list()` instead of full objects, consider extracting to utility function | # Development Guidelines Codebase-Specific Guidelines [#codebase-specific-guidelines] These guidelines reflect patterns and decisions specific to our frontend codebase and must be followed unless there is a strong, reviewed reason not to. Routing & Code Splitting [#routing--code-splitting] * All newly added routes **must use lazy loading** * Routes requiring access control should use `AuthRoute` with appropriate `labUser` and `labFeature` checks, where applicable Multi-Build Routing (react-app-rewired) [#multi-build-routing-react-app-rewired] Our frontend supports multiple build targets using `react-app-rewired`. Each build either loads a single module or the full SPA. * **Single-module builds** load only one module and rely on that module’s routing configuration. * **SPA build (`CrelioDashboard`)** loads all modules together and acts as the unified application shell. Build entry points are defined in `app-config.js`: ```js const allBuilds = { crm: "src/modules/crm/index.tsx", Admin: "src/modules/Admin/index.tsx", Finance: "src/modules/Finance/index.tsx", LabAdmin: "src/modules/LabAdmin/index.tsx", inventory: "src/modules/inventory/index.tsx", Accession: "src/modules/Accession/index.tsx", Operations: "src/modules/Operations/index.tsx", DoctorLogin: "src/modules/DoctorLogin/index.tsx", crmDashboard: "src/modules/crmDashboard/index.tsx", Registration: "src/modules/Registration/index.tsx", ReferralLogin: "src/modules/ReferralLogin/index.tsx", MarketingLogin: "src/modules/MarketingLogin/index.tsx", CrelioDashboard: "src/modules/CrelioDashboard/index.tsx", centerdashboard: "src/modules/centerdashboard/index.tsx", OrganisationLogin: "src/modules/OrganisationLogin/index.tsx", }; ``` **Adding a New Route (Mandatory Dual Registration)** * Module-level routing (single-module build) * Add the route in the module’s routing file so it works correctly in the module-specific build. ```bash src/modules//routes.tsx ``` * SPA routing & menu configuration (CrelioDashboard) * Add the same route in the corresponding RoutesAndMenus file so it is available in the SPA build. ```bash src/modules/CrelioDashboard/RoutesAndMenus/RoutesAndMenu.tsx ``` **Example (Operations module)**: ```bash # Module-level routing (single-module build) src/modules/Operations/routes.tsx # SPA routing & menu configuration (CrelioDashboard) src/modules/CrelioDashboard/RoutesAndMenus/operationsRoutesAndMenu.tsx ``` {" "} Failing to add the route in both locations can lead to inconsistent behavior: The page may work in a single-module build but be missing in the SPA - Or appear in the SPA while breaking module-specific builds{" "} JSX & Layout [#jsx--layout] * Avoid unnecessary wrapper `
` elements * Avoid excessive usage of layout components such as `` / `` when not required * Be mindful that certain layout components introduce implicit margin and padding State Management Decisions [#state-management-decisions] * Choose consciously between local state (`useState`) and Redux * Data required only within a component or between a parent and its immediate children (one-level nesting) **usually does not require Redux** * Avoid premature global state introduction Redux Usage [#redux-usage] * Avoid using `genericState` for large or complex data structures (objects or arrays of objects) * For substantial or domain-specific data, define a dedicated Redux slice with: * Clearly typed state * Explicit actions * Predictable reducers * `genericState` should be used only for: * Temporary flags * Quick, short-lived state * Where possible, ensure `genericState` entries are cleaned up on component unmount or after logic completion Component Size & Responsibility [#component-size--responsibility] * Files containing main UI rendering logic (`.jsx` / `.tsx`) should be **finite and readable** * As a guideline, keep such files within **\~250–300 lines** * Complex data transformation, mapping, or conditional logic should be extracted into: * Helper functions * Custom hooks * Utility files * Rendering logic should be easy to visually identify, allowing new contributors to quickly understand: * Where rendering starts * Where rendering ends # Frontend Review Guidelines Frontend Code Review Guidelines [#frontend-code-review-guidelines] Complete end-to-end code review process for frontend development, covering developer responsibilities before raising a PR and reviewer expectations during review. *** Part 1: Developer Guidelines (Before Creating PR) [#part-1-developer-guidelines-before-creating-pr] 1. Self-Review Responsibilities [#1-self-review-responsibilities] Before creating a PR, frontend developers **must self-review their changes** to ensure correctness, readability, performance, and long-term maintainability. Mandatory Self-Review Checklist [#mandatory-self-review-checklist] | # | Check | What to Verify | | - | --------------------- | ------------------------------------------------------------------- | | 1 | Code readability | Logic is easy to follow and well structured | | 2 | Naming clarity | Components, hooks, variables, and functions are meaningful | | 3 | No dead code | Remove unused imports, commented code, console logs | | 4 | Single responsibility | Components/hooks do one thing only | | 5 | Reusability | Common logic extracted into utility/helper functions or hooks | | 6 | Error handling | API failures and edge cases handled | | 7 | UI states | Loading, error, empty, and success/failure states covered | | 8 | Type safety | Types should be clear, consistent, and appropriate for the use case | *** 2. Code Quality Guidelines [#2-code-quality-guidelines] General Standards [#general-standards] * Use **TypeScript strictly** * Avoid `any`,`JsonObject` unless explicitly justified * Follow **camelCase** for variables/functions * Use **PascalCase** for components * Avoid deeply nested conditionals and JSX * Prefer early returns over complex branching * Keep files focused; split large components React Guidelines [#react-guidelines] * Prefer **functional components** * Keep components small and focused * Avoid unnecessary or duplicate `useEffect` hooks * Ensure `useEffect` dependency arrays are correct * Memoization (`useMemo`, `useCallback`) only when required * Avoid deeply nested JSX; extract subcomponents when needed * Use key attribute when rendering lists State Management [#state-management] * Prefer local state over global state * Avoid duplicated or derived state * Keep reducers pure and predictable * Do not perform heavy transformations inside JSX *** 3. Performance & UX Guidelines [#3-performance--ux-guidelines] | Area | Expectations | | --------------- | ------------------------------------------------------- | | Rendering | Avoid unnecessary re-renders | | Lists | Always use stable keys (avoid index where data changes) | | Expensive logic | Memoize when justified | | API calls | No duplicate or parallel calls without reason | | UX feedback | Loaders, disabled states, or skeletons provided | *** 4. Security & Safety Guidelines [#4-security--safety-guidelines] * Avoid exposing secrets, tokens, or sensitive internal identifiers in the UI where possible * Perform basic validation of user inputs before submitting requests (required fields, format checks) * Do not rely solely on backend responses for simple input correctness or user feedback * Be cautious when rendering dynamic or user-provided content *** Part 2: PR Review Guidelines (Reviewer Responsibilities) [#part-2-pr-review-guidelines-reviewer-responsibilities] Category 1: Code Quality (Blocking) [#category-1-code-quality-blocking] | Check | Expectation | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✅ Clear naming | No vague or misleading names | | ✅ Readable JSX | JSX not deeply nested or cluttered | | ✅ Clean code | No unused imports or variables | | ✅ Proper typing | Types are meaningful and correct | | ✅ Formatting | Matches project conventions | | ✅ No debug artifacts | No `console.log` or `debugger` statements. `console.warn` / `console.error` allowed only for intentional, meaningful logging (e.g. third-party integrations or complex API flows). | *** Category 2: Architecture & Patterns (Blocking) [#category-2-architecture--patterns-blocking] | Check | Expectation | | -------------------------- | ------------------------------------- | | ✅ Component responsibility | Components are not overloaded | | ✅ Separation of concerns | UI, logic, and data clearly separated | | ✅ Folder structure | Files placed in correct domains | | ✅ API abstraction | No API calls directly inside JSX | *** Category 3: Performance (Blocking for High Impact) [#category-3-performance-blocking-for-high-impact] | Check | Expectation | | --------------------- | ------------------------------------ | | ✅ Re-render control | Props/state usage reviewed | | ✅ Memoization | Used only where beneficial | | ✅ List efficiency | Stable keys and optimized rendering | | ✅ Effects correctness | No missing or excessive dependencies | *** Category 4: UX & Accessibility (Blocking for User-Facing Changes) [#category-4-ux--accessibility-blocking-for-user-facing-changes] | Check | Expectation | | -------------------- | ----------------------------------- | | ✅ Loading states | Clear feedback during async actions | | ✅ Error states | Helpful and actionable messages | | ✅ Empty states | Graceful handling of no data | | ✅ Visual consistency | Matches existing UI/design system | *** Category 5: Security (Blocking) [#category-5-security-blocking] | Check | Expectation | | -------------------- | ---------------------------------- | | ✅ No secrets exposed | Tokens, keys, internal data hidden | | ✅ Input safety | Inputs handled defensively | | ✅ Auth boundaries | Frontend does not bypass auth | # General Code Review Guidelines Code Review Guidelines - General [#code-review-guidelines---general] This document contains **general code review rules** applicable across all teams and tech stacks. *** Part 1: Review Process Workflow [#part-1-review-process-workflow] *** Part 2: Issue Severity Classification [#part-2-issue-severity-classification] | Severity | Definition | Action | Examples | | ----------------- | ------------------------------------------ | ----------------------------------- | ----------------------------------------------------- | | **🔴 Critical** | Security risk, data loss, production break | Block merge | SQL injection, exposed secrets, missing auth | | **🟠 Major** | Violates architecture, significant bugs | Block merge | Missing base class inheritance, N+1 queries, no tests | | **🟡 Minor** | Style violations, minor improvements | Request change, can merge after fix | Naming inconsistency, missing docstring | | **🔵 Suggestion** | Nice-to-have, refactoring ideas | Comment only, don't block | Alternative approach, future improvement | *** Part 3: Approval Requirements [#part-3-approval-requirements] | Change Type | Minimum Reviewers | Senior Approval Required | | -------------------------- | ----------------- | ------------------------ | | Bug fix (small) | 1 | No | | Feature (new) | 2 | Yes | | Database/Migration changes | 2 | **Yes (mandatory)** | | Security-related | 2 | **Yes (mandatory)** | | Core/Shared utilities | 2 | **Yes (mandatory)** | | Hotfix (production) | 1 | Yes (post-merge review) | *** Part 4: PR Template [#part-4-pr-template] ```markdown ## Summary ## Type of Change - [ ] Bug fix (non-breaking change fixing an issue) - [ ] New feature (non-breaking change adding functionality) - [ ] Breaking change (fix or feature causing existing functionality to change) - [ ] Refactor (code change that neither fixes a bug nor adds a feature) - [ ] Documentation update ## Related Ticket ## Testing Performed - [ ] Manual testing completed - [ ] Tested on local environment ## Screenshots (if UI changes) ## Breaking Changes ## Developer Checklist - [ ] Pre-commit hooks pass - [ ] Self-review of code completed - [ ] No debug code or print statements - [ ] Module owner added as reviewer (if applicable) ``` *** Part 5: Review Comment Guidelines [#part-5-review-comment-guidelines] Comment Format [#comment-format] Use prefixes to indicate severity: * `[CRITICAL]` - Must fix before merge * `[MAJOR]` - Should fix before merge * `[MINOR]` - Nice to fix, can merge after * `[SUGGESTION]` - Optional improvement * `[QUESTION]` - Seeking clarification * `[PRAISE]` - Positive feedback Example Comments [#example-comments] **Good:** ``` [MAJOR] This query inside the loop will cause N+1 issues. Consider using eager loading on line 45. ``` **Bad:** ``` Fix this. ``` Response SLAs [#response-slas] | PR Size | Initial Review | Re-review | | ---------------------- | -------------- | --------- | | Small (\<100 lines) | 4 hours | 2 hours | | Medium (100-500 lines) | 8 hours | 4 hours | | Large (>500 lines) | 24 hours | 8 hours | *** Part 6: Reviewer Assignment Rules [#part-6-reviewer-assignment-rules] Weekly Rotating Review Group [#weekly-rotating-review-group] A **rotating group of 3-4 engineers** is assigned weekly to handle PR reviews: | Rule | Description | | ------------------------ | ---------------------------------------------------------- | | **Group Size** | 3-4 engineers per week | | **Rotation** | Changes every Monday | | **Team Representation** | At least one member from each team for fair representation | | **Notification Channel** | `#eng-pr-reviews` Slack channel | | **Responsibility** | Group ensures all assigned PRs are reviewed and merged | **Weekly Rotation Schedule Example:** ``` Week 1: @dev-A (Team Alpha), @dev-B (Team Beta), @dev-C (Team Gamma) Week 2: @dev-D (Team Alpha), @dev-E (Team Beta), @dev-F (Team Gamma) Week 3: @dev-G (Team Alpha), @dev-H (Team Beta), @dev-I (Team Gamma) ... (continues rotating) ``` Module Owner Requirement (Developer Responsibility) [#module-owner-requirement-developer-responsibility] Adding the module owner as a reviewer is the **developer's sole responsibility**. Developers MUST add the relevant **module owner** to their PR based on the areas of code changed. **If the developer fails to add the module owner:** * The developer is **solely responsible** for any issues that arise * PRs may be reverted if module owner review was skipped * Repeated violations will be flagged in performance reviews Review Assignment Checklist (for Developers) [#review-assignment-checklist-for-developers] ```markdown Before requesting review, ensure: - [ ] At least 1 reviewer from the weekly rotating group is added - [ ] Module owner is added (if applicable to changed areas) - [ ] Posted PR link in #eng-pr-reviews channel ``` General Rules [#general-rules] 1. **Author cannot approve:** Developer cannot approve their own PR 2. **Escalation:** If no review within SLA, post reminder in `#eng-pr-reviews` 3. **Blocking reviews:** Module owner approval is required for merge (when applicable) 4. **Load balancing:** Rotating group helps distribute review load fairly *** Part 7: Pre-Submission Checklist (All Teams) [#part-7-pre-submission-checklist-all-teams] | # | Check | Verification | | - | --------------------- | -------------------------------------------------------------- | | 1 | Pre-commit hooks pass | Must show all green | | 2 | Tests added/updated | New features require tests; bug fixes require regression tests | | 3 | Self-review completed | Read your own diff line-by-line before submission | | 4 | No debug code | Remove `print()`, `console.log()`, `debugger`, commented code | | 5 | Documentation updated | Docstrings/comments for new classes/complex methods | PR Title Format [#pr-title-format] `[TYPE] Brief description` where TYPE = `FEAT`, `FIX`, `REFACTOR`, `DOCS`, `TEST`, `CHORE` Required PR Sections [#required-pr-sections] * Summary (what and why) * Type of change * Testing performed * Related ticket/issue link * Checklist confirmation * Dependent PRs if any * Dependent SQL Queries if any # Features Features [#features] Feature-focused documentation for business domains and workflows built across product surfaces. # Utils Utils [#utils] This section contains reusable utilities, conventions, and implementation standards shared across teams. What this consists of [#what-this-consists-of] # Debug Setup Debug Setup [#debug-setup] HyperDX debugging should start from a trace, not from unrelated logs. The goal is to follow one request from entrypoint to dependency calls and understand what happened without guessing. What Was Missing Before [#what-was-missing-before] | Previous Gap | Debugging Impact | | ------------------------------- | ------------------------------------------------------------------------------------------ | | No reliable request timeline | Engineers could not see the exact order of API, auth, database, cache, and search work | | No child spans for dependencies | Slow requests were hard to split between application code and external systems | | Weak log correlation | Logs existed, but matching them to one request required manual timestamp matching | | Missing domain context | A trace or log did not always explain which lab, route, query shape, or scope was involved | | No consistent trace ID handoff | Frontend, backend, logs, and dashboards were not always connected by one identifier | Required Request Flow [#required-request-flow] Every instrumented service should make this path possible: | Step | Where | Requirement | | ---- | ------------------------ | ----------------------------------------------------------------------- | | 1 | Client or caller | Capture the request that failed or was slow | | 2 | Response headers or logs | Find the request trace ID, preferably `X-Trace-Id` for HTTP APIs | | 3 | HyperDX | Search the trace ID | | 4 | Trace timeline | Inspect the root route span and child spans | | 5 | Dependency span | Check Redis, MySQL, Elasticsearch, HTTP, queue, or other external calls | | 6 | Domain attributes | Confirm the business context that shaped the behavior | What To Inspect In HyperDX [#what-to-inspect-in-hyperdx] | Signal | What It Answers | | ----------------------- | ---------------------------------------------------------------------------- | | Root span duration | How long the API request or job took end to end | | Child span duration | Which dependency or internal block consumed time | | Span status | Whether the failure is attached to a specific span | | Error events | Exception type, message, and where the error was recorded | | Trace ID | Shared correlation key for frontend, logs, traces, and backend investigation | | Service name | Which service emitted the span | | Route or operation name | Which endpoint, worker job, or operation ran | | Span attributes | Domain-specific context needed to explain the behavior | Debugging Common Issues [#debugging-common-issues] | Symptom | First Check | Follow-up | | ---------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | Slow API request | Compare root route span duration with dependency span durations | If dependencies are fast, inspect auth/session work, response shaping, serialization, and runtime overhead | | Empty or wrong search result | Inspect domain attributes and search dependency spans | Confirm query shape, routing, filters, hit count, and target index | | Auth failure | Inspect auth/session span and logs for the same trace | Confirm token/session values, Redis lookup, lab context, and rejected reason | | `5xx` response | Open the failed trace by trace ID | Find the span with error status, then inspect logs and Sentry for the same trace ID | | Stale data | Check freshness metrics and the request trace together | Confirm whether the read path is healthy before moving to CDC or sync runbooks | | Dependency timeout | Inspect dependency span status and duration | Check retry behavior, timeout config, and downstream health dashboards | Phoenix Search Example [#phoenix-search-example] Phoenix Search should expose the trace ID on search responses and emit enough span context to explain slow, empty, or failed searches. Important Phoenix Search attributes: | Attribute | Why It Matters | | ------------------------ | -------------------------------------------- | | `search.lab_id` | Confirms the lab context used by the request | | `search.search_key` | Shows which search mode was requested | | `search.query_shape` | Explains how the input was classified | | `search.routing` | Confirms Elasticsearch routing | | `search.is_multi_center` | Shows whether related lab scope was used | | `search.hit_count` | Shows how many results came back | | `search.zero_results` | Makes empty-result traces searchable | Important Elasticsearch span fields: | Field | Why It Matters | | ------------------------------ | ------------------------------------------------------------ | | `db.system.name` | Confirms the dependency is Elasticsearch | | `db.operation.name` | Confirms the operation, usually `search` | | `db.operation.parameter.index` | Confirms the target index | | `db.query.text` | Shows the generated query body when query capture is enabled | | `db.response.status_code` | Separates Elasticsearch failures from application failures | Debug Setup Checklist [#debug-setup-checklist] | Check | Expected Result | | ------------------------------------- | ----------------------------------------------------------------------- | | Service exports OTLP | Traces appear in HyperDX under the correct `OTEL_SERVICE_NAME` | | Framework instrumentation is enabled | HTTP route spans are created automatically | | Dependency instrumentation is enabled | Redis, MySQL, Elasticsearch, HTTP, or queue spans appear as child spans | | Logs include trace context | Logs carry `trace_id` and `span_id` | | HTTP response exposes trace ID | Operators can copy `X-Trace-Id` from the failing request | | Domain attributes are added | HyperDX traces explain business context, not only technical timing | | Errors are recorded on spans | Failed traces show the failing span and exception context | Do not call the setup complete until a real request can be opened in HyperDX and the timeline shows the route span, dependency spans, logs, and required domain attributes. # ClickStack HyperDX Infra ClickStack HyperDX Infra [#clickstack-hyperdx-infra] ClickStack HyperDX Infra is CrelioHealth's self-hosted observability service for OpenTelemetry data. Application services send traces, metrics, and logs to the OpenTelemetry collector, HyperDX provides the debugging UI, and ClickHouse stores the telemetry data. This service exists so engineers can debug production behavior from one request timeline instead of manually matching logs, timestamps, dashboards, and dependency calls. Repository Information [#repository-information] | Property | Value | | ------------------ | -------------------------------------------------------------------- | | **Repository** | [clickstack-infra](https://github.com/CrelioHealth/clickstack-infra) | | **Local Path** | `/Users/saitharun/Documents/hyperdx` | | **Runtime** | AWS ECS Fargate + EC2 ClickHouse | | **Region** | `ap-south-1` | | **Primary UI** | HyperDX | | **Ingestion** | OpenTelemetry collector over OTLP gRPC and OTLP HTTP | | **Storage** | ClickHouse | | **Metadata Store** | MongoDB | What This Service Provides [#what-this-service-provides] | Capability | Why It Matters | | ----------------------- | -------------------------------------------------------------------------- | | Trace timelines | Shows the full request path with parent and child spans | | Span duration breakdown | Separates application time from dependency time | | Log correlation | Connects logs to `trace_id` and `span_id` | | Metrics | Tracks service, dependency, and domain health over time | | Error investigation | Starts from a failed request and follows the trace to the failing span | | Service context | Adds domain fields such as lab, route, query shape, routing, and hit count | Why We Added OpenTelemetry [#why-we-added-opentelemetry] Before this setup, debugging depended mostly on logs, dashboards, and manual correlation. That made incidents slow to reason about because the team could not reliably see the exact request timeline, which dependency took time, which span failed, or which domain context was attached to the failing request. OpenTelemetry fixes that by sending structured traces, metrics, and logs from each service into HyperDX. A single trace can show the route span, auth/session work, Redis calls, MySQL calls, Elasticsearch calls, errors, timings, and service-specific attributes for the same user request. Current Example Integration [#current-example-integration] Phoenix Search is the first documented integration. It emits request traces, dependency spans, search attributes, Elasticsearch query context, CDC freshness metrics, and trace-aware logs. Use these pages for implementation and debugging details: * [Debug Setup](/docs/services/clickstack-hyperdx/debug-setup) * [Service Integration](/docs/services/clickstack-hyperdx/service-integration) * [Phoenix Search API Debugging](/docs/services/phoenix-search/operate/debugging) Debugging Principle [#debugging-principle] For every production issue, start with the most specific request identifier available: | Starting Point | Next Step | | ----------------------------------- | ------------------------------------------------------------------------------ | | Browser response has `X-Trace-Id` | Search that trace ID in HyperDX | | Backend log has `trace_id` | Open the matching trace and inspect spans | | Dashboard shows latency or errors | Filter by service, route, and time window, then inspect a representative trace | | User reports empty or wrong results | Inspect domain attributes and dependency spans for that request | The expected outcome is a clear answer to where the time, error, or wrong behavior came from. # Service Integration Service Integration [#service-integration] Every service should integrate with ClickStack HyperDX in the same basic shape: configure OpenTelemetry, enable automatic instrumentation, add domain attributes, and verify the result in HyperDX with a real request. Required Configuration [#required-configuration] | Setting | Purpose | | ----------------------------- | ------------------------------------------------------- | | `OTEL_SERVICE_NAME` | Stable service name shown in HyperDX | | `OTEL_EXPORTER_OTLP_ENDPOINT` | OpenTelemetry collector endpoint | | `OTEL_RESOURCE_ATTRIBUTES` | Environment, deployment, region, and ownership metadata | | Service log format | Must include trace and span IDs when a span is active | | Trace sampling policy | Must be explicit per environment | Collector endpoints are provided by the ClickStack HyperDX infra: | Protocol | Internal Endpoint | Port | | --------- | ------------------------------------- | ------ | | OTLP gRPC | `otel-collector.hdx-infra.local:4317` | `4317` | | OTLP HTTP | `otel-collector.hdx-infra.local:4318` | `4318` | Use the internal endpoint for services running inside the VPC. If an external ingest domain is configured, use the DNS name managed by the ClickStack infra instead of the Cloud Map name. Instrumentation Requirements [#instrumentation-requirements] | Area | Requirement | | ----------------- | ---------------------------------------------------------------------------- | | Entrypoints | Instrument HTTP routes, worker jobs, consumers, scheduled jobs, or CLI tasks | | Dependencies | Instrument database, cache, search, queue, and outbound HTTP clients | | Errors | Record exceptions on the active span before returning or retrying | | Logs | Inject `trace_id` and `span_id` into structured logs | | Metrics | Emit service and domain metrics using stable names | | Trace propagation | Forward incoming trace context and return a trace ID for support/debugging | Domain Context Requirements [#domain-context-requirements] Automatic spans are not enough. Each service must add the fields that explain why a request behaved the way it did. Examples of useful domain attributes: | Service Type | Useful Attributes | | ------------------- | --------------------------------------------------------------------------- | | API service | route, authenticated account, organization, lab, feature flag, request mode | | Search service | search key, query shape, routing, target index, hit count, zero-result flag | | Worker service | queue, job type, retry count, tenant, source event ID | | Sync service | source table, destination table, lag, batch size, offset, DLQ status | | Integration service | vendor, message type, external request ID, response code, retry state | Do not add secrets, tokens, PHI, raw patient payloads, or unrestricted request bodies as span attributes. Query capture must be explicitly controlled by service configuration. Implementation Checklist [#implementation-checklist] | Step | Done When | | --------------------------------- | -------------------------------------------------------------------------- | | Configure exporter | Service sends OTLP to the ClickStack HyperDX collector | | Set service identity | HyperDX shows the expected `OTEL_SERVICE_NAME` | | Enable framework instrumentation | Requests or jobs create root spans | | Enable dependency instrumentation | Downstream calls create child spans | | Add domain attributes | A trace explains tenant/scope/query/job context | | Correlate logs | Logs from the request appear with trace and span IDs | | Return trace ID | HTTP APIs expose `X-Trace-Id` or an equivalent support-safe correlation ID | | Validate in HyperDX | A real request shows timeline, spans, logs, metrics, and errors if present | Rollout Validation [#rollout-validation] Before marking a service as integrated: | Validation | Expected Result | | --------------- | ----------------------------------------------------------------- | | Healthy request | Root span is visible with correct service name and route/job name | | Dependency call | Child span appears with duration and status | | Error path | Failed request records error status and exception context | | Log correlation | Logs are searchable from the trace | | Domain context | Required service attributes are present | | Support handoff | Trace ID from response or logs opens the same request in HyperDX | Phoenix Search Baseline [#phoenix-search-baseline] Phoenix Search is the reference integration for this setup. It should be used as the baseline for new services because it connects the browser request, `X-Trace-Id`, API route span, Elasticsearch span, logs, metrics, and search-specific attributes in one debugging flow. For the current Phoenix Search flow, see [Phoenix Search API Debugging](/docs/services/phoenix-search/operate/debugging). # Crelio AI Crelio AI Service [#crelio-ai-service] Repository Information [#repository-information] * **Repository**: [crelio-ai](https://bitbucket.org/creliohealth-repo/crelio-ai) * **Language**: Python * **Main Branch**: `develop` * **Project**: CrelioHealth-For-Doctors (CREL) * **Created**: December 2024 Overview [#overview] AI service for intelligent healthcare solutions and automation. Clone Repository [#clone-repository] ```bash git clone git@bitbucket.org:creliohealth-repo/crelio-ai.git cd crelio-ai git checkout develop ``` # Crelio App Crelio App Service [#crelio-app-service] Overview [#overview] Backend service for the Crelio App, built with Django. This service handles patient registration, billing processes, lab reports, and external integrations. Repository Information [#repository-information] * **Repository**: [crelio-app](https://bitbucket.org/creliohealth-repo/crelio-app) * **Language**: Python (Django) * **Main Branch**: `develop` * **Project**: CrelioHealth-For-Doctors (CREL) * **Created**: November 2021 Quick Start [#quick-start] ```bash git clone git@bitbucket.org:creliohealth-repo/crelio-app.git cd crelio-app git checkout develop ``` *** Documentation [#documentation] Architecture [#architecture] Comprehensive internal architecture documentation covering design patterns, data flow, and app-specific details. * **[System Architecture](./architecture/00-system-architecture)**: High-level overview and domain boundaries. * **[Fat Models Design](./architecture/01-fat-models-design)**: Understanding the core design philosophy. * **[Data Flow](./architecture/02-data-flow-and-lifecycle)**: Request lifecycle and integration flows. * **[App Specifics](./architecture/apps)**: Deep dive into Core, Patient, Finance, Report, and other apps. # API Reference Fusion API Reference [#fusion-api-reference] Fusion exposes HTTP endpoints for enqueueing and managing background tasks. All endpoints accept JSON payloads. *** Queue Tasks [#queue-tasks] POST /queue/ [#post-queue] Immediately enqueue one or more tasks for processing. **Request Body:** ```json [ { "task": 1, "task_type": 1, "data": { "contact": 9876543210, "message": "Hello World", "countryCode": 91, "senderId": "CRELIO" }, "log_tag": "SMS_OTP", "headers": {}, "subscribe": "https://callback.url/notify", "failed_task_handler": "https://callback.url/failed", "success_task_handler": "https://callback.url/success", "override_priority": 2 } ] ``` **Parameters:** | Field | Type | Required | Description | | ---------------------- | ------ | -------- | ------------------------------------------------------- | | `task` | int | Yes | Task type (1=SMS, 2=Email, 3=PN, 4=Webhook, 6=WhatsApp) | | `task_type` | int | Yes | Sub-type of the task | | `data` | object | Yes | Task-specific payload | | `log_tag` | string | No | Custom tag for logging | | `headers` | object | No | HTTP headers for webhooks | | `subscribe` | string | No | URL for result callbacks | | `failed_task_handler` | string | No | URL to call when task fails | | `success_task_handler` | string | No | URL to call when task succeeds | | `override_priority` | int | No | Override default priority (0, 1, or 2) | | `retry` | int | No | Number of retry attempts (max 10) | **Response:** ```json { "code": 200, "job_ids": ["job-uuid-1", "job-uuid-2"] } ``` *** Task Types Reference [#task-types-reference] Task 1: SMS [#task-1-sms] | task\_type | Name | Priority | Function | | ---------- | -------------- | ------------ | ---------------------- | | 1 | OTP | HIGH (2) | `tasks.sms.sendOTPSMS` | | 2 | App Links | LOW (0) | `tasks.sms.sendSMS` | | 3 | Welcome | MODERATE (1) | `tasks.sms.sendSMS` | | 4 | Billing Report | MODERATE (1) | `tasks.sms.sendSMS` | | 5 | Appointment/HC | MODERATE (1) | `tasks.sms.sendSMS` | | 6 | Campaign | LOW (0) | `tasks.sms.sendSMS` | | 7 | Reminder | HIGH (2) | `tasks.sms.sendSMS` | **SMS Payload:** ```json { "contact": 9876543210, "message": "Your OTP is 1234", "countryCode": 91, "senderId": "CRELIO", "otp": 1234, "config": { "provider": "msg91" }, "callback_url": "https://..." } ``` Task 2: Email [#task-2-email] | task\_type | Name | Priority | Function | | ---------- | -------------- | ------------ | ------------------------------------------------ | | 1 | Welcome | MODERATE (1) | `tasks._email_v2.send_plain_email` | | 2 | Billing Report | MODERATE (1) | `tasks._email_v2.send_plain_email` | | 3 | Admin | MODERATE (1) | `tasks._email_v2.send_plain_email` | | 4 | Misc | LOW (0) | `tasks._email_v2.send_plain_email` | | 5 | Template | LOW (0) | `tasks._email_v2.send_template_attachment_email` | | 6 | Attachment | LOW (0) | `tasks._email_v2.send_template_attachment_email` | **Email Payload:** ```json { "to": ["user@example.com"], "cc": [], "bcc": [], "subject": "Email Subject", "message": "Email body", "sender_name": "CrelioHealth", "sender_email": "no-reply@creliohealth.com", "files": ["https://s3.amazonaws.com/.../file.pdf"], "template": "Template HTML", "is_template": 1, "callback_url": "https://..." } ``` Task 3: Push Notification [#task-3-push-notification] | task\_type | Name | Priority | | ---------- | -------------- | ------------ | | 1 | Reports | MODERATE (1) | | 2 | Appointment/HC | MODERATE (1) | | 3 | Promotions | MODERATE (1) | | 4 | Campaign | LOW (0) | | 5 | Feedback | LOW (0) | | 6 | Reminders | HIGH (2) | Task 4: Webhook [#task-4-webhook] | task\_type | Name | Priority | Function | | ---------- | ------ | ------------ | ----------------------- | | 1 | GET | MODERATE (1) | `tasks.webhooks.GET` | | 2 | POST | MODERATE (1) | `tasks.webhooks.POST` | | 3 | PUT | MODERATE (1) | `tasks.webhooks.PUT` | | 4 | DELETE | MODERATE (1) | `tasks.webhooks.DELETE` | **Webhook Payload:** ```json { "url": "https://api.example.com/endpoint", "args": {"key": "value"}, "isBody": 1, "timeout": 15, "callback": 1, "callback_url": "https://callback.url", "headers": { "Authorization": "Bearer token" } } ``` Task 6: WhatsApp [#task-6-whatsapp] | task\_type | Name | Priority | Function | | ---------- | -------------- | ------------ | -------------------------------------- | | 1 | Welcome | MODERATE (1) | `tasks.whatsapp.sendWhatsapp` | | 2 | Billing Report | MODERATE (1) | `tasks.whatsapp.sendWhatsapp_withFile` | *** Schedule Tasks [#schedule-tasks] POST /schedule/ [#post-schedule] Schedule tasks for future execution. **Request Body:** ```json [ { "sch_time": "2024-01-15T10:30:00", "task": 1, "task_type": 2, "data": {...}, "subscribe": "https://...", "headers": {} } ] ``` **Parameters:** | Field | Type | Required | Description | | ----------------- | ------ | -------- | ---------------------------------------------------- | | `sch_time` | string | Yes | Scheduled time in `YYYY-MM-DDTHH:MM:SS` format (UTC) | | `task` | int | Yes | Task type | | `task_type` | int | Yes | Sub-type of the task | | `data` | object | Yes | Task-specific payload | | `window_delay` | string | No | Key for window-based consolidation | | `override_window` | int | No | 1 to override existing window | **Response:** ```json { "code": 200, "job_ids": ["job-uuid-1"] } ``` *** Job Management [#job-management] POST /jobInfo/ [#post-jobinfo] Get information about a specific job. **Request Body:** ```json { "id": "job-uuid" } ``` POST /cancelJob/ [#post-canceljob] Cancel one or more scheduled jobs. **Request Body:** ```json { "job_id": ["job-uuid-1", "job-uuid-2"] } ``` **Response:** ```json {"code": 200} // Success {"code": 401} // Job not found {"code": 500} // Error ``` POST /rescheduleJob/ [#post-reschedulejob] Reschedule a job to a new time. **Request Body:** ```json { "job_id": "job-uuid", "resc_at": "2024-01-16T14:00:00" } ``` **Response:** ```json {"code": 200} // Success {"code": 401} // Job not found or already executed ``` POST /retryTask/ [#post-retrytask] Retry a failed task with a 2-minute delay. **Request Body:** ```json [ {"job_id": "job-uuid"} ] ``` *** Utility Endpoints [#utility-endpoints] POST /mapping/ [#post-mapping] Queue dictionary mapping tasks for processing. **Request Body:** ```json [ {"keyword": "term1", "data": {...}}, {"keyword": "term2", "data": {...}} ] ``` POST /update_consent/ [#post-update_consent] Update WhatsApp consent for a user. **Request Body:** ```json { "type": "consent_type", "cache_key": "unique_key", "details_obj": {...} } ``` *** Error Codes [#error-codes] | Code | Description | | ---- | --------------------- | | 200 | Success | | 401 | Job not found | | 500 | Internal server error | # Fusion Fusion Service [#fusion-service] Fusion is a **Flask-based task queue API server** that serves as the central hub for scheduling and routing background tasks in CrelioHealth. It provides HTTP endpoints for enqueueing tasks like SMS, Email, Push Notifications, Webhooks, and WhatsApp messages into Redis queues for processing by Fusion Worker. Repository Information [#repository-information] | Property | Value | | ----------------- | -------------------------------------------------------- | | **Repository** | [fusion](https://bitbucket.org/creliohealth-repo/fusion) | | **Language** | Python 3.6+ | | **Main Branch** | `main` | | **Project** | CrelioHealth-Fusion (CFN) | | **Framework** | Flask | | **Queue Backend** | Redis (RQ - Redis Queue) | Clone Repository [#clone-repository] ```bash git clone git@bitbucket.org:creliohealth-repo/fusion.git ``` *** Tech Stack [#tech-stack] | Category | Technology | Purpose | | ------------------- | ----------------- | ------------------------------- | | **Runtime** | Python 3.6+ | Application runtime | | **Web Framework** | Flask 0.12 | HTTP API server | | **Queue Backend** | Redis 3.5 | Job queue storage | | **Queue Library** | RQ (Custom Fork) | Task queue management | | **Scheduler** | RQ Scheduler | Delayed/Cron job scheduling | | **AWS SDK** | Boto3 | AWS Secrets Manager integration | | **Logging** | Elasticsearch 6.x | Centralized logging | | **Monitoring** | Raven (Sentry) | Error tracking | | **Process Manager** | Supervisor | Production process management | | **WSGI Server** | Gunicorn + Gevent | Production HTTP server | *** System Architecture Overview [#system-architecture-overview] *** Key Components [#key-components] 1. Flask API Server (app.py) [#1-flask-api-server-apppy] The main entry point that exposes the following HTTP endpoints: | Endpoint | Method | Description | | ------------------ | ------ | ---------------------------------------- | | `/queue/` | POST | Immediately enqueue tasks for processing | | `/schedule/` | POST | Schedule tasks for future execution | | `/jobInfo/` | POST | Get information about a specific job | | `/cancelJob/` | POST | Cancel scheduled jobs | | `/rescheduleJob/` | POST | Reschedule a job to a new time | | `/mapping/` | POST | Queue dictionary mapping tasks | | `/retryTask/` | POST | Retry a failed task | | `/update_consent/` | POST | Update WhatsApp consent status | 2. Priority Queue (wrappers/PriorityQueue.py) [#2-priority-queue-wrapperspriorityqueuepy] A custom priority-based queue implementation using Redis Sorted Sets (ZSET): ```python # Priority Levels PRIORITY_HIGH = 2 # OTP, Reminders PRIORITY_MODERATE = 1 # Standard messages PRIORITY_LOW = 0 # Campaigns, Bulk operations ``` Jobs are stored with their priority as the score, allowing higher-priority jobs to be processed first using `ZREVRANGE`. 3. RQ Scheduler (rq_scheduler/scheduler.py) [#3-rq-scheduler-rq_schedulerschedulerpy] Handles time-based job scheduling: * **`enqueue_at()`** - Schedule a job for a specific datetime * **`enqueue_in()`** - Schedule a job after a time delta * **`cron()`** - Schedule recurring jobs using cron syntax * Jobs are stored in `rq:scheduler:scheduled_jobs` sorted set 4. Task Types and Routing [#4-task-types-and-routing] ```python # Task to Queue Mapping Task 1 (SMS) → Queue: 'sms' → tasks.sms.sendSMS Task 2 (EMAIL) → Queue: 'email' → tasks._email_v2.send_plain_email Task 3 (PN) → Queue: 'pn' → tasks.push_notification.sendNotification Task 4 (WEBHOOK) → Queue: 'webhook' → tasks.webhooks.GET/POST/PUT/DELETE Task 6 (WHATSAPP) → Queue: 'whatsapp' → tasks.whatsapp.sendWhatsapp ``` *** Directory Structure [#directory-structure] ``` fusion/ ├── app.py # Main Flask application ├── config.py # AWS Secrets Manager configuration ├── run_worker.py # Worker entry point ├── run_schduler.py # Scheduler entry point ├── run_dashboard.py # RQ Dashboard entry point ├── rq/ # Custom RQ implementation │ ├── queue.py # Queue class with priority support │ ├── job.py # Job class with retry & callbacks │ ├── worker.py # Worker class │ ├── connections.py # Redis connection management │ └── cli/ # CLI commands ├── rq_scheduler/ # Scheduler implementation │ ├── scheduler.py # Main scheduler logic │ └── utils.py # Utility functions ├── wrappers/ # Utility wrappers │ ├── PriorityQueue.py # Priority queue implementation │ ├── Logger.py # Elasticsearch logging │ ├── scheduler_task.py # Cron job definitions │ └── _utils.py # Helper functions ├── instance/ # Environment configs │ ├── production.py # Production settings │ └── local.py # Local development settings └── dashboard/ # RQ Dashboard ``` *** Running Fusion [#running-fusion] Local Development [#local-development] ```bash # Create virtual environment python3.6 -m venv venv source venv/bin/activate # Install dependencies pip install -r req.txt # Set environment variables export CRELIO_DEPLOYMENT_ZONE="IN" export CRELIO_DEPLOYMENT_MODE="LOCAL" # Run Flask server flask run -h 0.0.0.0 -p 8002 --reload # Run scheduler (separate terminal) python run_schduler.py # Run dashboard (separate terminal) python run_dashboard.py ``` Using Procfile [#using-procfile] ```bash # Web server web: CRELIO_DEPLOYMENT_ZONE="IN" CRELIO_DEPLOYMENT_MODE="LOCAL" FLASK_APP=app.py flask run -h 0.0.0.0 -p 8002 --reload # Scheduler scheduler: python run_schduler.py # Dashboard dashboard: python run_dashboard.py ``` *** Configuration [#configuration] Environment Variables [#environment-variables] | Variable | Description | Example | | ------------------------ | ----------------- | ------------------------------ | | `CRELIO_DEPLOYMENT_ZONE` | Deployment region | `IN`, `US`, `EU`, `UAE`, `KSA` | | `CRELIO_DEPLOYMENT_MODE` | Deployment mode | `prod`, `staging`, `LOCAL` | Secrets Management [#secrets-management] Secrets are fetched from AWS Secrets Manager based on the deployment zone: ```python # Secret naming convention secret_name = f"{deployment_mode}/{deployment_region}/fusion".lower() # Example: "prod/in/fusion" ``` For local development, secrets are loaded from `secrets.json` in the parent directory. # RQ System Architecture RQ System Architecture [#rq-system-architecture] Fusion uses a **custom fork of RQ (Redis Queue)** with priority queue support and enhanced job management features. This document provides a deep technical understanding of how the queue system works. *** Core Concepts [#core-concepts] Redis as Queue Backend [#redis-as-queue-backend] All queues are stored in Redis using specific key patterns: | Key Pattern | Data Structure | Purpose | | ----------------------------- | ----------------- | --------------------------------------- | | `rq:queue:` | Sorted Set (ZSET) | Priority queues for tasks | | `rq:job:` | Hash | Job data and metadata | | `rq:job::dependents` | Set | Jobs dependent on this job | | `rq:scheduler:scheduled_jobs` | Sorted Set | Scheduled jobs (score = unix timestamp) | | `rq:worker:` | Hash | Worker registration and heartbeat | | `rq:workers` | Set | All registered workers | | `rq:queues` | Set | All known queue keys | *** Priority Queue Implementation [#priority-queue-implementation] Unlike standard RQ which uses Redis Lists (FIFO), Fusion uses **Sorted Sets (ZSET)** for priority-based processing. How It Works [#how-it-works] Priority Levels [#priority-levels] ```python class PriorityQueue: PRIORITY_HIGH = 2 # OTP, Reminders - processed first PRIORITY_MODERATE = 1 # Standard messages PRIORITY_LOW = 0 # Campaigns, bulk operations - processed last ``` Atomic Pop Operation [#atomic-pop-operation] To prevent race conditions when multiple workers are running: ```python def pop(self): try: # Get highest priority job (highest score) _item = self.connection.zrevrange(self._key, 0, 0)[0] # Atomically remove it - if ZREM returns 1, we got it if self.connection.zrem(self._key, _item) == 1: return _item else: # Another worker got it first, retry return self.pop() except IndexError: return None # Queue is empty ``` *** Job Lifecycle [#job-lifecycle] Job Statuses [#job-statuses] | Status | Description | | ---------- | ---------------------------------- | | `queued` | Job is waiting in queue | | `started` | Worker is executing the job | | `finished` | Job completed successfully | | `failed` | Job execution failed | | `deferred` | Waiting for dependency to complete | *** Job Data Structure [#job-data-structure] Each job is stored as a Redis Hash with the following fields: ```python { 'data': , 'created_at': '2024-01-15T10:30:00', 'origin': 'sms', # Queue name 'original_queue': 'sms', # Original queue (for requeue) 'description': 'tasks.sms.sendSMS(...)', 'enqueued_at': '2024-01-15T10:30:00', 'started_at': '2024-01-15T10:30:01', 'ended_at': '2024-01-15T10:30:02', 'status': 'finished', 'result': , 'exc_info': , 'timeout': 180, 'result_ttl': 500, # Custom Fusion fields 'priority': 2, 'retryCount': 7, 'log_tag': 'SMS_OTP', 'failed_task_handler': 'https://...', 'success_task_handler': 'https://...' } ``` *** Retry Mechanism [#retry-mechanism] Fusion implements automatic retry with exponential backoff: Retry Configuration [#retry-configuration] ```python # Default retry count job.retryCount = 7 # Retryable exceptions - ClientError (AWS) - ConnectTimeout - ConnectionError - ReadTimeout # Non-retryable (immediate failure) - All other exceptions ``` *** Scheduler System [#scheduler-system] The scheduler runs as a separate process, periodically checking for jobs that are due: Scheduler Flow [#scheduler-flow] Scheduling Options [#scheduling-options] 1. **enqueue\_at()** - Execute at specific datetime 2. **enqueue\_in()** - Execute after time delta 3. **cron()** - Recurring execution using cron syntax Cron Jobs [#cron-jobs] Defined in `wrappers/scheduler_task.py`: ```python # Run maintenance every 5 minutes add(scheduler, {}, '*/5 * * * *', 'tasks.maintenance.run_maintenance_on_failed_queue', id='redis_maintenance_on_failed_queue') ``` *** Worker Process Model [#worker-process-model] Workers use a **fork-based execution model** for isolation: Worker States [#worker-states] | State | Description | | ----------- | ---------------------- | | `started` | Worker just started | | `idle` | Waiting for jobs | | `busy` | Executing a job | | `suspended` | Paused by `rq suspend` | Heartbeat [#heartbeat] Workers send heartbeats to Redis to indicate they're alive: ```python def heartbeat(self, timeout=0): timeout = max(timeout, self.default_worker_ttl) # 420 seconds self.connection.expire(self.key, timeout) ``` *** Failed Queue [#failed-queue] Failed jobs are moved to a special queue for inspection and retry: ```python # Queue name rq:queue:failed # Requeue operations failed_queue.requeue(job_id) # To original queue (List) failed_queue.requeue_set(job_id) # To priority queue (Sorted Set) ``` *** Connection Management [#connection-management] Fusion uses a connection stack pattern for managing Redis connections: ```python # Push connection for current context push_connection(redis_conn) # Get current connection conn = get_current_connection() # Pop when done pop_connection() # Context manager with Connection(redis_conn): # Use connection pass ``` *** Logging & Monitoring [#logging--monitoring] Elasticsearch Logging [#elasticsearch-logging] All job events are logged to Elasticsearch: ```python # Index: active-fusion-logs # Type: logs { "job_id": "uuid", "task": "SMS", "task_type": "OTP", "status": "queued|processing|completed|failed|retry", "host": "worker-hostname", "log_time": "2024-01-15T10:30:00", "message": "Status message", "payload": "...", "job_result": "..." } ``` Log Events [#log-events] | Event | Method | Status | | ------------------- | ------------------------- | ------------ | | Job enqueued | `Log.start()` | `queued` | | Execution started | `Log.execution_start()` | `processing` | | Execution succeeded | `Log.execution_success()` | `completed` | | Execution failed | `Log.execution_failed()` | `failed` | | Retry scheduled | `Log.execution_retry()` | `retry` | *** Best Practices [#best-practices] 1. Choosing Priority [#1-choosing-priority] ```python # HIGH (2) - User-facing, time-sensitive - OTP SMS - Password reset emails - Critical reminders # MODERATE (1) - Standard operations - Report delivery - Appointment confirmations - Standard webhooks # LOW (0) - Background, bulk operations - Marketing campaigns - Batch notifications - Analytics webhooks ``` 2. Setting Timeouts [#2-setting-timeouts] ```python # Default timeout: 180 seconds # Webhook default: 15 seconds # For slow operations, increase timeout: payload = { 'url': '...', 'timeout': 60 # 60 seconds } ``` 3. Using Callbacks [#3-using-callbacks] ```python # For tracking delivery status: payload = { 'callback': 1, 'callback_url': 'https://your-api.com/delivery-status' } # For error handling: data = { 'failed_task_handler': 'https://your-api.com/handle-failure', 'success_task_handler': 'https://your-api.com/handle-success' } ``` # Worker Architecture Fusion Worker Architecture [#fusion-worker-architecture] Fusion Worker is a **multi-process, fork-based task execution system** that consumes jobs from Redis priority queues and executes them in isolated child processes. *** High-Level Architecture [#high-level-architecture] *** Process Model [#process-model] Fusion Worker uses a **fork-based execution model** for process isolation and safety. Why Fork? [#why-fork] 1. **Isolation** - Each job runs in its own process; crashes don't affect the main worker 2. **Memory Safety** - Child process memory is freed on exit 3. **Timeout Enforcement** - Parent can kill hung child processes 4. **Clean State** - Each job starts with a fresh process state Fork Flow [#fork-flow] *** Worker State Machine [#worker-state-machine] Worker States [#worker-states] | State | Description | Redis Key Value | | ----------- | ----------------------- | ------------------ | | `started` | Worker just initialized | `state: started` | | `idle` | Waiting for jobs | `state: idle` | | `busy` | Executing a job | `state: busy` | | `suspended` | Paused by admin | `state: suspended` | *** Queue Processing [#queue-processing] Priority Queue Mechanics [#priority-queue-mechanics] Unlike standard RQ which uses Redis Lists (FIFO), Fusion Worker uses **Sorted Sets (ZSET)** for priority-based processing. Atomic Dequeue [#atomic-dequeue] To prevent race conditions with multiple workers: ```python def work(self): while True: # Get highest priority job _items = connection.zrevrange(current_queue, 0, 0) if len(_items) > 0: _item = _items[0] else: continue # Queue empty # Atomically remove - only one worker can succeed if connection.zrem(current_queue, _item) == 1: # We got the job result = _item else: # Another worker got it first continue ``` Queue Round-Robin [#queue-round-robin] Workers iterate through all queues in round-robin fashion: ```python queue_keys = [q.key for q in self.queues] queues_length = len(queue_keys) current_queue_index = -1 while True: current_queue_index += 1 if current_queue_index >= queues_length: current_queue_index = 0 current_queue = queue_keys[current_queue_index] # Try to get job from this queue # If empty, move to next queue ``` Special Scheduler Queue Handling [#special-scheduler-queue-handling] The scheduler queue uses list-based (FIFO) processing: ```python if current_queue == 'rq:queue:scheduler': # Use BLPOP for scheduler queue _item = self.lpop([current_queue], timeout, connection) else: # Use ZREVRANGE for priority queues _items = connection.zrevrange(current_queue, 0, 0) ``` *** Job Execution Pipeline [#job-execution-pipeline] Preparation Phase [#preparation-phase] ```python def prepare_job_execution(self, job): timeout = (job.timeout or 180) + 60 with self.connection.pipeline() as pipeline: # Mark worker as busy self.set_state(WorkerStatus.BUSY, pipeline=pipeline) # Track current job self.set_current_job_id(job.id, pipeline=pipeline) # Extend worker TTL self.heartbeat(timeout, pipeline=pipeline) # Add to started registry registry = StartedJobRegistry(job.origin, self.connection) registry.add(job, timeout, pipeline=pipeline) # Update job status job.set_status(JobStatus.STARTED, pipeline=pipeline) pipeline.execute() ``` Execution Phase (Work Horse) [#execution-phase-work-horse] ```python def perform_job(self, job, queue): self.prepare_job_execution(job) push_connection(self.connection) try: with self.death_penalty_class(job.timeout or 180): rv = job.perform() # Execute the actual task job.ended_at = utcnow() job._result = rv self.handle_job_success(job, queue, started_job_registry) except (ClientError, ConnectTimeout, ConnectionError, ReadTimeout): # Retryable exceptions self.handle_job_failure(job, queue, retry_flag=True) except Exception: # Non-retryable - move to failed queue self.handle_job_failure(job, queue, retry_flag=False) finally: pop_connection() ``` *** Retry System [#retry-system] Retry Implementation [#retry-implementation] ```python def handle_job_failure(self, job, queue, retry_flag=False): if job.retryCount < 10 and retry_flag: # Schedule retry job.retryCount += 1 job.save() # Call Fusion to schedule retry in 2 minutes requests.post( settings.FUSION_SERVER + '/retryTask/', json=job._args ) # Log retry event Log.execution_retry( job_id=job.id, message=f'Job failed at retry attempt {job.retryCount}' ) else: # Max retries exceeded or non-retryable if job.failed_task_handler: self.failed_task_callback(job) # Move to failed queue self.move_to_failed_queue(job, *sys.exc_info()) ``` *** Signal Handling [#signal-handling] Parent Process (Worker) [#parent-process-worker] ```python def _install_signal_handlers(self): signal.signal(signal.SIGINT, self.request_stop) # Ctrl+C signal.signal(signal.SIGTERM, self.request_stop) # kill ``` Child Process (Work Horse) [#child-process-work-horse] ```python def setup_work_horse_signals(self): # Ignore Ctrl+C - parent handles it signal.signal(signal.SIGINT, signal.SIG_IGN) # Default SIGTERM handling signal.signal(signal.SIGTERM, signal.SIG_DFL) ``` Warm Shutdown [#warm-shutdown] *** Heartbeat System [#heartbeat-system] Workers send periodic heartbeats to prevent being marked as dead: ```python def heartbeat(self, timeout=0): timeout = max(timeout, self.default_worker_ttl) # 420 seconds self.connection.expire(self.key, timeout) ``` Worker Registration [#worker-registration] ```python # Worker key structure rq:worker:. # Hash fields { 'birth': '2024-01-15T10:30:00Z', 'queues': 'sms,email,pn,webhook,whatsapp', 'state': 'idle', 'current_job': 'job-uuid-if-busy' } ``` *** Monitoring & Observability [#monitoring--observability] Key Redis Keys to Monitor [#key-redis-keys-to-monitor] | Key | Description | | ------------------ | ----------------------------- | | `rq:workers` | Set of all registered workers | | `rq:worker:` | Individual worker status | | `rq:queue:` | Queue length (ZCARD) | | `rq:queue:failed` | Failed jobs count | Elasticsearch Logs [#elasticsearch-logs] All job events are logged: ```json { "index": "active-fusion-logs", "events": [ "queued", "processing", "completed", "failed", "retry" ] } ``` CLI Commands [#cli-commands] ```bash # View queue status python run_worker.py info # View workers python run_worker.py info -W # Suspend all workers python run_worker.py suspend # Resume workers python run_worker.py resume # Requeue failed jobs python run_worker.py requeue --all ``` *** Scaling Considerations [#scaling-considerations] Horizontal Scaling [#horizontal-scaling] Run multiple worker processes on the same or different machines: ```bash # Machine 1 python run_worker.py worker -c instance --customq all # Machine 2 python run_worker.py worker -c instance --customq all # Machine 3 (specialized for webhook queue only) python run_worker.py worker -c instance --customq webhook ``` Queue Isolation [#queue-isolation] For high-volume queues, run dedicated workers: ```bash # SMS-only workers (high priority) python run_worker.py worker -c instance --customq sms # Email workers python run_worker.py worker -c instance --customq email # Webhook workers python run_worker.py worker -c instance --customq webhook ``` Resource Tuning [#resource-tuning] | Setting | Default | Description | | -------------- | ------- | ---------------------------------- | | `worker_ttl` | 420s | Worker heartbeat TTL | | `result_ttl` | 500s | How long to keep job results | | `job.timeout` | 180s | Max job execution time | | Sleep interval | 3s | Time to wait when queues are empty | # Fusion Worker Fusion Worker [#fusion-worker] Fusion Worker is the **background task processing service** that consumes jobs from Redis queues and executes the actual work - sending SMS, emails, push notifications, making webhook requests, and sending WhatsApp messages. Repository Information [#repository-information] | Property | Value | | --------------- | ----------------------------------------------------------------------- | | **Repository** | [fusion\_worker](https://bitbucket.org/creliohealth-repo/fusion_worker) | | **Language** | Python 3.6+ | | **Main Branch** | `main` | | **Project** | CrelioHealth-Fusion (CFN) | | **Created** | April 2021 | Clone Repository [#clone-repository] ```bash git clone git@bitbucket.org:creliohealth-repo/fusion_worker.git ``` *** Tech Stack [#tech-stack] | Category | Technology | Purpose | | ---------------------- | ----------------- | ---------------------------------- | | **Runtime** | Python 3.6+ | Application runtime | | **Queue Backend** | Redis 2.10+ | Job queue storage | | **Queue Library** | RQ (Custom Fork) | Task queue processing | | **AWS SDK** | Boto3 | SES, SNS, S3 integration | | **HTTP Client** | Requests | Webhook calls | | **Logging** | Elasticsearch 5.x | Centralized logging | | **Error Tracking** | Raven (Sentry) | Exception monitoring | | **SMS** | 50+ Provider APIs | SMS delivery (MSG91, Twilio, etc.) | | **Email** | AWS SES | Email delivery | | **Push Notifications** | FCM / APNs | Mobile push notifications | *** System Role [#system-role] *** Directory Structure [#directory-structure] ``` fusion_worker/ ├── run_worker.py # Worker entry point ├── config.py # Configuration loader ├── rq/ # Custom RQ implementation │ ├── worker.py # Enhanced worker with retry logic │ ├── job.py # Job class │ ├── queue.py # Queue class │ ├── connections.py # Redis connection management │ └── cli/ # CLI commands │ └── cli.py # Worker CLI ├── tasks/ # Task implementations │ ├── sms.py # SMS sending logic │ ├── _email_v2.py # Email via AWS SES │ ├── push_notification.py # Push notifications │ ├── webhooks.py # HTTP webhooks │ ├── whatsapp.py # WhatsApp messaging │ ├── plugin_sms.py # SMS provider plugins │ ├── plugin_whatsapp.py # WhatsApp plugins │ ├── sms_gateways.py # SMS gateway definitions │ └── maintenance.py # Queue maintenance tasks ├── wrappers/ # Utility wrappers │ ├── Logger.py # Elasticsearch logging │ └── _utils.py # Helper functions ├── certificates/ # SSL certificates └── instance/ # Environment configs ├── production.py # Production settings └── local.py # Local development settings ``` *** Running the Worker [#running-the-worker] Local Development [#local-development] ```bash # Create virtual environment python3.6 -m venv venv source venv/bin/activate # Install dependencies pip install -r requirement.txt # Set environment variables export CRELIO_DEPLOYMENT_ZONE="IN" export CRELIO_DEPLOYMENT_MODE="LOCAL" # Run worker for all queues python run_worker.py worker -c instance --customq all ``` Using Procfile [#using-procfile] ```bash worker: CRELIO_DEPLOYMENT_ZONE="IN" CRELIO_DEPLOYMENT_MODE="LOCAL" python run_worker.py worker -c instance --customq all ``` CLI Options [#cli-options] ```bash python run_worker.py worker [OPTIONS] Options: -c, --config TEXT Module containing RQ settings (e.g., 'instance') --customq TEXT Specific queue to work on, or 'all' for all queues -b, --burst Run in burst mode (quit when queues are empty) -n, --name TEXT Worker name --logging_level TEXT Logging level (default: INFO) -v, --verbose Show more output -q, --quiet Show less output --results-ttl INTEGER Result TTL in seconds --worker-ttl INTEGER Worker TTL in seconds ``` *** Worker Execution Flow [#worker-execution-flow] *** Key Differences from Standard RQ [#key-differences-from-standard-rq] 1. Priority-Based Dequeuing [#1-priority-based-dequeuing] Standard RQ uses LPOP (FIFO), but Fusion Worker uses: ```python # Get highest priority job _items = connection.zrevrange(current_queue, 0, 0) # Atomic removal if connection.zrem(current_queue, _item) == 1: # Successfully claimed the job result = _item ``` 2. Scheduler Queue Handling [#2-scheduler-queue-handling] The scheduler queue uses a different mechanism (list-based): ```python if current_queue == 'rq:queue:scheduler': _item = self.lpop([current_queue], timeout, connection) else: _items = connection.zrevrange(current_queue, 0, 0) ``` 3. Enhanced Retry Logic [#3-enhanced-retry-logic] ```python def handle_job_failure(self, job, queue, started_job_registry=None, retry_flag=False): if job.retryCount < 10 and retry_flag: # Schedule retry job.retryCount += 1 job.save() requests.post(settings.FUSION_SERVER + '/retryTask/', json=job._args) else: # Call failure callback and move to failed queue if job.failed_task_handler: self.failed_task_callback(job) self.custom_failed_handle(job, started_job_registry) ``` 4. Callback System [#4-callback-system] ```python # On success def success_task_callback(self, job): job_meta = { 'description': job.description, 'job_id': job.id, 'status': job._status, 'retry_attempts': job.retryCount } requests.post(job.success_task_handler, data=job_meta) # On failure def failed_task_callback(self, job): job_meta = { 'description': job.description, 'failure_trace': job.exc_info, 'job_id': job.id, 'status': job._status } requests.post(job.failed_task_handler, data=job_meta) ``` *** Queues Configuration [#queues-configuration] By default, workers process all queues except `dictionary` and `report`: ```python queues = list(set(queues) - set(['dictionary', 'report'])) ``` Available Queues [#available-queues] | Queue | Description | | -------------- | ---------------------------------------- | | `sms` | SMS messages | | `email` | Email messages | | `pn` | Push notifications | | `webhook` | HTTP webhook requests | | `whatsapp` | WhatsApp messages | | `scheduler` | Scheduled jobs (special handling) | | `schedulerSet` | Scheduler sorted set | | `dictionary` | Dictionary mapping (excluded by default) | | `report` | Report processing (excluded by default) | *** Error Handling [#error-handling] Retryable Exceptions [#retryable-exceptions] These exceptions trigger automatic retry: * `ClientError` (AWS) * `ConnectTimeout` * `ConnectionError` * `ReadTimeout` Non-Retryable Exceptions [#non-retryable-exceptions] All other exceptions move the job directly to the failed queue. Whitelist Checking [#whitelist-checking] For webhook tasks, URLs are validated against a whitelist: ```python def isWhitelisted(job): if job._args[0]['payload']['task'] != 4: return 1 # Non-webhook tasks always pass test = re.match(settings.regex2, job._args[0]['payload']['url']) if 'staging' in job._args[0]['payload']['url']: job.retryCount += 10 # Allow more retries for staging return 1 # ... additional validation ``` *** Maintenance Tasks [#maintenance-tasks] The worker runs periodic maintenance every 24 hours: ```python @property def should_run_maintenance_tasks(self): if self.last_cleaned_at is None: return True if (utcnow() - self.last_cleaned_at) > timedelta(hours=24): return True return False ``` What Maintenance Does [#what-maintenance-does] 1. **Clean registries** - Remove stale jobs from started/finished registries 2. **Remove failed jobs** - Clean up old entries from failed queue ```python def clean_registries(self): def remove(job_id): if self.connection.delete(f"rq:job:{job_id}"): self.connection.lrem("rq:queue:failed", 0, job_id) for queue in self.queues: clean_registries(queue) self.last_cleaned_at = utcnow() ``` # Task Implementations Task Implementations [#task-implementations] This document details all the task handlers implemented in Fusion Worker, including their payloads, providers, and behavior. *** SMS Tasks (tasks/sms.py) [#sms-tasks-taskssmspy] Overview [#overview] SMS tasks are routed to different providers based on configuration. The system supports 50+ SMS providers across different regions. Functions [#functions] sendSMS(payload) [#sendsmspayload] Main SMS sending function that routes to the appropriate provider. **Payload Structure:** ```json { "payload": { "contact": 9876543210, "message": "Your message here", "countryCode": 91, "senderId": "CRELIO", "config": { "provider": "msg91" }, "provider_id": "provider-uuid", "credit_deducted": 1, "callback_url": "https://..." } } ``` sendOTPSMS(payload) [#sendotpsmspayload] Specialized function for OTP delivery with fast channel support. **Special Handling:** * Uses MSG91's OTP API template for Indian numbers * Supports `isOTP` flag for instant delivery * Supports `context` and `flowAuth` for transactional SMS Provider Routing [#provider-routing] ```python { "sns": send_sms_sns, # AWS SNS (International) "msg91": send_sms_msg91, # India "twilio_sms": send_twilio_sms, # International "msegat": send_sms_msegat, # Saudi Arabia "textlocal": send_sms_textlocal, # UK/India "clicksend": send_sms_clicksend, # Australia # ... 50+ more providers } ``` Country Code Handling [#country-code-handling] Non-Indian numbers automatically switch to AWS SNS: ```python if str(country_code) != "91": if obj.get("config", {}).get("provider") == "msg91": obj.get("config", {}).update({"provider": "sns"}) ``` Provider Whitelist [#provider-whitelist] Providers can be restricted by country code: ```python provider_mapping = json.loads(conn.hget(cache_key, provider_id)) if country_code not in provider_mapping: return "countryCode is not whitelisted for provider" ``` *** Email Tasks (tasks/_email_v2.py) [#email-tasks-tasks_email_v2py] Overview [#overview-1] Emails are sent via **AWS SES** (Simple Email Service) with support for templates, attachments, and tracking. Functions [#functions-1] send_plain_email(payload) [#send_plain_emailpayload] Send HTML emails without attachments. **Payload Structure:** ```json { "payload": { "to": ["recipient@example.com"], "cc": [], "bcc": [], "subject": "Email Subject", "message": "Email body", "sender_name": "CrelioHealth", "config": { "config_fields": [ {"name": "sender_id_name", "value": "Sender Name"}, {"name": "sender_id_email", "value": "sender@domain.com"}, {"name": "reply_to", "value": "reply@domain.com"} ] }, "callback_url": "https://..." } } ``` send_template_attachment_email(payload) [#send_template_attachment_emailpayload] Send emails with attachments and/or templates. **Additional Fields:** ```json { "files": [ "https://s3.amazonaws.com/.../file1.pdf", "https://s3.amazonaws.com/.../file2.xlsx" ], "template": "Template HTML", "is_template": 1 } ``` Email Validation [#email-validation] Emails are validated and cleaned before sending: ```python EMAIL_REGEX = r"^\w+([\.-]?\w+)*@\w+([\.-]?\w+)*(\.\w{2,3})+$" BLACKLIST_EMAILS = [ 'lifecare1@gmail.com', 'na@na.com', # System and test emails ] ``` Attachment Handling [#attachment-handling] 1. Files are downloaded from S3 2. Attached using MIME multipart 3. Deleted after sending (both locally and from S3) SES Configuration [#ses-configuration] ```python email_client = boto3.client('ses', region_name='us-east-1', aws_access_key_id=settings.AWS_ACCESS_KEY_ID, aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY ) # Uses configuration set for tracking ConfigurationSetName='lh_email_tracker' ``` Error States [#error-states] | Error | Description | | ---------------------------------- | -------------------------- | | `Message Rejected by SES` | SES rejected the email | | `Mail From domain is not verified` | Domain not verified in SES | | `Configuration set sending paused` | Config set is paused | | `Account sending paused exception` | SES account is paused | | `Server Error while sending Email` | Generic error | | `Published` | Email sent successfully | *** Push Notification Tasks (tasks/push_notification.py) [#push-notification-tasks-taskspush_notificationpy] Overview [#overview-2] Push notifications are sent to Android via **FCM** (Firebase Cloud Messaging) and iOS via **APNs** (Apple Push Notification service). Function [#function] sendNotification(payload) [#sendnotificationpayload] **Payload Structure:** ```json { "payload": { "android": { "collapse_key": "notification_type", "api_key": "FCM_SERVER_KEY", "registration_ids": ["device_token_1", "device_token_2"], "data": { "message": "Notification message", "category": "Category", "custom_data": {} } }, "ios": { "category": "notification_type", "certificate": "certificate.pem", "tokens": ["device_token_1"], "aps": { "alert": "Notification message", "badge": 1, "sound": "default" } } } } ``` *** Webhook Tasks (tasks/webhooks.py) [#webhook-tasks-taskswebhookspy] Overview [#overview-3] Webhook tasks make HTTP requests to external endpoints with support for all HTTP methods. Functions [#functions-2] GET(meta) [#getmeta] ```python def GET(meta): payload = meta['payload'] headers = payload.get('headers', {}) timeout = payload.get('timeout', 15) r = requests.get( payload['url'], params=payload['data'], timeout=timeout, headers=headers ) ``` POST(meta) [#postmeta] Supports both query params and body: ```python if int(payload['isBody']): r = requests.post(url, data=json.dumps(payload['args']), headers=headers) else: r = requests.post(url, data=payload['args'], headers=headers) ``` PUT(meta) and DELETE(meta) [#putmeta-and-deletemeta] Similar structure to GET/POST. Payload Structure [#payload-structure] ```json { "payload": { "url": "https://api.example.com/endpoint", "data": {"param": "value"}, "args": {"body_key": "body_value"}, "isBody": 1, "timeout": 15, "callback": 1, "callback_url": "https://callback.url", "headers": { "Authorization": "Bearer token", "Content-Type": "application/json" }, "body": { "callback_headers": {}, "callback_payload": {} } } } ``` Integration Payload [#integration-payload] For integration tracking, include: ```json { "args": { "integration_payload": { "integration_callback_url": "https://...", "metadata": {} } } } ``` Error Handling [#error-handling] 5xx errors trigger a retry: ```python if r.status_code in range(499, 599): raise ReadTimeout(message) ``` Callback Response [#callback-response] When `callback=1`, the response is sent to `callback_url`: ```python requests.post( url=payload['callback_url'], data={ 'resp': r.text, 'status_code': r.status_code } ) ``` *** WhatsApp Tasks (tasks/whatsapp.py) [#whatsapp-tasks-taskswhatsapppy] Overview [#overview-4] WhatsApp messages are sent through various WhatsApp Business API providers. Functions [#functions-3] sendWhatsapp(payload) [#sendwhatsapppayload] Send text-only WhatsApp messages. sendWhatsapp_withFile(payload) [#sendwhatsapp_withfilepayload] Send WhatsApp messages with media attachments. **Payload Structure:** ```json { "payload": { "contact": 9876543210, "countryCode": 91, "message": "Hello!", "template_name": "template_id", "template_params": ["param1", "param2"], "media_url": "https://...", "config": { "provider": "provider_name" } } } ``` *** Maintenance Tasks (tasks/maintenance.py) [#maintenance-tasks-tasksmaintenancepy] Function [#function-1] run_maintenance_on_failed_queue() [#run_maintenance_on_failed_queue] Scheduled to run every 5 minutes, this task cleans up the failed queue. ```python # Scheduled via cron add(scheduler, {}, '*/5 * * * *', 'tasks.maintenance.run_maintenance_on_failed_queue', id='redis_maintenance_on_failed_queue') ``` *** Adding a New SMS Provider [#adding-a-new-sms-provider] To add a new SMS provider: 1. **Create the provider function** in `tasks/plugin_sms.py`: ```python def send_sms_newprovider(obj): contact = obj.get('contact') message = obj.get('message') country_code = obj.get('countryCode') # API call to provider response = requests.post( 'https://api.newprovider.com/send', json={ 'to': f'+{country_code}{contact}', 'message': message, 'api_key': settings.NEW_PROVIDER_API_KEY } ) return response.json() ``` 2. **Add to provider mapping** in `tasks/sms.py`: ```python return { # ... existing providers "newprovider": send_sms_newprovider, }[sms_provider["provider"]](obj) ``` 3. **Import the function**: ```python from tasks.plugin_sms import ( # ... existing imports send_sms_newprovider, ) ``` *** Logging [#logging] All tasks log to Elasticsearch via the `Log` class: ```python from wrappers.Logger import Log Log = Log() # Log events Log.execution_start(job_id, payload, task_type, task, log_tag) Log.execution_success(job_id, payload, task_type, task, result, log_tag) Log.execution_failed(job_id, payload, task_type, task, result, message, log_tag) Log.execution_retry(job_id, payload, task_type, task, result, message, log_tag) Log.execution_warnings(job_id, task_type, task, payload, message, log_tag) ``` Log Structure [#log-structure] ```json { "job_id": "uuid", "task": "SMS", "task_type": "OTP", "status": "completed", "host": "worker-hostname", "log_time": "2024-01-15T10:30:00", "message": "Execution Successful", "job_result": "Response from provider", "tags": "SMS_OTP" } ``` # LiveHealth Frontend LiveHealth Frontend [#livehealth-frontend] Repository Information [#repository-information] * **Repository**: [livehealth-frontend](https://bitbucket.org/creliohealth-repo/livehealth-frontend) * **Language**: TypeScript * **Main Branch**: `main` * **Project**: Reusable UI Components (RUC) * **Description**: This repository contains all the frontend code Overview [#overview] Frontend application for the LiveHealth platform. Clone Repository [#clone-repository] ```bash git clone git@bitbucket.org:creliohealth-repo/livehealth-frontend.git ``` # LiveHealthApp LiveHealthApp [#livehealthapp] Repository Information [#repository-information] * **Repository**: [livehealthapp](https://bitbucket.org/creliohealth-repo/livehealthapp) * **Language**: Python * **Main Branch**: `main` * **Project**: CrelioHealth-For-Doctors (CREL) * **Created**: April 2021 Overview [#overview] Python-based application service. Clone Repository [#clone-repository] ```bash git clone git@bitbucket.org:creliohealth-repo/livehealthapp.git ``` # Architecture Phoenix Search Architecture [#phoenix-search-architecture] Phoenix Search has two tightly coupled planes: | Plane | Primary Job | Main Runtime | | ---------- | ------------------------------------------------------------- | ----------------------------------------------------------------- | | Read plane | Serve authenticated user search and user-detail lookup | FastAPI API service | | Sync plane | Keep `user_details` in Elasticsearch fresh from MySQL changes | Debezium, Redpanda, CDC consumer, `user_meta`, ES ingest pipeline | The API reads from Elasticsearch for search, but MySQL remains the source of truth for patient details and search scope resolution. CDC is the write-side projection system that keeps Elasticsearch usable for low-latency search. *** High-Level Architecture [#high-level-architecture] System Boundaries [#system-boundaries] | Boundary | Owned By Phoenix Search | External Dependency | | ------------------- | -------------------------------------------------------------------------------- | ---------------------------------------------- | | HTTP API | FastAPI app, auth middleware integration, user search routes, health and metrics | ALB / caller applications | | Search index | `user_details` index shape, query builder, routing strategy | Elasticsearch cluster | | Source data | Read access to `userDetails` for details and resolver lookups | LiveHealth MySQL schema and application writes | | CDC materialization | Consumer handlers, `user_meta` projection contract, ingest pipeline | Debezium Connect and Redpanda infrastructure | | Sessions | Session model validation and scope extraction | Django session data in Redis | | Operations | `cdc-ctl`, backfill, dashboards, runbooks | ECS, EC2, MySQL, Redpanda, ES, HyperDX | *** Runtime Topology [#runtime-topology] | Runtime | Process | Key Entrypoint | Main Dependencies | | ---------------- | ------------------------------------ | --------------------------------------------- | --------------------------------------- | | API | FastAPI / Uvicorn or Gunicorn worker | `search.web.application:get_app` | Redis, MySQL, Elasticsearch | | CDC consumer | Python async Kafka consumer | `cdc/consumers/consumer.py` | Redpanda, MySQL, Elasticsearch | | Debezium Connect | Kafka Connect worker | Connector JSON in `tools/cdc-ctl/connectors/` | MySQL binlog, Redpanda | | Backfill | Go binary | `backfill/main.go` | MySQL, Elasticsearch | | cdc-ctl | Go operator CLI | `tools/cdc-ctl/main.go` | Kafka Connect REST, Redpanda, MySQL, ES | The API and CDC consumer are separate runtimes. They share dependencies and the same target index, but they do not call each other directly. The API only observes CDC through Elasticsearch freshness metrics and health checks. *** High-Level Data Flow [#high-level-data-flow] Read Path [#read-path] Write / Sync Path [#write--sync-path] *** Low-Level API Architecture [#low-level-api-architecture] Application Assembly [#application-assembly] The FastAPI app is assembled in `search/web/application.py`. | Step | Code | Responsibility | | ---- | ----------------------------------------------- | ------------------------------------------------------------------------- | | 1 | `configure_logging()` | Configure process logging | | 2 | `sentry_sdk.init(...)` | Enable Sentry when `SEARCH_SENTRY_DSN` exists | | 3 | `FastAPI(..., lifespan=lifespan_setup)` | Create the application with startup/shutdown lifecycle | | 4 | `app.include_router(monitoring_router)` | Register `/`, `/health`, `/health/live`, `/health/ready`, `/metrics` | | 5 | `app.include_router(api_router, prefix="/api")` | Register `/api/v1/users/search` routes | | 6 | `register_exception_handlers(app)` | Normalize application errors | | 7 | `SessionAuthMiddleware` | Authenticate non-guest routes | | 8 | `CORSMiddleware` | Apply CORS policy | | 9 | `StripPrefixMiddleware` | Remove `/phoenix-search` before route matching | | 10 | `trace_id_header` middleware | Refresh rate-limit cache, stamp load-test attributes, return `X-Trace-ID` | Starlette middleware runs in reverse add order, so `StripPrefixMiddleware` is the first custom middleware to see production requests with the ALB prefix. Startup and Shutdown [#startup-and-shutdown] `search/web/lifespan.py` owns process lifecycle. Route Registration [#route-registration] | File | Route Layer | Registered Paths | | -------------------------------------- | ------------------------- | ----------------------------------------------------------- | | `search/web/api/router.py` | Versioned API router | `/api/v1/*` | | `search/domains/user_search/router.py` | User search domain router | `/api/v1/users/search` | | `search/web/api/monitoring/views.py` | Monitoring router | `/`, `/health`, `/health/live`, `/health/ready`, `/metrics` | Development and test environments also register debug routes under `/api/debug`. *** Search Request Internals [#search-request-internals] Search Context [#search-context] `SearchContext` is the immutable bundle that moves through the search pipeline: | Field | Source | Why It Matters | | -------------------- | ---------------------- | ----------------------------------------- | | `query` | Sanitized request body | Drives query shape classification | | `size` | `SEARCH_DEFAULT_SIZE` | Caps hit count | | `search_key` | Request body | Selects a `SEARCH_MAPPING` field set | | `filters` | Session scope | Enforces lab/org/branch/referral access | | `routing` | Resolved lab IDs | Targets Elasticsearch shards | | `search_fields` | Optional request body | Narrows allowed searchable fields | | `date_format_locale` | Session | Parses DOB queries in lab-specific format | Scope Resolution [#scope-resolution] | Login / Search Shape | Scope Behavior | Code | | --------------------------- | ------------------------------------------------------------------- | ------------------------------------------- | | Normal lab user | `term lab_id = resolved lab` | `build_session_filters` | | Doctor login | `referral_ids` when referral session is present | `_extra_filter` | | Branch login | `branch_ids` or `org_ids` depending on `search_type` | `_branch_login_filter` | | Collection center org login | `org_ids` using org plus sub-org expansion | `_resolve_org_ids`, `_extra_filter` | | Multi-center search | `terms lab_id` across related labs, still with login-specific scope | `_resolve_lab_ids`, `build_session_filters` | The caller cannot choose `lab_id` directly. Lab and org scope always comes from the authenticated session and MySQL lookup helpers. Query Builder [#query-builder] `search/domains/user_search/query.py` is intentionally shape-routed: | Shape | Main Clause Families | Avoids | | --------- | ---------------------------------------------------------------- | -------------------- | | `phone` | Contact exact/prefix, identity, buckets, patient ID, numeric IDs | Broad name matching | | `numeric` | Patient ID, identity, bucket IDs, numeric IDs, DOB | Broad name matching | | `alpha` | Patient ID, identity, buckets, full name | Numeric-only clauses | | `mixed` | Structured IDs, buckets, full name, DOB if parseable | Wildcards | Important query rules: * `SEARCH_MAPPING` selects the allowed logical fields for each `search_key`. * `search_fields` intersects with the selected mapping; no overlap returns an empty result. * There are no wildcard queries and no fallback catch-all query. * Search results sort by `_score` and then `last_updated_time desc`. * `matched_field` comes from ES highlights or named queries. * Buckets come from named query tags, then the service splits multi-center hits into `other_labs`. Elasticsearch Repository [#elasticsearch-repository] `search/domains/user_search/repository.py` owns the ES call: | Concern | Behavior | | ----------------- | --------------------------------------------------------------------- | | Circuit breaker | `_es_search` is wrapped by `es_breaker` | | Trace propagation | Active trace ID is passed as `opaque_id` | | Routing | Lab routing is passed to ES when available | | Timeout | Uses `SEARCH_ES_SEARCH_TIMEOUT` | | Metrics | Records query duration, query count, hit count, and zero-result count | | Circuit open | Returns empty hits rather than failing the API response | | Other ES failures | Propagate after recording error metrics | *** User Detail Lookup Internals [#user-detail-lookup-internals] User detail lookup is intentionally MySQL-backed, not ES-backed. This path uses MySQL so detail views are source-of-truth even if Elasticsearch is temporarily stale. *** Low-Level CDC Architecture [#low-level-cdc-architecture] Connector Layer [#connector-layer] | Connector | Captures | Topic Prefix | Partition Routing | Purpose | | --------------------------- | --------------------------------------------- | ------------ | -------------------------- | -------------------------------------------- | | `phoenix-source-existing` | `userDetails`, `billing`, `labReportRelation` | `phoenix` | `id` or `userDetailsId_id` | Source table changes into Redpanda | | `phoenix-source-projection` | `user_meta` | `phoenix` | `user_details_id` | Projection changes into Redpanda for ES sync | Production connector properties that define the architecture: | Property | Why It Matters | | --------------------------------------------------- | ------------------------------------------------------------------------- | | `database.include.list = livehealthapp` | Connector reads the source database binlog | | `table.include.list` | Restricts emitted records to search-relevant tables | | `snapshot.mode = when_needed` | Recovers schema/offset gaps without normal full table snapshots | | `snapshot.select.statement.overrides ... WHERE 1=0` | Captures schema without letting Debezium own historical data load | | `PartitionRouting` SMT | Keeps all events for the same user on the same partition number | | `signal.enabled.channels = source` | Enables controlled Debezium signal-table actions | | `heartbeat.action.query` | Updates `debezium_heartbeat` so operators can detect stalled binlog reads | | `errors.deadletterqueue.topic.name` | Sends connector/SMT failures to the Kafka Connect DLQ | Consumer Layer [#consumer-layer] The consumer is at-least-once. Offsets are stored only after successful handling or after DLQ publication. Replays are expected, so handlers use idempotent writes and ordering guards. Phase Routing [#phase-routing] | Phase | Detection | Source Topic Behavior | Projection Topic Behavior | | ----------- | ---------------------------- | ------------------------------------------------------ | -------------------------------------------------------- | | `migration` | `user_meta.full_name` exists | Materialize full denormalized rows into `user_meta` | Forward projection row to ES through ingest pipeline | | `running` | `user_meta.full_name` absent | Keep slim projection updated and resolve identity live | Compose full ES doc from projection + live `userDetails` | `CDC_PHASE_OVERRIDE` can force a phase for recovery. The override must be `migration` or `running`. Source Table Handler Paths [#source-table-handler-paths] | Topic | Running Handler | MySQL Write | Direct ES Write | | ------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------- | | `userDetails` | `_process_running_user_details` | Upsert `lab_id`, `last_updated_time`, and selected projection metadata | Yes, after `FieldResolver` composes a full document | | `billing` | `_process_running_source_mysql` -> `_handle_billing` | Append `lab_bill_ids`, `order_numbers`, `referral_ids`, `org_ids`, `branch_ids` | No | | `labReportRelation` | `_process_running_source_mysql` -> `_handle_lab_report` | Append `manual_sample_ids` | No | | `user_meta` | `_process_running_projection` | None | Yes, after identity is resolved from live `userDetails` | Field Resolver [#field-resolver] `cdc/consumers/field_resolver.py` composes ES documents during running phase. | Event Source | Identity Fields | Aggregate Fields | Reason | | ------------------- | ------------------------- | -------------------------- | ------------------------------------------------------ | | `userDetails` event | Live `userDetails` lookup | Current `user_meta` lookup | Avoid stale envelope identity on DLQ replay | | `user_meta` event | Live `userDetails` lookup | Event `after` image | Projection event is the aggregate change being indexed | The resolver treats `userDetails.labId_id = -1` as missing. That prevents merged-away patient rows from leaking sentinel data into Elasticsearch. Retry and Ordering Model [#retry-and-ordering-model] | Mechanism | Code Path | Purpose | | -------------------- | ------------------------------------------- | ----------------------------------------------------- | | Partition routing | Debezium `PartitionRouting` SMT | Same user, same Kafka partition across all CDC topics | | Per-partition worker | `consumer.py::_partition_worker` | Sequential handling within a partition | | Handler retry | `consumer.py::_handle_with_retries` | Recover transient MySQL/ES/Kafka issues | | MySQL deadlock retry | `router.py::_retry_on_deadlock` | Retry OperationalError 1213 with jittered backoff | | Consumer DLQ | `consumer.py::_send_to_dlq` | Preserve poison messages after retries | | CAS guard | `handlers.py::_handle_user_details_running` | Reject stale identity updates by `last_updated_time` | | CSV dedupe | `_upsert_csv_field`, `_append_csv_fields` | Make repeated billing/sample events safe | | Tombstone guard | `_is_tombstoned_user`, `FieldResolver` | Prevent merged-away patients from being recreated | *** Data Architecture [#data-architecture] Source Tables and Projection [#source-tables-and-projection] | Store | Object | Role | | ------------- | ------------------- | ---------------------------------------------------------------- | | MySQL | `userDetails` | Authoritative identity, demographics, lab routing, patient state | | MySQL | `billing` | Order, bill, referral, org, and branch aggregate source | | MySQL | `labReportRelation` | Manual sample ID aggregate source | | MySQL | `user_meta` | Search projection table used by CDC and backfill | | Elasticsearch | `user_details` | Search-optimized document index | Field Ownership [#field-ownership] | Field Family | Source of Truth | Projection / Index Behavior | | ----------------------------- | --------------------------------- | ---------------------------------------------------------------- | | Identity and demographics | `userDetails` | Indexed directly by CDC resolver or backfill join | | `lab_id` | `userDetails.labId_id` | Used as ES routing and access filter | | Patient IDs and identity IDs | `userDetails` | Searched through exact, prefix, suffix, or segment fields | | Billing IDs and order numbers | `billing` | Aggregated into CSV in `user_meta`, transformed to arrays for ES | | Org, referral, branch IDs | `billing` | Aggregated into recency-ordered CSV fields | | Manual sample IDs | `labReportRelation` via `billing` | Aggregated into `manual_sample_ids` | | CDC freshness | ES newest `last_updated_time` | Probed by API background task and exposed in `/health` | Elasticsearch Document Shape [#elasticsearch-document-shape] The ES document combines identity plus aggregate arrays: ```json { "id": 101, "lab_id": 1, "full_name": "John Doe", "lab_patient_id": "P-1001", "contact": "9999999999", "manual_sample_ids": ["S-1001"], "order_numbers": ["ORD-1001"], "lab_bill_ids": ["5001"], "org_ids": [20], "referral_ids": [77], "branch_ids": [10], "last_updated_time": "2024-01-02T10:00:00Z" } ``` The API search path depends on ES routing by `lab_id`. If a patient moves labs, CDC deletes the old routed document and indexes the new routed document. Elasticsearch Routing Model [#elasticsearch-routing-model] Phoenix Search uses Elasticsearch custom routing on the `user_details` index. The index mapping declares `_routing.required = true`, so every write, delete, and point lookup must pass the same routing key that was used when the document was indexed. | Concern | Behavior | | --------------------------- | ------------------------------------------------------------ | | ES document ID | `user_details_id`, stored in ES as `id` | | ES routing key | `lab_id` as a string | | Source of routing | `userDetails.labId_id`, denormalized into `user_meta.lab_id` | | Normal search routing | Current session lab ID | | Multi-center search routing | Comma-separated related lab IDs resolved from MySQL | | Filter endpoint routing | Current session lab ID | | Detail endpoint | MySQL-backed, not an ES point lookup | Routing is not the only security boundary. The API also adds `lab_id` / org / branch / referral filters from the authenticated session. Routing targets the relevant ES shard or shards; filters enforce the allowed result scope. Write paths must use the same rule: | Writer | Routing Behavior | | ------------------------ | ------------------------------------------------------------------------------- | | CDC userDetails handler | Resolves full ES doc, indexes with `routing=str(lab_id)` | | CDC user\_meta handler | Resolves projection doc, indexes/deletes with `routing=str(lab_id)` | | CDC lab move / reroute | Deletes old doc with old `lab_id`, then indexes new doc with new `lab_id` | | CDC merge tombstone | Uses the previous `lab_id` from the `before` image to delete the old routed doc | | Backfill bulk indexer | Sets bulk item `Routing` from `user_meta.lab_id` | | Backfill targeted repair | Calls ES index with `WithRouting(lab_id)` | | Backfill verify | Reads `GET /user_details/_doc/?routing=` | Debugging must include routing. A document can exist under one routing key and appear missing under another: ```bash curl -s -u elastic: \ "https://:9200/user_details/_doc/?routing=" ``` *** Consistency Model [#consistency-model] | Concern | Guarantee | | ------------------ | ------------------------------------------------------------------------------------- | | Search freshness | Eventually consistent from MySQL through CDC into ES | | Detail lookup | Stronger source-of-truth read from MySQL | | Per-user ordering | Preserved by partition routing and per-partition workers | | Global ordering | Not guaranteed across different users or partitions | | Duplicate delivery | Expected; handlers are designed to be idempotent | | CDC outage | API can still serve existing ES results, but freshness age increases | | ES outage | Search API readiness degrades; CDC retries then DLQs failed writes | | Redis outage | Auth/session and rate-limit flows are affected | | MySQL outage | Detail lookup, scope resolution, CDC materialization, and resolver reads are affected | The important rule is that Elasticsearch is a projection, not the source of truth. When ES and MySQL disagree, MySQL wins and CDC/backfill should repair ES. *** Failure Domains [#failure-domains] | Failure | Immediate Symptom | First Debug Page | | -------------------------------- | ----------------------------------------------------------------- | -------------------------------------------------------------------------- | | API dependency down | `/health/ready` returns 503 | [Operations](/docs/services/phoenix-search/operate/operations) | | Search stale | `/health` has stale CDC body status | [CDC](/docs/services/phoenix-search/sync-path/cdc) | | Debezium connector failed | Redpanda topics stop receiving source events | [CDC Tools and Backfill](/docs/services/phoenix-search/sync-path/backfill) | | Consumer lag high | `cdc_consumer_lag` rises | [Operations](/docs/services/phoenix-search/operate/operations) | | Projection corrupt or incomplete | ES disagrees with `user_meta` / MySQL | [CDC Tools and Backfill](/docs/services/phoenix-search/sync-path/backfill) | | Query returns unexpected results | Wrong `search_key`, `search_fields`, session scope, or ES routing | [API Reference](/docs/services/phoenix-search/read-path/api-reference) | *** Source References [#source-references] | Area | Files | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | App assembly | `search/web/application.py`, `search/web/api/router.py` | | Startup and dependency lifecycle | `search/web/lifespan.py`, `search/services/*/lifespan.py` | | Auth and session context | `search/services/auth/middleware.py`, `search/services/auth/dependencies.py`, `search/services/auth/schemas.py` | | Search service path | `search/domains/user_search/router.py`, `service.py`, `repository.py`, `query.py`, `filters.py`, `context.py` | | MySQL lookup path | `search/domains/user_search/queries.py` | | CDC consumer orchestration | `cdc/consumers/consumer.py` | | CDC routing and handlers | `cdc/consumers/router.py`, `handlers.py`, `field_resolver.py`, `phase_detector.py` | | Debezium connector configs | `tools/cdc-ctl/connectors/source-connector-existing.production.json`, `source-connector-projection.production.json` | | Backfill architecture | `backfill/main.go`, `scanner.go`, `indexer.go`, `migrate.go`, `row.go`, `transform.go` | # Phoenix Search Phoenix Search [#phoenix-search] Phoenix Search is a **FastAPI search microservice** for patient and user lookup. It serves authenticated search requests from LiveHealth surfaces, queries the `user_details` Elasticsearch index, and uses MySQL plus Redis for source data, session context, rate limiting, and operational checks. CDC is part of Phoenix Search. The CDC pipeline keeps the Elasticsearch projection in sync from MySQL through Debezium, Redpanda, the Phoenix CDC consumer, the `user_meta` projection table, and the Elasticsearch ingest pipeline. Start with [Start Here](/docs/services/phoenix-search/start-here) for the reading order. For the full high-level and low-level system view, see [Architecture](/docs/services/phoenix-search/architecture). For frontend endpoint and ephemeral-token auth, see [Auth and Endpoints](/docs/services/phoenix-search/read-path/auth-and-endpoints). Repository Information [#repository-information] | Property | Value | | ------------------ | ---------------------------------------------------------------- | | **Repository** | [phoenix-search](https://github.com/CrelioHealth/phoenix-search) | | **Language** | Python 3.13+ | | **Main Branch** | `main` | | **Framework** | FastAPI | | **Search Backend** | Elasticsearch | | **Primary Index** | `user_details` | | **CDC Runtime** | Python consumer + Debezium + Redpanda | Production Snapshot [#production-snapshot] Latest saved dashboard screenshots for production IN show Phoenix Search handling the observed traffic window with healthy service and Elasticsearch signals. The full migration evidence, old-vs-new index comparison, and OpenTelemetry notes are in [Post-Migration Results](/docs/services/phoenix-search/operate/post-migration-results). | Signal | Observed Production Value | | ----------------------- | ------------------------------------------------------------------------------------ | | API request error rate | `0%` in the HTTP service dashboard | | Main endpoint | `POST /api/v1/users/search`, about `97%` of endpoint time | | Search endpoint latency | Median about `22.76 ms`, p95 about `47.67 ms` | | ES query latency | p50 about `2.5 ms`, p95 about `4.75 ms`, p99 about `4.95 ms` | | CDC health | `1`, with ES document freshness around `3.6s` | | ES cluster | `GREEN`, `3` active data nodes | | `user_details` index | About `157.44M` docs, `67.61 GB` primary, `135.75 GB` total | | Storage reduction | About `47%` lower than the old `userdetails` index, roughly `50%` in practical terms | The detailed dashboard snapshot and migration results live in [Post-Migration Results](/docs/services/phoenix-search/operate/post-migration-results). Clone Repository [#clone-repository] ```bash git clone git@github.com:CrelioHealth/phoenix-search.git ``` *** Tech Stack [#tech-stack] | Category | Technology | Purpose | | -------------------- | ------------------------------------------ | ---------------------------------------------------- | | **Runtime** | Python 3.13+ | API and CDC consumer runtime | | **Web Framework** | FastAPI | HTTP API server | | **Search** | Elasticsearch | Patient and user search index | | **Cache / Sessions** | Redis | Django session lookup, KV support, rate-limit config | | **Database** | MySQL | Source-of-truth user data and `user_meta` projection | | **CDC Broker** | Redpanda | Kafka-compatible event transport | | **CDC Source** | Debezium MySQL Connector | MySQL binlog capture | | **CLI / Packaging** | uv, Typer | Local commands and project execution | | **Observability** | Prometheus, OpenTelemetry, HyperDX, Sentry | Metrics, traces, logs, and errors | *** System Architecture Overview [#system-architecture-overview] *** Key Components [#key-components] 1. FastAPI Application (search/web/application.py) [#1-fastapi-application-searchwebapplicationpy] The app factory registers monitoring routes at the root and API routes under `/api`. Production traffic can arrive behind the ALB path prefix `/phoenix-search`; `StripPrefixMiddleware` removes that prefix before FastAPI routing. | Route Area | Registered Path | Notes | | --------------- | ----------------------------------------------- | ------------------------------ | | Home and health | `/`, `/health`, `/health/live`, `/health/ready` | Monitoring routes are public | | Metrics | `/metrics` | Prometheus exposition endpoint | | OpenAPI | `/api/openapi.json` | FastAPI OpenAPI document | | Docs | `/docs` | FastAPI Swagger UI | | User Search | `/api/v1/users/search` | Authenticated user search API | 2. User Search Domain (search/domains/user_search/) [#2-user-search-domain-searchdomainsuser_search] The domain owns its router, schemas, service, repository, filters, and Elasticsearch query builders. | File | Role | | --------------- | ---------------------------------------------------------------------- | | `router.py` | FastAPI endpoints and dependencies | | `schemas.py` | Request and response models | | `service.py` | Session-aware search orchestration, hit bucketing, MySQL detail lookup | | `repository.py` | Elasticsearch query execution and circuit-breaker handling | | `constant.py` | `SEARCH_MAPPING` keys and searchable fields | Search requests are scoped by the authenticated session. The service resolves lab, branch, organization, referral, and multi-center constraints before querying Elasticsearch, sends ES routing from the resolved `lab_id` values, then groups hits into `registered`, `samples`, `orders`, and `other_labs`. 3. Authentication and Session Context [#3-authentication-and-session-context] `SessionAuthMiddleware` authenticates every non-guest route. It supports: | Client Type | Auth Mechanism | | ------------------ | ------------------------------------------------------------- | | Web app | `sessionid` cookie resolved from Redis-backed Django sessions | | Mobile app | Bearer JWT when `X-App-Name` is present | | Ephemeral web flow | Bearer JWT without `X-App-Name` | The validated session is stored on request state and in a context variable so route dependencies can resolve `lab_id`, organization scope, and search restrictions. 4. Lifecycle and Observability (search/web/lifespan.py) [#4-lifecycle-and-observability-searchweblifespanpy] Startup initializes OpenTelemetry, Prometheus instrumentation, Elasticsearch, MySQL, Redis, and rate-limit Redis. A background CDC freshness probe periodically queries the newest `last_updated_time` in Elasticsearch and exposes freshness through `/health` plus metrics. 5. CDC Subsystem (cdc/) [#5-cdc-subsystem-cdc] CDC is documented in this section because it is the write-side sync path for Phoenix Search. It is not a separate service in the docs navigation. See [Architecture](/docs/services/phoenix-search/architecture) for the system view, [CDC](/docs/services/phoenix-search/sync-path/cdc) for the pipeline, and [Operations](/docs/services/phoenix-search/operate/operations) for runbooks. *** Local Setup [#local-setup] Prerequisites [#prerequisites] | Tool | Version / Requirement | | ------ | ---------------------------------------------------------------- | | Python | 3.13+ | | uv | Latest | | Docker | Latest | | MySQL | 8.x source database; hosted externally for dev, Docker for tests | Quick Start [#quick-start] ```bash make install cp .env.example .env cp docker/.env.example docker/.env make up-dev make seed make run ``` The API runs on `http://localhost:8000`, and FastAPI docs are available at `http://localhost:8000/docs`. Services and Ports [#services-and-ports] | Service | Dev | Test | Description | | ------------- | -------- | -------- | --------------------------------- | | API | `8000` | `8010` | FastAPI server | | Elasticsearch | `9210` | `9212` | Search backend | | Kibana | `5601` | - | Elasticsearch dashboard | | Redis | internal | internal | Cache, sessions, rate-limit state | | MySQL | `3306` | `3308` | External dev DB, Docker test DB | | HyperDX | `8080` | - | Observability UI | *** Core Flows [#core-flows] Search Request [#search-request] Detail Lookup [#detail-lookup] MySQL to Elasticsearch Sync [#mysql-to-elasticsearch-sync] *** Make Targets [#make-targets] | Command | What it does | | ------------------------ | ------------------------------------------ | | `make install` | Install Python dependencies | | `make run` | Start the API server | | `make up-dev` | Start Redis, Elasticsearch, and Kibana | | `make up-all` | Start all local infra plus HyperDX | | `make down` | Stop all containers | | `make test` | Run unit tests without Docker | | `make test-docker` | Run the full suite in Docker | | `make lint` | Run Ruff and mypy | | `make seed` | Seed 50 users across 2 labs | | `make seed-reset` | Drop, recreate, and seed fresh local data | | `make register-pipeline` | Register the Elasticsearch ingest pipeline | | `make run-cdc` | Start the local CDC stack | | `make connector-status` | Show Debezium connector health | | `make backfill-migrate` | Populate the MySQL projection table | | `make backfill-run` | Push projected data into Elasticsearch | *** Source References [#source-references] | Concern | Source | | --------------------------------------- | ------------------------------------------ | | App factory and ALB prefix handling | `search/web/application.py` | | Route registration | `search/web/api/router.py` | | User search endpoints | `search/domains/user_search/router.py` | | User search request and response models | `search/domains/user_search/schemas.py` | | Search orchestration | `search/domains/user_search/service.py` | | Elasticsearch query execution | `search/domains/user_search/repository.py` | | Query shape routing | `search/domains/user_search/query.py` | | CDC runbook | `cdc/RUNBOOK.md` | | CDC settings | `cdc/consumers/config.py` | # Start Here Start Here [#start-here] Use this page first. Phoenix Search has a read side and a sync side, and mixing them makes the docs hard to follow. | If You Need To Understand | Read | | ---------------------------------------------- | -------------------------------------------------------------------------------------- | | What Phoenix Search is and where it fits | [Phoenix Search Overview](/docs/services/phoenix-search) | | How the real system is wired | [Architecture](/docs/services/phoenix-search/architecture) | | How frontend authentication and endpoints work | [Auth and Endpoints](/docs/services/phoenix-search/read-path/auth-and-endpoints) | | How search requests and query shapes work | [API Reference](/docs/services/phoenix-search/read-path/api-reference) | | How MySQL changes reach Elasticsearch | [CDC](/docs/services/phoenix-search/sync-path/cdc) | | How data is migrated, repaired, or reindexed | [CDC Tools and Backfill](/docs/services/phoenix-search/sync-path/backfill) | | How to run, monitor, or debug production | [Operations](/docs/services/phoenix-search/operate/operations) | | How to debug one API request end to end | [API Debugging](/docs/services/phoenix-search/operate/debugging) | | What improved after migration | [Post-Migration Results](/docs/services/phoenix-search/operate/post-migration-results) | *** Mental Model [#mental-model] The frontend does not call a Phoenix Search login endpoint. It calls crelio-app to mint a short-lived Phoenix Search token, then uses that token against the Phoenix Search API. The API does not own source-of-truth patient data. It searches Elasticsearch and uses MySQL for detail lookup and session-scope resolution. Elasticsearch documents are routed by `lab_id`. Search requests derive routing from the authenticated session, CDC writes use the document's current `lab_id`, and backfill writes use `user_meta.lab_id`. CDC is not a separate service in the docs. It is the write-side sync subsystem that keeps Phoenix Search's Elasticsearch index current. *** Phase 1 Why / Context [#phase-1-why--context] This is the Phase 1 reasoning the docs assume. The goal is not only to move code into a new service; it is to make preview user detail search simpler to operate, easier to debug, and cheaper to serve at production scale. | Question | Answer | | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Why was Phoenix Search introduced in Phase 1? | Preview user detail search had too much fan-out. One user search could require roughly `6-7` lookup paths across patient identity, LRF / lab-report identifiers, manual sample ID, bill ID, and order identifiers. Phoenix Search turns that into one routed Elasticsearch search over `user_details`. | | Which legacy search paths are replaced first? | The Phase 1 target is preview user detail search: patient identity lookup plus identifier lookups that previously needed separate backend/search calls and merge behavior. | | Why use one `user_details` Elasticsearch document? | The search input can match many identifiers, but the user-facing result is still a user/patient entity. Denormalizing the searchable identifiers, org/referral/branch filters, and display fields into one projection lets the API return bucketed results without calling multiple lookup APIs. | | Why use analyzer-backed mappings? | Exact keyword matching alone is not enough for patient and identifier search. The mapping supports exact, prefix, suffix, segment, and `search_as_you_type` fields so Phoenix Search can handle partial names, phone-like input, lab patient IDs, manual sample IDs, bills, orders, and national IDs in one query shape. | | Why is CDC required? | Elasticsearch is a read projection, not the source of truth. MySQL remains the source. CDC keeps `user_meta` and `user_details` current after the initial backfill without making the frontend wait on source-table joins during every search. | | Why Debezium, Redpanda, and the Phoenix CDC consumer? | Debezium reads MySQL binlog changes, Redpanda gives a replayable Kafka-compatible transport, and the Python consumer owns Phoenix-specific materialization rules. That split lets operators inspect connector state, topic lag, DLQ records, consumer health, and Elasticsearch sync separately. | | Why is backfill still needed if CDC exists? | CDC keeps changes current from a point in time. Backfill migrates the existing historical dataset into `user_meta` and Elasticsearch so the index starts complete, then CDC handles ongoing inserts, updates, deletes, and lab reroutes. | | Why is Elasticsearch routing by `lab_id` required? | Search scope is lab/session driven. Routing documents by `lab_id` keeps reads and writes scoped to the same shard route, makes point lookups deterministic, and avoids cross-lab scatter for the hot search path. Any direct ES document check must include `routing=`. | | Why use ephemeral tokens from crelio-app? | LiveHealth already has the user session. crelio-app mints a short-lived Phoenix Search token containing session and lab context, and Phoenix Search validates that token plus Redis session state. The frontend does not need a separate Phoenix login flow. | | Why is the Phase 1 endpoint `https://phoenix-search-in.crelio.solutions/api`? | The current frontend fallback and crelio-app token response point to the IN Phoenix Search base URL. The docs therefore describe the real Phase 1 endpoint instead of inventing region segregation that is not present in the current integration. | | Why OpenTelemetry now? | The new service boundary makes tracing valuable. A single request can expose the browser `X-Trace-Id`, API route, search attributes, Elasticsearch query span, MySQL dependency timing, errors, CDC freshness, and logs around the same trace. | | What proves Phase 1 worked? | The post-migration evidence shows one ES search request replacing the previous multi-lookup pattern, roughly `50%` lower index storage, `0%` observed HTTP request error rate, ES p50 around `2.5 ms`, ES p95 around `4.75 ms`, and trace-level debugging from frontend request to ES query. | Known Phase 1 implementation facts from code: | Fact | Evidence | | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | Frontend gets an ephemeral Phoenix token from crelio-app | `livehealth-frontend/src/services/phoenixSearch/axios/tokenManager.ts` | | crelio-app signs the token using `PHOENIX_SEARCH_JWT_SECRET` | `crelio-app/core/views/phoenix_search_token.py` | | Phoenix Search validates it using `SEARCH_EPHEMERAL_JWT_SECRET` | `search/search/services/auth/ephemeral.py` | | Frontend default Phoenix Search URL is `https://phoenix-search-in.crelio.solutions/api` | `livehealth-frontend/src/services/phoenixSearch/axios/config.ts` | | Search calls use `/v1/users/search` under that base URL | `livehealth-frontend/src/services/phoenixSearch/search.ts` | *** Folder Structure [#folder-structure] ```text phoenix-search/ ├── index # service overview ├── start-here # reading path + Phase 1 context ├── architecture # high-level and low-level system design ├── read-path/ │ ├── auth-and-endpoints │ └── api-reference ├── sync-path/ │ ├── cdc │ └── backfill └── operate/ ├── operations ├── debugging └── post-migration-results ``` Keep new docs in the same split: | Content Type | Folder | | ---------------------------------------------------------- | ------------------ | | Frontend integration, auth, endpoints, request shapes | `read-path/` | | Debezium, Redpanda, consumer behavior, migration, backfill | `sync-path/` | | On-call, metrics, dashboards, deploy, debug | `operate/` | | System-wide design and boundaries | `architecture.mdx` | # Add Test to Bill Add Test to Bill [#add-test-to-bill] This operation appends tests from an inbound HL7 order to an existing active bill. It is handled synchronously inside `hl7/2_3/createOrder/`. When This Flow Runs [#when-this-flow-runs] All conditions must be true: * `orderNumber` is present. * An active bill exists for the same lab and order number, or for the numeric `labBillId` equivalent of the order number. * `addTestToBill = 1`. * `uniqueOrderNo = 0`. * `testList` contains at least one test. If `testList` is empty, the router returns `400 Invalid Test codes`. Implementation Path [#implementation-path] | Step | Function | Description | | -------------------- | -------------------------------------- | ------------------------------------------------------------- | | Existing bill lookup | `hl7/2_3/createOrder/` | Finds active matching bill. | | Test resolution | `hl7/2_3/createOrder/` | Resolves inbound test codes against `allTests`. | | Add-test execution | `bill_add_test_third_party` | Calls the standard add-test controller. | | Billing update | `commonBillAddTestController` | Adds tests, billing rows, reports, samples, and related data. | | AOE persistence | `create_question_values_from_aoe_data` | Saves inbound AOE values after the add-test operation. | Configuration [#configuration] | Key | Required value | Description | | -------------------- | -------------- | --------------------------------------------------------------------- | | `addTestToBill` | `1` | Enables this operation. | | `uniqueOrderNo` | `0` | Allows an existing order number to be used for add-on tests. | | `add_duplicate_test` | Optional `1` | Adds inbound tests even when the bill already contains the same test. | | `orgPriceList` | Optional `1` | Uses the existing bill organization's price list for added tests. | | `is_aoe_store` | Optional `1` | Stores inbound OBX AOE values after tests are added. | Test Selection [#test-selection] The router collects `testCode` values from the inbound `testList` and fetches active `allTests` records for the lab. For each resolved test: * The test is skipped when it already exists on the bill, unless `add_duplicate_test = 1`. * The test amount comes from the organization price list when `orgPriceList = 1` and a list price is available. * Otherwise, the master test amount is used. * `report_level_tags` from the inbound test item is preserved. Internal Add-Test Request [#internal-add-test-request] The router creates an internal request object for `bill_add_test_third_party`. | Field | Value | | ------------------- | ----------------------------------------------------------------------------- | | `billAddType` | `2`, indicating add tests to an existing bill. | | `labBillId` | Existing bill's `labBillId`. | | `testList` | Resolved tests that should be added to the bill. | | `billTotalAmount` | Sum of the resolved amounts for tests being added. | | `paymentMode` | Payment mode from the existing bill's organization. | | `billAdvance` | `0`, because this integration add-test path does not collect advance payment. | | `smsFlag` | `0`, patient SMS disabled for this internal call. | | `orgSMSFlag` | `0`, organization SMS disabled for this internal call. | | `sampleAccession` | Lab feature flag used by the standard add-test controller. | | `batchManagement` | Lab feature flag used by the standard add-test controller. | | `cashBoxManagement` | Lab feature flag used by the standard add-test controller. | | `loginUser` | Lab user ID from the inbound token. | Each resolved test item includes Crelio test fields such as `testId`, `testName`, `isProfile`, `testAmount`, `sampleId`, `testQuantity`, outsource fields, and `report_level_tags`. Side Effects [#side-effects] After the add-test controller completes, the router: * Saves AOE values using `create_question_values_from_aoe_data`. * Writes an activity log in category `194`. * Returns a successful response even if no new tests were eligible because all inbound tests already existed on the bill. Responses [#responses] When at least one test is added: ```json { "status": 200, "Message": "[Add Tests to Bill] Tests have been added to Bill(ID: ) for Patient (ID: ) through HL7 integration" } ``` When the inbound tests are duplicates and no new test is added: ```json { "status": 200, "Message": "No new tests were added to the existing Bill(ID: ) for Patient (ID: )" } ``` Common Issues [#common-issues] | Issue | Check | | ------------------------------------- | -------------------------------------------------------------------------------------------------- | | Add-test flow is not triggered | Confirm `addTestToBill = 1` and `uniqueOrderNo = 0`. | | Endpoint returns `Invalid Test codes` | Confirm OBR-4 is parsed into `testList`. | | Tests are skipped | Confirm whether the bill already has the tests and whether `add_duplicate_test` should be enabled. | | Test amount is unexpected | Check `orgPriceList` and organization list priority. | # Bill Cancel Bill Cancel [#bill-cancel] This operation cancels an existing active bill when the inbound HL7 order sends a cancellation status. It is handled synchronously inside `hl7/2_3/createOrder/`. When This Flow Runs [#when-this-flow-runs] All conditions must be true: * `orderNumber` is present. * An active bill exists for the same lab and order number, or for the numeric `labBillId` equivalent of the order number. * `uniqueOrderNo = 1`. * `cancelBill = 1`. * `orm_status` is `CA`, case-insensitively. `orm_status` is parsed from `ORC-1`. Implementation Path [#implementation-path] | Step | Function | Description | | ----------------------------- | -------------------------- | ---------------------------------------------------------------------------- | | Existing bill lookup | `hl7/2_3/createOrder/` | Finds active matching bill. | | Cancellation payload creation | `hl7/2_3/createOrder/` | Builds the cancellation payload from the existing bill and inbound comments. | | Bill cancellation | `cancelBillCommonFunction` | Applies standard Crelio bill cancellation behavior. | Internal Cancellation Payload [#internal-cancellation-payload] The router creates an internal cancellation payload for `cancelBillCommonFunction`. | Field | Value | | --------------- | ------------------------------------------------------------- | | `labBillId` | Existing bill's `labBillId`. | | `labId` | Lab ID resolved from `authKey`. | | `labUserId` | Existing bill user ID, falling back to token user ID. | | `deductionFlag` | `0`. This integration path does not request payment deletion. | | `billComment` | Inbound bill comments, usually from NTE. | | `labUserName` | Lab user name from the inbound token. | | `fromApi` | `0`. | | `orgSMSFlag` | `0`. | Cancellation Effects [#cancellation-effects] `cancelBillCommonFunction` applies the normal product cancellation flow: * Marks the bill as cancelled with `isCancel = 1`. * Clears refund and write-off flags on the bill. * Updates `lastEditedBy` and `lastUpdatedTime`. * Appends cancellation details to the bill comments. * Updates organization due and ledger entries where applicable. * Marks related `billingInfo` rows as dismissed. * Marks related `labReportRelation` rows as dismissed. * Clears rack/location fields on related collected samples. * Updates report/sample search indexes through existing helpers. * Calls `cancelBillTaskCall` for asynchronous cancellation side effects. Response [#response] ```json { "status": 200, "Message": "Bill canceled successfully for Order Number : " } ``` Important Notes [#important-notes] * This flow cancels the full bill. * It does not selectively cancel only the tests listed in OBR. * Test/sample-level cancellation through sample status is documented in [Update Order](./Update%20Order). * If `uniqueOrderNo = 0`, this bill-cancel branch is not used. * If `cancelBill = 0`, the same inbound `CA` message can be acknowledged as a duplicate order instead of cancelling the bill. Common Issues [#common-issues] | Issue | Check | | ------------------------------------- | --------------------------------------------------------------------------------------------- | | Cancellation is not triggered | Confirm `uniqueOrderNo = 1`, `cancelBill = 1`, and `ORC-1 = CA`. | | Bill is not found | Confirm the inbound `orderNumber` matches an active bill order number or numeric `labBillId`. | | Request returns `Bill already exists` | The bill was found, but cancellation conditions were not all met. | | Only one test should be cancelled | Use a test/sample cancellation flow, not the full bill cancel configuration. | # Create New Order Create New Order [#create-new-order] This page documents the new-order path of `hl7/2_3/createOrder/`. Related operations are documented separately: * [Update Order](./Update%20Order) * [Add Test to Bill](./Add%20Test%20to%20Bill) * [Bill Cancel](./Bill%20Cancel) When This Flow Runs [#when-this-flow-runs] A new order is created when the inbound message is parsed successfully and the router does not handle it as a duplicate, add-test, or cancellation request. Typical conditions: * `authKey` should be valid. * An enabled integration exists with `actionCategoryListId = 51`. * The HL7 data should be valid. * No active bill is found for `orderNumber`, or no order number is parsed. * The HL7 status should not contain bill cancellation status (CA in ORC). * The HL7 data should not contains same order Number with non billed Tests that means `Add test to bill` workflow. Implementation Path [#implementation-path] | Step | Function | Description | | ------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | Validate request | `hl7/2_3/createOrder/` | Validates token and integration configuration. | | Parse HL7 | `orm_hl7_parsing_api_corrohealth` or `orm_hl7_parsing_function_tox` | Converts HL7 segments into the inbound order payload. | | Enqueue downstream create | `Fusion.webhook` | Posts the inbound order payload to `/create_order_in_crelio/`. | | Create patient and bill | `create_order_in_crelio` | Creates or updates patient, resolves org/referral/tests, and creates bill. | | Create billing records | `commonBillingMasterFunction` | Creates bill, billing info, reports, samples, payments, and related records. | Router Response [#router-response] The router does not create the bill directly. It enqueues a Fusion job and returns the accepted order reference. ```json { "status": 200, "jobId": "", "orderNumber": "" } ``` The `jobId` means the request was accepted for downstream processing. It does not by itself guarantee that bill creation succeeded. Parsed Payload Used For Creation [#parsed-payload-used-for-creation] | Parsed field | HL7 source | Downstream use | | ------------------------- | ------------------------------------------- | ----------------------------------------------- | | `fullName` | `PID-5` | Patient name. | | `firstName` | `PID-5.2` | Patient first name. | | `middleName` | `PID-5.3` | Patient middle name. | | `lastName` | `PID-5.1` | Patient last name. | | `gender` | `PID-8` | Patient gender. | | `dob` | `PID-7` | Patient date of birth. | | `mobile` | `PID-13` | Patient contact number. | | `labPatientId` | `PID-2` | External patient identifier. | | `nationalId` | `PID-19` | National identity number. | | `passportNo` | `PID-20` | Passport number. | | `insurance_details` | `IN1` | Patient insurance creation/update. | | `aoe_data` | `OBX`, when `is_aoe_store` is enabled | AOE question values. | | `test_aoe_mapping` | OBX values grouped by current OBR test code | AOE-to-test mapping. | | `bill_attachment_details` | OBX attachment data | Bill attachment upload. | | `shared_providers` | OBX question `REFERRED_TO_DOCTOR` | Shared provider records. | | `orderNumber` | OBR/ORC-derived order identifier | Saved on the bill and used for future matching. | | `orgCode` | PV1/MSH/vendor-specific | Organization lookup. | | `doctorCode` | `OBR-16` or `PV1-7` for ADT | Referral lookup. | | `testList` | `OBR-4` | Tests to bill. | | `billDate` | `OBR-7` | Bill date. | | `sampleDate` | `OBR-7`, when `collectionTime = 1` | Collection date/time. | | `comments` | `NTE` | Bill comments. | | `icd_details` | `DG1` | Bill ICD details. | Patient Handling [#patient-handling] `create_order_in_crelio` first attempts to match an existing patient by `labPatientId`. When `patient_strict_check = 1`, the match also includes first name, middle name, last name, date of birth, and gender. If a patient is found, selected demographics are updated from the inbound payload. If no patient is found, the downstream function validates required fields and creates a patient through `commonPatientRegistrationFunction`. Required patient fields for a new patient: * Patient name * Gender * Age or date of birth * Valid organization/referral context Organization And Referral Handling [#organization-and-referral-handling] Organization is resolved from: 1. `organizationIdLH` 2. `orgCode` 3. `organizationName` or `organisationName` 4. Lab default organization fallback Referral is resolved from: 1. `doctorCode` 2. `referralIdLH` 3. `referralName` 4. Lab default referral fallback When a non-empty organization or referral name is supplied and no matching record exists, the downstream function can create a new organization or referral using the existing billing helpers. Test Handling And Pricing [#test-handling-and-pricing] Each inbound test is matched against active `allTests` records in this order: 1. `testCode` 2. `testID` 3. `testName` For matched tests, the downstream payload includes the internal test ID, resolved amount, quantity, concession, sample ID, sample type, report comments, report-level tags, and ICD code ID. Pricing priority: 1. Organization price list, when available. 2. Doctor price list, when no organization price applies. 3. Master test amount. 4. If `orgPriceList = 1`, organization price-list priority is explicitly used. Unmatched tests are returned as `unbilled_tests`. If no inbound test matches master data, the bill is not created. Bill Creation Side Effects [#bill-creation-side-effects] After `commonBillingMasterFunction` creates the bill: * Home collection can be booked when `isHomeCollection = 1`. * AOE values are saved through `create_question_values_from_aoe_data`. * Test-level clinical information is saved through `create_test_clinical_info`. * Shared providers are created when provided in the inbound message. * Accession/manual sample update is attempted unless `test_wise_accession = 1`. * Base64 bill attachments are uploaded when present. * Tox/micro orders enqueue drug setup through `/integration/tox/add_drugs_into_report/`. Downstream Success Response [#downstream-success-response] ```json { "code": 200, "is_tox": 0, "unbilled_tests": [], "Message": "Bill Generated successfully with billId Id : " } ``` For tox/micro orders: ```json { "code": 200, "unbilled_tests": [], "Message": "Tox Bill Generated successfully with billId Id : " } ``` Common Failure Responses [#common-failure-responses] | Response | Meaning | | ---------------------------------------------------------------------------- | --------------------------------------------------------------------- | | `Invalid auth key` | Downstream payload did not contain a valid token. | | `Mandatory fields are Missing e.g. Name, Gender, Age/DOB, Test Details etc` | Patient or test data required for registration/billing is incomplete. | | `Wrong Organisation or Referral details` | Organization/referral lookup failed during patient creation. | | `Patient not registered. Combination Mismatched of Patient ID / Contact No.` | Existing patient matching failed and registration did not complete. | | `Test codes are not available in master data - ...` | None of the inbound tests matched active master tests. | # Inbound HL7 Overview Inbound HL7 Overview [#inbound-hl7-overview] `hl7/2_3/createOrder/` is the inbound HL7/SFTP router for order-related workflows. It validates the integration token, parses the raw HL7 payload, checks the external order number, and then routes the request to the correct operation. Entry Point [#entry-point] | Item | Value | | --------------------------------- | ------------------------------------------------------------------------------------------------------- | | Function | `hl7/2_3/createOrder/` | | Source file | [livehealthapp/labs/views.py](https://github.com/CrelioHealth/livehealthapp/blob/develop/labs/views.py) | | Parser, standard orders | `orm_hl7_parsing_api_corrohealth` | | Parser, tox/micro orders | `orm_hl7_parsing_function_tox` | | Downstream bill creation endpoint | `/create_order_in_crelio/` | | Required integration category | `actionCategoryListId = 51` | Request Contract [#request-contract] | Field | Location | Required | Description | | ---------------------- | ------------ | -------- | ---------------------------------------------------------------------- | | `authKey` | Query string | Yes | Validates `requestToken` and resolves lab, lab user, and lab timezone. | | `integration` | Query string | No | Vendor identifier passed to the parser for vendor-specific mapping. | | `patient_strict_check` | Query string | No | Enables stricter downstream patient matching by demographic fields. | | HL7 message | Request body | Yes | Raw HL7 payload parsed into the inbound order payload. | Routing Summary [#routing-summary] | Operation | Routing condition | Documentation | | ------------------------ | ------------------------------------------------------------------------------------------------- | -------------------------------------------- | | Create new order | No active bill exists for the inbound order number, or no order number is parsed. | [Create Order](./Create%20Order) | | Update order / accession | Existing bill or report is found downstream and the request carries accession/sample identifiers. | [Update Order](./Update%20Order) | | Add tests to bill | Existing active bill, `addTestToBill = 1`, and `uniqueOrderNo = 0`. | [Add Test to Bill](./Add%20Test%20to%20Bill) | | Cancel bill | Existing active bill, `uniqueOrderNo = 1`, `cancelBill = 1`, and `ORC-1 = CA`. | [Bill Cancel](./Bill%20Cancel) | Request Routing Flow [#request-routing-flow] Common Integration Configuration [#common-integration-configuration] These keys are read from the integration configuration. | Key | Default | Used by | Description | | ------------------------------ | ------- | ---------------------------------- | ---------------------------------------------------------------------------------------------- | | `uniqueOrderNo` | `1` | Create, cancel, duplicate handling | Enforces one active bill per external order number. | | `cancelBill` | `0` | Bill cancel | Allows full bill cancellation when the inbound order status is `CA`. | | `addTestToBill` | `0` | Add test to bill | Allows appending inbound tests to an existing bill when duplicate order numbers are permitted. | | `add_duplicate_test` | `0` | Add test to bill | Allows adding a test even if the bill already has that test. | | `toxMicroIntegration` | `0` | Create order | Switches to the tox/micro parser and post-bill drug setup. | | `orgPriceList` | `0` | Create, add test | Uses organization price-list amounts where applicable. | | `replace_insurance_details` | `0` | Create order | Replaces active patient insurance records before saving inbound insurance. | | `auto_add_insurance` | `0` | Create order | Creates insurance master data from inbound `IN1` if the code is not found. | | `prescription_reflex_at_order` | `0` | Tox/micro create order | Controls prescription reflex setup after tox bill creation. | | `is_aoe_store` | unset | Create, add test | Stores OBX question/value data as AOE values. | Parsed Fields Used For Routing [#parsed-fields-used-for-routing] | Parsed field | HL7 source | Description | | ------------------ | -------------------------------------------------------- | ------------------------------------------------------------------------------ | | `orm_status` | `ORC-1` | Order control/status. `CA` is used for cancellation. | | `orderNumber` | OBR placer/filler fields, with vendor-specific overrides | External order number used for duplicate, update, add-test, and cancel lookup. | | `testList` | `OBR-4` | Inbound tests to create or append. | | `lab_report_id` | `OBR-3` | Report identifier used by accession/update flows. | | `manual_sample_id` | `ZCT-12` | Manual sample/accession identifier used by update flows. | Duplicate Order Lookup [#duplicate-order-lookup] When the inbound `orderNumber` is available, the router searches active bills in the same lab where: * `isCancel = 0` * `isRefund = 0` * `isWriteOff = 0` * `orderNumber` matches the inbound order number, or the inbound order number is numeric and matches `labBillId` That lookup decides whether the request becomes add-test, bill cancel, duplicate acknowledgement, or create order. Response Timing [#response-timing] Create-order requests are forwarded to Fusion and return a `jobId` immediately. Add-test and bill-cancel requests are handled synchronously in the router. Update/accession behavior happens inside `/create_order_in_crelio/` when the downstream function detects an existing order or report. # Update Order Update Order [#update-order] The inbound create-order endpoint also supports update-like behavior. This does not call the normal UI bill-update API. Instead, `/create_order_in_crelio/` detects existing bills or reports and updates accession/sample state before attempting new bill creation. When This Flow Runs [#when-this-flow-runs] This flow runs inside `create_order_in_crelio` when the parsed payload points to an existing active bill or report. Supported identifiers: | Parsed field | Typical HL7 source | Lookup behavior | | ------------------ | ------------------------------ | ---------------------------------------------- | | `orderNumber` | OBR/ORC-derived order number | Finds active bill by order number. | | `lab_report_id` | `OBR-3` | Finds the active lab report relation. | | `manual_sample_id` | `ZCT-12` | Finds active report by manual sample ID. | | `bill_id` | Vendor-specific, mainly Cerner | Finds report/bill for Cerner accession update. | Implementation Path [#implementation-path] | Step | Function | Description | | ------------------------ | ------------------------------------- | ---------------------------------------------------------------------------------- | | Existing order detection | `create_order_in_crelio` | Checks order number, report ID, and manual sample ID before patient/bill creation. | | Accession parsing | `sample_accession_hl7_payload` | Re-parses the original HL7 payload for accession fields. | | Accession update | `common_function_to_update_sample_id` | Updates report and collected sample accession data. | | Test/sample dismissal | `dismissTestAPI` | Used when inbound sample status indicates cancellation. | Accession Parser Fields [#accession-parser-fields] `sample_accession_hl7_payload` reads a smaller payload from the same raw HL7 message. | Parsed field | HL7 source | Description | | ------------------------- | -------------------------- | ------------------------------------------------------------------------------- | | `lab_report_id` | `OBR-3` | Report identifier. | | `manual_sample_id` | `ZCT-12` | New manual sample/accession number. | | `test_code` | `OBR-4` | Test code used as a fallback report lookup. | | `orc_status` | `ORC-1` | Order control/status. | | `sample_status` | `ORC-5` | Sample state such as `in-lab`, `completed`, `collected`, or cancel-like values. | | `lrr_list` | All `OBR-3` values | Multi-report update list, used especially for Cerner. | | `sample_location_mapping` | Cerner ZCT/location fields | Sample rack/location mapping. | Update Behavior [#update-behavior] When a matching report/sample is found, the update flow can: * Update the report manual sample ID. * Update the collected sample accession number. * Mark accession as done on the collected sample. * Update report/sample timestamps. * Update sample collected or received flags when `sample_status` is `in-lab`, `completed`, or `collected`. * Update Elasticsearch report/sample fields. * Write an activity log for the accession update. Sample-Level Cancellation [#sample-level-cancellation] If `sample_status` contains `cancel`, the flow dismisses the matching report/test instead of updating accession fields. The dismissal comment is: ```text Test dismissed through Integration by user ``` This is test/sample-level cancellation. Full bill cancellation is documented in [Bill Cancel](./Bill%20Cancel). Cerner-Specific Behavior [#cerner-specific-behavior] When `integration = cerner`, the update flow supports additional behavior: * It can update an existing bill's `orderNumber`. * It can update sample received state for all tests listed in `OBR` segment. * It can match tests by lab report ID, test code, or test name. * It can update rack/location information from Cerner-specific ZCT data. Responses [#responses] | Response | Meaning | | ------------------------------------------- | --------------------------------------------------------------- | | `Manual sampleId updated` | Existing bill/report was found and accession update succeeded. | | `Manual sampleId failed to update` | Existing bill/report was found but update failed. | | `Accession Number was updated successfully` | Cerner accession update succeeded. | | `Sample Received successfully!` | Cerner sample received update succeeded. | | `Accession number should not be blank` | Non-Cerner update did not include a manual sample/accession ID. | | `No records found` | No bill/report identifier could be resolved for update. | | `Can not find the Lab Report...` | Report lookup failed for the bill/test combination. | # ADT HL7 Outbound ADT [#hl7-outbound-adt] ADT outbound messages transmit patient and visit/event information. They can include PV1, OBX, and NTE content depending on the configured segment list. Generation Flow [#generation-flow] Implementation Details [#implementation-details] The ADT branch is selected when `payloadType` is `ADT` in `/integration/sftp/hl7/2_3/send_hl7_data/`. It emits an `ADT^A04` style patient registration/event message using the report's bill, patient, and visit context. Request Contract [#request-contract] | Field | Location | Required | Notes | | -------------------------------- | -------- | -------: | ------------------------------------------------------------------------------------------------------- | | `payloadType` | Root | Yes | Must be `ADT`. | | `labReportDetails[].labReportId` | Root | Yes | First report id is used to load patient and bill context. `testDetails` can be used as fallback. | | `integrationDetails` | Root | No | Controls sender/receiver identifiers, Cerner formatting, AOE, custom segment metadata, and TCP routing. | Segment Construction [#segment-construction] | Segment | When emitted | Notes | | ------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `MSH` | Always | Message type is `ADT^A04`; version is read from `versionId` with legacy default behavior. | | `PID` | Cerner branch | Cerner-specific PID includes MRN, CMRN, FIN, national id/worker code, nationality, address, and contact format. | | `PV1` | Cerner branch and configured visit context | Contains outpatient/laboratory context and bill date. | | `AL1` | `AL1` in `segments` | Adds patient allergy segment. | | `OBX` | Cerner or `AOE` enabled | Cerner sends medical history and height/weight observations; AOE sends question/value rows from bill questions. | | `CST` | `CST` in `segments` | Sends operational metadata such as ward, worker code, doctor code, aadhar, allergy, bill remark, agent data, height/weight, lab name, and sponsorship. | Cerner-Specific Behavior [#cerner-specific-behavior] When `integration_name` is `cerner`, the branch changes patient identifiers and visit fields substantially. It chooses a CMRN value from national identity when present, builds a FIN from bill id and sending facility, formats contact as home phone, and sends medical history/height/weight as OBX values. Failure Cases [#failure-cases] | Case | Result | | --------------------------- | ---------------------------------- | | Missing report id | HTTP 400. | | No message generated | JSON status `400`, `Invalid Data`. | | Socket/processing exception | JSON status `500` with traceback. | ADT Configuration [#adt-configuration] | Key | Label | Type | Default | Description | | ---------------------- | --------------------- | -------------- | --------: | ------------------------------------------------------ | | `pv1.2` | Patient Class | text | `""` | Patient class code for `PV1-2`. | | `sendingApplication` | Sending Application | text | `""` | Populates `MSH-3`. | | `receivingApplication` | Receiving Application | text | `""` | Populates `MSH-5`. | | `receivingFacility` | Receiving Facility | text | `""` | Populates `MSH-6`. | | `integration_name` | Integration Name | text | `""` | Enables ADT-specific formatting. | | `labName` | Lab Name | text | `""` | Display lab name. | | `segments` | HL7 Segments | checkbox group | `["PV1"]` | Optional ADT segments. Supported: `PV1`, `OBX`, `NTE`. | | `sendingFacility` | Sending Facility | text | `""` | Populates `MSH-4`. | | `versionId` | HL7 Version | dropdown | `2.4` | HL7 message version. | | `host` | Host | text | `""` | TCP destination host. | | `port` | Port | text/number | `""` | TCP destination port. | | `AOE` | Enable AOE | toggle | `0` | Sends Ask-On-Entry questions. | | `AOE_code` | AOE Code Field | text | `""` | Field/code mapping for AOE questions. | Minimal Example [#minimal-example] ```json { "payloadType": "ADT", "labReportDetails": [ { "labReportId": 123456 } ], "integrationDetails": { "sendingApplication": "Creliohealth", "receivingApplication": "MIRTH", "receivingFacility": "MAIN", "versionId": "2.4", "host": "127.0.0.1", "port": 5000, "segments": ["PV1"] } } ``` # DFT HL7 Outbound DFT [#hl7-outbound-dft] DFT outbound messages transmit billing and charge data as `DFT^P03`. DFT can be generated through the generic SFTP endpoint, the bill-wise forwarding path, or the direct 2.3 and 2.4 DFT APIs. Generation Flow [#generation-flow] Implementation Details [#implementation-details] The SFTP DFT path is selected when `payloadType` is `DFT` in `/integration/sftp/hl7/2_3/send_hl7_data/`. The endpoint resolves the report from `labReportDetails[0].labReportId` or `testDetails[0].labReportId`, loads the bill, patient, provider, ICD, insurance, and billing-test context, then generates a `DFT^P03` message. Request Contract [#request-contract] | Field | Location | Required | Notes | | -------------------------------- | -------- | ----------------: | ------------------------------------------------------------------------------------------- | | `payloadType` | Root | Yes | Must be `DFT`. | | `labReportDetails[].labReportId` | Root | Yes | First report id is used to load the bill/report context. `testDetails` is used as fallback. | | `host` | Root | Only for TCP send | The SFTP DFT branch reads this from the root request, not `integrationDetails`. | | `port` | Root | Only for TCP send | If zero/missing, generated HL7 is returned but not sent. | | `integrationDetails` | Root | No | Controls insurance, receiving app/facility, vendor rules, and FT1 mapping. | Generated Segment Structure [#generated-segment-structure] | Segment | Source data | Behavior | | ------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `MSH` | Integration identifiers and generated timestamp/control id | Message type is `DFT^P03`; sending application defaults to `Creliohealth`; version defaults to `2.3.1` in this branch. | | `PID` | Patient ids, name, DOB, gender, address, contact | Patient name is rebuilt from stored first/middle/last name where available. | | `PV1` | Patient class, address, provider, bill paid amount | Provider data comes from the bill doctor. | | `IN1` | Patient or bill-level insurance | Emitted when insurance exists. Athena gets relation-specific IN1 behavior. | | `FT1` | Each requested report/test | One or more charge rows are generated from integration code, procedure code, unique test id, or test code depending on flags and vendor. | Vendor and Mapping Behavior [#vendor-and-mapping-behavior] | Integration | Behavior | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ATHENA` | Uses outsource category/name in FT1 where available and relation-aware IN1 formatting. If `integrationCode` contains comma-separated values, each value becomes an FT1 row. | | `EMED` | Prefers report integration codes; falls back to procedure code when configured. | | Default | Emits one FT1 row per report using unique test id when `unique_testcode` is enabled, otherwise the report test code. | Bill-Wise DFT Forwarding [#bill-wise-dft-forwarding] When `payloadType` is `DFT` in `/integration/sftp/hl7/2_3/send_billwise_hl7_data/`, the function does not build the final HL7 message directly. It loads the bill, serializes all synced reports, creates a webhook payload, and forwards it to `integrationDetails.url` or the default `/integration/hl7/send_dft_hl7_data/` URL. The forwarded payload includes patient demographics, bill amount, advance/due amount, order number, report details, `host`, `port`, and `integrationDetails`. Failure and Skip Cases [#failure-and-skip-cases] | Case | Result | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | Missing report id | HTTP 400: `labReportId is not valid`. | | `is_insurance_enable` is true and no insurance is available | JSON status `200` with message `No Insurance details are available for this patient` in the SFTP DFT branch. | | Message cannot be generated | JSON status `400`, `Invalid Data`. | | Socket/processing exception | JSON status `500` with traceback text. | DFT Configuration [#dft-configuration] DFT through /integration/sftp/hl7/2_3/send_hl7_data/ [#dft-through-integrationsftphl72_3send_hl7_data] | Key | Label | Type | Default | Description | | ---------------------- | --------------------- | ----------- | ------: | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `is_insurance_enable` | Enable Insurance | toggle | `0` | Requires/transmits insurance information. If enabled and insurance is missing, generation is stopped or returns a no-insurance response depending on branch. | | `BillingInsurance` | Use Billing Insurance | toggle | `0` | Uses bill-level insurance records instead of patient insurance. | | `receivingApplication` | Receiving Application | text | `""` | Receiving application identifier for `MSH-5`. | | `receivingFacility` | Receiving Facility | text | `""` | Receiving facility identifier for `MSH-6`. | | `integration_name` | Integration Name | text | `""` | Enables integration-specific DFT formatting such as Athena or EMED rules. | | `unique_testcode` | Use Unique Test Code | toggle | `0` | Uses unique test ids/codes for FT1 procedure mapping in supported paths. | | `testdetails_flag` | Include Test Details | toggle | `1` | Includes test code and name in FT1 where supported. | | `FT1_testCode` | FT1 Test Code Mapping | text/toggle | `""` | Includes the test code in FT1 mapping where supported. | DFT 2.3 direct API [#dft-23-direct-api] API: `/integration/hl7/send_dft_hl7_data/` | Key | Label | Type | Default | Description | | ---------------------- | ------------------------- | ------ | -------------: | ---------------------------------------------------------------------------------- | | `integration_name` | Integration Name | text | `""` | Controls direct DFT 2.3 formatting, including NextGen-specific segment behavior. | | `sendingApplication` | Sending Application | text | `Creliohealth` | Populates `MSH-3`. | | `receivingApplication` | Receiving Application | text | `""` | Populates `MSH-5`. | | `receivingFacility` | Receiving Facility | text | `""` | Populates `MSH-6`. | | `versionId` | HL7 Version | text | `2.3.1` | Populates `MSH-12`. | | `is_insurance_enable` | Require Insurance Details | toggle | `0` | If enabled and no insurance is available, returns `400` and does not generate DFT. | | `BillingInsurance` | Use Bill-Level Insurance | toggle | `0` | Uses `UserBillInsurance` records for the bill. | | `lab_time_zone` | Use Lab Time Zone | toggle | `0` | Converts report, collection, accession, and update timestamps to the lab timezone. | | `FT1_testCode` | Include Test Code in FT1 | toggle | `0` | Includes test code in the FT1 transaction segment where supported. | DFT 2.4 direct API [#dft-24-direct-api] API: `/integration/hl7/2_4/send_dft_hl7_data/` | Key | Label | Type | Default | Description | | -------------------------- | ----------------------------------------- | ------ | -------------: | ---------------------------------------------------------------------------------------------------- | | `integration_name` | Integration Name | text | `""` | Controls DFT 2.4 vendor-specific formatting, including PGM, Athena, Cermak, and custom FT1 behavior. | | `sendingApplication` | Sending Application | text | `Creliohealth` | Populates `MSH-3`. | | `receivingApplication` | Receiving Application | text | `""` | Populates `MSH-5`. | | `receivingFacility` | Receiving Facility | text | `""` | Populates `MSH-6`. | | `versionId` | HL7 Version | text | `2.3.1` | Populates `MSH-12`; set to `2.4` when required by the destination. | | `host` | TCP Host | text | `""` | Destination TCP host. | | `port` | TCP Port | number | `0` | Destination TCP port fallback. | | `` | Integration-Specific Port | number | `0` | Dynamic port lookup. The system checks `integrationDetails[integration_name]` before `port`. | | `is_insurance_enable` | Require Insurance Details | toggle | `0` | If enabled and insurance is missing, returns `400` and does not generate DFT. | | `BillingInsurance` | Use Bill-Level Insurance | toggle | `0` | Uses bill-level insurance mapping instead of patient insurance records. | | `outsource_name` | Outsource Name | text | `""` | Name used in outsourced FT1/PV1 formatting. | | `outsource` | Enable Outsource Mapping | toggle | `0` | Enables outsource-based mapping for provider/service details. | | `outsource_category` | Use Outsource Category | toggle | `0` | Uses outsource category for provider code/name. | | `lab_name` | Lab Name Override | text | `""` | Overrides or supplies lab name in PV1/FT1 fields. | | `disabled_integrations` | Disabled Integration Identifiers | array | `[]` | Skips DFT generation for matching integration identifiers. | | `parameter_tests_list` | Parameter-Level Test Codes | array | `[]` | Test codes that generate FT1 entries from report parameter integration codes. | | `segment_sequence` | HL7 Segment Sequence | array | `[]` | Custom segment ordering such as `["EVN", "PID", "PV1", "IN1", "GT1", "FT1", "DG1"]`. | | `int_code_over_proc_code` | Prefer Integration Code | toggle | `0` | Uses test integration code in FT1 when available instead of procedure code. | | `testdetails_flag` | Include Test Details | toggle | `0` | Includes test code and test name in FT1. | | `base64_result` | Include Base64 Report | toggle | `0` | Appends report PDF as base64 ED OBX when available. | | `orgName` | Use Organization Name as Sending Facility | toggle | `0` | Uses organization name instead of organization code in sending facility. | | `serviceDate` | Use Report Date as Service Date | toggle | `0` | Uses report date as FT1 service date instead of bill date. | | `sampleDate` | Use Sample Date as Service Date | toggle | `0` | Uses each report sample date as FT1 service date where available. | | `profile_icd_code_mapping` | Profile ICD Code Mapping | object | `{}` | Adds configured ICD codes for matching profile test codes. | | `orderId` | Use Bill ID as Visit ID | toggle | `0` | Uses lab bill id instead of order number as PV1 visit id. | | `FT1_testCode` | Include Test Code in FT1 | toggle | `0` | Includes test code in FT1 where supported. | | `drug_wise_ft1` | Generate Drug-Wise FT1 | toggle | `0` | Generates additional FT1 segments for extracted drug results. | | `ignore_parameters` | Ignored Parameters | array | `[]` | Parameter names excluded during drug-wise FT1 extraction. | Minimal Example [#minimal-example] ```json { "payloadType": "DFT", "labReportDetails": [ { "labReportId": 123456 } ], "integrationDetails": { "integration_name": "PGM", "sendingApplication": "Creliohealth", "receivingApplication": "BILLING", "receivingFacility": "MAIN", "versionId": "2.4", "host": "127.0.0.1", "port": 5000, "is_insurance_enable": 1, "BillingInsurance": 1, "testdetails_flag": 1 } } ``` # ORM HL7 Outbound ORM [#hl7-outbound-orm] ORM outbound messages transmit order data as `ORM^O01`. They are generated from report/order context and can include visit, insurance, diagnosis, medication, custom, attachment, and AOE data depending on the configured segment list. Generation Flow [#generation-flow] Implementation Details [#implementation-details] The ORM path is selected when `payloadType` is `ORM` in `/integration/sftp/hl7/2_3/send_hl7_data/`. It sends order context for one or more tests on the bill, usually after bill creation, order update, cancellation, collection, or dispatch webhooks. Request Contract [#request-contract] | Field | Location | Required | Notes | | -------------------- | -------- | -------: | ---------------------------------------------------------------------------------------------------------------- | | `payloadType` | Root | Yes | Must be `ORM`. | | `labReportDetails[]` | Root | Yes | Supplies the report/test list and the first `labReportId` used to load canonical bill context. | | `webhookId` | Root | No | Influences `ORC-1` action code and some Cerner collection/cancel/dispatch behavior. | | `integrationDetails` | Root | No | Controls identifiers, optional segments, test filtering, branch/outsource behavior, transport, and vendor rules. | Action Mapping [#action-mapping] The ORM branch maps webhook context into order-control behavior: | Condition | ORC action meaning | | -------------------------------------- | ------------------------------------------------------------------------------- | | `webhookId` in `[2, 18]` | Update/change order (`XO`) for the default branch. | | `webhookId` in `[3, 8, 14]` | Cancel order (`CA`) for the default branch. | | Other values | New order (`NW`) for the default branch. | | Cerner collection/outsourcing webhooks | Uses Cerner-specific statuses such as `Collected`, `Canceled`, or `Dispatched`. | Segment Construction [#segment-construction] | Segment | When emitted | Notes | | ----------- | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `MSH` | Always | Uses configured sending/receiving application/facility and `versionId`. | | `PID` | Always | Cerner gets MRN/CMRN/FIN-specific PID formatting; other integrations use standard patient identifiers and demographics. | | `PV1` | Always in the current report-wise branch | Carries patient class, sending facility/lab, provider, visit id, billing type, and bill source where enabled. | | `AL1` | `AL1` in `segments` | Uses patient allergy text. | | `IN1` | `IN1` in `segments` and insurance exists | Uses patient insurance or bill-level insurance when `BillingInsurance` is enabled. | | `GT1` | `GT1` in `segments` | Uses default guarantor family relation. | | `ORC`/`OBR` | Per eligible test/profile | Test selection obeys `testCodes`, profile handling, Cerner formatting, and outsource routing flags. | | `FT1`/`DG1` | Segment enabled | Uses procedure codes/integration codes and ICD mappings. `TestWiseICD=2` scopes ICDs to the current test. | | `OBX` | Segment/feature enabled | Carries clinical history, AOE, lab form values, risk/type values, component-type values, attachments, or TRF depending on flags. | | `CST` | `CST` in `segments` | Carries operational metadata such as ward, worker code, doctor code, branch, bill comments, organization notes, height/weight, passport, and batch details. | Test Selection [#test-selection] `testCodes` acts as a filter over requested report/test codes. If it is empty, all eligible tests in the request are considered. If it is non-empty and no report/test passes the filter, the API returns JSON status `209` with `Test codes are not configured.` Vendor-Specific Behavior [#vendor-specific-behavior] | Integration | Behavior | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `cerner` | Uses Cerner-specific PID, PV1, ORC, and OBR fields; supports alternate HLA message output on `HLA_port` for outsourced analyzer flow. | | `cpl` | Calculates billing type as insurance, credit, or patient pay. | | `siglo` | Prefixes contact with country code and prefers national id for SSN. | | `tribal` | Updates bill order number to the manual sample id and changes OBR placer/filler ordering. | Transport Behavior [#transport-behavior] The normal ORM message is sent to `integrationDetails.host:integrationDetails.port` when `port` is non-zero. If Cerner/HLA outsource behavior is active, a second message can be sent to `HLA_port`. Messages are still returned in the JSON response, and `sent_to_mirth` indicates whether a socket send occurred. ORM Configuration [#orm-configuration] Message: `ORM^O01` | Key | Label | Type | Default | Description | | ----------------------------- | ----------------------------- | -------------- | ------------------------------------------: | --------------------------------------------------------------------------------------------------- | | `integration_name` | Integration Name | text | `""` | Enables ORM-specific formatting such as Cerner, CPL, Siglo, Tribal, and test-wise ORM behavior. | | `pv1.2` | Patient Class | text | `OPD` | Patient class used in `PV1-2`. | | `sendingApplication` | Sending Application | text | `Creliohealth` | Populates `MSH-3`. | | `receivingApplication` | Receiving Application | text | `""` | Populates `MSH-5`. | | `receivingFacility` | Receiving Facility | text | `""` | Populates `MSH-6`. | | `sendingFacility` | Sending Facility | text | org code | Populates `MSH-4`; defaults to organization code. | | `labName` | Lab Name | text | `""` | Display lab name used in PV1/CST and related context. | | `testCodes` | Test Codes Filter | array | `[]` | Restricts transmitted tests. Empty includes all eligible tests. | | `segments` | HL7 Segments | checkbox group | `[]` | Optional segments. Supported: `PV1`, `RXO`, `AL1`, `IN1`, `FT1`, `DG1`, `CST`, `OBX`, `NTE`, `GT1`. | | `outsource_field` | Include Outsource Information | toggle | `0` | Includes outsource vendor details in ORC/OBR context. | | `versionId` | HL7 Version | dropdown | `2.3.1` | Populates `MSH-12`. Common supported values are `2.3`, `2.3.1`, `2.4`, `2.5`, `2.5.1`. | | `host` | Host | text | `""` | TCP host for ORM transmission. | | `port` | Port | text/number | `0` | Primary TCP port. | | `HLA_port` | HLA Port | text/number | `0` | Alternate port for outsourced/HLA analyzer message flow. | | `isProfileCode` | Use Profile Codes | toggle | `0` | Sends profile codes where applicable. | | `AOE` | Enable AOE | toggle | `0` | Sends Ask-On-Entry questions as OBX segments. | | `AOE_code` | AOE Code Field | text/toggle | `0` | Adds question code in AOE OBX where enabled. | | `Labform` | Include Lab Form | toggle | `0` | Sends additional patient/lab form values as OBX. | | `nm_field_types` | Numeric Field Types | array | `["pin", "number", "float", "phonenumber"]` | Field types treated as numeric OBX values. | | `dt_field_types` | Date Field Types | array | `["datetime", "date"]` | Field types treated as date OBX values. | | `outsource_id` | Outsource ID Field | text/number | `0` | Used for outsource-specific test filtering in selected webhooks. | | `TestWiseICD` | Test-wise ICD Codes | toggle/mode | `0` | Includes ICD codes per test. Mode `2` restricts ICD mapping to the current test. | | `TestWiseORM` | Test-wise ORM Formatting | toggle | `0` | Sends test-specific ORM messages. | | `branchIntegration` | Enable Branch Integration | toggle | `0` | Uses branch details in CST/custom context where applicable. | | `attachments` | Include Attachments | toggle | `0` | Sends bill attachments and patient proofs as ED OBX. | | `billSource` | Bill Source Code | text/toggle | `0` | Includes bill source in PV1/FT1 context. | | `accessionDate` | Accession Date Field | text/toggle | `0` | Uses accession/sample receipt date where enabled. | | `lab_time_zone` | Use Lab Timezone | toggle | `0` | Converts dates to lab timezone. | | `prescribed_drugs` | Send Prescribed Medication | toggle | `0` | Includes prescribed drugs in RXO. | | `include_obx_component_type` | Include OBX Component Type | toggle | `0` | Adds OBX segments for configured component types. | | `allowed_obx_component_types` | Allowed OBX Component Types | array | `["NM", "ST", "TX", "CE"]` | Allowed OBX component value types. | | `enable_obx_risk_and_type` | Include Risk and Patient Type | toggle | `0` | Adds risk level and patient type OBX values. | | `BillingInsurance` | Use Billing Insurance | toggle | `0` | Uses bill-level insurance records. | | `trf_flag` | Include TRF | toggle | `0` | Adds TRF PDF as base64 ED OBX. | Minimal Example [#minimal-example] ```json { "payloadType": "ORM", "labReportDetails": [ { "labReportId": 123456 } ], "integrationDetails": { "sendingApplication": "Creliohealth", "receivingApplication": "MIRTH", "receivingFacility": "MAIN", "versionId": "2.4", "host": "127.0.0.1", "port": 5000, "segments": ["PV1", "IN1", "DG1"] } } ``` # ORU HL7 Outbound ORUR01 [#hl7-outbound-orur01] ORUR01 outbound messages transmit results, usually as `ORU^R01`. The report-wise path uses `labReportId`; the bill-wise path uses `bill_id` and can generate a consolidated result message for all synced, non-dismissed reports on the bill. Report-Wise Generation Flow [#report-wise-generation-flow] Bill-Wise Generation Flow [#bill-wise-generation-flow] Implementation Details [#implementation-details] ORUR01 is the most feature-rich outbound branch. It converts submitted report results into `ORU^R01` by default, with report-wise and bill-wise variants. Report-Wise Request Contract [#report-wise-request-contract] | Field | Location | Required | Notes | | ----------------------- | -------- | ----------: | ---------------------------------------------------------------------------------------------------------------------------- | | `payloadType` | Root | Yes | Must include `ORUR01`. | | `labReportId` | Root | Yes | Used directly to load the source report. | | `dictionaryId` | Root | Conditional | Used by legacy dictionary gates unless `disable_dict` bypasses them. | | `reportFormatAndValues` | Root | Usually | Source of result rows for report-wise OBX/NTE generation. | | `reportBase64` | Root | Conditional | Used when `base64_result` is enabled and `billwise_base64` is not used. | | `integrationDetails` | Root | No | Controls result formatting, optional segments, routing, base64, insurance, tox/micro behavior, and vendor-specific behavior. | Bill-Wise Request Contract [#bill-wise-request-contract] | Field | Location | Required | Notes | | -------------------- | -------- | -------: | --------------------------------------------------------------------------------- | | `payloadType` | Root | Yes | Must be `ORUR01`. | | `bill_id` | Root | Yes | Loads all synced, non-dismissed, non-redraw reports on the bill. | | `billId` | Root | No | External/lab bill id used in message context. | | `reportDetails` | Root | Usually | Bill-wise result payload grouped into profiles/tests. | | `final_report` | Root | No | Sets `OBR-25` to `F` when true, otherwise `P`. | | `integrationDetails` | Root | No | Controls bill-wise routing, grouping, base64, attachments, and segment inclusion. | Result Generation Model [#result-generation-model] | Stage | Report-wise behavior | Bill-wise behavior | | ------------------ | ------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | Context loading | Starts from one `labReportRelation` and uses its bill/patient context. | Starts from bill id and loads all synced reports on the bill. | | Result rows | Reads `reportFormatAndValues` for the report. | Reads `reportDetails`, profile/test groupings, and report format values for each report. | | OBX values | Strips HTML, normalizes spaces, calculates reference ranges and abnormal flags. | Same concepts, but repeated for each profile/test block. | | Descriptive values | `descriptionFlag` values become `NTE` chunks rather than normal result `OBX`. | Same behavior per grouped result. | | Specimen | Adds `SPM` with accession/sample details. | Adds `SPM` per grouped report/test block where applicable. | Segment Construction [#segment-construction] | Segment | When emitted | Notes | | ----------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- | | `MSH` | Always when dictionary/disable check allows generation | `msgType` defaults to `ORU^R01`; sender/facility may be vendor-specific. | | `PID` | Always | Includes patient identifiers, demographics, race/ethnicity mappings, address, contact, and SSN/passport fields. | | `PV1` | `PV1` in `segments` | Carries visit/facility/provider context. | | `AL1` | `AL1` in `segments` | Patient allergies. | | `IN1` | `IN1` in `segments` and insurance exists | Uses patient insurance or bill-level insurance with `BillingInsurance`. | | `FT1`/`DG1` | Segment enabled | Uses bill date, address, ICD code, accession, and procedure/test code. | | `ORC`/`OBR` | Per result block | Uses order number, accession, test/profile code/name, dates, department/category, report status, and provider. | | `OBX` | Per result value | Handles normal, base64, AOE, clinical history, risk/type, tox/micro, antibiotic, and document values. | | `NTE` | Comments/descriptive results/users/doctors | Controlled by comments flags and signing/submitted/saved user flags. | | `SPM` | Result/specimen block | Includes accession/sample and collection/accession dates. | | `CST`/`ZBR` | Feature enabled | Custom operational metadata and mark-as-done linked report details. | Validation and Business Skips [#validation-and-business-skips] | Case | Result | | -------------------------------------------------------------- | ------------------------------------------------------------------------- | | Missing `labReportId` in report-wise flow | HTTP 400. | | Missing `bill_id` in bill-wise flow | HTTP 400. | | Insurance required but unavailable | JSON status `400`, `No Insurance details are available for this patient`. | | `restrict_outsourced_reports` and outsourced flag is `3` | Report-wise skips with status `200`; bill-wise skips individual reports. | | `restrict_outsourced_data` and outsourced flag is enabled | JSON status `400`, `Outsourced Data`. | | Athena integration identifier mismatch | JSON status `209`, `Data is not related to Athena!`. | | `only_positive_results` enabled and no positive result matched | Message is blanked and the endpoint returns `Invalid Data`. | Dynamic Routing [#dynamic-routing] The branch derives an `integration_identifier` from the bill order number using `integration_differentiator` (default `-`) unless `integration_identifier` is explicitly configured. If the main `port` is not set, the code can look up a dynamic port from `integrationDetails[integration_identifier]`. Some integrations, such as EMDEON/DOH-style flows, also use a secondary `doh_port`. Integration Name Specifications [#integration-name-specifications] ORUR01 uses both `integration_name` and, in selected report-wise flows, `integration_identifier`. `integration_name` changes the segment layout; `integration_identifier` usually selects a dynamic destination and can enable receiver-specific ORC/OBR variants inside the EMDEON-style branch. | Integration name / identifier | Flow | Segment and routing behavior | | ----------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ATHENA` | Report-wise and bill-wise | Validates that the routing identifier is Athena before generation. Sender application/facility can be overridden from configuration, provider name is formatted as last/first, result/status formatting is Athena-specific, and bill-wise output can use `athena_port`. | | `EMED` | Report-wise and bill-wise | Uses `lab_name` as sending application and maps sending facility to ordering facility. In bill-wise ORU, receiving facility is aligned to the sending facility. | | `EMDEON` | Report-wise | Builds the EMDEON/DOH-style ORU envelope with receiving application `FILE`, receiving facility from the ordering facility, sender fallback `ECL`, CLIA-aware sending facility values, dynamic identifier ports, and optional `doh_port` routing. | | `expirity` | Report-wise | Changes ORC placement so placer/filler identifiers are sent in the receiver-specific order expected by Expirity. | | `ellkay` | Report-wise | Changes OBR layout and method/comment behavior for Ellkay-style result delivery. | | `CALREDIE` | Report-wise | Uses public-health segment formatting with CLIA/LIMS identifiers, provider NPI format, normalized race/ethnicity, optional SFT, and public-health OBX/SPM structure. | | `LOUSIANA` | Report-wise | Shares the public-health branch with CALREDIE and adds Louisiana-specific phone/address formatting, LOINC suffixing, and abnormal flag values. The implementation expects the spelling `LOUSIANA`. | | `ILLINOIS` | Report-wise | Adjusts CLIA/facility formatting and supports software information through `SFT` segment configuration. | | `TEXAS` | Report-wise | Uses Texas-specific MSH/PID/ORC/OBR structure with CLIA/OID identifiers, contact formatting, and LOINC-aware test descriptions. | | `MARYLAND` | Report-wise | Prefixes the sending application with organization context and uses Maryland-specific PID/ORC/OBR/test-description structure. | | `NABIDH` | Report-wise | Uses report integration code where present, sends nationality/national-id visit context, formats sample type as code/name, and changes base64/CAP-accreditation OBX handling. | | `CATALYST` | Report-wise and bill-wise | Uses the lab bill id as the placer/order identifier where Catalyst expects bill-level identifiers. | | `ECW` | Report-wise and bill-wise | Adds pathologist name and report comments as `NTE` content after result or document OBX generation. | Tox and Microbiology Notes [#tox-and-microbiology-notes] Toxicology and microbiology payloads can send list-valued result parameters rather than one scalar result. The branch supports selecting result keys with `result_name`/`results`, filtering by `parameter_list`, adding drug-level ORC/OBR grouping with `Drug_ORC`, producing Quan/Cutoff sub-parameters with `sub_param`, and changing abnormal flag behavior based on prescribed/non-prescribed status. ORUR01 Configuration [#orur01-configuration] Message: `ORU^R01` by default. Result Content and Validation [#result-content-and-validation] | Key | Label | Type | Default | Description | | ----------------------- | -------------------------------- | ------ | ------: | ---------------------------------------------------------------------------------------------------- | | `integration_name` | Integration Name | text | `""` | Enables ORU-specific formatting, including Athena, EMED, ECW, Catalyst, tox, and micro behavior. | | `param_dict_id` | Disable Dictionary ID Validation | toggle | `1` | Disables mandatory parameter dictionary id validation. | | `disable_dict` | Disable Dictionary Check | toggle | `1` | Bypasses dictionary validation for test code mapping. | | `test_params` | Send Parameter Name | toggle | `1` | Includes detailed test parameters and names in OBX. | | `test_description` | Test Description | string | `""` | Overrides OBR/OBX test descriptor. If blank, falls back to `testCode^testName`. | | `test_category` | Test Category | toggle | `0` | Adds test category into OBR context. | | `department` | Department | toggle | `0` | Adds department code/name into OBR context. | | `clinical_history_flag` | Clinical History Flag | toggle | `0` | Sends patient clinical history in OBX. | | `actual_result` | Actual Result | toggle | `0` | Used for COVID-style fixed results such as Detected, Not detected, Positive, Negative, Inconclusive. | | `test_code_check` | Test Codes Filter | list | `[]` | Fixed list of test codes to send; used with positive-result filtering. | | `only_positive_results` | Only Positive Results | toggle | `0` | Sends only positive results in OBX. | | `ref_range_flag` | Reference Ranges | toggle | `1` | Sends reference ranges in OBX. | | `criticalFlag` | Critical Flag | toggle | `1` | Sends critical-aware abnormal flags in OBX. | | `mark_as_done_reports` | Mark As Done Reports | toggle | `0` | Sends mark-as-done reports into ZBR segment. | Segment Controls [#segment-controls] | Key | Label | Type | Default | Description | | -------------------------------- | ----------------------- | -------------- | ------: | ------------------------------------------------------------------------------ | | `segments` | HL7 Segments | checkbox group | `[]` | Optional segments. Supported: `PV1`, `RXO`, `AL1`, `IN1`, `FT1`, `DG1`, `CST`. | | `ReportLevelTags` | Report Level Tags | toggle | `1` | Sends report-level tags in custom segment/content where available. | | `WardNumber` | Ward Number Flag | toggle | `1` | Sends ward number in custom segment. | | `signing_doctor1` | Signing Doctor 1 | toggle | `0` | Adds first signing doctor name into NTE. | | `signing_doctor2` | Signing Doctor 2 | toggle | `0` | Adds second signing doctor name into NTE. | | `reportsaveduser` | Report Saved User | toggle | `0` | Adds the lab user who saved the report into NTE. | | `submitted_user` | Submitted User | toggle | `0` | Adds the lab user who submitted the report into NTE. | | `order_comments_flag` | Order Comments Flag | toggle | `1` | Sends bill comments in NTE. | | `test_comments_flag` | Test Comments Flag | toggle | `1` | Sends report comments in NTE. | | `AOE` | AOE Enable | toggle | `0` | Adds Ask-On-Entry values in OBX. | | `prescribed_drugs` | Prescribed Drugs | toggle | `0` | Sends prescribed drugs into RXO. | | `enable_obx_risk_and_type` | Risk and Patient Type | toggle | `0` | Sends risk level and patient type into OBX. | | `enable_prescription_brand_name` | Prescription Brand Name | toggle | `0` | Sends medication brand name into RXO. | | `Antibiotics` | Antibiotics | toggle | `0` | Sends antibiotics and organism details into OBX. | Documents and Base64 [#documents-and-base64] | Key | Label | Type | Default | Description | | ------------------------ | ------------------------- | ------ | ------: | --------------------------------------------------------------------------------------------- | | `base64_result` | Send Results as Base64 | toggle | `1` | Sends full report as base64 PDF in OBX. | | `billwise_base64` | Bill-wise Base64 Encoding | toggle | `0` | Sends consolidated bill-wise report PDF when enabled with `base64_result`. | | `signed_result` | Include Signed Result | toggle | `0` | With `billwise_base64`, sends signed reports when enabled; otherwise sends submitted reports. | | `attachments` | Send Attachment | toggle | `0` | Sends bill attachments and patient id proofs into OBX. | | `trf_flag` | TRF Flag | toggle | `0` | Sends TRF PDF into OBX. | | `base64_supported_tests` | Base64 Supported Tests | list | `[]` | Restricts base64 behavior to configured test codes where supported. | Routing and Transport [#routing-and-transport] | Key | Label | Type | Default | Description | | ---------------------------- | -------------------------- | ----------- | --------: | ---------------------------------------------------------------------------- | | `sendingApplication` | Sending Application | text | `""` | Populates `MSH-3`; integration-specific overrides may apply. | | `sendingFacility` | Sending Facility | text | `""` | Populates `MSH-4`; integration-specific overrides may apply. | | `receivingApplication` | Receiving Application | text | `""` | Populates `MSH-5`. | | `receivingFacility` | Receiving Facility | text | `""` | Populates `MSH-6`. | | `host` | Host | text | `""` | TCP destination host. | | `port` | Port | text/number | `0` | Primary TCP destination port. | | `dynamic_port` | Port Dynamic Routing | text/number | `""` | Dynamic routing key through `integrationDetails[integration_identifier]`. | | `integration_differentiator` | Integration Differentiator | string | `-` | Delimiter used to derive integration identifier from order number. | | `integration_identifier` | Integration Identifier | text | `""` | Overrides derived dynamic routing identifier. | | `versionId` | HL7 Version | dropdown | `2.4` | HL7 version in `MSH-12`. | | `msgType` | Message Type | text | `ORU^R01` | HL7 message type in MSH. | | `msh21` | MSH-21 Encoding Characters | text | `""` | Custom value for `MSH-21` where supported. | | `lab_time_zone` | Lab Timezone Dates | toggle | `0` | Converts dates to lab timezone. Default sends UTC-style dates in some paths. | Insurance, Branch, and Outsource [#insurance-branch-and-outsource] | Key | Label | Type | Default | Description | | ----------------------------- | --------------------------- | ------- | ------: | ----------------------------------------------------------------------------- | | `is_insurance_enable` | Is Insurance Enable | boolean | `0` | If enabled and insurance is missing, returns `400` and does not generate HL7. | | `BillingInsurance` | Billing Insurance | toggle | `0` | Sends insurance information used for the report/bill. | | `branchIntegration` | Enable Branch Integration | toggle | `0` | Uses branch facility details instead of main lab details. | | `outsource_flag` | Outsource Flag | toggle | `0` | Sends outsource information in OBR. | | `restrict_outsourced_reports` | Restrict Outsourced Reports | toggle | `0` | Unidirectional mode: skips outsourced tests from HL7. | | `restrict_outsourced_data` | Restrict Outsourced Data | toggle | `0` | Bidirectional mode: skips outsourced tests from HL7. | Device, Lab, and LOINC Metadata [#device-lab-and-loinc-metadata] | Key | Label | Type | Default | Description | | --------------- | ------------------- | ------ | ------: | ------------------------------------------------------ | | `device_id` | Device ID | text | `""` | Analyzer/device identifier. | | `device_mfr` | Device Manufacturer | text | `""` | Analyzer manufacturer name. | | `device_abr` | Device Abbreviation | text | `""` | Analyzer abbreviation. | | `lab_clia_no` | Lab CLIA Number | text | `""` | CLIA number for the lab. | | `lab_name` | Lab Name | text | `""` | Display lab name used in facility and segment context. | | `lab_address` | Lab Address | text | `""` | Complete lab address. | | `lims_clia` | LIMS CLIA Number | text | `""` | CLIA number from LIMS. | | `lims_name` | LIMS Name | text | `""` | LIMS name identifier. | | `sub_id` | Subscription ID | text | `""` | Subscription/account id or OBX sub-id context. | | `LOINC_support` | LOINC Support | toggle | `0` | Enables LOINC code mapping. | Toxicology and Microbiology Result Options [#toxicology-and-microbiology-result-options] | Key | Label | Type | Default | Description | | -------------------- | ------------------------ | ------ | -------------------------: | --------------------------------------------------------------------------- | | `result_name` | Result Field Name | text | `result_2` | Selects which result key is sent in OBX for tox/micro payloads. | | `results` | Results List | list | `[]` | Sends multiple configured result keys in OBX. | | `parameter_filter` | Parameter Filter | toggle | `0` | Filters parameters using `parameter_list`. | | `parameter_list` | Parameter List | list | `[]` | Allowed parameter names, for example prescription, confirmation, screening. | | `sub_param` | Sub Parameters | toggle | `0` | Adds sub-parameter behavior such as Quan and CutOff. | | `Drug_ORC` | Drug ORC Segment | toggle | `0` | Adds ORC/OBR grouping for each report parameter. | | `prescribed_check` | Prescribed Check | toggle | `0` | Adds prescribed/non-prescribed context and affects abnormal flag logic. | | `descriptive_result` | Descriptive Result Field | text | `result_2` | Result key used in prescribed-check abnormal flag evaluation. | | `positive_results` | Positive Results | list | `["positive", "detected"]` | Values treated as positive/abnormal in tox flows. | ORUR01 Bill-Wise Configuration [#orur01-bill-wise-configuration] API: `/integration/sftp/hl7/2_3/send_billwise_hl7_data/` The bill-wise ORU branch shares most ORUR01 options above and adds bill-level routing and grouping behavior. | Key | Type | Default | Description | | ----------------------------- | ------: | --------------: | ------------------------------------------------------------------------------- | | `host` | string | `""` | TCP host used to send bill-wise ORU. | | `port` | number | `0` | Primary TCP port. Also forwarded for bill-wise DFT when payload type is DFT. | | `url` | string | default DFT API | Forwarding URL used when generating bill-wise DFT. | | `integration_name` | string | `""` | Controls integration-specific behavior such as ATHENA, EMED, ECW, and Catalyst. | | `integration_differentiator` | string | `-` | Splits order number to derive integration identifier. | | `integration_identifier` | string | `""` | Overrides derived integration identifier. | | `athena_port` | number | `0` | Overrides final socket port when `integration_name` is ATHENA. | | `versionId` | string | `2.3` | HL7 version used in MSH. | | `base64_result` | boolean | `0` | Adds ED/OBX base64 PDF block. | | `billwise_base64` | boolean | `0` | Fetches consolidated bill-wise PDF instead of request reportBase64. | | `receivingApplication` | string | `""` | MSH receiving application. | | `receivingFacility` | string | `""` | MSH receiving facility. | | `disable_dict` | boolean | `1` | Bypasses dictionary-id gate. | | `segments` | list | `[]` | Optional segments: `AL1`, `PV1`, `IN1`, `FT1`, `DG1`, `RXO`, `CST`. | | `is_insurance_enable` | boolean | `0` | Requires insurance before generation. | | `BillingInsurance` | boolean | `0` | Uses bill-level insurance mapping. | | `restrict_outsourced_reports` | boolean | `0` | Skips reports where `outsourcedReportFlag == "3"`. | | `profile_with_test` | boolean | `0` | Creates OBR at profile + test granularity. | | `priority_flag` | boolean | `0` | Uses print priority in OBR and reorders ORU blocks by priority. | | `criticalFlag` | boolean | `0` | Uses critical-sensitive abnormal flags. | | `attachments` | boolean | `0` | Appends merged proof/bill attachments as ED OBX. | | `trf_flag` | boolean | `0` | Appends TRF PDF as ED OBX. | Minimal Example [#minimal-example] ```json { "payloadType": "ORUR01", "labReportId": 123456, "dictionaryId": 0, "reportDetails": [], "integrationDetails": { "integration_name": "ATHENA", "sendingApplication": "Creliohealth", "receivingApplication": "ATHENA", "receivingFacility": "MAIN", "versionId": "2.4", "host": "127.0.0.1", "port": 5000, "base64_result": 1, "segments": ["PV1", "IN1"] } } ``` # OUL HL7 Outbound OUL [#hl7-outbound-oul] OUL outbound messages transmit specimen/result data with OUL formatting. Unlike most outbound branches, OUL configuration primarily uses snake\_case keys. Generation Flow [#generation-flow] Implementation Details [#implementation-details] The OUL branch is selected when `payloadType` is `OUL` in `/integration/sftp/hl7/2_3/send_hl7_data/`. It is a specimen/result-style message with OUL R21 structure and snake\_case configuration keys. Request Contract [#request-contract] | Field | Location | Required | Notes | | ----------------------- | -------- | ----------: | -------------------------------------------------------------------------------------------------- | | `payloadType` | Root | Yes | Must be `OUL`. | | `labReportId` | Root | Yes | Used directly to load the source report. | | `reportFormatAndValues` | Root | Usually | Source rows for OUL result OBX values. | | `reportBase64` | Root | Conditional | Used when `base64_result` is enabled. | | `integrationDetails` | Root | No | Uses snake\_case sender/facility keys and controls reference range, base64, segments, and routing. | Segment Construction [#segment-construction] | Segment | Source | Notes | | ----------- | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `MSH` | `sending_application`, `sending_facility`, receiving fields, generated control id | Message type is `OUL^R21^OUL_R21`; default version is `2.4`. | | `PID` | Patient demographics | Same canonical patient context as other report-wise branches. | | `SAC` | Manual sample/accession and container type | Includes accession and container metadata for specimen handling. | | `ORC` | Bill/order/provider organization context | Uses action type `RE` and bill/order timestamps. | | `OBR` | Test/sample/report context | Uses integration code/test name, collection/accession/report dates, sample type, provider, and report corrected status. | | `OBX` | `reportFormatAndValues` | Skips email-only parameters, strips HTML, emits non-description values, reference ranges, units, and abnormal flag. | | `TCD`/`SID` | Result metadata | `TCD` follows result OBX; `SID` is added when configured in `segments`. | | `CST` | `CST` in `segments` | Adds patient/bill operational metadata. | Reference Range and Result Rules [#reference-range-and-result-rules] The branch calculates reference range from male/female lower/upper bounds when present. If `ref_range_flag` is enabled, ranges are formatted as `lower-upper`; otherwise only the lower bound is sent. If a result parameter is marked as a description, it is not emitted as a normal numeric OBX. Dynamic Routing [#dynamic-routing] The branch derives an identifier from the order number using `integration_differentiator` and can override `port` from `integrationDetails[integration_identifier]`. This is one of the branches where the configuration key style differs from most other outbound flows. Failure Cases [#failure-cases] | Case | Result | | --------------------------- | ---------------------------------- | | Missing `labReportId` | HTTP 400. | | No generated message | JSON status `400`, `Invalid Data`. | | Socket/processing exception | JSON status `500` with traceback. | OUL Configuration [#oul-configuration] OUL uses snake\_case keys. | Key | Label | Type | Default | Description | | ---------------------------- | -------------------------- | -------------- | ---------------: | ---------------------------------------------------------------------------- | | `sending_application` | Sending Application | text | `""` | Populates sending application in MSH. | | `receiving_application` | Receiving Application | text | `""` | Receiving application identifier. | | `receiving_facility` | Receiving Facility | text | `""` | Receiving facility identifier. | | `integration_name` | Integration Name | text | `""` | Enables OUL-specific formatting and processing. | | `segments` | HL7 Segments | checkbox group | `["SPM", "OBX"]` | Optional OUL segments. Supported: `SPM`, `OBX`, `NTE`, `CST`. | | `sending_facility` | Sending Facility | text | `""` | Populates sending facility in MSH. | | `versionId` | HL7 Version | dropdown | `2.4` | HL7 message version. | | `host` | Host | text | `""` | TCP destination host. | | `port` | Port | text/number | `""` | TCP destination port. | | `integration_differentiator` | Integration Differentiator | radio | `-` | Delimiter used to parse order number for dynamic routing. | | `integrationDetails` | Integration Details | object | `{}` | Dynamic routing object. Port override can be read by integration identifier. | | `ref_range_flag` | Include Reference Ranges | toggle | `1` | Includes reference ranges in OBX. | | `base64_result` | Send Results as Base64 | toggle | `1` | Sends full report as base64 PDF in OBX. | | `lab_name` | Lab Name | text | `""` | Display lab name. | Minimal Example [#minimal-example] ```json { "payloadType": "OUL", "labReportId": 123456, "integrationDetails": { "sending_application": "Creliohealth", "receiving_application": "MIRTH", "receiving_facility": "MAIN", "versionId": "2.4", "host": "127.0.0.1", "port": 5000, "segments": ["SPM", "OBX", "NTE"] } } ``` # Outbound HL7 Overview HL7 Outbound Overview [#hl7-outbound-overview] Outbound HL7 integrations generate billing, order, patient, specimen, and result messages from LiveHealth/Crelio data. The detailed configuration for each message family now lives in its own page. Scope [#scope] This section documents the legacy outbound HL7 APIs implemented in [livehealthapp/labs/views.py](https://github.com/CrelioHealth/livehealthapp/blob/develop/labs/views.py). The APIs are used by SFTP/webhook-style integrations where LiveHealth prepares an HL7 message and optionally pushes it to a downstream listener such as Mirth over TCP. The documentation focuses on two high-use endpoints: | Endpoint | Function | Purpose | | --------------------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------ | | `/integration/sftp/hl7/2_3/send_hl7_data/` | `sftp_HL7_payload_generation_function` | Report-wise HL7 generation for `DFT`, `ORM`, `OUL`, `ADT`, and `ORUR01`. | | `/integration/sftp/hl7/2_3/send_billwise_hl7_data/` | `sftp_billwise_HL7_payload_generation_function` | Bill-wise `ORUR01` generation and bill-wise `DFT` forwarding. | These endpoints are legacy in naming, but still contain the active compatibility behavior for multiple vendor integrations. They combine HL7 generation, vendor-specific mapping, socket delivery, activity logging, and response rendering in one flow. Mental Model [#mental-model] A request to these APIs is not a raw HL7 payload. It is a JSON request that identifies a report or bill, includes the serialized report/result data, and supplies `integrationDetails` flags. The API then reloads canonical database records, builds the HL7 segments, normalizes the message, and either returns it or sends it to the configured TCP destination. Data Sources Used During Generation [#data-sources-used-during-generation] The API reloads most business context from the database even when similar values are present in the request. The request is primarily used to choose the flow, supply result arrays/base64 payloads, and override integration behavior. | Data area | Source used by the API | Used for | | -------------------------- | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Report identity | `labReportId`, `labReportDetails`, or `testDetails` | Resolving the first `labReportRelation` record for report-wise generation. | | Bill identity | `bill_id` | Loading all synced, non-dismissed reports for bill-wise ORU and creating bill-wise DFT forwarding payloads. | | Patient demographics | `labReportRelation.userDetailsId` and selected request fallbacks | `PID`, contact, address, DOB, gender, race, ethnicity, national id, passport, patient ids. | | Bill/order data | `labReportRelation.billId`, `billingInfo`, `BillingICD` | `ORC`, `OBR`, `FT1`, `DG1`, visit id, order number, bill source, ICD and procedure mapping. | | Organization/provider data | `orgId`, `docId`, branch records | Sending/ordering facility, provider identifiers, organization address/contact, branch override behavior. | | Results | `reportFormatAndValues`, `reportDetails`, report format metadata | `OBX`, `NTE`, abnormal flags, reference ranges, result status, base64 report blocks. | | Insurance | `PatientInsurance` or `UserBillInsurance` | `IN1` segments and insurance-required validation. | | Attachments/TRF | Attachment models and TRF endpoints | ED `OBX` blocks for PDF/image/base64 documents. | Runtime Contract [#runtime-contract] | Step | Behavior | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | Request parsing | `sftp_HL7_payload_generation_function` accepts either a JSON object or a one-item JSON array and uses the first object when an array is received. | | Message control id | A 14-digit value is derived from `uuid.uuid4().int`; `processingId` defaults to `P`. | | Timestamp format | HL7 timestamps are built as `YYYYMMDDHHMMSS`; some branches convert report, bill, sample, and accession dates to the lab timezone. | | Message cleanup | Before return/send, line breaks and selected HTML tokens are normalized; `
` tags are removed or converted depending on branch. | | ID enrichment | `update_message_with_ids(jsonData, message, "HL7")` is applied before final response/send in the main outbound paths. | | Transport | If a non-zero port is resolved, the message is sent to `host:port` by TCP socket. If not, the generated HL7 is returned with `sent_to_mirth: false`. | | Logging | Successful generation writes activity log entries for webhook triggering and generated HL7 text. | | Error handling | Unhandled exceptions return status `500` with the generated message-so-far and traceback text. | Report-Wise vs Bill-Wise Behavior [#report-wise-vs-bill-wise-behavior] | Behavior | Report-wise endpoint | Bill-wise endpoint | | ------------------ | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Primary identifier | `labReportId` for `ORUR01`/`OUL`; first `labReportDetails[].labReportId` or `testDetails[].labReportId` for others. | `bill_id`. | | Supported payloads | `DFT`, `ORM`, `OUL`, `ADT`, `ORUR01`. | `ORUR01` and `DFT`. | | Report selection | Starts from one report and often uses its bill context. | Loads all reports on the bill where `dismissed=0`, `sampleRedrawFlag=0`, and `isSynced=1`. | | DFT behavior | Builds DFT in-process for the report/bill context. | Builds a webhook payload and forwards it to the configured DFT URL; it does not directly build the final DFT message in this function. | | ORU behavior | Builds one report-oriented result message. | Builds a consolidated bill-wise ORU from the supplied/report-derived result details. | Response Shape [#response-shape] Successful generation returns JSON similar to: ```json { "status": 200, "message": "MSH|^~\\&|...", "file Name": "", "sent_to_mirth": true } ``` For report-wise generation the response may also include `MSH_1` and `MSH_2`. Bill-wise DFT forwarding returns the generated forwarding payload instead of the final HL7 text: ```json { "status": 200, "message": "DFT HL7 sent successfully", "sent_to_mirth": false, "payload": { "Status": "Bill Generation HL7 for SFTP", "bill_id": 12345, "payloadType": "DFT" } } ``` Operational Cautions [#operational-cautions] * The `DFT` branch in the report-wise SFTP endpoint reads `host` and `port` from the root request, while most other branches read them from `integrationDetails`. * `integration_name` is case-normalized in many branches and activates destination-specific behavior. Treat it as a behavior switch, not only a label. * `segments` is an allow-list for optional segment groups. Some base segments are always produced even if the list is empty. * Some branches return status `200` for business skips, for example no insurance in the DFT SFTP branch. Consumers should inspect both HTTP status and response body. * Because socket delivery happens synchronously in the request, an unreachable host or port can raise an exception and return status `500`. Integration Name Specifications [#integration-name-specifications] The outbound generators use `integration_name` as a compatibility switch for receiver-specific HL7 structure. ORU flows may also derive an `integration_identifier` from the order number with `integration_differentiator`; that identifier is mainly used for dynamic port selection and a few receiver-specific ORU layouts. | Integration name / identifier | Applies to | Specification | | ----------------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ATHENA` | `DFT`, report-wise `ORUR01`, bill-wise `ORUR01` | DFT uses Athena-aware insurance relation handling and can split comma-separated report integration codes into multiple `FT1` rows. ORU validates that the derived/configured identifier belongs to Athena, allows sender/facility overrides, formats provider name as last/first, applies Athena-specific result/specimen behavior, and can route bill-wise output through `athena_port`. | | `EMED` | `DFT`, report-wise `ORUR01`, bill-wise `ORUR01` | DFT prefers report integration code and falls back to procedure/test code. ORU uses the configured lab name as the sending application, maps sending facility to the ordering facility, and uses the sending facility as the receiving facility in the bill-wise branch. | | `CERNER` | `ORM`, `ADT` | ORM uses Cerner-specific patient, visit, order, and observation identifiers including MRN/CMRN/FIN-style fields, collection/cancel/dispatch statuses, and optional HLA/outsourced message routing through `HLA_port`. ADT uses Cerner-specific PID/PV1 construction and can add medical history, height, and weight observations. | | `CPL` | `ORM` | Sets billing type as insurance, credit, or patient pay based on insurance and bill-payment context, and adds configured order comments as `NTE` content. | | `SIGLO` | `ORM` | Prefixes patient contact with country code and prefers national id/SSN formatting expected by the destination. | | `TRIBAL` | `ORM` | Updates the bill order number from the manual sample/accession identifier and changes OBR placer/filler ordering for the receiver. | | `EMDEON` | report-wise `ORUR01` | Builds an EMDEON/DOH-style ORU with receiving application `FILE`, sending application fallback `ECL`, CLIA-aware sending facility values, dynamic port lookup by integration identifier, and optional secondary DOH routing through `doh_port`. | | `expirity` identifier | report-wise `ORUR01` | Uses a derived/configured `integration_identifier` to change ORC placement and report-order formatting inside the EMDEON-style ORU flow. | | `ellkay` identifier | report-wise `ORUR01` | Uses a derived/configured `integration_identifier` to change OBR layout and method/comment handling inside the EMDEON-style ORU flow. | | `CALREDIE` | report-wise `ORUR01` | Applies public-health style MSH/PID/ORC/OBR formatting with CLIA/LIMS identifiers, provider NPI formatting, normalized race/ethnicity fields, and public-health OBX/SPM construction. | | `LOUSIANA` | report-wise `ORUR01` | Uses the same public-health branch as CALREDIE with Louisiana-specific contact formatting, address shaping, LOINC suffixing, and abnormal-flag vocabulary. The code uses the spelling `LOUSIANA`; configuration must match that value unless the implementation is changed. | | `ILLINOIS` | report-wise `ORUR01` | Adjusts CLIA/facility formatting and can emit `SFT` software details when configured. | | `TEXAS` | report-wise `ORUR01` | Uses Texas-specific MSH/PID/ORC/OBR structure, CLIA/OID identifier placement, provider/contact formatting, and LOINC-oriented test descriptions. | | `MARYLAND` | report-wise `ORUR01` | Prefixes sending application with organization context and uses Maryland-specific PID, ORC, OBR, and LOINC/test-description formatting. | | `NABIDH` | report-wise `ORUR01` | Uses integration code as the test code where present, sends nationality/national-id visit context, formats sample type with code/name pairs, and has special base64/CAP-accreditation OBX behavior. | | `CATALYST` | report-wise `ORUR01`, bill-wise `ORUR01` | Uses the lab bill id as the placer/order identifier in ORU segments where this receiver expects bill-level identifiers instead of report-level ids. | | `ECW` | report-wise `ORUR01`, bill-wise `ORUR01` | Appends pathologist name and report comments as `NTE` content after result/document OBX generation. | | default / blank | All supported payloads | Uses the generic outbound segment templates and direct `host`/`port` routing from the applicable request location. Configure explicit integration names only when the receiver needs one of the behaviors above. | Configuration Guidance [#configuration-guidance] | Field | Guidance | | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `integration_name` | Use the exact receiver switch expected by the branch. The implementation compares most values case-insensitively, but spelling still matters for values such as `LOUSIANA`. | | `integration_identifier` | Use when one integration setup needs dynamic ORU routing by destination identifier. If blank, the report-wise ORU branch derives it from the order number prefix before `integration_differentiator`. | | `integration_differentiator` | Defaults to `-`; use it only when order numbers encode destination routing, for example `ATHENA-12345`. | | `` port keys | ORU dynamic routing can look up `integrationDetails[integration_identifier]`; some branches also use dedicated keys such as `athena_port`, `doh_port`, and `HLA_port`. | | `sendingApplication` / `sendingFacility` | These are normal MSH fields, but several integration names override them. Verify the generated `MSH` after enabling a receiver-specific branch. | Message Type Pages [#message-type-pages] | Payload type | HL7 message | Documentation | Typical trigger | | ------------ | --------------------------- | ------------------ | ----------------------------------------------- | | `ADT` | `ADT` | [ADT](./ADT) | Patient/event information outbound | | `ORM` | `ORM^O01` | [ORM](./ORM) | Order creation, update, cancellation, or resend | | `ORUR01` | `ORU^R01` by default | [ORUR01](./ORUR01) | Result submission | | `OUL` | OUL result/specimen message | [OUL](./OUL) | Specimen/result outbound with OUL formatting | | `DFT` | `DFT^P03` | [DFT](./DFT) | Billing/charge outbound | Outbound Routing Flow [#outbound-routing-flow] Purpose [#purpose] This document describes the outbound HL7 generation APIs used by the livehealthapp repository for billing, order, patient, specimen, and result transmission. It covers the following implementation entry points: | API | Function | File | Primary use | | --------------------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `/integration/sftp/hl7/2_3/send_hl7_data/` | `sftp_HL7_payload_generation_function` | [labs/views.py](https://github.com/CrelioHealth/livehealthapp/blob/develop/labs/views.py) | SFTP-triggered HL7 generation for `DFT`, `ORM`, `OUL`, `ADT`, and `ORUR01`. | | `/integration/sftp/hl7/2_3/send_billwise_hl7_data/` | `sftp_billwise_HL7_payload_generation_function` | [labs/views.py](https://github.com/CrelioHealth/livehealthapp/blob/develop/labs/views.py) | Bill-wise ORU generation and bill-wise DFT forwarding. | | `/integration/hl7/send_dft_hl7_data/` | `DFT_HL7_payload_generation_function` | [labs/integration\_functions.py](https://github.com/CrelioHealth/livehealthapp/blob/develop/labs/integration_functions.py) | Direct DFT P03 generation for HL7 2.3 style billing integrations. | | `/integration/hl7/2_4/send_dft_hl7_data/` | `DFT_HL7_2_4_payload_generation_function` | [labs/integration\_functions.py](https://github.com/CrelioHealth/livehealthapp/blob/develop/labs/integration_functions.py) | Direct DFT P03 generation for HL7 2.4 style billing integrations and vendor-specific DFT behavior. | Message Types [#message-types] | Payload type | HL7 message | Typical trigger | Main segments | | ------------ | --------------------------- | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | `ADT` | `ADT` | Patient/event information outbound | `MSH`, `PID`, optional `PV1`, `OBX`, `NTE` | | `ORM` | `ORM^O01` | Order creation, update, cancellation, or resend | `MSH`, `PID`, `PV1`, `ORC`, `OBR`, optional `AL1`, `IN1`, `GT1`, `RXO`, `OBX`, `FT1`, `DG1`, `CST`, `NTE` | | `ORUR01` | `ORU^R01` by default | Result submission | `MSH`, `PID`, `ORC`, `OBR`, `OBX`, `SPM`, optional `PV1`, `AL1`, `IN1`, `FT1`, `DG1`, `RXO`, `CST`, `NTE` | | `OUL` | OUL result/specimen message | Specimen/result outbound with OUL formatting | `MSH`, specimen and result segments, optional `SPM`, `OBX`, `NTE`, `CST` | | `DFT` | `DFT^P03` | Billing/charge outbound | `MSH`, `EVN`, `PID`, `PV1`, `FT1`, `DG1`, optional `IN1`, `GT1`, `OBX`, custom vendor segments | Request Shape [#request-shape] Most outbound calls expect JSON in the request body. The common structure is: ```json { "payloadType": "ORUR01", "labReportId": 123456, "labReportDetails": [], "reportDetails": [], "dictionaryId": 0, "integrationDetails": { "host": "127.0.0.1", "port": 5000, "versionId": "2.4" } } ``` Bill-wise ORU/DFT uses `bill_id` as the primary identifier: ```json { "payloadType": "ORUR01", "bill_id": 12345, "billId": 1001, "reportDetails": [], "integrationDetails": { "host": "127.0.0.1", "port": 5000 } } ``` Common Runtime Behavior [#common-runtime-behavior] * If `port` is configured and non-zero, the generated HL7 is sent over TCP socket to `host:port`. * If `port` is missing or zero, the API can still return the generated HL7 message without sending it to Mirth or another TCP listener. * `update_message_with_ids(jsonData, message, "HL7")` is applied before final return/send in the main outbound paths. * HTML line break tags and some report formatting tokens are normalized before sending. * `integrationDetails` is the main configuration object for flags. * Some functions read `host` and `port` from the root payload, while others read them from `integrationDetails`. See the API-specific notes below. API-Specific Notes [#api-specific-notes] /integration/sftp/hl7/2_3/send_hl7_data/ [#integrationsftphl72_3send_hl7_data] Function: `sftp_HL7_payload_generation_function` Supported `payloadType` values: * `DFT` * `ORM` * `OUL` * `ADT` * `ORUR01` Identifier rules: * `ORUR01` and `OUL` read `labReportId` from the root payload. * Other payloads read the first `labReportId` from `labReportDetails`; if empty, `testDetails` is used as a fallback. * If no report id can be resolved, the API returns `400` with `labReportId is not valid`. Transport rules: * `ORM`, `OUL`, `ADT`, and `ORUR01` primarily read `host` and `port` from `integrationDetails`. * The `DFT` branch in this function reads `host` and `port` from the root payload. /integration/sftp/hl7/2_3/send_billwise_hl7_data/ [#integrationsftphl72_3send_billwise_hl7_data] Function: `sftp_billwise_HL7_payload_generation_function` Supported bill-wise behavior: * `ORUR01`: Generates one bill-wise ORU message using all synced, non-dismissed reports for the bill. * `DFT`: Builds a bill-wise payload and forwards it to a configured URL, defaulting to `/integration/hl7/send_dft_hl7_data/`. Identifier rules: * Requires `bill_id`. * If `bill_id` is missing, the API returns `400` with `bill_id is not valid`. DFT forwarding rules: * Reads `host`, `port`, and optional `url` from `integrationDetails`. * Sends a webhook payload containing patient, billing, order, report, and integration details to the configured DFT URL. /integration/hl7/send_dft_hl7_data/ [#integrationhl7send_dft_hl7_data] Function: `DFT_HL7_payload_generation_function` Supported payload: * `payloadType: "DFT"` Behavior: * Generates `DFT^P03`. * Uses HL7 version `2.3.1` by default. * Supports insurance validation, bill-level insurance, timezone conversion, and NextGen-specific segment ordering/formatting. * Reads `host` and `port` from the root payload. /integration/hl7/2_4/send_dft_hl7_data/ [#integrationhl72_4send_dft_hl7_data] Function: `DFT_HL7_2_4_payload_generation_function` Supported payload: * `payloadType: "DFT"` Behavior: * Generates `DFT^P03`. * Supports vendor-specific DFT behavior for integrations such as `PGM`, `ATHENA`, `CERMAK`, and custom dynamic segment sequencing. * Reads `host`, `port`, and dynamic integration-specific port keys from `integrationDetails`. * Supports additional DFT options such as outsource mapping, custom segment order, base64 result attachment, profile ICD mapping, and drug-wise FT1 generation. Configuration Key Naming [#configuration-key-naming] Different branches expect different key styles. | Branch | Expected style | Examples | | ----------------------------- | ---------------------------------------- | ------------------------------------------------------------------------ | | `ORM`, `ADT`, `ORUR01`, `DFT` | camelCase and mixed legacy keys | `sendingApplication`, `receivingFacility`, `labName`, `BillingInsurance` | | `OUL` | snake\_case | `sending_application`, `receiving_facility`, `lab_name` | | DFT 2.4 dynamic port | integration name as key | `PGM`, `ATHENA`, or any configured `integration_name` | | ORU bill-wise dynamic port | lower-case integration identifier as key | `integrationDetails[integration_identifier]` | If the wrong key style is sent, the code usually falls back to blank/default values. Common Configuration Keys [#common-configuration-keys] | Key | Type | Default | Used by | Description | | ----------------------- | ------------: | ---------------: | --------------------- | ------------------------------------------------------------------------------- | | `integration_name` | string | `""` | All major branches | Enables vendor-specific formatting and validation. | | `sendingApplication` | string | varies | DFT, ORM, ADT, ORUR01 | Populates `MSH-3`. | | `sending_application` | string | `""` | OUL | OUL snake\_case equivalent of sending application. | | `sendingFacility` | string | org code / `""` | DFT, ORM, ADT, ORUR01 | Populates `MSH-4` or related sending facility fields. | | `sending_facility` | string | `""` | OUL | OUL snake\_case equivalent of sending facility. | | `receivingApplication` | string | `""` | DFT, ORM, ADT, ORUR01 | Populates `MSH-5`. | | `receiving_application` | string | `""` | OUL | OUL snake\_case equivalent of receiving application. | | `receivingFacility` | string | `""` | DFT, ORM, ADT, ORUR01 | Populates `MSH-6`. | | `receiving_facility` | string | `""` | OUL | OUL snake\_case equivalent of receiving facility. | | `host` | string | `""` | Most outbound paths | TCP destination host. | | `port` | number/string | `0` or `""` | Most outbound paths | TCP destination port. | | `versionId` | string | `2.3.1` or `2.4` | Most outbound paths | HL7 version populated in `MSH-12`. | | `segments` | list | varies | ORM, OUL, ADT, ORUR01 | Optional segment inclusion list. | | `lab_time_zone` | boolean | `0` | DFT, ORM, ORUR01 | Converts dates to the configured lab timezone before HL7 formatting. | | `BillingInsurance` | boolean | `0` | DFT, ORM, ORUR01 | Uses bill-level insurance mapping instead of default patient insurance records. | | `is_insurance_enable` | boolean | `0` | DFT, ORUR01 | Requires insurance data before generating HL7. | Recommended Segment Configuration [#recommended-segment-configuration] Use the smallest segment list required by the receiving system. | Use case | Suggested segment list | | ------------------------------------ | -------------------------------------------------------- | | Basic ORM order | `["PV1"]` or default mandatory segments only | | ORM with insurance and diagnoses | `["PV1", "IN1", "DG1"]` | | ORM with medications | `["PV1", "RXO"]` plus `prescribed_drugs: 1` | | ORM with custom operational metadata | `["PV1", "CST"]` | | Basic ORU result | `[]` plus default ORC/OBR/OBX/SPM behavior | | ORU with visit and insurance | `["PV1", "IN1"]` | | ORU with attachments/TRF | `attachments: 1` and/or `trf_flag: 1` | | DFT with insurance | `is_insurance_enable: 1`, `BillingInsurance` as required | Validation and Failure Cases [#validation-and-failure-cases] | Case | Behavior | | ----------------------------------------------------------------- | --------------------------------------------------------------------------- | | Missing `labReportId` in report-wise endpoint | Returns HTTP 400. | | Missing `bill_id` in bill-wise endpoint | Returns HTTP 400. | | `is_insurance_enable` enabled and no insurance is available | DFT/ORU branches return a failure response and do not generate/send HL7. | | `testCodes` configured for ORM but no matching tests are eligible | ORM can return status `209` with "Test codes are not configured." | | Disabled DFT integration identifier in DFT 2.4 | Returns status `209` and skips DFT generation. | | Athena ORU with mismatched integration identifier | Returns status `209` and skips generation for non-Athena data. | | `port` empty or zero | Message is generated and returned where supported, but TCP send is skipped. | # Labcorp Labcorp JSON Integration [#labcorp-json-integration] This document describes the Labcorp JSON integration implemented in `crelio-app/integration/Labcorp`. The integration submits outbound order data from Crelio to Labcorp, receives requisition documents, attaches those documents to the bill, and stores Labcorp accession numbers against the matching report records. Overview [#overview] The Labcorp integration handles the following responsibilities: * Generate and cache a Labcorp OAuth access token. * Build a Labcorp order payload from bill, patient, provider, insurance, AOE, specimen, and lab data. * Submit the JSON order payload to the configured Labcorp order endpoint. * Read Labcorp requisition responses, including requisition PDFs and ABNs. * Upload returned PDFs as billing attachments through the Crelio attachment API. * Update `LabReportRelation.ReportLevelTags` with Labcorp accession numbers. * Log success and failure activity for audit and troubleshooting. Implementation Reference [#implementation-reference] | File | Purpose | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | [crelio-app/integration/Labcorp/views.py](https://github.com/CrelioHealth/crelio-app/blob/develop/integration/Labcorp/views.py) | Order submission view, token generation, payload mapping, response handling, attachment job creation, state normalization, and toxicology AOE helpers. | | [crelio-app/integration/Labcorp/constants.py](https://github.com/CrelioHealth/crelio-app/blob/develop/integration/Labcorp/constants.py) | USPS state and territory mapping used by the Labcorp payload. | | [crelio-app/integration/Labcorp/**init**.py](https://github.com/CrelioHealth/crelio-app/blob/develop/integration/Labcorp/__init__.py) | Package marker. | | [crelio-app/integration/urls.py](https://github.com/CrelioHealth/crelio-app/blob/develop/integration/urls.py) | Registers the Labcorp endpoint as `labcorp/send-order/`. | Endpoint [#endpoint] Send Order to Labcorp [#send-order-to-labcorp] * **URL**: `/api-v3/integration/labcorp/send-order/` * **Method**: `POST` * **View**: `CreateOrderIntoLabcorp` * **Access**: Guest endpoint, CSRF exempt * **Primary trigger**: Outbound JSON integration webhook for Labcorp order submission Workflow [#workflow] Request Payload [#request-payload] The endpoint receives the standard Crelio bill webhook payload with Labcorp configuration inside `integrationDetails`. At least one item must be present in `labReportDetails`; the first item is used to resolve the lab, bill, patient, and report context. ```json { "billId": 12345, "Patient Name": "Jane Patient", "Patient Age": "35 years", "Patient Dob": "1991-01-01", "Patient gender": "Female", "Patient Id": 98765, "Mobile Number": "5551234567", "patient_email": "jane@example.com", "labPatientId": "LP-98765", "labReportDetails": [ { "labReportId": 456, "labId": { "labId": 10, "labName": "Main Lab" } } ], "integrationDetails": { "authUrl": "https://labcorp.example.com/oauth/token", "orderUrl": "https://labcorp.example.com/orders", "client_id": "client-id", "client_secret": "client-secret", "account_number": "123456", "site_name": "MAIN SITE", "labState": "NY", "receiving_application": "1100", "receiving_facility": "TA", "sending_application": "CRELBASP", "sending_facility": "TE043987", "lab_time_zone": 1, "relation": { "self": "1", "spouse": "2", "parent": "3" }, "gender_identities": { "Male": "M", "Female": "F", "Non-binary": "N", "Other": "OTH" }, "race_options": { "white": "W", "black": "B", "asian": "A" }, "sexual_orientations": { "Heterosexual": "HET", "Other": "OTH" }, "bill_source_json": { "insurance": "T", "online": "C" } } } ``` Configuration [#configuration] Integration URL [#integration-url] Configure the Labcorp integration URL in `labIntegration.url` for the webhook action that should submit Labcorp orders: ```text {domain}/api-v3/integration/labcorp/send-order/ ``` Integration Details [#integration-details] | Field | Required | Description | Default or behavior | | ----------------------- | ----------: | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | `authUrl` | Yes | Labcorp OAuth token URL. | Token generation fails if empty or invalid. | | `orderUrl` | Yes | Labcorp order submission URL. | Order submission fails if empty or invalid. | | `client_id` | Yes | Labcorp client ID. | Sent as form data during token generation and as `Client-id` header during order submission. | | `client_secret` | Yes | Labcorp client secret. | Sent during token generation. | | `account_number` | Yes | Labcorp account number for the sending lab. | Sent as `account.number`. | | `site_name` | Yes | Labcorp site name. | Sent as `account.site_name`. | | `labState` | Recommended | State code for the lab account address. | Falls back to lab area when blank. | | `receiving_application` | No | Labcorp receiving application. | `1100` | | `receiving_facility` | No | Labcorp receiving facility. | `TA` | | `sending_application` | No | Crelio sending application. | `CRELBASP` | | `sending_facility` | No | Crelio sending facility. | `TE043987` | | `lab_time_zone` | No | When truthy, collection and injury dates are converted to `lab.labTimeZone`. | `0` | | `relation` | Recommended | Relationship mapping used for guarantor and insurance relationship values. | Guarantor defaults to `3`; insurance defaults to `1`. | | `gender_identities` | Recommended | Maps additional patient gender identity answers to Labcorp codes. | Defaults to `ASKU`. | | `race_options` | Recommended | Maps patient race values to Labcorp race codes. | Blank when no mapping is found. | | `sexual_orientations` | Recommended | Maps additional patient sexual orientation answers to Labcorp codes. | Defaults to `UNK`. | | `bill_source_json` | Recommended | Maps Crelio bill source to Labcorp bill type. | Defaults to `P`. | Payload Mapping [#payload-mapping] The integration builds one Labcorp order payload and converts all string values to uppercase before submission. Account [#account] The `account` object is built from `integrationDetails` and the lab record: * `number`: `integrationDetails.account_number` * `site_name`: `integrationDetails.site_name` * `phone_number`: `lab.labContact` * `address.street_address`: lab address * `address.city`: `lab.labCity` * `address.state`: `integrationDetails.labState` or lab area * `address.postal_code`: `lab.pincode` Patient [#patient] The `patient` object is built from the bill payload, `LabReportRelation.userDetailsId`, and additional patient information: * `external_id`: webhook `Patient Id` * `alternate_id`: webhook `labPatientId`, or `Patient Id` when lab patient ID is empty * `last_name`, `first_name`, `middle_name` * `address`, including USPS-normalized state when possible * `date_of_birth`: webhook `Patient Dob` * `gender`: first character of `Patient gender` when it is `M` or `F`; otherwise `N` * `race`: mapped from `race_options`, or overridden by Race AOE answers when present * `ethnicity`: first character of patient ethnicity, default `U`, or Hispanic AOE answer when present * `patient_class`: additional patient info answer containing `patient class`, default `N` * `gender_identities`: mapped from additional patient info question containing `gender identities` * `sexual_orientations`: mapped from additional patient info question containing `sexual orientation` When a mapped gender identity or sexual orientation value is `OTH`, the original answer is included as the description. Ordering and Authorizing Provider [#ordering-and-authorizing-provider] The same provider object is sent as both `ordering_provider` and `authorizing_provider`. | Labcorp field | Crelio source | | ------------------ | ----------------------------------------------------------- | | `ids.npi` | `billing.docId.docRegNo` | | `ids.local_id` | `billing.docId_id` | | `ids.provider_num` | `billing.docId.docComments` | | `last_name` | First token from `billing.docId.docFullName` | | `first_name` | Second token from `billing.docId.docFullName`, when present | Guarantor [#guarantor] The guarantor is read from the patient's default `FamilyRelation`. If the relationship is `self`, patient details are used. Otherwise, the configured guarantor record is used. Mapped fields include: * Name, address, phone number, and middle initial. * `patient_relationship` from `integrationDetails.relation`. * `charge_adjustment_code` from the guarantor option code. * `employer_name` when the guarantor relation is `other` and the order is workers compensation. If no default family relation exists, `guarantor` is sent as an empty object. Next of Kin [#next-of-kin] `next_of_kin` is included when the patient age parsed from `Patient Age` is less than 17. Values are read from bill question answers under the `Next of Kin` subprocess. Expected question labels include: * `Last Name` * `First Name` * `Street Address` * `Line2` * `City` * `State` * `Postal Code` * `Phone Number` The state is normalized through the USPS state mapping when possible. Insurance [#insurance] The integration evaluates active `PatientInsurance` records for the patient and deduplicates them by insurance ID. Insurance details are included in the final payload only when the Labcorp bill type is `T`. When `bill_type` is `T`: * `primary_insurance` is populated from the first active insurance record. * `secondary_insurance` is populated from the second active insurance record when present. Mapped insurance fields include: * Payer address and payer name. * `patient_relationship` from `integrationDetails.relation`. * `payer_code` from insurance network or insurance code. * `policy_number`, blank for government-tagged insurance. * `group_number`, blank for government-tagged insurance. * `workers_comp`: `Y` or `N`. * `id`: insurance code. * `policy_holder` details from the guarantor/patient context. * `date_of_injury`, converted to lab timezone when `lab_time_zone` is enabled. Bill Type [#bill-type] `bill_type` is selected from `integrationDetails.bill_source_json` using `billing.source`. If no mapping exists, it defaults to `P`. Common examples: | Crelio bill source | Labcorp bill type | | ------------------------------------ | ----------------- | | Patient/self-pay source | `P` | | Client/account billing source | `C` | | Insurance/third-party billing source | `T` | Confirm the source-to-type mapping with Labcorp during implementation. The integration uses configured source mappings rather than fixed bill source names. Specimens and Tests [#specimens-and-tests] The integration loops over non-dismissed `LabReportRelation` rows for the bill and lab. For each test or CAP-accredited profile, the payload includes: * Test `id` from `testCode` * Test `name` from `testName` * `diagnoses` from bill-level ICD codes and test/profile-specific `BillingInfo` ICD codes * `aoes` from bill question answers and optional prescribed drug details Specimens include: * `collected_in_house`: `true` * `alternate_id`: bill ID from the webhook * `test_codes`: test or profile test payloads * `collection_date`: `sampleDate` formatted as `YYYY-MM-DDTHH:mm` * `clinical_comments`: bill comments plus patient clinical history When a profile test has CAP accreditation enabled, specimen characteristics are grouped under profile-level specimens. Specimen characteristic answers are read from the `specimen characteristics` subprocess. Supported specimen characteristic labels include: * `description` * `source_id` * `source_site_modifier` `source_site_modifier` can contain comma-separated values. Each value is sent as a separate identifier. AOE Answers [#aoe-answers] AOE values are built from `QuestionValue` records for the report or profile. Questions under the `specimen characteristics` subprocess are excluded from standard AOE mapping. Each AOE includes: * `id`: `process.description` when present, otherwise `question.question_code` * `text`: question text * `responses`: one response per comma-separated answer value AOE answers also influence patient demographics: * Questions containing `race` can override the patient race value. * Questions containing `hispanic` can override ethnicity. Prescribed Drugs for Toxicology [#prescribed-drugs-for-toxicology] When `LabReportRelation.store_values_to_document_db` is enabled, the integration reads report values and finds report formats where the parameter name or integration code is `prescription`. Selected drugs are loaded from the `Drug` table and converted into Labcorp toxicology AOE values. Generated AOE IDs include: | AOE ID | Meaning | | -------- | ------------------- | | `PMSNUM` | Sequence number | | `PMCODE` | RX Norm code | | `PMTEXT` | RX Norm text | | `PMFTDN` | Free text drug name | | `PMSRCE` | RX Norm source | If the drug comments contain `RXNORM`, RX Norm fields are sent. Otherwise, the free-text drug name AOE is used. Courtesy Copy [#courtesy-copy] Courtesy copy entries can come from additional patient information or from the most recent shared provider relation. Additional patient info: * If the answer to a question containing `courtesy_copy` is `P`, the integration sends a patient-copy entry. * Otherwise, it sends a fax-style entry using the organization fax number and organization full name. Fallback shared provider: * Uses `DoctorReportRelation` for the patient and lab. * Sends type `F` when the referral has a fax number, otherwise `P`. * Sends the referral fax number and referral full name when available. Token Generation and Caching [#token-generation-and-caching] The integration uses client credentials to generate a Labcorp access token. ```text POST {authUrl} Content-Type: application/x-www-form-urlencoded grant_type=client_credentials&client_id={client_id}&client_secret={client_secret} ``` The generated token is cached with key: ```text labcorp_access_token_{lab_id} ``` The cache timeout is 3500 seconds. If token generation fails or the response does not contain an access token, the endpoint returns: ```json { "code": 400, "access_token": null, "error": "Token generation failed" } ``` Labcorp Order Submission [#labcorp-order-submission] After payload generation, the integration posts to `integrationDetails.orderUrl`. ```text POST {orderUrl} Client-id: {client_id} Content-Type: application/json Accept: application/json Authorization: Bearer {access_token} ``` If Labcorp returns a non-200 response, the endpoint: * Logs a failure activity with category ID `193`. * Returns the Labcorp response content. * Returns the generated outbound payload for troubleshooting. ```json { "code": 400, "response": "Labcorp response body", "payload": { "account": {}, "patient": {}, "specimens": [] } } ``` Successful Response Handling [#successful-response-handling] On HTTP 200 from Labcorp, the endpoint expects a response containing `requisitions`. ```json { "requisitions": { "ACCESSION123": { "pdf": "base64-pdf", "abn": "base64-abn-pdf", "sourceOrder": { "specimen": { "test_codes": [ { "id": "80053" } ] } } } } } ``` For each requisition, the integration performs the following actions: * Creates an attachment entry for the requisition PDF. * Creates another attachment entry for the ABN PDF when `abn` is present. * Reads test codes from `sourceOrder.specimen.test_codes`. * Updates matching report rows by test code with `ReportLevelTags = accession_no`. Billing Attachment Upload [#billing-attachment-upload] The integration uploads returned PDFs through the Crelio billing attachment endpoint. Before creating upload jobs, it retrieves the latest lab `RequestToken` that is associated with a lab user. If no token is found, the endpoint returns: ```json { "code": 400, "error": "Invalid Token" } ``` When a token exists, each attachment is queued through the Fusion client as a webhook job: ```json { "url": "{settings.WEBHOOK_URL}/uploadBillingAttachment/?authKey={auth_key}", "body": { "base64_data": "data:application/pdf;base64,{pdf}", "billId": 111, "ext": "pdf", "fileName": "ACCESSION123.pdf" } } ``` The bill `orderNumber` is updated with the Labcorp bill ID from the webhook, and the endpoint returns the queued Fusion job IDs. Success Response [#success-response] ```json { "code": 200, "response": { "requisitions": { "ACCESSION123": { "pdf": "base64-pdf", "abn": "base64-abn-pdf", "sourceOrder": { "specimen": { "test_codes": [ { "id": "80053" } ] } } } } }, "payload": { "account": {}, "patient": {}, "specimens": [] }, "job_ids": ["job-id-1"] } ``` Data Normalization [#data-normalization] Uppercase Conversion [#uppercase-conversion] Before submission, the integration recursively uppercases all string values in dictionaries and lists. This applies to patient names, addresses, provider names, AOE text, insurance values, comments, and most configured identifiers. State Normalization [#state-normalization] The helper `get_state_ny` accepts either two-letter USPS abbreviations or full state/territory names. It returns a valid USPS abbreviation or blank when the value is not recognized. Supported values include all US states and territories configured in `US_STATES`, such as: | Input | Output | | ---------------------- | ------ | | `new york` | `NY` | | `California` | `CA` | | `district of columbia` | `DC` | | `Puerto Rico` | `PR` | Date and Time Handling [#date-and-time-handling] When `integrationDetails.lab_time_zone` is truthy: * Sample collection dates are converted to `lab.labTimeZone`. * Insurance injury dates are converted to `lab.labTimeZone`. When the flag is false, dates are formatted directly from the stored datetime. Activity Logging [#activity-logging] Successful and failed submissions are logged through `ActivityLog` with `log_category_id = 193`. Each log entry includes the patient name, bill ID, lab ID, lab name, and the submission outcome. Use these activity logs as the first reference when validating whether an order was submitted to Labcorp successfully. Error Handling [#error-handling] | Condition | Response | Notes | | --------------------------------- | ------------------------------------------------- | ---------------------------------------------------------- | | Missing first `labReportId` | `400` with `labReportId is not valid` | The endpoint cannot resolve the bill and report context. | | Token generation failure | `400` with `Token generation failed` | Check `authUrl`, `client_id`, and `client_secret`. | | Labcorp order API returns non-200 | `400` with Labcorp response and generated payload | Failure activity is logged. | | Missing upload `RequestToken` | `400` with `Invalid Token` | Required to upload returned PDFs into billing attachments. | | Unhandled exception | `500` with generic error | Exception is captured in Sentry. | Operational Checklist [#operational-checklist] * Confirm the Labcorp URL is configured as `/api-v3/integration/labcorp/send-order/`. * Confirm `integrationDetails.authUrl`, `orderUrl`, `client_id`, and `client_secret`. * Confirm `account_number`, `site_name`, `labState`, sending and receiving applications/facilities, and bill source mapping with Labcorp. * Confirm patient demographics include DOB, gender, address, city, state, and pincode. * Confirm provider NPI and provider number values are present where Labcorp requires them. * Confirm test codes and profile test codes match Labcorp orderable codes. * Confirm ICD codes are present for tests that require diagnoses. * Confirm AOE question codes or process descriptions match Labcorp expected AOE IDs. * Confirm specimen characteristic questions are configured for CAP-accredited profile tests that require them. * Confirm active insurance records are complete when `bill_type` resolves to `T`. * Confirm a valid `RequestToken` exists for the lab so requisition PDFs can be attached back to the bill. * Confirm Fusion is available for queued attachment upload jobs. Troubleshooting [#troubleshooting] | Symptom | Likely cause | What to check | | ------------------------------------------- | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `labReportId is not valid` | `labReportDetails[0].labReportId` is missing or zero. | Webhook payload and bill-report relation. | | `Token generation failed` | Invalid OAuth URL or credentials. | `authUrl`, `client_id`, `client_secret`, and Labcorp credential status. | | Labcorp returns validation errors | Required order fields or mappings are missing. | Returned `payload`, test codes, provider fields, patient address, bill type, insurance fields, and AOE IDs. | | Requisition PDFs are not attached | Missing upload token or Fusion job failure. | `RequestToken`, `settings.WEBHOOK_URL`, Fusion job IDs, and billing attachment API logs. | | Accession number is not visible on a report | Labcorp returned test code does not match `reportID.testCode`. | `sourceOrder.specimen.test_codes` in Labcorp response and local test code configuration. | | Incorrect state in outbound payload | Input state is not recognized by `US_STATES`. | Patient, guarantor, insurance, next-of-kin, and lab state values. | | Insurance is not sent | `bill_type` did not resolve to `T`. | `billing.source` and `integrationDetails.bill_source_json`. | Implementation Notes [#implementation-notes] * The endpoint reads the first item in `labReportDetails` to determine the lab and report context, then queries all non-dismissed report relations for that bill and lab. * The token cache is scoped by lab ID, not by client ID or endpoint URL. * Only the first two unique active insurance records are sent in the final payload, as primary and optional secondary insurance. * The generated `control_id` is a 14-digit prefix of a UUID integer. * All returned requisition and ABN attachments use the filename `{accession_no}.pdf`. * Sentry captures unhandled exceptions, while Labcorp non-200 responses are returned directly with the generated outbound payload. # IT Dose IT Dose Integration [#it-dose-integration] This document provides an overview of the IT Dose integration, including API configurations, database table configurations, and detailed functionality descriptions. Overview [#overview] The IT Dose integration handles the following functionalities: * New order generation * Order cancel status update * Sample collection status update * Sample receive status update * Sample dismiss/test dismiss * Sending report result details Workflow [#workflow] Below is a flow diagram representing the integration process:
Bill Operation ActionIds - 2, 3, 6, 30, 8, 14
Report Submission ActionId - 50 Architecture Flow [#architecture-flow] Crelio API Configurations [#crelio-api-configurations] Endpoints [#endpoints] 1. **Create Order in IT Dose** * **URL**: `/api-v3/integration/it-dose/create-order/` * **Method**: POST * **Payload**: Webhook payload of bill generation, bill update, bill cancel, sample collection, sample receive, sample dismiss/test dismiss will be sent to this endpoint.
**for example:** ```json { "orderNumber": "", "billComments": "", "dueAmount": 100.0, "Patient Name": "ABC PQR XYZ", "billConcession": 0.0, "labPatientId": "", "patient_email": "", "port": "0", "Status": "Bill Generation HL7 for SFTP", "billReferral": "Dr Referral doctor", "billTotalAmount": 100.0, "billId": 3, "bill_id" : 8, "Patient City": "Pune", "billRemark": "", "billReferralId": 13788, "Patient Address": "", "Patient Dob": "1986-03-02", "username": "livehealth", "Mobile Number": "", "Patient gender": "Male", "billTime": "2021-09-09T10:02:27Z", "Patient Pincode": "411233", "host": "if.myadsc.com", "Patient Age": "35 years", "Patient Id": 1889028480, "Patient Area": "Pashan", "labId" : 9, "labReportDetails": [ { "reportDate": "2022-05-18T07:07:50Z", "labReportIndex": 48420, "sampleDate": "2022-05-17T18:30:00Z", "labId": { "labName": "Demonstration Diagnostics - Hyderabad", "labId": 9 }, "orgId": { "orgId": 264517, "orgFullName": "DIRECT" }, "reportID": { "testAmount": "1776", "testCode": "DHEA SULPHATE (DHEAS),SERUM", "testCategory": "-", "procedureCode_id": null, "sampleId": { "type": "Urine", "name": "Urine for Micro" }, "testName": "DHEA SULPHATE (DHEAS),SERUM", "testID": 144, "integrationCode": "12345,2345,6533", "dictionaryId": null, "departmentId": { "id": 1055, "name": "Pathology" } }, "labReportId": 10, "collectedSampleId": { "collectionTime": "2022-05-18T07:07:50Z", "accessionNo": "D32400113822MR", "toBatchProcessing": 0, "lastUpdated": "2022-05-18T07:07:53Z", "batchId": null, "isRejected": 0, "comments": "", "dismissed": 0, "redraw": 0, "id": 85972426 } } ], "billAdvance": 0, "webhookId": 49 } ``` 2. **Send Result to IT Dose** * **URL**: `/api-v3/integration/it-dose/send-result/` * **Method**: POST * **Payload**: ```json { "Signing Doctor": [ { "Signing Doctor 1": "Strange" } ], "status": "Report Submit (With Values)", "orderNumber": "3", "Patient Name": "test teest", "fileInputReport": 0, "smart_report_links": [], "payloadType": "ORUR01", "labPatientId": "4105", "Test Name": "Glucose Random", "CentreReportId": 8, "integrationCode": null, "port": 0, "referralId": 13877, "Patient Contact": "8329777358", "billReferral": "Ogunlade", "labId": 9, "Sample Date": "2021-11-02T07:01:13Z", "billId": 3, "dictionaryId": 1515, "Patient Alternate Contact": "", "isSigned": 1, "reportFormatAndValues": [ { "highlight": 0, "value": "FLU A = NEGATIVE, FLU B = NEGATIVE", "reportFormat": { "id" : 11111, "lowerBoundFemale": "-", "lowerBoundMale": "-", "upperBoundFemale": "-", "otherMale": "-", "upperBoundMale": "-", "emailFlag": 0, "integrationCode": "", "descriptionFlag": 1, "criticalLowerMale": "-", "dictionaryId": null, "fileInput": 0, "method": "", "otherFemale": "-", "criticalUpperFemale": "-", "listField": 0, "otherFlag": 0, "testUnit": "#", "criticalUpperMale": "-", "isImage": 0, "highlightFlag": 0, "testName": " ", "criticalLowerFemale": "-", "dictionaryTestUnit": "", "order": 15 } }, { "highlight": 1, "value": "21", "reportFormat": { "lowerBoundFemale": "2", "lowerBoundMale": "2", "upperBoundFemale": "20", "otherMale": ">20.0 mg/dL", "upperBoundMale": "20", "emailFlag": 0, "integrationCode": "", "descriptionFlag": 0, "criticalLowerMale": "1", "dictionaryId": null, "fileInput": 0, "method": "", "otherFemale": ">20.0 mg/dL", "criticalUpperFemale": "30", "listField": 0, "otherFlag": 1, "testUnit": "mg/dL", "criticalUpperMale": "-", "isImage": 0, "highlightFlag": 0, "testName": "Creatinine", "criticalLowerFemale": "1", "dictionaryTestUnit": "", "order": 2 } }, { "highlight": 0, "value": "9", "reportFormat": { "lowerBoundFemale": "4", "lowerBoundMale": "4", "upperBoundFemale": "9", "otherMale": "4.0 - 9.0", "upperBoundMale": "9", "emailFlag": 0, "integrationCode": "", "descriptionFlag": 0, "criticalLowerMale": "-", "dictionaryId": null, "fileInput": 0, "method": "", "otherFemale": "4.0 - 9.0", "criticalUpperFemale": "-", "listField": 0, "otherFlag": 1, "testUnit": "-", "criticalUpperMale": "-", "isImage": 0, "highlightFlag": 0, "testName": "pH", "criticalLowerFemale": "-", "dictionaryTestUnit": "", "order": 3 } } ], "Report Date": "2021-11-02T07:04:55Z", "username": "username", "testCode": "8800", "userDetailsId": 20791157, "Gender": "Female", "Age": "27 years", "labName": "Chennai", "Accession Date": "2021-11-02T07:03:25Z", "labCity": "Pune", "Report Id": 50, "host": "127.0.0.1", "webhookId": 50, "Patient Id": 7, "password": "password", "labReportId": 8, "reportDate": "2021-11-02T07:06:04Z", "alternateEmail": null, "Approval Date": "2021-11-02T07:04:55Z", "Contact No": "8329777358", "orgId": 5, "testID": 209, "reportBase64" : "" } ``` IT Dose API Requests [#it-dose-api-requests] Operation Type Mapping [#operation-type-mapping] The IT Dose integration uses specific operation type codes to define the nature of the request. The following table outlines the mapping between Webhook IDs and their corresponding operation types used in API requests: | **Webhook ID** | **Action Name** | **Operation Type Code** | **Operation Description** | | -------------- | ---------------- | ----------------------- | --------------------------------------------------------------------- | | 2 | Bill Update | RE | Add New Request / Add On Request (Additional tests to existing order) | | 3 | Bill Cancel | OC | Order Cancel (Cancel specific tests in an order) | | 6 | Sample Receive | DR | Department Received (Sample received at the laboratory) | | 8 | Sample Dismiss | RE | Add New Request / Add On Request (Retest/Redraw request) | | 14 | Test Dismiss | RE | Add New Request / Add On Request (Specific test retest request) | | 30 | Sample Collect | SC | Status Change (Update sample collection status) | | 18 | Add Test To Bill | RE | Add New Request / Add On Request (Add new test to existing order) | Operation Type Definitions [#operation-type-definitions] * **OC (Order Cancel)**: Used to cancel one or more tests within an existing order. This operation marks the specified tests as canceled in the IT Dose system. * **DR (Department Received)**: Used when a physical sample arrives and is received by the laboratory department. This updates the sample status to indicate it has been received and is ready for processing. * **SC (Status Change)**: Used to update the overall status of a sample collection. This includes marking samples as collected, processed, or moving them through different stages of the collection workflow. * **RE (Add New Request / Add On Request)**: Used for multiple scenarios including adding new tests to an existing order, requesting a redraw of a sample, requesting a retest, or adding supplementary tests to an incomplete order. The following API requests are integrated with the IT Dose system [#the-following-api-requests-are-integrated-with-the-it-dose-system] 1. **Create Order** * **Endpoint**: [https://limsuat.maxlab.co.in/maxlab/api/CrelioLimsIntegration/NewBookingData](https://limsuat.maxlab.co.in/maxlab/api/CrelioLimsIntegration/NewBookingData) * **Method**: POST * **Headers**: ```json { "Content-Type": "application/json", "Authorization": "Bearer " } ``` * **Payload**: ```json { "CrelioOrderID": "MLJK23501", "InterfaceCompanyName": "Span_BLR", "Title": "", "FirstName": "", "MiddleName": "", "LastName": "", "landmark": "", "House_No": "", "Mobile": "", "Pincode": "", "Email": "", "Gender": "", "DOB": "", "Patient_ID": "ML8SIK3469", "CentreCode": "2278", "Type": "NW", "TestDetail": [ { "PackageCode": "", "PackageName": "", "TestCode": "15732", "ItemName": "" }, { "PackageCode": "", "PackageName": "", "TestCode": "1234", "ItemName": "" } ] } ``` 2. **Update Order** * **Endpoint**: `https://limsuat.maxlab.co.in/maxlab/api/CrelioLimsIntegration/` * **Method**: POST * **Headers**: ```json { "Content-Type": "application/json", "Authorization": "Bearer " } ``` * **Payload**: ```json { "CrelioOrderID": "COI123454", "InterfaceCompanyName":"Span_BLR", "TestDetail": [ { "Type":"RE", // Refer to operation type mapping for details "PackageCode": "", "TestCode": "LV026", "Remark":"" }, { "Type":"RE", // Refer to operation type mapping for details "PackageCode": "", "TestCode": "P032", "Remark":"" } ] } ``` 3. **Cancel Order** * **Endpoint**: `https://limsuat.maxlab.co.in/maxlab/api/CrelioLimsIntegration/` * **Method**: DELETE * **Headers**: ```json { "Content-Type": "application/json", "Authorization": "Bearer " } ``` * **Payload**: ```json { "CrelioOrderID": "COI123454", "InterfaceCompanyName":"Span_BLR", "TestDetail": [ { "Type":"OC", // Refer to operation type mapping for details "PackageCode": "1234", "TestCode": "P032", "Remark":"" }, { "Type":"OC", // Refer to operation type mapping for details "PackageCode": "1234", "TestCode": "LV026", "Remark":"" } ] } ``` 4. **Submit Results** * **Endpoint**: `https://limsuat.maxlab.co.in/maxlab/API/LimsToCrelioIntegration/LimsToCrelioResultEntry` * **Method**: POST * **Headers**: ```json { "Content-Type": "application/json", "Authorization": "Bearer " } ``` * **Payload**: ```json { "tests": [ { "ledgerTransactionNo": "23343545343422", "testUniqueId": "23343545343422", "PrintNabl": "0", "testName": "5834~LIPASE", "testCode": "5834", "attachment": "", "attachmentExtension": "", "addReport": "", "resultStatus": "Approved", "approvedByDoctorId": "1813", "approvedByDoctorName": "ABC", "approvedDate": "2025-11-01 00:00:00", "resultEnteredById": "1813", "resultEnteredByName": "ABC", "resultDateTime": "2025-11-01 00:00:00", "interpretation": "", "testComment": "", "parameters": [ { "parameterId": "232", "parameterName": "Lipase", "value": "234", "parameterComment": "", "unit": "", "minValue": "", "maxValue": "", "printSequenceNo": "", "biologicalReferenceRange": "", "method": "", "isCritical": "", "minCritical": "", "maxCritical": "", "flag": "" }, { "parameterId": "233", "parameterName": "Amylase", "value": "120", "parameterComment": "", "unit": "", "minValue": "", "maxValue": "", "printSequenceNo": "", "biologicalReferenceRange": "", "method": "", "isCritical": "", "minCritical": "", "maxCritical": "", "flag": "" } ] }, { "ledgerTransactionNo": "23343545343422", "testUniqueId": "23343545343422", "testName": "5834~LIPASE", "testCode": "5834", "attachment": "", "attachmentExtension": "", "addReport": "", "resultStatus": "Approved", "approvedByDoctorId": "1813", "approvedByDoctorName": "ABC", "approvedDate": "2025-11-01 00:00:00", "resultEnteredById": "1813", "resultEnteredByName": "ABC", "resultDateTime": "2025-11-01 00:00:00", "interpretation": "", "testComment": "", "parameters": [ { "parameterId": "232", "parameterName": "Lipase", "value": "234", "parameterComment": "", "unit": "", "minValue": "", "maxValue": "", "printSequenceNo": "", "biologicalReferenceRange": "", "method": "", "isCritical": "", "minCritical": "", "maxCritical": "", "flag": "High" } ] } ] } ``` API Conditions [#api-conditions] ITDoseCreateOrderView [#itdosecreateorderview] This API handles the creation of orders in IT Dose. Below are the conditions: 1. **Order Payload**: * The payload is prepared using the `prepare_order_payload` function. * If the payload is not generated, return a `400` status with an error message. 2. **Endpoint URL**: * For `webhookId` 49, use the `url` from `integrationDetails`. * For other webhook IDs: * If `bill_source` is `MAXHEALTH`, use `max_health_url`. * Otherwise, use `update_url`. 3. **Authentication**: * Fetch the authentication token using `get_auth_token`. * Use the token in the `Authorization` header for API requests. 4. **Response Handling**: * Log the activity in the `ActivityLog` table. * Return the response from IT Dose along with the payload and status code. ITDoseSendResultView [#itdosesendresultview] This API handles the submission of report results to IT Dose. Below are the conditions: 1. **Payload Extraction**: * Extract specific keys from the request payload using the `extract_payload` function. 2. **Validation**: * Validate the `labReportId` and fetch the corresponding `LabReportRelation` object. * If the `labReportId` is invalid, return a `400` status with an error message. 3. **Endpoint URL**: * If `bill_source` is `MAXHEALTH`, use `max_health_url`. * Otherwise, use `update_url`. 4. **Result Payload**: * Prepare the result payload using the `prepare_result_payload` function. * If the payload contains an error, return a `400` status with the error message. 5. **Response Handling**: * Log the activity in the `ActivityLog` table. * Return the response from IT Dose along with the payload and status code. ITDoseAsyncCalls [#itdoseasynccalls] This API handles asynchronous calls to IT Dose. Below are the conditions: 1. **Validation**: * Validate the `endpoint_url`, `headers`, and `payload` in the request payload. * If any of these are missing, return a `400` status with the request payload. 2. **Response Handling**: * Return the response from IT Dose along with the payload and status code. Database Table Configurations [#database-table-configurations] labIntegration Table [#labintegration-table] * **Purpose**: Stores integration details for IT Dose. * **Key Fields**: * `url`: Endpoint URL for IT Dose APIs. * `integrationExtraDetails`: JSON field containing additional configuration details. Example integrationExtraDetails Structure [#example-integrationextradetails-structure] ```json { "auth_url": "https://itdose-auth-url.com", "url": "https://itdose-order-url.com", "result_url": "https://itdose-result-url.com", "auth_token": "", "max_health_url": "https://maxhealth-url.com", "update_url": "https://update-url.com" } ``` Functionality Details [#functionality-details] New Order Generation [#new-order-generation] * **Trigger**: Webhook ID 49 * **Process**: 1. Fetch integration details from the `labIntegration` table. 2. Prepare the order payload using `prepare_order_payload`. 3. Send the payload to the IT Dose API. 4. Log the activity in the `ActivityLog` table. Report Submission [#report-submission] * **Trigger**: Webhook ID 50 * **Process**: 1. Extract payload details using `extract_payload`. 2. Prepare the result payload using `prepare_result_payload`. 3. Send the payload to the IT Dose API. 4. Log the activity in the `ActivityLog` table. Authentication [#authentication] * **Process**: 1. Fetch the authentication token using `get_auth_token`. 2. Use the token in the `Authorization` header for API requests. Error Handling [#error-handling] * Invalid tokens return a `400` status with an error message. * Missing or incomplete integration configurations return appropriate error responses. Webhook Action IDs Reference [#webhook-action-ids-reference] The IT Dose integration utilizes the following webhook action IDs to trigger various operations. For a comprehensive list of all available webhook actions in the system, refer to the [Webhook Actions](../../prerequisites/webhook-actions#list-of-webhook-actions) documentation. IT Dose Action ID Mapping [#it-dose-action-id-mapping] The IT Dose integration supports various actions, each identified by a unique webhook ID. The following table outlines the mapping between webhook IDs and their corresponding actions: * The trigger\_status is determined dynamically based on the webhook ID received in the request.
* This trigger is used to update the data in ITDOSE according to the corresponding status.
* This mapping ensures that the appropriate action is executed for each webhook event, maintaining consistency and accuracy in the integration process. | **Webhook ID** | **Action Name** | **IT Dose Functionality** | **Reference** | | -------------- | ------------------------- | ------------------------------- | -------------------------------------------------------------------------------- | | 2 | Bill Update | Order Update Status | [See Webhook ID 2](../../prerequisites/webhook-actions#list-of-webhook-actions) | | 3 | Bill Cancel | Order Cancellation | [See Webhook ID 3](../../prerequisites/webhook-actions#list-of-webhook-actions) | | 6 | Sample Receive | Sample Receive Status Update | [See Webhook ID 6](../../prerequisites/webhook-actions#list-of-webhook-actions) | | 8 | Sample Dismiss | Sample Dismissal Status Update | [See Webhook ID 8](../../prerequisites/webhook-actions#list-of-webhook-actions) | | 14 | Test Dismiss | Test Dismissal Status Update | [See Webhook ID 14](../../prerequisites/webhook-actions#list-of-webhook-actions) | | 30 | Sample Collect | Sample Collection Status Update | [See Webhook ID 30](../../prerequisites/webhook-actions#list-of-webhook-actions) | | 49 | Bill Generation Webhook | New Order Generation | [See Webhook ID 49](../../prerequisites/webhook-actions#list-of-webhook-actions) | | 50 | Report Submission Webhook | Report Result Submission | [See Webhook ID 50](../../prerequisites/webhook-actions#list-of-webhook-actions) | Detailed Action ID Descriptions [#detailed-action-id-descriptions] Webhook ID 2 - Bill Update [#webhook-id-2---bill-update] * **Action**: Triggered when an existing bill is updated. * **IT Dose Operation**: Updates the corresponding order status in IT Dose. * **When Used**: When bill details are modified post-creation but before settlement. Webhook ID 3 - Bill Cancel [#webhook-id-3---bill-cancel] * **Action**: Triggered when a bill is canceled. * **IT Dose Operation**: Initiates order cancellation process in IT Dose. * **When Used**: When a bill needs to be canceled entirely. Webhook ID 6 - Sample Receive [#webhook-id-6---sample-receive] * **Action**: Triggered when a sample is received in the lab. * **IT Dose Operation**: Updates the sample receive status in IT Dose. * **When Used**: When a physical sample arrives at the laboratory. Webhook ID 8 - Sample Dismiss [#webhook-id-8---sample-dismiss] * **Action**: Triggered when a sample is dismissed. * **IT Dose Operation**: Marks the sample as dismissed in IT Dose. * **When Used**: When a sample is no longer valid or cannot be processed. Webhook ID 14 - Test Dismiss [#webhook-id-14---test-dismiss] * **Action**: Triggered when a specific test is dismissed. * **IT Dose Operation**: Marks the specific test as dismissed in IT Dose. * **When Used**: When a particular test within a sample needs to be dismissed. Webhook ID 30 - Sample Collect [#webhook-id-30---sample-collect] * **Action**: Triggered when a sample is collected from the patient. * **IT Dose Operation**: Updates the sample collection status in IT Dose. * **When Used**: When a sample is successfully collected from the patient. Webhook ID 49 - Bill Generation Webhook [#webhook-id-49---bill-generation-webhook] * **Action**: Triggered for new bill/order generation. * **IT Dose Operation**: Creates new orders in IT Dose. * **When Used**: When a new bill is generated and needs to be sent to IT Dose. Webhook ID 50 - Report Submission Webhook [#webhook-id-50---report-submission-webhook] * **Action**: Triggered for report submission. * **IT Dose Operation**: Sends report results and test values to IT Dose. * **When Used**: When a report is finalized and results need to be submitted to IT Dose. # Odoo (Nerva) Odoo (Nerva) [#odoo-nerva] This document provides an overview of the Odoo (Nerva) integration, including API configurations and the flow of data between the system and Odoo. Overview [#overview] The Odoo integration handles the following functionalities: * Updating payment data for labs. * Creating partners and organizations in Odoo. * Logging integration activities. Work Flow Diagram [#work-flow-diagram] Below is the updated flow diagram representing the integration process, including Webhook IDs and the condition for payment updates: Client API details [#client-api-details] Endpoints [#endpoints] 1. **Create Partner in Odoo** * **URL**: Provided in `add_partner_url`. * **Method**: POST * **Payload**: ```json { "params": { "id": "", "name": "", "email": "", "phone": "", "organization_id": "" } } ``` 2. **Create Organization in Odoo** * **URL**: Provided in `add_organisation_url`. * **Method**: POST * **Payload**: ```json { "params": { "id": "", "name": "", "organization_code": "", "contact": "", "address": "
", "payment_mode": "", "analytic_account_id": "" } } ``` 3. **Add Payment for Order** * **URL**: Provided in `payment_url`. * **Method**: POST * **Payload**: ```json { "params": { "user_id": "", "id": "", "notes": "", "cart_items": [ { "id": "", "item_id": "", "quantity": "", "price": "", "analytic_id": "", "discount": "", "tax_id": "" } ] } } ``` Configurations [#configurations] URL Configuration [#url-configuration] * **URL**: `{domain}/api-v3/integration/odoo/update-details/` * This URL should be configured in the database under the `labIntegration` table under the `url` column against below Webhook Action Ids. Webhook Action IDs [#webhook-action-ids] The `labIntegration` table contains multiple webhook `actionId` values that are configured as follows: * **2**: Bill update * **17**: Bill settlement * **49**: Bill generation for Integrations * **52**: Organisation create and update Extra Configurations for Action ID 49 [#extra-configurations-for-action-id-49] For `actionId` 49, additional configuration is required in the `integrationExtraDetails` field. The structure is as follows:
(Note: The URLs below may change depending on the Odoo configuration and can differ between staging and production environments.) ```json { "integration" : "Odoo", "add_partner_url": "https://nervateameg-balance-labs-sand-box-25617602.dev.odoo.com/api/v1/addPartner", "payment_url": "https://nervateameg-balance-labs-sand-box-25617602.dev.odoo.com/api/v1/addPayment", "add_organisation_url": "https://nervateameg-balance-labs-sand-box-25617602.dev.odoo.com/api/v1/addOrganization", "api_key": "123" } ``` The `api_key` will be provided by the third party and will be used to authenticate and authorize access to their APIs. Ensure that the client URLs are added to the `integrationExtraDetails` field for `actionId` 49. Webhook Action IDs Reference [#webhook-action-ids-reference] The Odoo integration utilizes the following webhook action IDs to trigger various operations. For a comprehensive list of all available webhook actions in the system, refer to the [Webhook Actions](../../prerequisites/webhook-actions#list-of-webhook-actions) documentation. Odoo Action ID Mapping [#odoo-action-id-mapping] | **Webhook ID** | **Action Name** | **Odoo Functionality** | **Reference** | | -------------- | --------------------------------------- | --------------------------- | -------------------------------------------------------------------------------- | | 2 | Bill Update | Update Bill Details | [See Webhook ID 2](../../prerequisites/webhook-actions#list-of-webhook-actions) | | 17 | Bill Settlement | Settlement Process | [See Webhook ID 17](../../prerequisites/webhook-actions#list-of-webhook-actions) | | 49 | Bill Generation HL7 for SFTP (Outbound) | Payment Data Update in Odoo | [See Webhook ID 49](../../prerequisites/webhook-actions#list-of-webhook-actions) | | 52 | Organisation Update | Organization Create/Update | [See Webhook ID 52](../../prerequisites/webhook-actions#list-of-webhook-actions) | Detailed Action ID Descriptions [#detailed-action-id-descriptions] Webhook ID 2 - Bill Update [#webhook-id-2---bill-update] * **Action**: Triggered when an existing bill is updated. * **Odoo Operation**: Updates the bill details in Odoo. * **Condition**: If bill is settled or completed, proceed with payment update. * **When Used**: When bill information is modified after initial creation. Webhook ID 17 - Bill Settlement [#webhook-id-17---bill-settlement] * **Action**: Triggered when a bill is settled. * **Odoo Operation**: Initiates payment update process in Odoo when bill is settled. * **Condition**: Mandatory trigger for payment data update. * **When Used**: When a bill payment is finalized and settled. Webhook ID 49 - Bill Generation HL7 for SFTP (Outbound) [#webhook-id-49---bill-generation-hl7-for-sftp-outbound] * **Action**: Triggered for new bill generation via HL7 for SFTP. * **Odoo Operation**: Creates payment records and updates payment data in Odoo. * **Condition**: If bill is settled or completed, proceed with payment update. * **When Used**: When a new bill is generated and payment data needs to be synced with Odoo. Webhook ID 52 - Organisation Update [#webhook-id-52---organisation-update] * **Action**: Triggered to update or create organization details. * **Odoo Operation**: Creates or updates partner/organization information in Odoo. * **Condition**: No payment update required, only organization sync. * **When Used**: When organization or partner information is created or modified in the system. Payment Update Logic [#payment-update-logic] The Odoo integration follows a specific payment update condition across multiple webhook IDs. The payment update process is triggered based on the following cases: **Case 1: Bill Update (Webhook ID 2)** * When a bill is updated and the bill status is settled or completed * Action: Proceed with payment data update to Odoo * Otherwise: Return success response without updating payment data **Case 2: Bill Settlement (Webhook ID 17)** * When a bill is explicitly settled * Action: Mandatory payment data update to Odoo * This is the primary trigger for finalizing payment records **Case 3: Bill Generation (Webhook ID 49)** * When a new bill is generated via HL7 for SFTP * Check: If bill is settled or completed at the time of generation * Action: Proceed with payment data update to Odoo if condition is met * Otherwise: Return success response without updating payment data **Case 4: Organization Update (Webhook ID 52)** * When organization or partner information is created or updated * Action: Sync organization/partner details to Odoo * Note: No payment update required for this webhook ID, only organization synchronization Payment Update Conditions Summary [#payment-update-conditions-summary] The system ensures that payment data is only synchronized with Odoo under the following conditions: * The bill must be in a **completed** or **settled** state * All billing information must be **finalized** * Payment can be **safely processed** in the Odoo system * The transaction has passed all **validation checks** Payment Update Condition [#payment-update-condition] In all the above cases, if the bill is settled or completed (both terms are equivalent), proceed with the payment update into Odoo. Constants [#constants] * **TAX\_ID\_MAPPER**: Maps tax percentages to IDs.
Note : We will add more mapping in the future | **Tax Percentage** | **Tax ID** | | ------------------ | ---------- | | 0% | 1 | | 15% | 2 | Error Handling [#error-handling] * Invalid tokens return a `400` status with an error message. * Missing or incomplete integration configurations return appropriate error responses. Activity Logging [#activity-logging] Integration activities are logged using the `ActivityLog` model, which records details such as lab ID, bill ID, and activity text. *** This concludes the documentation for the Odoo integration. # Design Decisions Design Decisions [#design-decisions] *** Key Design Decisions & Constraints [#key-design-decisions--constraints] **Simplified view over Ledger comprehensiveness** * Account Statement intentionally shows less than the Ledger — only three entry types (Bill, Payment, Advance/Manual) * The goal is cash-flow visibility, not a complete audit trail * Ledger remains the source of truth for running balance; Account Statement is a complementary view **No new database tables** * Account Statement reads from three existing tables: `Billing`, `Payments`, `OrganizationTransaction` * No new model — zero schema surface to maintain * Trade-off: data quality depends entirely on the correctness of the upstream tables **Prepaid and WalkIn only — PostPaid excluded** * PostPaid organizations use the Invoice + Ledger workflow for settlement tracking * Showing Account Statement alongside Invoices would create confusion about which view to trust * The `orgPaymentType` check (`[ORG.PREPAID, ORG.WALKIN].includes(...)`) enforces this at the tab level **Data starts from 10th March 2026 — because `transaction_category` was nullable before that** * The `transaction_category` column was added by migration `0028` on **13th February 2026**, but with `null=True, db_default=None` — meaning all pre-migration rows and any rows created before each write path was deployed still have `NULL` in this column * Without `transaction_category`, the API cannot distinguish an advance entry from a billing-related ledger entry (e.g., bill settlement credit-back) — both appear as `OrganizationTransaction` rows * **10th March 2026** was chosen as the safe cutoff: by that date, both `livehealthapp` (`add_organization_advance`, `Organization.advanceCollection`) and `crelio-app` (`manual_ledger_entry`) were fully deployed and setting `transaction_category` on every new row * `ACCOUNT_STATEMENT_MIN_DATE = moment("2026-03-10")` is hardcoded in the frontend; the UI date guard prevents API calls for earlier ranges and shows the warning: *"Account Statement is valid after 10th March 2026."* *** Architectural Rationale [#architectural-rationale] **Direct source queries, no running balance** * Ledger failures cascade because each entry's balance depends on the previous one * Account Statement avoids this entirely by computing totals independently per request * Each of the three queries (`get_bills`, `get_payments`, `get_advance_collection`) is independent — a failure in one does not affect the others **Conversion-aware window clamping** * When an organization converts payment type (e.g., WalkIn → Prepaid), the financial relationship changes fundamentally * Mixing pre- and post-conversion data in one statement would show incorrect totals * The API uses `TransactionCategoryEnum.ORGANIZATION_CONVERSION` to detect the event and clamp `startDate` automatically, with no manual intervention required **`ADVANCE` payment type excluded from payments** * When the lab uses an org's existing advance to settle a bill, a `Payments` record is created with `paymentType = "ADVANCE"` * Including this would double-count: the advance was already tracked in `OrganizationTransaction` * Exclusion via `.exclude(paymentType="ADVANCE")` ensures each rupee appears only once **`billDue` computed on frontend, not backend** * `billDue = billTotalAmount - billAdvance` is calculated in `mapAccountStatementResponseToRows()` * The backend returns raw fields; the frontend owns the presentation logic * Keeps the API contract simple and allows the frontend to evolve the display (e.g., show 0 instead of negative) without backend changes **`statementId` resolved by payment type on frontend** * For card payments → `cardNo`; cheque → `chequeNo`; online → `transactionId` * The backend returns all three fields; the frontend picks the appropriate reference * Avoids a backend decision about which field is "the" reference for a given payment mode *** Reusability & Extensibility [#reusability--extensibility] **`mapAccountStatementResponseToRows()` is the single transformation boundary** * All three API lists are normalized into `AccountStatementRow` in one function * Adding a new entry type (e.g., credit notes) requires only a new mapping branch here * The AG Grid component (`AccountStatementTab`) requires no changes for new entry types — only `entryType` badge styling needs updating **`groupDateKey` as a stable grouping key** * Row grouping by `YYYY-MM-DD` (in lab timezone) is handled entirely by the hidden `groupDateKey` field * Changing timezone logic or date format requires updating only `toGroupKey()` in `helpers.ts` **Tab lazy rendering** * `AccountStatementTab` receives `null` when the tab is not active, preventing API calls on unrelated tab switches * This pattern can be reused for any tab that should not fetch data until selected # Overview Related JIRA [#related-jira] * [EN-11316](https://crelio.atlassian.net/browse/EN-11316) Whimsical [#whimsical] * [Account Statement UI/UX](https://whimsical.com/account-statement-walkin-prepaid-flexi-prepaid-QKNNk6vkjgPDBUSvniqjhp) *** Account Statement [#account-statement] Account Statement is a **period-based cash-flow view** for an organization: money the organization **provided** to the lab (advances, direct payments) versus money **utilized** against bills. It is built for finance and operations teams who need clarity without opening the full Ledger. For a **step-by-step UI walkthrough** (screenshots), see the [Workflow Guide](/docs/product-engineering/features/account-statement/workflow-guide). Data is available from **10th March 2026** onwards. Before this date, the `transaction_category` column did not exist on `organizationTransaction`, so advance entries cannot be reliably distinguished from other ledger events. See [Data model](/docs/product-engineering/features/account-statement/backend/data-model) for `transaction_category` context. *** Use cases — why organizations need it [#use-cases--why-organizations-need-it] | Scenario | How Account Statement helps | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Period reconciliation** | Finance can answer “What did we pay or add as advance, and what bills landed in the same window?” without tracing Ledger running balances. | | **Advance vs bills** | Prepaid orgs see **advance top-ups** and **bills** in one timeline, so they can see whether utilization matches expectations. | | **Walk-in settlement** | WalkIn orgs see **org payments** and **bills** together for a date range — useful for branch or corporate billing reviews. | | **Audits and sharing** | A single grid for a chosen range is easier to export or explain to non-technical stakeholders than a long Ledger with mixed event types. | | **Trust without “Verify Ledger”** | Numbers are **recomputed from source rows** (bills, org payments, categorized advance/correction transactions), not from a chain that can drift after one bad entry. | These use cases assume the organization is **Prepaid** or **WalkIn**; PostPaid continues to use invoices and Ledger (see below). *** Where it sits in the product [#where-it-sits-in-the-product] *** Why Account Statement vs Ledger? [#why-account-statement-vs-ledger] The **Ledger** records every financial movement with a running balance. That is powerful but brittle: one wrong or duplicate line skews everything below it; webhooks can double-post or miss; staff spend time on **Verify Ledger** fixes. Account Statement **does not replace** Ledger for compliance or full audit trails. It **complements** it with a read-only, **source-based** snapshot for a chosen period. *** Who sees Account Statement? [#who-sees-account-statement] The tab appears in Organization Management only for: | Organization type | `orgPaymentType` | Visibility | | ----------------- | ---------------- | ------------------------------------------ | | **Prepaid** | `1` | Shown | | **WalkIn** | `2` | Shown | | **PostPaid** | `0` | Hidden — PostPaid uses invoices and Ledger | *** What it shows (conceptually) [#what-it-shows-conceptually] Three kinds of **business events** appear on one date-grouped timeline (how rows are built and filtered: [Statement behavior](/docs/product-engineering/features/account-statement/backend/statement-behavior), [API](/docs/product-engineering/features/account-statement/backend/api-references)): | Entry type | Meaning | | ----------------------------------- | ------------------------------------------------------------------------------------------------- | | **Bill** | Active bills for the org in the selected period | | **Payment** | Organization-category collections (cash, card, cheque, online, etc.) | | **Advance / manual credit / debit** | Advance collected and staff manual corrections (only where categorized correctly in backend data) | `billDue` (bill total minus advance on bill) is computed on the **frontend** for display. *** What it excludes (product view) [#what-it-excludes-product-view] | Excluded | Why it matters to orgs | | -------------------------------------- | ----------------------------------------------------------------------- | | Cancelled, refunded, written-off bills | Only “live” billing counts toward utilization in the statement window | | Patient-side payments | Statement is **org ↔ lab** cash flow, not patient wallet | | Advance-type payment lines | Avoids double-counting with advance rows from `OrganizationTransaction` | | PostPaid invoice settlement | Different product path; tab is not offered for PostPaid | *** Feature scope summary [#feature-scope-summary] * **Prepaid** and **WalkIn** only; PostPaid uses existing invoice and Ledger flows * **Data from 10th March 2026** — enforced in UI and aligned with `transaction_category` population (see Backend) * **Conversion-aware date window** — if the org changed payment type, the API may clamp the range; see [Statement behavior](/docs/product-engineering/features/account-statement/backend/statement-behavior) * **No new product tables** — projection over existing data; see [Data model](/docs/product-engineering/features/account-statement/backend/data-model) *** Document map [#document-map] * [Workflow Guide](/docs/product-engineering/features/account-statement/workflow-guide) — step-by-step UI walkthrough with screenshots * [Backend — overview](/docs/product-engineering/features/account-statement/backend) — request pipeline * [Backend — data model](/docs/product-engineering/features/account-statement/backend/data-model) — linked source tables, `transaction_category` * [Backend — statement behavior](/docs/product-engineering/features/account-statement/backend/statement-behavior) — conversion window, inclusion rules * [Backend — API](/docs/product-engineering/features/account-statement/backend/api-references) — HTTP contract and payload fields * [Frontend — overview](/docs/product-engineering/features/account-statement/frontend) — component tree * [Frontend — container](/docs/product-engineering/features/account-statement/frontend/container) — tab, date guard, fetch flow * [Frontend — mapping](/docs/product-engineering/features/account-statement/frontend/mapping) — API → rows * [Frontend — grid](/docs/product-engineering/features/account-statement/frontend/grid) — AG Grid * [Frontend — types](/docs/product-engineering/features/account-statement/frontend/types) — TypeScript & constants * [Design Decisions](/docs/product-engineering/features/account-statement/design-decisions) # Workflow Guide Account Statement Workflow Guide [#account-statement-workflow-guide] This guide walks the **lab finance / admin** flow end to end: from Organization Management to a loaded statement grid. For UI implementation details, see [Frontend](/docs/product-engineering/features/account-statement/frontend); for API rules, see [Backend](/docs/product-engineering/features/account-statement/backend). *** End-to-end workflow [#end-to-end-workflow] *** 1. Open Organization Management [#1-open-organization-management] Go to **Finance → Organization Management** and open the organization you want to review. Organization Management What you should see [#what-you-should-see] * The organization list and detail area for the selected org * Standard organization tabs (details, contacts, etc.) — **Account Statement** appears only for the right payment types (next step) *** 2. Preview Organization Details [#2-preview-organization-details] * Click on any organization row to navigate to its detailed view. * Review organization information, including details, contacts, and eligibility for the Account Statement tab. *** 3. Open the Account Statement tab [#3-open-the-account-statement-tab] Select **Account Statement**. The tab body mounts only when this tab is active (other tabs do not trigger statement fetches). Account Statement tab On load [#on-load] * A **blue info banner** reminds you that statement data is valid from **10 March 2026** onward (aligned with backend `transaction_category` — see [Overview](/docs/product-engineering/features/account-statement) callout and [Data model](/docs/product-engineering/features/account-statement/backend/data-model)). * The grid is empty until you apply a valid date range and a successful fetch completes. *** 4. Set the date range and load data [#4-set-the-date-range-and-load-data] Date rules (product) [#date-rules-product] * **Start date before 10 March 2026** — warning is shown, rows are cleared, and the API is **not** called. * **Valid range** — the client calls `GET /api-v3/organization/account-statement` and maps the three lists into one timeline (see [Container](/docs/product-engineering/features/account-statement/frontend/container) and [Row mapping](/docs/product-engineering/features/account-statement/frontend/mapping)). *** 5. Read the statement grid [#5-read-the-statement-grid] Rows are **grouped by calendar day**. Each group header shows the date, row count, and **Collection** vs **Billing** subtotals (see [Grid](/docs/product-engineering/features/account-statement/frontend/grid)). | Entry type (badge) | Meaning | | ------------------------- | ------------------------------- | | **Bill** | Bill in range | | **Payment** | Org payment collected | | **Advance** | Advance / top-up line | | **Manual Credit / Debit** | Staff correction on org account | Optional columns (source, order number, comment) are available from the grid **column tool panel**. *** 6. When something looks wrong [#6-when-something-looks-wrong] | Situation | What to check | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **No tab** | Organization is PostPaid — not in scope for Account Statement | | **Warning + no rows** | Start date is before **10 March 2026** — widen range forward | | **Empty grid after fetch** | No activity in range, or **204** from API (e.g. conversion window entirely before current payment model — see [Statement behavior](/docs/product-engineering/features/account-statement/backend/statement-behavior)) | | **Toast: fetch failed** | Network or server error — retry; if it persists, check session and org id | *** Related docs [#related-docs] * [Overview](/docs/product-engineering/features/account-statement) — use cases and scope * [Frontend](/docs/product-engineering/features/account-statement/frontend) — components, mapping, grid * [Backend](/docs/product-engineering/features/account-statement/backend) — data sources and API # Backend import Image from 'next/image'; import screenshot3 from '@/images/antimicrobiogram/Screenshot_2026-05-25_at_12.41.53_PM-6e7c1f47-d0bc-4f24-a953-cb1ee1f48550.png'; import screenshot4 from '@/images/antimicrobiogram/Screenshot_2026-05-25_at_12.41.20_PM-c0992b42-1358-4c7a-ba7f-8fa8ba9b6193.png'; Backend [#backend] Antimicrobiogram spans feature flags, login/session propagation, reporting models, read APIs, scheduled jobs, repair triggers, repair queue processing and migration tracking The easiest way to understand the backend is to start from the core principle: > Antimicrobiogram is a precomputed reporting feature. The backend does not build the matrix directly from live transactional joins on every request. Instead, it converts source report values into a dedicated summary table and then serves that table through a focused reporting API. System design [#system-design] Core tables [#core-tables] 1. OrganismAntibioticSummary [#1-organismantibioticsummary] This is the master reporting table. It stores one denormalized row per organism-antibiotic result item that should contribute to Antimicrobiogram analytics. The table is intentionally wide because it needs to support: * raw result browsing * sensitivity aggregation * organization scoping * patient and order context * date-based filtering This table is the read source for both the raw list view and the grouped sensitivity view. 2. OrganismAntibioticSummaryRepairQueue [#2-organismantibioticsummaryrepairqueue] This table is the historical repair queue. Each row represents: * one lab * one date window * one repair lifecycle Typical statuses: * `PENDING` * `PROCESSING` * `COMPLETED` * `FAILED` This queue is fed in two ways: * automatically by SQL triggers * manually by the repair API 3. crelio_data_migrations [#3-crelio_data_migrations] This table is the central tracker for scheduled backfill jobs. For Antimicrobiogram, the relevant rows are: * `migration_type = 'ANTIMICROBIOGRAM'` This table tells support and engineering: * whether a backfill was queued * whether it is scheduled * whether it is running * how long it took 4. labFeatures [#4-labfeatures] The important flag is: * `is_antimicrobiogram_enabled` This flag is referenced in three places: * Support Dashboard configuration * session payload generation * repair trigger eligibility Session propagation [#session-propagation] The feature flag is exposed to the frontend during login/session setup. The backend places `is_antimicrobiogram_enabled` into the session payload during login/session setup, and the React app reads that value from Redux session state. Read API [#read-api] The main read endpoint is: * `GET /api-v3/report/organism-antibiotic-results/` This endpoint can return: * raw summary rows * grouped sensitivity rows * or both Important request params [#important-request-params] * `organization_id` * `result_type` * `report_date_from` * `report_date_to` * `bill_time_from` * `bill_time_to` * `sample_type` * `service_name` * `response_type` Response behavior [#response-behavior] The view returns: ```json { "results": [...], "total": 123, "sensitivity": [...] } ``` This is a very practical response shape because the frontend can use the same endpoint for raw browsing and for matrix rendering. POST control API [#post-control-api] The same view also exposes a control entrypoint: * `POST /api-v3/report/organism-antibiotic-results/?type=...` Supported job types: * `DAILY` * `MIGRATION` * `REPAIR` * `ALL` This is the orchestration-friendly endpoint that lets the scheduler or an internal runner invoke the processors in a controlled way. Manual repair API [#manual-repair-api] The manual repair endpoint is: * `POST /api-v3/report/organism-antibiotic-summary/repair/` This endpoint accepts a lab-local window payload such as: ```json { "start_date": "2026-04-08T00:00:00", "end_date": "2026-04-08T23:59:59" } ``` The view: 1. resolves lab id from session 2. reads the lab timezone 3. parses the incoming datetimes 4. normalizes them 5. enqueues a repair queue row This is useful for support or engineering when a manual historical rerun is needed. Summary-building pipeline [#summary-building-pipeline] The core worker is: ```python OrganismAntibioticSummary._process_window(lab_id, start_date, end_date) ``` This is the heart of the feature. Step 1: Identify reports for the requested window [#step-1-identify-reports-for-the-requested-window] The processor begins by selecting `labReportRelation` ids where: * `labId_id = lab_id` * `billId.billTime >= start_date` * `billId.billTime < end_date` This means the processing window is anchored on `billing.billTime`. That is important. All three job types - daily, repair, and migration - use the same billing-date-driven window model. Step 2: Clear old summary rows for those reports [#step-2-clear-old-summary-rows-for-those-reports] Once the report ids are known, the processor deletes existing summary rows for the same lab/report set so the window can be rebuilt cleanly. This is what makes the process idempotent in practice. The rebuild path does not append blindly. It replaces the affected slice. Step 3: Load report values [#step-3-load-report-values] The processor uses: * `ReportValue.get_values(reportForId_id__in=report_ids)` This abstraction matters because report values may live across SQL-backed and document-backed storage, and `ReportValue.get_values(...)` already knows how to fetch them correctly. Step 4: Keep only Antimicrobiogram-relevant component types [#step-4-keep-only-antimicrobiogram-relevant-component-types] The allowed component types are: * `microbiology` * `organism` * `antibiotic resistance` That means the feature is deliberately scoped to organism-antibiotic result content and ignores unrelated report components. Step 5: Group report values [#step-5-group-report-values] The values are grouped by: * `(reportForId_id, profileTestId)` That grouping is helpful because one report can have multiple related component payloads that need to be processed together. Step 6: Load related report context [#step-6-load-related-report-context] The processor builds an `lrr_map` using `select_related(...)` for: * patient * billing * report/test * profile test * organization * lab This gives the builder everything it needs to create a rich denormalized row. Step 7: Prepare common fields [#step-7-prepare-common-fields] The method `_prepare_common_fields(...)` extracts the shared metadata used by both microbiology and molecular rows. This is the shared envelope around every summary row. Microbiology row building [#microbiology-row-building] Microbiology rows come from report values where: * `component_type == "microbiology"` For each item in the component `value` list, the builder creates one `OrganismAntibioticSummary` row. Every microbiology organism-antibiotic result item becomes a directly readable summary row with all the important metadata attached.
Antimicrobiogram patient list
Raw Microbiology results in grid form
Molecular row building [#molecular-row-building] Molecular rows are built a little differently because the payload is split across: * `organism` component values * `antibiotic resistance` component values
Antimicrobiogram raw rows
Raw Molecular results in grid form
The builder: 1. collects the organism list 2. collects the antibiotic resistance list 3. reads `organism_mappings` from each antibiotic resistance item 4. creates one summary row for each allowed organism-antibiotic pairing This makes molecular data behave like microbiology data from the reporting layer's perspective, even though the source payload shape is different. Sensitivity calculation [#sensitivity-calculation] The backend calculates grouped sensitivity using: * `OrganismAntibioticSummary.calculate_sensitivity(qs)` The grouping keys are: * `organism` * `organism_category` * `antibiotic` For each group, the backend computes: * `total_tested` * `sensitive_count` * `resistant_count` * `intermediate_count` * `sensitivity_pct` * `resistant_pct` * `intermediate_pct` This grouped payload is exactly what the frontend needs to draw the heatmap. Daily job [#daily-job] The daily processor is: * `OrganismAntibioticSummary.run_daily_antimicrobiogram_jobs()` What it does [#what-it-does] 1. fetches all labs where `is_antimicrobiogram_enabled = True` 2. resolves each lab's previous local day 3. converts that day into UTC 4. calls `_process_window(...)` Why this is nice [#why-this-is-nice] The logic is lab-timezone aware and scales cleanly across many labs. Every enabled lab gets yesterday's slice rebuilt using its own timezone rather than a shared UTC-only calendar day. Migration job [#migration-job] The migration processor is: * `CrelioDataMigrations.run_scheduled_antimicrobiogram_jobs()` How rows are selected [#how-rows-are-selected] The processor looks for: * `migration_type = 'ANTIMICROBIOGRAM'` * `is_scheduled = True` * `job_status = 'Pending'` What it does [#what-it-does-1] 1. resolves the requested processing window 2. marks the row as running 3. calls `_process_window(...)` 4. updates job tracking with completion status and time taken 5. removes the job from active scheduled processing once the run is complete This is the first-load historical backfill engine for newly enabled labs. Repair queue processor [#repair-queue-processor] The repair processor lives in: * `OrganismAntibioticSummaryRepairQueue.process_queue(...)` Queue execution flow [#queue-execution-flow] 1. selects `PENDING` rows using `select_for_update(skip_locked=True)` 2. marks a row as `PROCESSING` 3. resolves the lab timezone 4. calls `_process_window(...)` 5. marks the row as `COMPLETED` If an exception happens, the row is marked `FAILED` with the error message. This is a solid queue design because it is: * safe under concurrency * explicit about job lifecycle * simple to inspect operationally Billing repair trigger [#billing-repair-trigger] The billing-side historical repair trigger is: * `trg_antimicrobiogram_billing_update` Trigger purpose [#trigger-purpose] When a historical billing record changes in a way that affects Antimicrobiogram membership, the feature immediately queues a repair for that billed day. What it checks [#what-it-checks] 1. lab feature flag is enabled 2. `isCancel` changed 3. the affected day is not the current day What it enqueues [#what-it-enqueues] It inserts a row into: * `OrganismAntibioticSummaryRepairQueue` with: * `start_date = affected day 00:00:00` * `end_date = affected day 23:59:59` * `status = 'PENDING'` If the same day is already represented in the queue, the row is reactivated through the `ON DUPLICATE KEY UPDATE status = 'PENDING'` pattern. LabReportRelation repair trigger [#labreportrelation-repair-trigger] The report-side historical repair trigger is: * `trg_antimicrobiogram_lrr_update` Trigger purpose [#trigger-purpose-1] This trigger captures report-level changes that affect the summary content or attribution of a historical day. Watched columns [#watched-columns] * `dismissed` * `sampleRedrawFlag` * `orgId_id` * `reportID_id` * `reportFormatId_id` * `completedTests` * `isSigned` * `isSynced` * `isPartialFill` What it does [#what-it-does-2] 1. verifies the feature is enabled for the lab 2. compares old and new values for the watched fields 3. fetches the associated `billTime` and `labTimeZone` 4. builds the historical one-day repair window 5. ignores present-day changes because those are covered by the daily job 6. writes or re-queues a `PENDING` repair row This is the key automation that keeps historical Antimicrobiogram data trustworthy without requiring manual intervention for every report-side change. Why the repair model is strong [#why-the-repair-model-is-strong] The repair model is strong because it is split correctly: * daily job handles normal ingestion * migration job handles first-time or large-window history * triggers handle targeted historical corrections That means the system can stay both fast and correct. Debugging and verification SQL [#debugging-and-verification-sql] Check feature enablement [#check-feature-enablement] ```sql SELECT labForId_id AS lab_id, is_antimicrobiogram_enabled FROM labFeatures WHERE labForId_id = ?; ``` Check migration rows [#check-migration-rows] ```sql SELECT id, lab_id, is_scheduled, start_date, end_date, job_status, time_taken FROM crelio_data_migrations WHERE lab_id = ? AND migration_type = 'ANTIMICROBIOGRAM' ORDER BY id DESC; ``` Check repair queue rows [#check-repair-queue-rows] ```sql SELECT id, lab_id, start_date, end_date, status, error_message, created_at, updated_at FROM OrganismAntibioticSummaryRepairQueue WHERE lab_id = ? ORDER BY id DESC; ``` Check summary rows [#check-summary-rows] ```sql SELECT organization_id, organization_name, organism, antibiotic, interpretation, COUNT(*) AS row_count FROM OrganismAntibioticSummary WHERE lab_id = ? AND report_date >= ? AND report_date < ? GROUP BY organization_id, organization_name, organism, antibiotic, interpretation ORDER BY organization_name, organism, antibiotic; ``` Short mental model for backend engineers [#short-mental-model-for-backend-engineers] If you want the shortest accurate backend model, use this: 1. enablement sets the lab feature and queues migration 2. migration loads history 3. daily job loads yesterday 4. triggers queue historical day repairs 5. repair processor rebuilds those days 6. reads come from `OrganismAntibioticSummary` 7. sensitivity is grouped at read time from the summary rows That is the backend in one chain. # Design Decisions Design decisions [#design-decisions] Key architectural choices and trade-offs in one place: summary table, job split (daily/repair/migration), trigger-driven repairs, billing-time anchor, and export-first design. 1. A summary table was chosen instead of live report aggregation [#1-a-summary-table-was-chosen-instead-of-live-report-aggregation] Antimicrobiogram could have been implemented as a live aggregation over: * `billing` * `labReportRelation` * report values * patient and organization joins That would have made reads expensive and repetitive. Instead, the system stores a denormalized row set in `OrganismAntibioticSummary`. Why this is a good choice [#why-this-is-a-good-choice] * the same processed row can support both raw result browsing and grouped matrix rendering * the frontend stays fast because it reads from one purpose-built table * historical rebuilds become operationally straightforward * the feature is easier to export because the data is already shaped for reporting This is exactly the kind of tradeoff you want in a reporting-heavy feature. 2. Organization-wise reporting was made the default [#2-organization-wise-reporting-was-made-the-default] The feature is intentionally organization-wise. That is not just a UI preference. It is a domain decision. Why it is the right call [#why-it-is-the-right-call] * local susceptibility patterns matter at organization level * combining all organizations too early can hide important variation * users usually want a clean, scoped answer they can trust immediately The frontend reinforces this by auto-selecting the first organization with data. 3. Three job types were used instead of one generic batch [#3-three-job-types-were-used-instead-of-one-generic-batch] The feature uses different jobs because each job type solves a different operational need. Daily jobs [#daily-jobs] * standard freshness * small previous-day windows Repair jobs [#repair-jobs] * targeted historical corrections * automatically fixing old days after meaningful source updates Migration jobs [#migration-jobs] * first-time onboarding * long-window historical backfill Trying to force all three use cases into a single generic runner would make the feature harder to operate and harder to reason about. 4. Repair is trigger-driven [#4-repair-is-trigger-driven] The feature watches changes through SQL triggers on: * `billing` * `labReportRelation` This is a very pragmatic design. Why trigger-driven repair works well here [#why-trigger-driven-repair-works-well-here] * the system notices source-of-truth changes immediately * historical days can be repaired automatically * support does not need to manually detect every meaningful update * rebuilds stay focused on the exact day that changed This is the kind of operational automation that makes a reporting feature feel dependable. 5. Present-day updates are handled differently from historical updates [#5-present-day-updates-are-handled-differently-from-historical-updates] The daily job owns the previous day. The repair triggers focus on prior days. That separation is clean because present-day data is still moving, while historical days should be stable and correctable. This keeps the system from doing unnecessary noisy repairs for data that has not yet rolled into the normal daily reporting window. 6. Billing date was chosen as the processing anchor [#6-billing-date-was-chosen-as-the-processing-anchor] All rebuild windows are based on `billing.billTime`. This is a strong operational anchor because: * it aligns with historical day-based processing * it gives one consistent way to select the affected summary slice * it makes daily, repair, and migration jobs all speak the same window language When a system has multiple job types, using one shared window anchor reduces confusion. 7. One endpoint serves both raw rows and grouped sensitivity [#7-one-endpoint-serves-both-raw-rows-and-grouped-sensitivity] The read API returns: * `results` * `sensitivity` This is a very engineer-friendly contract. Why it is good [#why-it-is-good] * raw views and heatmap views stay backed by the same source query * the frontend does not need separate orchestration logic for two different APIs * debugging is easier because the detailed rows and the grouped matrix come from the same request family That is a strong interface design for a feature like this. 8. Microbiology and molecular data share one reporting model [#8-microbiology-and-molecular-data-share-one-reporting-model] The source payloads are not identical: * microbiology rows come from `microbiology` * molecular rows come from `organism` and `antibiotic resistance` But the feature still stores them in one summary table with a `result_type` discriminator. Why this is a good design [#why-this-is-a-good-design] * users see one coherent reporting surface * the backend can reuse the same filtering and export behavior * the frontend can switch between raw and matrix views without jumping across totally different models This is a great example of unifying different source shapes behind one reporting contract. 9. Timezone-aware windows are essential [#9-timezone-aware-windows-are-essential] The feature processes days in the lab's local timezone and then converts them to UTC for querying. This matters a lot. If reporting days were treated as plain UTC days, labs in different regions would see confusing edge behavior around midnight. By resolving each daily or repair window in lab-local time first, the feature stays aligned with business reality. That is the correct design for multi-timezone reporting. 10. Excel export is part of the core design, not an afterthought [#10-excel-export-is-part-of-the-core-design-not-an-afterthought] Export is built into the main Antimicrobiogram experience. That is important because reports like this are often: * reviewed outside the app * shared across teams * discussed in operational or clinical meetings The export includes matrix data, filter metadata, and legend styling, which makes it a true deliverable, not just a raw download. 11. Queue tables make operations transparent [#11-queue-tables-make-operations-transparent] Two separate operational trackers are used: * `OrganismAntibioticSummaryRepairQueue` * `crelio_data_migrations` This separation is a good design choice. Why it helps [#why-it-helps] * engineers can inspect historical repairs without mixing them with long-window migrations * support can understand whether a lab is waiting on onboarding history or a one-day fix * schedulers and processors have simpler contracts Operational clarity is underrated in reporting features, and this design gets it right. 12. The overall architecture is balanced [#12-the-overall-architecture-is-balanced] The strongest thing about the feature is balance. It balances: * detail and performance * automation and control * raw exploration and grouped reporting * onboarding backfill and daily freshness * historical correctness and runtime simplicity That balance is why the feature is not just technically complete, but also easy to operate and easy to explain. # Frontend Frontend [#frontend] This page explains the Antimicrobiogram frontend from the point of view of an engineer reading the React code. Primary files [#primary-files] | File | Responsibility | | :----------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------- | | `apps/livehealth-frontend/src/components/Finance/MIS/Components/TestValues/index.tsx` | Parent container, tab wiring, feature gating | | `apps/livehealth-frontend/src/components/Finance/MIS/Components/TestValues/components/OrganismAntibioticResultsView.tsx` | Main Antimicrobiogram UI | | `apps/livehealth-frontend/src/components/Finance/MIS/Components/TestValues/helpers.ts` | Fetch helpers, heatmap builders, export helpers | | `apps/livehealth-frontend/src/components/Finance/MIS/Components/TestValues/constants.ts` | Colors, labels, date format constants | | `apps/livehealth-frontend/src/components/Finance/MIS/Components/TestValues/utils/interface.ts` | Type contracts used by the screen | Feature gating [#feature-gating] The frontend checks one session-level switch: ```ts Boolean(sessionState?.is_antimicrobiogram_enabled) ``` That single flag controls whether the `Organism/Antibiotic Results` tab is added to the `Test Values` tab list. This is a clean product choice because it keeps the rest of the screen untouched for labs that do not use the feature. Parent screen behavior [#parent-screen-behavior] The Antimicrobiogram UI is mounted from `TestValues/index.tsx`. That parent file does three important things: 1. decides whether the `Organism/Antibiotic Results` main tab should exist 2. owns the sub-tab state for: * `Microbiology` * `Molecular` * `Antimicrobiogram` 3. hands rendering over to `OrganismAntibioticResultsView` when the tab is active This means the feature does not live in isolation. It is intentionally part of a broader organism/antibiotic result exploration surface. Main component state [#main-component-state] `OrganismAntibioticResultsView` owns the key runtime state for the feature. Data state [#data-state] `rawResults`, `sensitivityResults`, `dataLoading` Filter state [#filter-state] `selectedSample`, `selectedService`, `selectedOrganization`, `selectedPatient`, `dateField`, `startDate`, `endDate` Export state [#export-state] `exporting`, `downloadOpen` Grid state [#grid-state] * `antimicrobiogramGridApiRef` This is a good state split. It keeps fetching, filtering, and export concerns separate enough to reason about without making the component over-abstracted. Fetch lifecycle [#fetch-lifecycle] The fetch flow is driven by `loadOrganismAntibioticResults(...)`, which wraps `fetchOrganismAntibioticResults(...)`. Common request behavior [#common-request-behavior] Every request sends: * `response_type=both` * date range as Unix seconds * optional `sample_type` The endpoint is: * `/api-v3/report/organism-antibiotic-results/` Result-type behavior by sub-tab [#result-type-behavior-by-sub-tab] | Sub-tab | Request behavior | | :----------------- | :-------------------------------------------------------------------------- | | `Microbiology` | sends `result_type=MICROBIOLOGY` | | `Molecular` | sends `result_type=MOLECULAR` | | `Antimicrobiogram` | does not send `result_type`, so the backend can return both source families | Date behavior [#date-behavior] Raw result tabs can switch between: * `report_date` * `bill_time` The Antimicrobiogram heatmap always uses: * `report_date_from` * `report_date_to` That keeps the matrix experience simpler. useEffect-driven refresh model [#useeffect-driven-refresh-model] The screen refetches when important inputs change: * sub-tab * date field * date range * selected sample * selected service * selected organization This means the page behaves like a report screen, not like a static cached table. Filter behavior [#filter-behavior] Not every filter behaves the same way, and that is intentional. | Filter | Raw result tabs | Antimicrobiogram | | :----------- | :-------------- | :--------------- | | Date range | server refetch | server refetch | | Sample type | server refetch | server refetch | | Service | client filter | server refetch | | Patient | client filter | not shown | | Organization | not shown | server refetch | This pattern makes sense: * raw tabs behave like a data explorer * Antimicrobiogram behaves like a scoped report Default organization selection [#default-organization-selection] The Antimicrobiogram sub-tab is organization-wise, so the frontend needs a sensible default. The flow is: 1. fetch the result set 2. derive organization options from `organization_id` and `organization_name` 3. sort the options 4. if no organization is selected yet, select the first available one 5. refetch with `organization_id` This gives a nice first-load experience because the user does not land on an empty or ambiguous lab-wide view. The page immediately narrows itself to a specific organization with data. Option building [#option-building] The frontend builds filter options from the fetched rows using `buildUniqueOptions(...)`. It derives: * patient options * service options * sample options * organization options This means the filters are data-backed and only show values that actually exist in the currently fetched result set. Heatmap rendering [#heatmap-rendering] The Antimicrobiogram matrix is built from the backend `sensitivity` payload, then narrowed using the filtered result set. The transformation steps are: 1. gather all visible antibiotics across the sensitivity rows 2. sort antibiotic names alphabetically 3. assign synthetic column ids like `antibiotic_0`, `antibiotic_1`, and so on 4. create one row per organism 5. attach the full aggregated antibiotic cell object into the correct synthetic field Each cell object carries: * `name` * `total_tested` * `sensitive_count` * `resistant_count` * `intermediate_count` * `sensitivity_pct` * `resistant_pct` * `intermediate_pct` That is a nice design because the renderer can stay focused on display while the data object still keeps the deeper grouped values attached to it. Heatmap headers [#heatmap-headers] The heatmap has custom headers for both the organism column and the antibiotic columns. Organism header [#organism-header] The organism header shows: * `Organism Group` * `N=` Antibiotic headers [#antibiotic-headers] Each antibiotic column header shows the antibiotic name in a compact heatmap-style cell. This makes the matrix dense but still readable. Color system [#color-system] The heatmap colors are defined in `constants.ts`. | Range | Meaning | Color family | | :---------- | :----------------- | :----------- | | `>= 90%` | high sensitivity | green | | `70% - 89%` | medium sensitivity | amber | | `< 70%` | low sensitivity | red | | no data | empty | white | The same thresholds drive: * cell background colors * cell text colors * legend swatches * Excel export cell styles That consistency is important. The export looks and feels like the on-screen report instead of becoming a disconnected spreadsheet dump. Export behavior [#export-behavior] Excel export is implemented through the visible ag-Grid instance and `exportAntimicrobiogramToExcel(...)`. What gets exported [#what-gets-exported] * all heatmap columns * matrix cell values as formatted strings such as `46.02% (N=415)` * metadata rows * legend rows Metadata rows included [#metadata-rows-included] * organization name * period * sample type * service * generated-on timestamp Sheet details [#sheet-details] * file name prefix = `Antimicrobiogram_YYYYMMDD_HHmm` * sheet name = `Antimicrobiogram` This gives the exported sheet enough context to be meaningful even after it leaves the app. Why the frontend implementation is good [#why-the-frontend-implementation-is-good] The frontend design is strong for a few reasons: * it keeps feature gating simple through session state * it reuses one view for raw rows and matrix rendering * it uses the backend summary table rather than trying to aggregate raw report values in the browser * it makes organization scoping automatic * it keeps export logic close to the rendered grid In short, the frontend is doing exactly what it should do: present, filter, and export already shaped data instead of owning the reporting computation itself. Quick engineer mental model [#quick-engineer-mental-model] If you are new to the code, this is the easiest way to think about the UI: 1. `index.tsx` decides whether the feature exists on the page 2. `OrganismAntibioticResultsView.tsx` owns the runtime behavior 3. `helpers.ts` turns the API payload into a heatmap-friendly shape 4. ag-Grid renders the rows and columns 5. the same grid powers Excel export That is the full frontend story. # Overview import Image from 'next/image'; import screenshot1 from '@/images/antimicrobiogram/Screenshot_2026-05-25_at_12.42.15_PM-e3781579-308f-4a0e-9516-293c6b7d5c55.png'; Overview [#overview] Antimicrobiogram is an organization-scoped heatmap that summarizes organism vs antibiotic sensitivity from microbiology and molecular report payloads. It trades live-join complexity for predictable, fast reads by storing denormalized summary rows and computing grouped sensitivity at read time. Why it exists * Fast, repeatable reporting for stewardship and operations. * Exportable, lab-scoped view that’s easy to interpret and share. What users see * Operations → Operational Export → Organism/Antibiotic Results (sub-tabs: Microbiology, Molecular, Antimicrobiogram). * Matrix: rows = organism, columns = antibiotic, cell = sensitivity% (with N).
Antimicrobiogram heatmap overview
Why this feature matters [#why-this-feature-matters] Antimicrobiogram is useful for both operational and clinical reasons. Operationally, it gives the lab a reusable analytics view on top of already approved and bill-linked report data. That means the lab can study historical trends without reprocessing raw report payloads in the browser every time. Clinically, it helps teams understand local antibiotic sensitivity trends. That is useful for stewardship discussions, internal review, retrospective analysis, and exported sharing with stakeholders who need a quick organism-antibiotic picture. What the user sees [#what-the-user-sees] The feature appears inside the `Operational Export` module in Operations under `Organism/Antibiotic Results` Inside that main tab, the UI exposes three sub-tabs: * `Microbiology` * `Molecular` * `Antimicrobiogram` The first two sub-tabs help the user inspect detailed rows for raw results for Microbiology and Molecular test results. The third sub-tab is the true matrix-style report that the business refers to as Antimicrobiogram. Core product rules [#core-product-rules] | Rule | Meaning | | :-------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------- | | Antimicrobiogram is always organization-wise | The visible matrix is built for a single organization at a time | | Default selection is the first organization with available data | When the page opens, the frontend automatically selects the first organization derived from the fetched result set | | Rows are organisms | Each matrix row represents one detected organism | | Columns are antibiotics | Each matrix column represents one antibiotic | | Cell value is sensitivity percentage | The visible percentage is `Sensitive / Total Tested` | | Export is first-class | Users can download the matrix as Excel directly from the page | Sensitivity calculation [#sensitivity-calculation] The most important business rule in this feature is the sensitivity percentage calculation. For each organism + antibiotic combination: ```text Sensitivity % = (Sensitive count / Total tested count) * 100 ``` Example: * organism = `E. coli` * antibiotic = `Ampicillin` * tested count = `415` * sensitive = `191` * resistant = `200` * intermediate = `24` So: ```text Sensitivity % = 191 / 415 * 100 = 46.02% ``` That sensitivity percentage is what is displayed in the cell. The same grouped data also keeps the resistant and intermediate counts and percentages, which are useful for the tooltip/export layer and for future extension. Feature surfaces [#feature-surfaces] | Surface | Responsibility | | :---------------------------- | :------------------------------------------------------- | | Support Dashboard | Turns the feature on or off at lab level | | Login/session payload | Exposes `is_antimicrobiogram_enabled` to the frontend | | Operational Export page | Shows the `Organism/Antibiotic Results` tab when enabled | | OrganismAntibioticResultsView | Fetches, filters, renders, and exports the data | | Summary and queue tables | Store the processed rows and rebuild work | Core storage model [#core-storage-model] Antimicrobiogram is backed by three main operational tables plus the lab feature flag. | Layer | Table | Why it exists | | :---------------- | :---------------------------------------------------------------- | :---------------------------------------------------------------- | | Feature switch | `labFeatures.is_antimicrobiogram_enabled` | Controls availability and backfill onboarding | | Summary store | `OrganismAntibioticSummary` | Holds denormalized organism-antibiotic result rows used for reads | | Repair queue | `OrganismAntibioticSummaryRepairQueue` | Holds one-day historical windows that must be rebuilt | | Migration tracker | `crelio_data_migrations` with `migration_type='ANTIMICROBIOGRAM'` | Tracks large backfill jobs such as first-time enablement | The three job types [#the-three-job-types] The feature stays fresh using three complementary job types. 1. Daily job [#1-daily-job] This is the normal day-to-day loader. It runs once per day and processes the previous local day for each enabled lab. Example: * job execution time = `2026-04-09 02:00` local lab time * processed range = `2026-04-08 00:00:00` to `2026-04-08 23:59:59` local day 2. Repair job [#2-repair-job] This job exists for historical correction. If something important changes on a past bill or report, the feature does not wait for a full migration. Instead, it queues a one-day repair for the affected day and rebuilds just that slice. 3. Migration job [#3-migration-job] This is the historical backfill path. When a lab enables Antimicrobiogram, the system schedules a long-window backfill so the report is useful from day one instead of showing only newly arriving data. End-to-end data flow [#end-to-end-data-flow] At a high level, the flow looks like this: Why the architecture works well [#why-the-architecture-works-well] This feature works well because it separates concerns cleanly: * source systems continue to save report values normally * backend jobs turn those values into a reporting shape * reads are fast because they hit the summary table instead of the full transactional graph * repairs are precise because they are day-window driven * the frontend can stay simple and focus on filtering, matrix layout, and export Short mental model for engineers [#short-mental-model-for-engineers] If you want the simplest possible mental model, think of Antimicrobiogram like this: 1. report values are the raw truth 2. `OrganismAntibioticSummary` is the reporting truth 3. daily jobs keep yesterday ready 4. repair triggers keep historical days correct 5. migration jobs make newly enabled labs useful immediately 6. the frontend turns grouped summary rows into an organization-wise heatmap That is the full feature in one chain. # Workflow Guide import Image from 'next/image'; import screenshot1 from '@/images/antimicrobiogram/Screenshot_2026-05-25_at_12.42.15_PM-e3781579-308f-4a0e-9516-293c6b7d5c55.png'; import screenshot2 from '@/images/antimicrobiogram/Screenshot_2026-05-25_at_12.42.50_PM-0e605c01-54f7-415d-8e3e-09713934f86a.png'; import flagEnable from '@/images/antimicrobiogram/Flag-Enable.png'; Workflow Guide [#workflow-guide] This page explains the feature the way an engineer, support person, or product owner would mentally walk through it in real life. The goal here is not just to say what the code does. The goal is to make the lifecycle easy to follow: * how the feature gets enabled * how the first data arrives * how daily freshness is maintained * how historical corrections are repaired * how users finally read and export the report 1. Enabling the feature [#1-enabling-the-feature] The feature starts at the Support Dashboard. The toggle is backed by: * `labFeatures.is_antimicrobiogram_enabled` When support enables it for a lab, three things matter immediately. A. The lab now has access to the feature [#a-the-lab-now-has-access-to-the-feature] The flag becomes part of the lab feature configuration. During login/session preparation, the backend places `is_antimicrobiogram_enabled` into session payloads, and the frontend uses that flag to decide whether the `Organism/Antibiotic Results` tab should be shown. B. The feature becomes visible in Operational Export [#b-the-feature-becomes-visible-in-operational-export] Once the session carries the flag, the frontend adds the Antimicrobiogram-capable tab to the Operational Export tab in Operations. C. A historical backfill is queued [#c-a-historical-backfill-is-queued] Enabling the feature also creates a scheduled `crelio_data_migrations` entry with: * `migration_type = 'ANTIMICROBIOGRAM'` * `is_scheduled = 1` * `job_status = 'Pending'`
Feature flag enable flow
This is what makes the feature useful right after onboarding. The system does not expect the user to wait weeks for enough new daily data to accumulate. 2. Migration workflow [#2-migration-workflow] The migration job is the first big batch load for a lab. Business expectation [#business-expectation] If the lab enables the flag on `2026-04-08`, the system should prepare a one-year historical window: * start = `2025-04-08` * end = `2026-04-08` That range is queued through `crelio_data_migrations` and later executed by the Antimicrobiogram migration processor. What the migration processor does [#what-the-migration-processor-does] When the scheduled migration runner picks up that row: 1. it resolves the lab and the backfill date range 2. it marks the migration as running 3. it calls the summary builder for the requested window 4. it updates migration tracking so the job lifecycle is visible to support and engineering 5. once the run finishes, the queued migration work is treated as consumed and cleared from the active queue lifecycle In practical terms, migration is the "bring this lab up to speed" path. 3. Daily workflow [#3-daily-workflow] Daily processing is what keeps the feature current. The intended run model is straightforward: * run once every day * choose a scheduled time, for example `02:00` * convert the window using the lab timezone * process the previous local day only Example [#example] Say the lab is in a timezone where the daily job runs on: * `2026-04-09 02:00` The processed data window should be: `2026-04-08 00:00:00` through `2026-04-08 23:59:59` The backend converts that local-day window into UTC and uses it to select the relevant `billing.billTime` slice. Why this is a strong design [#why-this-is-a-strong-design] This keeps the job: * predictable * small * timezone-safe * easy to rerun if needed It also makes reasoning about freshness simple. If the user asks, "when does yesterday appear?", the answer is: after the next daily Antimicrobiogram run for that lab. 4. Repair workflow [#4-repair-workflow] Repair exists for historical corrections. This is the part of the feature that makes it feel production-grade. Historical summary tables are only trustworthy if they can respond to meaningful updates in past source records. Antimicrobiogram does that with trigger-driven repair queueing. 5. Billing trigger workflow [#5-billing-trigger-workflow] The billing trigger is: * `trg_antimicrobiogram_billing_update` It runs: * `AFTER UPDATE ON billing` What it watches [#what-it-watches] This trigger is intentionally focused. It queues repair when: * `isCancel` changes That is the right business choice. A bill cancellation can change whether the record should contribute to Antimicrobiogram analytics for that historical day. What it does step by step [#what-it-does-step-by-step] 1. checks whether `labFeatures.is_antimicrobiogram_enabled = 1` for the lab 2. checks whether `OLD.isCancel` and `NEW.isCancel` are different 3. derives the affected day from `NEW.billTime` 4. constructs: * `start_time = that day 00:00:00` * `end_time = that day 23:59:59` 5. skips present-day rows 6. inserts or re-queues a `PENDING` row into `OrganismAntibioticSummaryRepairQueue` Why this trigger matters [#why-this-trigger-matters] Without this trigger, a historical bill cancellation could leave the summary matrix out of sync with the real transactional state until someone manually repaired it. With the trigger in place, the repair path becomes automatic. 6. LabReportRelation trigger workflow [#6-labreportrelation-trigger-workflow] The report-side trigger is: * `trg_antimicrobiogram_lrr_update` It runs: * `AFTER UPDATE ON labReportRelation` What it watches [#what-it-watches-1] This trigger covers report-level changes that can affect whether a row should be part of the summary or how it should be attributed. The watched fields are: * `dismissed` * `sampleRedrawFlag` * `orgId_id` * `reportID_id` * `reportFormatId_id` * `completedTests` * `isSigned` * `isSynced` * `isPartialFill` Why these fields matter [#why-these-fields-matter] These fields directly affect reporting semantics: * whether a report should count * which organization it belongs to * whether it is complete and ready * whether it has been signed or synced * whether a redraw or dismissal changes its reporting meaning What it does step by step [#what-it-does-step-by-step-1] 1. checks whether the lab has Antimicrobiogram enabled 2. compares the watched old/new values 3. if none of them changed, exits quickly 4. fetches `billTime` and `labTimeZone` 5. derives the historical day window tied to that bill 6. skips present-day rows 7. inserts or re-queues a `PENDING` repair row in `OrganismAntibioticSummaryRepairQueue` 7. Why both triggers are needed [#7-why-both-triggers-are-needed] It is worth calling this out clearly. The billing trigger and the `labReportRelation` trigger are not duplicates. They cover different classes of historical change. | Trigger | Best at capturing | | :-------------- | :------------------------------------------------------------------------------- | | Billing trigger | Bill cancellation changes | | LRR trigger | Report completeness, ownership, visibility, signing, syncing, and redraw changes | Together they make the repair system robust. 8. Repair queue processing [#8-repair-queue-processing] Once the trigger writes a row to `OrganismAntibioticSummaryRepairQueue`, the repair processor takes over. That queue row contains: * `lab_id` * `start_date` * `end_date` * `status` * timestamps The queue lifecycle is: 1. `PENDING` 2. picked by queue processor 3. `PROCESSING` 4. summary window rebuilt 5. `COMPLETED` If a row is re-queued for the same window before processing, the trigger logic pushes it back to `PENDING`, which keeps the repair path simple and resilient. 9. One-day rebuild behavior [#9-one-day-rebuild-behavior] The rebuild unit for repair is intentionally one day. That is important because it gives a nice balance: * small enough to run quickly * large enough to fully repair the affected historical slice When a repair job runs for a day, it removes the existing summary content for that day and rebuilds it from source report values. That means the result is deterministic and easy to reason about. 10. User-facing report workflow [#10-user-facing-report-workflow] After data exists, the user-facing flow is simple. Step 1: Open Operations module [#step-1-open-operations-module] The user goes to Operations and opens: * `Operational Export` Step 2: Open Organism/Antibiotic Results [#step-2-open-organismantibiotic-results] The user opens: * `Organism/Antibiotic Results` Step 3: Choose the sub-tab [#step-3-choose-the-sub-tab] The screen offers: * `Microbiology` * `Molecular` * `Antimicrobiogram` The Antimicrobiogram sub-tab is the aggregated matrix view.
Antimicrobiogram heatmap overview
Figure 1 — Antimicrobiogram heatmap overview
Step 4: Default organization is selected [#step-4-default-organization-is-selected] The frontend loads the result set, derives the available organizations, and automatically selects the first organization that has data. This keeps the initial user experience smooth. The user lands on a meaningful view without needing an extra manual step. Step 5: Apply filters [#step-5-apply-filters] The user can refine the report using: * date range * sample type * service * organization Step 6: Read the heatmap [#step-6-read-the-heatmap] The matrix now shows:
Antimicrobiogram filtered view
Figure 2 — Recalculations on the basis of the Filtered results
* organisms as rows * antibiotics as columns * sensitivity percentage in each populated cell * N alongside the row/header context Step 7: Export [#step-7-export] The user can download the visible matrix as Excel. 11. Export workflow [#11-export-workflow] The export path is intentionally straightforward. When the user clicks `Download -> Excel`, the system exports: * the visible heatmap * organization metadata * selected date range * sample type * selected service * generated timestamp * heatmap legend This makes the export immediately shareable. It is not just a raw grid dump; it carries the context needed to understand the sheet later. 12. Full feature lifecycle in one story [#12-full-feature-lifecycle-in-one-story] Here is the whole feature as one operational story: 1. support enables Antimicrobiogram for a lab 2. the lab feature flag is saved 3. session payloads begin exposing `is_antimicrobiogram_enabled` 4. a migration row is queued to backfill history 5. the migration processor fills the summary table for the historical window 6. the daily processor keeps adding the previous day's report slice 7. if a historical bill or report changes, the billing/LRR triggers queue a one-day repair 8. the repair processor rebuilds that day 9. the frontend reads summary rows and sensitivity groups 10. the user sees a clean organization-wise heatmap and can export it That is the end-to-end workflow the feature was built to deliver. 13. Quick operational checklist [#13-quick-operational-checklist] If someone is validating the feature end to end, this is the shortest good checklist: 1. confirm `is_antimicrobiogram_enabled` for the lab 2. confirm the session carries the same flag 3. confirm migration or daily jobs have populated `OrganismAntibioticSummary` 4. confirm historical updates create repair queue entries 5. confirm the repair processor clears pending rows 6. confirm the frontend auto-selects an organization and renders the heatmap 7. confirm Excel export works with the expected metadata # Design Decisions Design decisions [#design-decisions] *** B2B Collection vs. other trip flows [#b2b-collection-vs-other-trip-flows] **B2B Collection** is the product for **organization-site sample collection logistics**—planned stops, routes, and trips for **B2B clients**. The same trip-management engine in **crelio-app** also powers **other** flows (for example **phlebotomist** trips that may be tied to a patient visit record). Those are **different products**; this doc only describes **`trip_type` = B2B / `b2b-logistics`**. *** B2B logistics trips (B2BTrip) [#b2b-logistics-trips-b2btrip] **B2B** logistics trips are `Trip` rows with **`trip_type = B2B_TRIP`**. The **`B2BTrip`** proxy manager targets trips used for **multi-stop organization routes**. Naming in URLs or legacy code may differ; this doc sticks to **`b2b-logistics`** and **`B2BTrip`** only. Keeping B2B logistics trips on **dedicated `b2b-logistics` URLs** lets web and mobile use separate serializers, auth (e.g. `b2b_logistics` JWT context), and lifecycle rules from phlebotomist flows. *** b2b-logistics in the URL [#b2b-logistics-in-the-url] The frontend passes **`b2b-logistics`** as `trip_type` so Django routes to the correct handlers under: `api-v3/trip-management/b2b-logistics/…` alongside other `trip_type` values for unrelated trip products. *** livehealthapp [#livehealthapp] Legacy **livehealthapp** UIs (e.g. organization finance) are **not** the home for **B2B Collection** logistics; that lives in **livehealth-frontend** under **Registration → B2B Collection**. # Overview B2B Collection [#b2b-collection] **B2B Collection** is how a **lab** plans and runs **sample collection at organization (B2B client) sites**: defining pickup **locations** (stops), building **routes** across those sites, and scheduling **logistics trips** so pickup staff visit clients in order—with maps, status, and trail data for operations. These pages document **B2B organization-site logistics** only—stops, routes, and trips under **`b2b-logistics`**. Other lab workflows and products are out of scope here. What it is for [#what-it-is-for] * **Organization sites as stops** — Register client locations (addresses, contacts) where the lab collects samples. * **Routes** — Order those stops into efficient paths for recurring or planned use. * **Trips** — Assign **pickup partners** (lab field staff), schedule run times, track progress stop-by-stop, cancel or adjust series, and view **trail** data for audit and maps. * **Field tools** — Mobile and web APIs under **`b2b-logistics`** power pickup-person apps and the Registration → B2B Collection screens. Prerequisites [#prerequisites] | Gate | Where it lives | Purpose | | ---------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `b2b_collection` | Session / `LabFeatures` (`crelio-app`) | Enables **Registration → B2B Collection** (Route Management & Trip Management) in `livehealth-frontend` | The sidebar entry **B2B Collection** appears when `isB2bCollectionEnabled()` is true (`session.b2b_collection`). How the pieces fit together [#how-the-pieces-fit-together] Related repositories [#related-repositories] | Repository | Role | | ----------------------- | ---------------------------------------------------------------------------------------------------------------- | | **livehealth-frontend** | **Registration → B2B Collection**: Route Management & Trip Management (`B2bCollection/`) | | **crelio-app** | Trip management module: `B2BTrip`, `api-v3/trip-management/b2b-logistics/…`, locations, routes, visiting persons | Documentation map [#documentation-map] * [Workflow guide](/docs/product-engineering/features/b2b-collection/workflow-guide) — Stops → routes → trips → monitoring * [Frontend](/docs/product-engineering/features/b2b-collection/frontend) — `livehealth-frontend` surfaces and APIs used by the UI * [Backend](/docs/product-engineering/features/b2b-collection/backend) — Trip models and URL patterns * [Design decisions](/docs/product-engineering/features/b2b-collection/design-decisions) — Why `b2b-logistics` differs from other trip types # Workflow Guide Workflow Guide [#workflow-guide] This page covers the **B2B Collection** workflow: **organization sites** → **routes** → **trips** → **monitoring**. *** 1. Enable access [#1-enable-access] Ensure the lab has **`b2b_collection`** enabled so **Registration → B2B Collection** appears for logistics staff. *** 2. Define stops (organization locations) [#2-define-stops-organization-locations] Navigate to **Registration → B2B Collection → Route Management**. 1. Create **stops** — named locations (typically **B2B client / organization sites**) with address, coordinates, and contact details. 2. Stops are stored via trip-management APIs: `api-v3/trip-management/b2b-logistics/location/…` and reused across routes. Stops can be linked to an **organization** record where the product supports it, so routes reflect real client sites. *** 3. Build routes [#3-build-routes] Under **Route Management**: 1. Create a **route**: an ordered list of stops with segment distances. 2. Edit, copy, or disable routes from the grid (soft-disable via PATCH where applicable). A route is a **reusable path template**; execution happens when you schedule a **trip**. *** 4. Plan trips [#4-plan-trips] Navigate to **Registration → B2B Collection → Trip Management**. 1. Filter by **date range** and optional **trip status** (pending, in progress, completed, cancelled, etc.). 2. **Create trip** — choose a **pickup partner**, schedule the run, and attach ordered **trip locations** (stops). 3. **Recurring / series** — multiple scheduled instances use `POST …/trips/new/bulk` when the UI sends `scheduled_trips`. 4. **Edit or cancel** — single occurrence vs whole series (`update` vs `update/bulk`). Pickup partners come from `api-v3/trip-management/b2b-logistics/pickup-persons/list`. *** 5. Monitor a trip [#5-monitor-a-trip] 1. Open **Trip details** from the list (URL includes `tripId` and date range for context). 2. Review **per-stop status**, comments, and any **sample** summaries returned by the API. 3. **Trail** — `GET …/trips/{tripId}/{pickupPersonId}/trail` for path history and maps. *** Related [#related] * [Frontend — Route & trip UI](/docs/product-engineering/features/b2b-collection/frontend/livehealth-frontend/registration-b2b-ui) * [API reference](/docs/product-engineering/features/b2b-collection/frontend/livehealth-frontend/api-reference) # Overview import Image from 'next/image'; import cellCounterImage from '@/images/cell-counter/cell_counter_image.png'; Related Documentation: [#related-documentation] [JIRA Ticket](https://crelio.atlassian.net/browse/EN-11386) [Whimsical](https://whimsical.com/cell-counter-Xni5QA6J1bNCjCYvGjv78k) Prerequisites [#prerequisites] The feature requires `cell_counter_enabled` flag to be True for a test. What is it for? [#what-is-it-for] It helps eliminate manual counting errors, speeds up the counting process under the microscope, and directly maps counted values to the appropriate test parameters on the report entry page. Physical or digital cell counters help laboratory professionals quickly and accurately count different cell types while examining samples under a microscope. These tools reduce manual counting errors, speed up the counting process, and simplify recording results for diagnostic reporting. We designed the interface similar to [Cell Counter Tool](https://www.cellcountr.com/). Key Benefits [#key-benefits] * Faster cell counting workflow * Reduced manual calculation errors * Seamless integration with report entry * Improved accuracy and efficiency for lab professionals Supported Capabilities [#supported-capabilities] The Cell Counter feature includes support for: * **Test-level configuration**\ Configure cell counting behavior specific to individual tests. * **Keyboard-based counting**\ Count cells efficiently using keyboard shortcuts. * **Rule handling**\ Apply rules such as count limits, abnormal value logic, and options for undo or reset. * **Direct data flow**\ Automatically map counted values to the relevant report parameters. * **Export capability**\ Export counted data for further analysis or record keeping. Cell Counter Interface # Claim Management The module provides the ability to: [#the-module-provides-the-ability-to] * Configure insurance price lists * Assign insurance to patients * Calculate patient and insurance payable amounts * Verify insurance eligibility * Submit and manage insurance claims This documentation is divided into: [#this-documentation-is-divided-into] * Functional overview * Frontend implementation * Backend architecture [KT Sessions](https://drive.google.com/drive/folders/1mO80yPI1rUcOs1aqdZWtDf75Ulq9f4y_?usp=drive_link) # Design Decisions Design Decisions & Architecture [#design-decisions--architecture] *** Key Design Decisions & Constraints [#key-design-decisions--constraints] Order-wise callout as the primary unit [#order-wise-callout-as-the-primary-unit] One callout action covers all critical reports in an order simultaneously. * Previously: user had to action each critical report separately * Now: `bill_id` is the root key across API, bulk manager, and modal * Schema change: `bill` FK added to `CriticalCallout`; `lab_report` made nullable Single bill-level draft record [#single-bill-level-draft-record] Exactly **one** `CriticalCallout` row is created per bill when `is_draft=True` (not one per report). * `lab_report = None` on the draft record * Selected test IDs stored in `critical_callout_meta.callout_for` for UI restore * Only one draft can ever exist per order — atomically replaced on every resubmit CALLOUT_ATTEMPTED as an explicit enum value [#callout_attempted-as-an-explicit-enum-value] `CALLOUT_ATTEMPTED = 4` is a first-class status in `CriticalValuesEnum`, not a flag. * Worklist can filter and display attempted orders separately from pending * Export includes it as a distinct `Callout Status` column value * Activity log carries `is_draft: true` in `dumped_json` * Flows through the same badge, tab-filter, and ES-sync paths as `PENDING` and `DONE` Draft deletion on every submit [#draft-deletion-on-every-submit] The existing draft record is always deleted inside `transaction.atomic()` before new records are created. * Ensures: at most one draft per bill at any time * Ensures: a completed callout can never coexist with a draft for the same bill Department-scoped report visibility [#department-scoped-report-visibility] `FetchCriticalRecordsByBillView` applies department filters per session type. * Doctor login: filtered by assigned departments or `billId__docId` * Lab user: filtered via `LabUserDepartmentRelation` * Support login: no department filter (elevated access) Unified modal — action + history in one surface [#unified-modal--action--history-in-one-surface] Callout history is visible in the right panel of the same modal used to perform the callout. * Avoids context switching during operations * History loaded via a parallel bulk logs API call on modal open * Users can see previous attempts before deciding on next action Communication channels as additive, config-controlled [#communication-channels-as-additive-config-controlled] SMS and WhatsApp were added without touching existing `critical_notification_settings` keys. * `is_email`, `always_notify`, `mandatory_comments` etc. are unchanged * New channels use the same `notify_to` payload structure * Future channels can be added the same way — no config schema migration needed always_notify enforcement at the surface level [#always_notify-enforcement-at-the-surface-level] `always_notify` is checked individually in `ReportEntryFooter` and `DoctorFooter`, not in shared middleware. * Each surface intercepts its own action (Save & Sign, Approve) * Condition: `completedTests === 1` AND `criticalValues ∈ CRITICAL_CALLOUT_VALUES` AND `always_notify === 1` AND no existing logs * Keeps enforcement logic close to the action it guards; easy to modify per surface Navigation state preservation on worklist [#navigation-state-preservation-on-worklist] Worklist filter state (date range, department, tab, search) is restored when user returns from patient overview. * Worklist users process many orders in sequence * Losing filters on every row click would break the operational flow *** Architectural Rationale [#architectural-rationale] CriticalReport as a proxy model, not a new model [#criticalreport-as-a-proxy-model-not-a-new-model] `CriticalReport` is a Django proxy of `LabReportRelation` — no extra DB table. * Inherits all `LabReportRelation` fields and relationships * Adds behaviour: parameter evaluation, email, activity log, ES sync * Avoids data duplication; keeps callout logic co-located with the report BulkCriticalCalloutManager as a standalone class [#bulkcriticalcalloutmanager-as-a-standalone-class] Bulk callout logic was extracted into a class rather than embedded in the view. * A single `process_callout()` call spans: parameter batching → email → DB transaction → ES sync → activity log * Classmethod `get_critical_report_params_mapper` is reused by both the manager and `FetchCriticalRecordsByBillView` * Improves testability — no HTTP layer needed to unit test the callout flow Three-query batch fetch for parameter evaluation [#three-query-batch-fetch-for-parameter-evaluation] `get_critical_report_params_mapper()` resolves critical parameters for all reports in an order using **3 DB queries** regardless of report count. 1. `ReportValue` — all values for the report IDs 2. `ReportFormat` — all formats for the format IDs 3. `ValueRanges` — age ranges (only if any formats have `ageRangeFlag`) Results are grouped in Python; zero N+1 risk. Runs on every modal open and every callout submit. Two-path ES sync strategy [#two-path-es-sync-strategy] | Path | Trigger | Scope | | ----------------------------------------- | ------------------------------ | ------------- | | `CriticalReport.after_save()` | Django `save()` hook | Single report | | `BillSplitManager.sync_lab_reports(bill)` | After bulk callout transaction | Full bill | The two-path design exists because `bulk_update` skips the Django `save()` hook — bill-level sync after the transaction guarantees ES consistency without re-implementing `after_save` for bulk operations. do_save=False for deferred bulk_create [#do_savefalse-for-deferred-bulk_create] `save_critical_callout(do_save=False)` returns an unsaved instance instead of calling `INSERT` immediately. * Bulk manager collects all instances, then calls `bulk_create` inside one atomic transaction * Avoids N separate `INSERT` statements * Partial failures roll back cleanly — no orphaned callout records *** Reusability & Extensibility [#reusability--extensibility] | What you can do | How | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | Add a new communication channel | Add to `COMMUNICATION_METHODS` (frontend) + `METHOD_KEY_VALUE_MAPPER` + `save_critical_callout` kwargs (backend) | | Embed the callout button on a new surface | Import `CriticalNotificationButton`, pass `labReport` + `onSuccess` + visibility condition — no modal changes | | Add a new callout status | Add to `CriticalValuesEnum` → `CRITICAL_CALLOUT_VALUES` → `labReportStatusUtils` → worklist tab (if needed) | | Extend draft restore fields | Add field to `CriticalCalloutMeta` interface → persist in `critical_callout_meta` JSON → restore in `getRestoredCalloutState()` | | Add a new recipient type | Add to `RECIPIENT_KEY_VALUE_MAPPER` in `BulkCriticalCalloutManager` + flag in `notify_by_email()` | | Change the email template | Edit `critical_notification.jinja` — context (`report_items`, `is_bulk`, `show_patient_name`, `comment`) is parameterised | | Add a new export column | Add field to `return` object in `getCriticalCalloutExportRows()` — SheetJS picks up headers dynamically | # Overview Bill-wise Critical Callout [#bill-wise-critical-callout] Critical Callout is the workflow that ensures lab staff or doctors communicate out of normal range, life-threatening test results to the responsible party — a referral doctor, the patient's organisation, or the patient — before or immediately after reports are finalised. This document redesigns callout from **report-wise** to **order-wise** so that all critical reports in a single order can be acted on together. *** Related JIRA [#related-jira] * [EN-11720](https://crelio.atlassian.net/browse/EN-11720) * [EN-11721](https://crelio.atlassian.net/browse/EN-11721) Whimsical [#whimsical] * [Critical Callout UI/UX](https://whimsical.com/critical-callout-ux-ELHxenKWFtbbSSLMEeSzx9) *** Why is Critical Callout needed? [#why-is-critical-callout-needed] Some test results fall outside safe clinical thresholds. When this happens, simply delivering a report is not enough — the result must be actively communicated to someone who can act on it immediately. Labs are operationally and legally expected to: * Identify which results are beyond normal **and** beyond critical bounds * Communicate those results to the right person (referral doctor, organisation, or patient) * Maintain a traceable record of that communication Before this redesign, callout was handled one report at a time, which meant: * A patient with three critical reports in one order required three separate callout actions * There was no centralised queue for operations teams to manage pending callouts * Incomplete communication attempts had no tracked state — they simply vanished * History was not visible alongside the action, forcing users to navigate away for context *** How critical status is determined [#how-critical-status-is-determined] A parameter is considered critical when its value falls outside both the **normal range** and the **critical range** configured in the report format. | Parameter | Normal Range | Critical Range | Patient Value | Result | | --------- | -------------- | -------------- | ------------- | ----------------- | | WBC Count | 4,000 – 11,000 | 2,000 – 25,000 | 1,500 | **Critical Low** | | WBC Count | 4,000 – 11,000 | 2,000 – 25,000 | 30,000 | **Critical High** | | WBC Count | 4,000 – 11,000 | 2,000 – 25,000 | 7,500 | Normal | For list-field (qualitative) parameters, criticality is determined by a flag (`is_critical = 2`) in the parameter's `defaultDescription` mapping rather than a numeric range. *** Who can perform a Critical Callout? [#who-can-perform-a-critical-callout] | Role | Where | | -------------------------- | ------------------------------------------------------------------ | | **Lab staff (lab login)** | Report Entry footer, Waiting List, Operations Worklist | | **Doctors (doctor login)** | Doctor Waiting List, Patient-wise Report List, Report Entry footer | Both roles have access to the same modal and the same action set (Save Draft / Notify and Mark as Done). Department-scope filtering is applied on the backend so each user only sees reports relevant to their assigned department. *** Is Critical Callout mandatory? [#is-critical-callout-mandatory] By default, it is **not mandatory**. It is an action that becomes available once a report has critical values. However, it can be made mandatory by enabling **"Always send notification when critical values are identified"** in the Notify Configuration tab. When this setting is on: * The callout step is enforced when signing reports with critical values * Exemptions apply automatically for: auto-approval, auto-dispatch, Sign All, and mobile app flows See [Notify Configuration](/docs/product-engineering/features/critical-callout/frontend/notify-configuration) for full settings reference. *** Data model — which fields hold the status [#data-model--which-fields-hold-the-status] Two database entities track callout state: * **`LabReportRelation.criticalValues`** — an integer field on every report row that tracks the current callout status: * **`CriticalCallout` table** — a separate audit table that stores one record per callout action. For draft callouts, `lab_report` is `null` and only `bill` is set, meaning one draft record represents the whole order. For completed callouts, one record is created per report. *** Where can a callout be triggered? [#where-can-a-callout-be-triggered] All surfaces open the same `CriticalNotificationModal`. See [Worklist Guide](/docs/product-engineering/features/critical-callout/workflow-guide) for a step-by-step walkthrough of the modal and the callout actions. *** Feature scope summary [#feature-scope-summary] * Callout is now **order-wise** — one action covers all critical reports in the order * **Callout Attempted** is a first-class draft status, not just a missed action * A new **Critical Callout Worklist** under Operations provides a centralised queue * Callout history is visible inside the same modal — no navigation required * All communication channels (Email, Fax, SMS, WhatsApp) are selectable per callout action * Notification configuration is unchanged for labs that have it already set up *** Document map [#document-map] * [Backend](/docs/product-engineering/features/critical-callout/backend) * [Frontend](/docs/product-engineering/features/critical-callout/frontend) * [Worklist Guide](/docs/product-engineering/features/critical-callout/workflow-guide) * [Design Decisions](/docs/product-engineering/features/critical-callout/design-decisions) # Worklist Guide Critical Callout Worklist Guide [#critical-callout-worklist-guide] This guide walks through the complete operational flow — from opening the worklist to completing or drafting a callout. *** End-to-end workflow [#end-to-end-workflow] *** 1. Access the worklist [#1-access-the-worklist] Navigate to **Operations → Critical Callout Worklist**. The list shows all orders where at least one report is marked critical. Critical Callout Worklist Visibility rules [#visibility-rules] * Orders are shown regardless of individual report status * Includes orders with dispatched reports * List is order-wise — not report-wise Tabs [#tabs] | Tab | Shows | | ------------------- | ------------------------------------------------------------------------ | | **All** | Every order with a critical report | | **Callout Pending** | Orders with at least one `CALLOUT_PENDING` or `CALLOUT_ATTEMPTED` report | | **Callout Done** | Orders where all critical reports are `CALLOUT_DONE` | *** 2. Open the callout modal [#2-open-the-callout-modal] Click the **Critical Callout** button on any row. The modal fetches: * All critical reports for the order * The existing draft (if any) to prefill state *** 3. What's inside the modal [#3-whats-inside-the-modal] The modal has two tabs: **Critical Callout** (action) and **Notify Configuration** (settings). Critical Callout Modal Critical Callout tab [#critical-callout-tab] The left panel is the action area, the right panel shows callout history. **Footer actions:** | Button | What it does | | --------------------------- | --------------------------------------------------------------------------------- | | **Save Draft** | Sets status to `Callout Attempted`; saves draft at bill level; prefills on reopen | | **Notify and Mark as Done** | Sends notification; sets status to `Callout Done`; writes audit entry | Callout history panel (right side) [#callout-history-panel-right-side] A vertical timeline of all previous callout actions for this order's reports, sorted most-recent first. Callout History Each entry shows: | Field | Content | | --------------- | ----------------------------------------------------------- | | **Title** | Callout Completed (green ✓) or Callout Attempted (yellow ✗) | | **Timestamp** | Date and time of action | | **By** | User or doctor who performed it | | **Callout for** | Test names included in this action | | **Method** | Email / Fax / SMS / WhatsApp | | **Recipient** | Organisation / Referral / Patient / Other | | **Comment** | Comment text if entered | *** 4. Attempted vs Done — decision guide [#4-attempted-vs-done--decision-guide] Use Save Draft when [#use-save-draft-when] * Recipient is busy or unavailable * Call goes unanswered * Line disconnected Saving draft: * Preserves selected reports, channels, recipients, and comment * All selections are restored when the modal is reopened * The attempt is recorded in the history panel Use Notify and Mark as Done when [#use-notify-and-mark-as-done-when] Communication was successful. The notification is sent and the status is finalised. *** 5. Callout status lifecycle [#5-callout-status-lifecycle] `CALLOUT_ATTEMPTED` is not a terminal state. The modal can be reopened and the callout completed at any time, cycling back through the decision. *** 6. Return to filtered worklist [#6-return-to-filtered-worklist] After completing an action, back navigation restores the worklist to the same filters and tab that were active before navigating to the order. This supports fast multi-order processing without reapplying filters manually. *** 7. Redirect from existing critical queues [#7-redirect-from-existing-critical-queues] From existing critical report drilldowns and waiting lists, use the **View in Critical Callout Worklist** link to open the new worklist directly for that order's context. *** 8. Operational checklist [#8-operational-checklist] * Attempted entries remain visible under **Callout Pending** * Done entries move to **Callout Done** * Reopening a drafted callout restores all previous selections * History panel records every attempt and completion with actor and timestamp * Back navigation restores worklist filter and search state # Backend Backend [#backend] This page consolidates **server-side responsibilities**: report conversion, presigned generation + reconcile, **`uploadFileTypeReportAPI`** branching (new vs legacy), mark-as-done, **Elasticsearch** and **Pusher** side effects, **print/render** special cases, and **PY-3** Storage Manager + Lambda-authenticated file-type-report helpers. Cross-links: [Overview](/docs/product-engineering/features/file-type-report-revamp/overview), [Workflow Guide](/docs/product-engineering/features/file-type-report-revamp/workflow-guide), [Design Decisions](/docs/product-engineering/features/file-type-report-revamp/design-decisions). Backend Technicalities [#backend-technicalities] livehealthapp (PY-2) [#livehealthapp-py-2] 1. Report conversion [#1-report-conversion] **File:** [`apps/livehealthapp/labs/API.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/labs/API.py)\ **Function:** `convertReportToFileOrNormal` **Responsibilities:** * Authorize conversion (permissions + radiology guard + session shape). * Delete existing `ReportValue` rows. * Toggle `fileInputReport` and related flags (`store_values_to_document_db` reset in this path). * Reset completion, approval, signing, partial sign, print-done fields. * Trigger **Pusher** + **patient report ES** reindex + **activity log**. 2. File upload finalizer [#2-file-upload-finalizer] **File:** [`apps/livehealthapp/reports/views.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/reports/views.py)\ **Function:** `uploadFileTypeReportAPI` (+ `save_file_type_report_to_report_value` on the write path) **Responsibilities:** * **Branch detection:** if `initialFilePath` is present and path-like → **presigned-aware fast path** (file already in storage). * **Legacy branch:** if `fileData` present → base64 decode path, `restruct_uploaded_files`, `SmStore`, then value write. * Update **`ReportValue.value`** to the stored path. * Mutate **`LabReportRelation`** completion / approval fields (e.g. `completedTests`, `isApproved`, timestamps, acting `labUserId`). * **`commonPusherFunctionForLabReportRelation`** (or equivalent) for waiting-list freshness. * **Patient report Elasticsearch** reindex for search surfaces. * **Upload activity log** entry. Report mutations (presigned-aware branch - conceptual) [#report-mutations-presigned-aware-branch---conceptual] | Field | Effect | | :--------------------------- | :---------------------------------------------------------------------- | | `ReportValue.value` | File path string | | `reportDate` / `lastUpdated` | Updated | | `isApproved` | Reset toward “needs review” posture (implementation-specific constants) | | `isPartialFill` | Reset | | `labUserId` | Acting user | | `completedTests` | Driven to completed posture for the file-report case | Exact field names should be verified in the implementation block inside `uploadFileTypeReportAPI` when debugging a specific deployment branch. 3. Legacy preprocessing [#3-legacy-preprocessing] **Same file:** `restruct_uploaded_files` in [`reports/views.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/reports/views.py) **Role:** normalize/validate uploads in legacy mode - images, damaged PDFs, EOF quirks, internal failure codes. Historically this is the kind of work the revamp tries to **move off** the synchronous PY-2 request path toward **async / infra-assisted** processing. 4. Storage Manager (PY-2) [#4-storage-manager-py-2] **File:** [`apps/livehealthapp/labs/storage_manager.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/labs/storage_manager.py) | Class / function | Responsibility | | :------------------------ | :-------------------------------------------------------------------- | | `SmGeneratePreSignedURL` | Issue direct-upload contract + create **Storage Manager log** | | `SmReconcilePreSignedURL` | `head_object`-style verification + reconcile flags + **size** on log | | `get_storage_paths` | Path structure generation | | Cleanup utilities | e.g. replace deleted file-type attachments with `file_not_exists.pdf` | **URL wiring:** [`apps/livehealthapp/livehealthapp/urls.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/livehealthapp/urls.py) (or equivalent project `urls.py` in your branch) - `sm_generate_presigned_url/`, `sm_reconcile_presigned_url/`, `uploadFileTypeReport/`, `uploadFileWithHeader/`. 5. Mark-as-done support [#5-mark-as-done-support] **File:** [`apps/livehealthapp/labs/newAPI.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/labs/newAPI.py)\ **Function:** `mark_report_as_done_api` Marks sibling operational reports complete with business rules (e.g. **at least one** operational report must remain incomplete in the bill), updates ES, triggers hooks/Pusher as implemented. 6. Downstream rendering [#6-downstream-rendering] | File | Why it matters | | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------- | | [`templates/labTemplates/fileInputReport.jinja`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/templates/labTemplates/fileInputReport.jinja) | Template helper for rendering file-input images in lab print HTML | | [`reports/v2_views/report_view.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/reports/v2_views/report_view.py) | Some print flows convert **image-backed** file input values into **generated PDF pages** | File-type reports are not “upload and forget”; **print/download** stacks branch on file paths vs parameter matrices. crelio-app (PY-3) [#crelio-app-py-3] 1. Storage Manager APIs and core implementation [#1-storage-manager-apis-and-core-implementation] | Area | Link | | :-------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | HTTP views | [`apps/crelio-app/core/views/storage_manager/storage_manager_views.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/apps/crelio-app/core/views/storage_manager/storage_manager_views.py) | | URL routing | [`apps/crelio-app/core/urls.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/apps/crelio-app/core/urls.py) | | Manager | [`apps/crelio-app/core/utils/storage_manager/manager.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/apps/crelio-app/core/utils/storage_manager/manager.py) | | Path + validation utilities | [`apps/crelio-app/core/utils/storage_manager/utils.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/apps/crelio-app/core/utils/storage_manager/utils.py) | Capabilities include presigned POST generation, presigned download, reconcile, canonical path generation, extension gates, soft-delete / cleanup behaviors. 2. Canonical path generation (documented shape) [#2-canonical-path-generation-documented-shape] ```text ////_. ``` Example: ```text IN/123/456456/FileTypeReport/16915654908422848526992_foo.jpeg ``` Both PY-2 and PY-3 utilities support **caller-supplied** `cloud_path` patterns; the **temp vs concrete** distinction matters when Lambda rewrites paths. 3. File-type-report Lambda-facing APIs [#3-file-type-report-lambda-facing-apis] **File:** [`apps/crelio-app/report/views/file_type_report_lambda.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/apps/crelio-app/report/views/file_type_report_lambda.py)\ **Routes:** see [`apps/crelio-app/report/urls.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/apps/crelio-app/report/urls.py) | Endpoint family | Responsibility | | :------------------------------------------------ | :-------------------------------------------------------------------------------------------------- | | `report-value/file-type/get/storage-manager/path` | Accept **temporary** path → return **canonical** concrete Storage Manager path | | `report-value/file-type/update` | Update `report_results` / lab report state with **final** file path + SM log path rewrite semantics | **`FetchFileTypeReportPath` (conceptual):** looks up SM log for temp path, pulls file metadata, resolves S3 account details, emits canonical path. **Important implementation detail from source analysis:** extension normalization may **force `.pdf`** in the helper path - treat this as a **normalization contract** when reasoning about mixed image/PDF inputs on Lambda-shaped flows. **`UpdateReportValueWithFilePath` (conceptual):** accepts `filePath`, `temporary_filepath`, `labReportId`; updates `report_results.value`, timestamps, completion flags; aligns SM log from temporary to final path. 4. Lambda authentication [#4-lambda-authentication] **File:** [`apps/crelio-app/core/utils/aws/authenticate_lambda_requests.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/apps/crelio-app/core/utils/aws/authenticate_lambda_requests.py) * Expects header **`X-Lambda-Token`** * Validates against **`settings.CRON_SIGNING_KEY`** (naming may vary by environment - confirm in deployment secrets) 5. File type reports beyond manual UI upload [#5-file-type-reports-beyond-manual-ui-upload] **Example:** [`apps/crelio-app/interfacing/models/device.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/apps/crelio-app/interfacing/models/device.py) - instrument/device flows can persist `file_category="FileTypeReport"` style outputs. **Format linkage:** [`apps/crelio-app/report/views/report_format_link.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/apps/crelio-app/report/views/report_format_link.py) - format type `4` correlates with `fileInput=1` semantics in that module’s conventions. DB and architecture [#db-and-architecture] Relational state, object storage, Storage Manager logs, and Elasticsearch documents together implement the feature; PY-2 remains the **observed** domain finalizer for the default frontend path. Core entities [#core-entities] | Entity | Why it matters | | :---------------------------------------- | :------------------------------------------------------------ | | `labReportRelation` / `LabReportRelation` | `fileInputReport`, `completedTests`, approval / signing flags | | `ReportValue` / `report_results` | **`value` holds the file path** for file-type reports | | `StorageManagerLog` | Presigned lifecycle, sizes, paths, reconcile markers | | Patient report ES document | Search + waiting-list propagation | What gets persisted where [#what-gets-persisted-where] | Layer | Stored value | | :------------------ | :----------------------------------------------------------- | | S3 | Binary PDF/image object | | Report DB | Path in `ReportValue.value` or `report_results.value` | | Report row | Completion / approval metadata | | Storage Manager log | Upload metadata, category, paths, reconcile state, file size | | Elasticsearch | Patient-report doc + SM log record | File path semantics [#file-path-semantics] | Path type | Typical example | Who creates it | | :------------------------- | :-------------------------------------------------------- | :---------------------------- | | Temporary caller-specified | `FileTypeReports/IN/28/123456/ab12cd/report.pdf` | Frontend | | Canonical SM | `IN/28/20798388/FileTypeReport/_report.pdf` | Storage Manager / PY-3 helper | | Compatible path | Human-stable equivalent without generated cloud id prefix | Storage Manager utilities | | Behavior | Explanation | | :---------------------------------- | :------------------------------------------------------------------------ | | Requested path can be honored as-is | Caller-supplied `cloud_path` supported in SM utilities | | Temp vs concrete can diverge | Basis for Lambda + PY-3 helper design | | Deleted attachments normalized | Cleanup may rewrite to `file_not_exists.pdf` | | PY-3 path helper may force `.pdf` | Normalization / downstream PDF assumption in Lambda-assisted architecture | Setup dependencies by layer [#setup-dependencies-by-layer] | Layer | Dependency | Why | | :------- | :------------------------------ | :------------------------------- | | Frontend | Session with lab context | SM + finalization are lab-scoped | | Frontend | Report already in file mode | Wrong mode → wrong UX path | | PY-2 | S3 account for `FileTypeReport` | presigned generation | | PY-2 | extension allowlist | SM rejects unsupported types | | PY-2 | SM ES logging | reconcile + observability | | PY-2 | patient report ES | searchable completion state | | PY-2 | Pusher / realtime | operator UI freshness | | PY-3 | Lambda token auth | secured helper endpoints | | AWS | S3 + optional event routing | direct upload + optional Lambda | Architectural caveats (backend-focused) [#architectural-caveats-backend-focused] 1. **Observed FE still finalizes in PY-2** - presigned + reconcile can be PY-2 while PY-3 exposes parallel helpers. 2. **Lambda source not in repo snapshot** - only contracts + auth + endpoints are verifiable here. 3. **Header/footer** remains **`uploadFileWithHeader`** / base64 transport on PY-2. 4. **Temporary path may be what PY-2 stores** in the observed path until/unless a later job rewrites - contrast with PY-3 “temp → canonical” helper intent. For rationale and decision framing, see [Design Decisions](/docs/product-engineering/features/file-type-report-revamp/design-decisions). # Design Decisions Design Decisions [#design-decisions] This page states **what was decided**, **what was traded away**, and **what must not be confused** when reading `livehealth-frontend`, `livehealthapp`, and `crelio-app` together. It complements [Overview](/docs/product-engineering/features/file-type-report-revamp/overview) facts with **engineering intent** and **deployment-shape caveats**. Architectural Rationale [#architectural-rationale] The legacy file-type upload path treated the **application server as a byte pipe**: * large POST bodies * memory spikes decoding base64 * CPU spent validating/restructuring binaries **inside** the web worker lifecycle * tighter effective ceilings on “how big a report PDF can be” because the **transport hop** was the bottleneck The revamp’s primary intent is **storage-transport decoupling**: > move the **binary** from “HTTP through Django” to “HTTP directly into object storage,” while keeping **domain transitions** (report value write, completion flags, audit, search, realtime) on trusted backend code. That is materially different from “make uploads async” or “delete backend work” - the backend still performs the **authoritative state mutation** in the observed stack. Robustness, Reusability, and Tradeoffs [#robustness-reusability-and-tradeoffs] | Dimension | How this design behaves | | :-------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Robustness** | **Explicit stages** (presign → S3 → reconcile → finalize) surface partial failures instead of silent inconsistency; reconcile closes the SM log loop; encrypted PDFs fail fast on the client. | | **Reusability** | **Storage Manager** centralizes presigned contracts, logging, path rules, and vendor abstractions so file-type upload is not a one-off S3 hack; **`FileTypeReport`** category is reused by device/interfacing flows, not only the modal. | | **Tradeoffs** | **More moving parts** in the browser (orchestration, retries, UX for orphan objects if finalize fails after S3 success); **header/footer** keeps a **legacy** high-payload path for parity; **PY-3 + Lambda** improve decoupling but **infra and auth** become part of the correctness story. | Architecture / Design Decisions [#architecture--design-decisions] Documented choices and implications for the revamp, the dual finalization surfaces (PY-2 vs PY-3/Lambda), and operator-visible forks (plain upload vs header/footer). 1: presigned POST as the default happy-path contract [#1-presigned-post-as-the-default-happy-path-contract] **Choice:** Storage Manager issues a **short-lived presigned upload** (fields + URL) rather than accepting the file on `uploadFileTypeReport` in the main UI flow. **Rationale:** * removes app servers from the **GB-scale** failure domain of buffering uploads * aligns with S3’s strengths (high-throughput object ingest) * centralizes **account selection**, **extension policy**, and **logging** at SM boundary **Tradeoff:** * **two-phase** client orchestration (presign → POST → reconcile → finalize) increases FE complexity and failure surfaces (partial success if finalize fails after S3 success). * **eventual consistency** window requires reconcile semantics and sometimes short delays. 2: keep PY-2 as the observed domain finalizer [#2-keep-py-2-as-the-observed-domain-finalizer] **Choice:** Current `livehealth-frontend` calls **`uploadFileTypeReport/`** on PY-2 with **`initialFilePath`**, not exclusively PY-3 report-value updaters. **Rationale:** * PY-2 already owned **Pusher**, **patient ES reindex**, **activity logs**, and **radiology / permission guards** in production-hardened code paths. * Presigned transport is orthogonal to **where** the domain transaction commits first in a phased rollout. **Implication for architecture discussions:** * “We moved to PY-3 for file reports” may be **true in some environments**, but is **not proven** by the inspected FE happy path alone. * Engineers should speak precisely: **“binary off PY-2 HTTP; domain finalize still PY-2 in observed FE.”** 3: introduce PY-3 Lambda-facing helpers without requiring Lambda in-repo [#3-introduce-py-3-lambda-facing-helpers-without-requiring-lambda-in-repo] **Choice:** Expose authenticated endpoints that: * translate **temporary** uploaded paths to **canonical** SM paths * update **`report_results`** and related lab report state * rewrite SM logs across temp → final **Rationale:** * enables **async validation / normalization** (virus scan, PDF repair, page rasterization) without blocking the browser session * enables **path normalization** (including extension policy) closer to Storage Manager v2 rules **Explicit non-decision in this repo snapshot:** * the **Lambda handler**, **IAM**, **S3 event wiring**, and **idempotency strategy** are **out of tree** here. * documentation describes **contracts**; runtime behavior requires the **infra repo** or AWS console truth. 4: retain a legacy header/footer path [#4-retain-a-legacy-headerfooter-path] **Choice:** “Apply header and footer” continues to use **`uploadFileTypeReportWithHeader`** → backend **`uploadFileWithHeader`** with **base64** payload semantics. **Rationale:** * header/footer composition may still depend on **server-side PDF manipulation** libraries and templates not ported to a pure-presigned flow. * feature parity > uniform transport for a **lower-volume** branch. **Tradeoff:** * this branch **reintroduces payload pressure**; large files will still stress the legacy path. * operators may report “presigned works for plain upload but not with headers” - that is **expected** given the fork. 5: client-side encrypted-PDF rejection [#5-client-side-encrypted-pdf-rejection] **Choice:** reject encrypted PDFs in the browser before presign. **Rationale:** * downstream storage and print pipelines historically assume **decrypt-at-rest is not required** for operator uploads. * failing fast avoids paying presign + S3 + reconcile costs for guaranteed-bad outcomes. **Tradeoff:** * false positives/negatives depend on the client PDF probe quality; edge PDFs may disagree with server tooling. 6: treat file-type reports as a cross-surface pattern [#6-treat-file-type-reports-as-a-cross-surface-pattern] **Choice:** reuse `file_category="FileTypeReport"` outside manual UI (e.g. interfacing/device flows). **Rationale:** * “file type report” is a **data shape** (path as result), not only a modal feature. **Implication:** * SM logs and S3 layout debugging must not assume **only** `UploadFileTypeReportModal` writers. Temporary path vs canonical path (design axis) [#temporary-path-vs-canonical-path-design-axis] **Temporary path** (frontend-generated `FileTypeReports/...` style): * easy to correlate with a **single upload attempt** * keeps presign logs **namespaced** away from patient-canonical prefixes until business finalization **Canonical path** (`IN///FileTypeReport/...` style): * aligns with **patient document** conventions, search, retention, and cross-feature linking **Observed PY-2 path** may persist the **temporary** path as `ReportValue.value` unless a later process rewrites - engineers must verify **actual stored strings** per deployment. **PY-3 helper path** is explicitly designed to **rewrite** temp → final and update logs accordingly - when Lambda is engaged. Failure-mode philosophy [#failure-mode-philosophy] The system prefers **explicit multi-step failure** over silent inconsistency: * presign failure → no S3 object; no finalize * S3 success + reconcile failure → logs incomplete; operators/debuggers look at SM ES * S3 success + finalize failure → **orphan object** risk; mitigation is operational (cleanup jobs) + client retry UX Documenting these stages separately (see [Workflow Guide](/docs/product-engineering/features/file-type-report-revamp/workflow-guide)) is intentional - it matches **observability boundaries**. Final design statement [#final-design-statement] The File Type Report Revamp is best categorized as: **Transport decoupling + preserved domain authority + optional async normalization layer.** It is **not** “remove Django from file reports.” It **is** “stop using Django as a CDN for multi-megabyte PDFs on the default operator path, while Django (PY-2 today) still decides what ‘done’ means for the lab report row and its indexes.” # Frontend Frontend (livehealth-frontend) [#frontend-livehealth-frontend] This page maps **where the feature lives in the React codebase**, how **`uploadFileTypeReport`** sequences presigned generation → S3 POST → reconcile → PY-2 finalization, and how **header/footer** deliberately diverges to the legacy path. Cross-links: [Overview](/docs/product-engineering/features/file-type-report-revamp/overview), [Workflow Guide](/docs/product-engineering/features/file-type-report-revamp/workflow-guide), [Backend](/docs/product-engineering/features/file-type-report-revamp/backend). What frontend owns [#what-frontend-owns] * **Operator UX** for conversion ([`ReportConversionModal`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Modals/ReportConversionModal/index.tsx)) and for upload ([`UploadFileButton`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/TestWaitingList/UploadFileButton.tsx), [`UploadFileTypeReportModal`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/TestWaitingList/UploadFileTypeReportModal.tsx)). * **Client-side validation** before network (encrypted PDF rejection, extension acceptance, session/lab guards in helpers). * **Orchestration of the revamp path:** build temp **`cloud_path`**, call **`sm_generate_presigned_url/`**, **`uploadFileOnPresigned`** (S3 POST), **`reconcileUploadedPresignedFile`**, then **`uploadFileTypeReport/`** with **`initialFilePath`** only (no raw file body on the happy path). * **Legacy branch** when header/footer is selected: base64 / `fileData` submission to **`uploadFileWithHeader`** via **`uploadFileTypeReportWithHeader`**. * **Optional post-upload** `mark_report_as_done_api` when the modal option is enabled. Frontend Technicalities [#frontend-technicalities] 1. Upload entry point [#1-upload-entry-point] **File:** [`src/components/reusable/TestWaitingList/UploadFileButton.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/TestWaitingList/UploadFileButton.tsx) **Role:** * Surfaces the **Upload File** action for file-enabled reports in the waiting list / operations surfaces. * Stashes selected `labReportDetails` in generic Redux state. * Opens the upload modal. 2. Upload modal [#2-upload-modal] **File:** [`src/components/reusable/TestWaitingList/UploadFileTypeReportModal.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/TestWaitingList/UploadFileTypeReportModal.tsx) **Responsibilities:** * File picker with extension acceptance aligned to backend/SM rules (PDF + common images). * Optional **Apply header and footer** - dispatches to **`uploadFileTypeReportWithHeader`** (legacy). * Optional **Mark all other reports in this bill as Done** - post-upload call to mark-as-done API. * Patient/report context from modal state (`labReportDetails`). 3. Main upload helper (revamp vs legacy) [#3-main-upload-helper-revamp-vs-legacy] **File:** [`src/utils/helpers.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/utils/helpers.ts) | Function | Role | | :------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------- | | `uploadFileTypeReport` | **Main revamp path** - presigned URL, direct S3 upload, reconcile, then `uploadFileTypeReport/` with metadata (`initialFilePath`, etc.). | | `uploadFileTypeReportWithHeader` | **Older path** - reads file to base64 / data URL and posts **`fileData`** to backend header/footer endpoint. | The revamp path’s practical contract: **metadata finalization** after the binary is already in object storage. 4. Presigned direct upload helper [#4-presigned-direct-upload-helper] **File:** [`src/redux/actions/PromotionActions.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/redux/actions/PromotionActions.ts) **Function:** `uploadFileOnPresigned(...)` **Responsibilities (conceptual):** * Convert local **data URL** back to `Blob` / `File`. * Submit **multipart/form-data** to the presigned POST target (S3). * Apply short **post-upload delay** where implemented (eventual consistency buffer). * Invoke **reconcile** helper after successful POST. This is the browser-side locus of the **performance breakthrough**: the application server is not the binary transport hop on the happy path. 5. Reconcile helper [#5-reconcile-helper] **File:** [`src/components/reusable/MultiAttachments/helper.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/MultiAttachments/helper.ts) **Function:** `reconcileUploadedPresignedFile` **Responsibilities:** * `POST` to **`sm_reconcile_presigned_url/`** with the uploaded `filepath`. * Normalize success/failure shape for upstream callers (modal + helpers). Reconcile is **not cosmetic** for operators of Storage Manager observability: it closes the presigned lifecycle in logs and records **authoritative size** after `head_object`-style verification on the PY-2 side. 6. Report conversion UI [#6-report-conversion-ui] **File:** [`src/components/reusable/Modals/ReportConversionModal/index.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Modals/ReportConversionModal/index.tsx) **Responsibilities:** * Confirmation UX (values cleared, signing reset). * Calls `convertReportToFileOrNormal/` with `labReportId`. * Radiology / permission failures surface per API responses. 7. Permission label (admin UI) [#7-permission-label-admin-ui] **File:** [`src/components/LabAdmin/UserManagement/utils/operationSettingConstants.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/LabAdmin/UserManagement/utils/operationSettingConstants.ts) **Permission label:** “Allow Convert to file Report” → backend flag **`allow_report_file_conversion`**. Frontend request contract summary [#frontend-request-contract-summary] | Operation | Endpoint | Payload type | | :-------------------------- | :----------------------------------- | :----------------------------------------------------------------------------- | | Convert report mode | `convertReportToFileOrNormal/` | `labReportId` | | Generate presigned upload | `sm_generate_presigned_url/` | `filename`, `file_category`, `cloud_path`, `patient_id`, `request_host_origin` | | Reconcile uploaded file | `sm_reconcile_presigned_url/` | `filepath` | | Finalize file report | `uploadFileTypeReport/` | `fileName`, `extension`, `initialFilePath`, `labReportId` | | Optional sibling completion | `mark_report_as_done_api/` | list of `lab_reports` + comments | | Header/footer (legacy) | `uploadFileWithHeader/` (via helper) | base64 / structured file payload | Frontend Engineering checklist [#frontend-engineering-checklist] | Gotcha | Why it matters | | :--------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------- | | Header/footer path still posts file to backend | Does **not** get the full presigned-scale benefit; payload size and app-server pressure return. | | Browser still builds **data URL / base64** before presigned POST | Memory pressure exists client-side; the win is **offloading backend transport**, not eliminating client-side reads. | | Images are first-class in accept lists | Downstream print pipelines sometimes **wrap images into PDF pages** - not “PDF-only” in code. | | Encrypted PDFs rejected client-side | Operators may see failure **before** any presigned call; distinguish from SM / S3 errors. | | “Lambda finishes everything” is not guaranteed by FE alone | Observed FE path **finalizes through PY-2** with `initialFilePath`; Lambda path is **parallel architecture**, not implied by FE source alone. | PY-2 legacy static references (if still deployed) [#py-2-legacy-static-references-if-still-deployed] Some deployments retain older static operations modals: * [`static/components/4.0/operations/modals/FileUploadModal.jsx`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/static/components/4.0/operations/modals/FileUploadModal.jsx) * [`static/components/4.0/operations/apiCalls.js`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/static/components/4.0/operations/apiCalls.js) Treat these as **historical / parallel surfaces** unless your lab still ships that bundle; the React paths above are the primary modern integration for file-type report upload in `livehealth-frontend`. # Overview Overview [#overview] Overview [#overview-1] This guide is written for engineers to read and learn about the **File Type Report Revamp**: why it exists, how it differs from the legacy server-mediated upload path, what the **observed** `livehealth-frontend` + `livehealthapp` (PY-2) happy path does today, and where **PY-3** + Lambda-facing surfaces extend the architecture without necessarily being the only production path visible in a given repo snapshot. Use the sibling pages for depth: * [Workflow Guide](/docs/product-engineering/features/file-type-report-revamp/workflow-guide) - step-by-step lifecycle, setup, verification, failure modes * [Frontend](/docs/product-engineering/features/file-type-report-revamp/frontend) - React entry points, helpers, presigned orchestration * [Backend](/docs/product-engineering/features/file-type-report-revamp/backend) - PY-2 finalization, Storage Manager, PY-3 helpers, persistence and indexing * [Design Decisions](/docs/product-engineering/features/file-type-report-revamp/design-decisions) - tradeoffs, dual paths, caveats Prerequisites [#prerequisites] | Requirement | Why it matters | | :-------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------- | | User with **`allow_report_file_conversion`** (or API-allowed doctor access) | Without it, **convert-to-file** is blocked in operations. | | Report **not radiology** | Conversion / upload paths reject radiology in PY-2 guards. | | **S3 + Storage Manager** mapping for `FileTypeReport` | Presigned generation needs account, bucket, region, and **extension allowlist**. | | PY-2 routes reachable from the browser | `sm_generate_presigned_url/`, `sm_reconcile_presigned_url/`, `uploadFileTypeReport/`, `convertReportToFileOrNormal/`. | | **Pusher** + **patient-report Elasticsearch** healthy | Completion is persisted even if UI looks stale when realtime/indexing fails. | | Optional **Lambda path** | S3 events, deployed Lambda, `X-Lambda-Token` / `CRON_SIGNING_KEY`, and PY-3 file-type endpoints, only if that deployment shape is used. | What Is It For [#what-is-it-for] * **Product:** Let a lab attach a **finished report file** (PDF or supported image) as the analytical result instead of typing every parameter. * **Operations:** Reduce friction for high-volume reporting where the authoritative artifact already exists outside the LIS parameter grid. * **Engineering:** Remove the **application server** from the **binary upload path** on the default flow so large multi-page PDFs are practical, while keeping **domain finalization** (report row, audit, search, realtime) on trusted backend code. Key Features [#key-features] | Topic | Summary | | :-------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | What is a file type report? | A report where the system stores a **file path** (typically PDF or image) instead of manually entered parameter values. | | Why was the revamp needed? | The old architecture pushed the **entire file payload through the backend**, creating payload-size and performance bottlenecks for larger lab reports. | | What changed? | The new architecture uses **Storage Manager presigned upload URLs** so the browser uploads **directly to object storage** instead of sending the raw file through the application backend on the main path. | | Biggest practical gain | File transfer is **offloaded from the app server**, reducing request pressure and earlier size restrictions tied to backend payload transport. | | Core business effect | Labs can attach a **finished report document** instead of keying every value manually. | | Important nuance from code | The current `livehealth-frontend` happy path **definitely** uses presigned upload plus **PY-2 finalization**. The repository also contains **PY-3 Lambda-facing endpoints** for a more decoupled path; **Lambda source is not in the inspected repo snapshot**. | | Another important nuance | The **“Apply header and footer”** flow still uses the **legacy server-mediated upload** path and does **not** use the presigned upload flow. | Glossary [#glossary] | Term | Meaning in this system | Typical place seen in code | | :---------------- | :----------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | File Type Report | A report whose result is a **file attachment path** rather than traditional report parameter values. | `fileInputReport`, `ReportValue.value` | | `fileInputReport` | Flag on `labReportRelation` / `LabReportRelation` indicating file-based report behavior. | PY-2 [`labs/API.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/labs/API.py), PY-3 [`report/models/lab_report_relation.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/apps/crelio-app/report/models/lab_report_relation.py) | | Report Value | DB row storing the effective value for a report parameter; for file reports, **`value` is the file path**. | PY-2 `save_file_type_report_to_report_value`, PY-3 `update_lab_report_and_report_value_with_path` | | Storage Manager | Shared file-storage abstraction: paths, presigned URLs, metadata logs, cleanup, download/reconcile. | PY-2 [`labs/storage_manager.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/labs/storage_manager.py), PY-3 [`core/utils/storage_manager/*`](https://github.com/CrelioHealth/crelio-app/tree/develop/apps/crelio-app/core/utils/storage_manager) | | Presigned URL | Short-lived **direct-upload contract** from backend so the client uploads straight to S3. | `sm_generate_presigned_url`, storage-manager generate-presigned-url | | Reconcile | Post-upload step verifying the object exists and updating Storage Manager logs with **final size / status**. | `sm_reconcile_presigned_url`, presigned reconcile | | Temporary path | Caller-specified path **before** canonical Storage Manager placement. | `FileTypeReports/...` prefix patterns | | Concrete path | Canonical Storage Manager path matching standard SM conventions. | `IN///FileTypeReport/_.pdf` style paths | | Pusher update | Real-time report / waiting-list refresh after upload or finalization. | `commonPusherFunctionForLabReportRelation` | | ES | Elasticsearch - patient/report search indexing **and** Storage Manager log records. | `LabReportRelationES`, `StorageManagerLog` | What problem the revamp solves [#what-problem-the-revamp-solves] Old behavior [#old-behavior] Under the older mechanism, file type reports were uploaded via the backend **store** path: * the browser read the file * the **raw payload** moved through the application request * the backend validated and stored the file * only then was the report updated Predictable pain points: * large request bodies * backend CPU and memory pressure * poor scaling for high-resolution and multi-page PDFs * stricter effective file-size limits because the **application server** sat in the middle of the upload stream New behavior [#new-behavior] The revamp shifts bulk transfer out of the application tier: * the frontend asks for a **presigned upload contract** * the backend generates an S3 target and logs the request * the frontend uploads **directly to S3** * the system then **reconciles** metadata and **finalizes** the report record Architecture-level impact [#architecture-level-impact] | Area | Old model | New model | | :----------------------------------- | :------------------------------- | :------------------------------------------------------------------------------ | | Raw file transport | Browser → Backend → Storage | Browser → Storage | | App-server involvement during upload | High | Low | | Payload bottleneck on app server | Yes | Largely removed | | File-size tolerance | Constrained by backend transport | Much better when S3 receives the binary directly | | Post-upload bookkeeping | Backend only | Split across frontend metadata calls, Storage Manager logs, report finalization | | Extensibility | Tight coupling | Better decoupling when Lambda / PY-3 helpers are used | Reality check from the codebase [#reality-check-from-the-codebase] The architecture described in product/technical context and the architecture visible in source code are closely related, but **not one single linear path** in every deployment. What is definitely live in the current frontend code [#what-is-definitely-live-in-the-current-frontend-code] The current `livehealth-frontend` upload flow does this: 1. Validate the selected file. 2. Request a presigned upload from PY-2 `sm_generate_presigned_url/`. 3. Upload the binary to S3 directly. 4. Reconcile via `sm_reconcile_presigned_url/`. 5. Call PY-2 `uploadFileTypeReport/` with **metadata only**, especially `initialFilePath`. 6. Let PY-2 update the report value, send Pusher updates, and re-index the report in ES. What also exists in the repository [#what-also-exists-in-the-repository] Broader Lambda / PY-3 support surfaces: * PY-3 Storage Manager APIs * Lambda auth via `X-Lambda-Token` * PY-3 endpoint to convert a **temporary** file path into a **concrete** Storage Manager path * PY-3 endpoint to update the report value with the **final** file path * the **presigned-upload revamp is implemented** in the FE + PY-2 path * the **PY-3 / Lambda support surfaces are present** in `crelio-app` * the **exact production Lambda implementation** is external to this repo snapshot or lives in another repo / deployment artifact Cross-repo map (repositories and ownership) [#cross-repo-map-repositories-and-ownership] | Repo / app | Layer | Why it matters for this revamp | | :--------------------------------------------------------------------------- | :-------------------------------------------- | :------------------------------------------------------------------------------------------------------------------- | | [`livehealth-frontend`](https://github.com/CrelioHealth/livehealth-frontend) | Browser UI and orchestration | Starts upload, requests presigned URL, uploads to S3, reconciles, triggers report finalization | | [`livehealthapp`](https://github.com/CrelioHealth/livehealthapp) | PY-2 operational backend | Owns the **currently observed** finalization endpoint, report conversion, Pusher trigger, patient-report ES re-index | | [`crelio-app`](https://github.com/CrelioHealth/crelio-app) | PY-3 services and newer Storage Manager stack | Modern Storage Manager endpoints, canonical path utilities, Lambda auth, file-type-report Lambda-facing helpers | Suggested mental model (Level 0) [#suggested-mental-model-level-0] 1. **Product meaning** - the report result is a **file**, not typed values. 2. **State meaning** - `fileInputReport = 1`; `ReportValue.value` stores a **path**. 3. **Upload meaning** - Storage Manager gives the browser a **direct lane** into S3. 4. **Domain finalization meaning** - backend still decides when the report becomes **complete and searchable**. 5. **Infra-assisted evolution** - PY-3 and Lambda support preprocessing and path normalization **more decoupled** than the older PY-2-only synchronous path. The technical essence: **storage-transport decoupling** with **report-state side effects** still owned by the domain layer. See [Design Decisions](/docs/product-engineering/features/file-type-report-revamp/design-decisions) for explicit tradeoffs and caveats. # Workflow Guide Workflow Guide [#workflow-guide] This page is the **operator- and engineer-oriented walkthrough**: conversion prerequisites, upload lifecycle (presigned + reconcile + finalization), branching between **observed PY-2 path** vs **Lambda-assisted PY-3 path**, setup dependencies, a practical verification checklist, and failure-mode triage. Implementation references live in [Frontend](/docs/product-engineering/features/file-type-report-revamp/frontend) and [Backend](/docs/product-engineering/features/file-type-report-revamp/backend). How to enable the feature [#how-to-enable-the-feature] There is no separate in-repo **feature flag** that toggles “presigned vs legacy” for the main modal path; enablement is **permission + report mode + infrastructure**. 1. Grant **`allow_report_file_conversion`** on the lab user so **Report conversion** is available in operations. 2. For each report, run **Convert to file report** first (`fileInputReport = 1`); upload is only the correct workflow after conversion. 3. Confirm the lab’s session and operations context are valid (non-radiology report, operation privileges where enforced). How to configure the feature [#how-to-configure-the-feature] Configuration is mostly **platform / infra**, not a single JSON toggle in this docs repo. | Area | What to configure | | :-------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Storage Manager / S3** | Active account mapping for category **`FileTypeReport`**, allowed extensions (PDF + images as required), credentials, region, bucket policy for presigned POST. | | **PY-2 HTTP** | Routes exposed for `sm_generate_presigned_url/`, `sm_reconcile_presigned_url/`, `uploadFileTypeReport/`, `uploadFileWithHeader/` (legacy header/footer), `convertReportToFileOrNormal/`. | | **Search + realtime** | Patient-report **Elasticsearch** indexing and **Pusher** (or equivalent) must be healthy for the waiting list to reflect completion. | | **Optional Lambda-assisted path** | S3 event on temp prefix, Lambda deployment, **`X-Lambda-Token`** aligned with **`CRON_SIGNING_KEY`**, PY-3 **`report-value/file-type/*`** routes reachable from Lambda’s network. | Additional setup for Lambda-assisted architecture [#additional-setup-for-lambda-assisted-architecture] | Setup item | Why it matters | | :---------------------------------------- | :------------------------------------------------- | | S3 event notification on temporary prefix | Triggers Lambda on object upload | | Lambda deployment | Async validation / preprocessing | | `X-Lambda-Token` header | PY-3 Lambda request authentication | | `CRON_SIGNING_KEY` | Validates Lambda-originated requests | | PY-3 file-type-report endpoints | Temp → canonical path + DB update | | Storage Manager log visibility | PY-3 path generation reads existing SM log records | **Caveat:** Lambda **handler** and **event binding** are not in the inspected repo snapshot; this section documents **contracts and surfaces**, not the runtime script. Navigate through the workflow [#navigate-through-the-workflow] 1. Read **Feature lifecycle at a glance** (diagram below) for the fork between **PY-2 finalization** vs **Lambda + PY-3**. 2. Follow **[End-to-end flow (numbered)](#end-to-end-flow-numbered)** - Steps **0–7** (conversion → modal → validate → presign → S3 → reconcile → finalize → PY-2 side effects). 3. Use **Happy-path sequence diagram**, **Validation and branching**, and **Report state machine** for reviews and runbooks. 4. Use **Basic local verification checklist** and **Failure modes and troubleshooting** when validating an environment. Feature lifecycle at a glance [#feature-lifecycle-at-a-glance] End-to-end flow (numbered) [#end-to-end-flow-numbered] The steps below are the detailed walkthrough for [Navigate through the workflow](#navigate-through-the-workflow). Step 0: A report must be a file report first [#step-0-a-report-must-be-a-file-report-first] This feature does **not** start with upload. It starts with **report-mode conversion**. * user opens the report conversion modal ([`ReportConversionModal`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Modals/ReportConversionModal/index.tsx)) * frontend posts to `convertReportToFileOrNormal/` * PY-2 flips `fileInputReport`, clears values, resets signing/approval state, triggers Pusher + ES What changes during conversion [#what-changes-during-conversion] | Field / behavior | Effect when converting to file report | | :-------------------------------------------------------- | :------------------------------------ | | `fileInputReport` | Set to `1` | | `store_values_to_document_db` | Set to `0` in the conversion path | | Existing `ReportValue` rows | Deleted | | `completedTests` | Reset to `0` | | Signing data | Cleared | | `isApproved` / `isSigned` / `partialSigned` / `printDone` | Reset | | Pusher / ES | Triggered after conversion | Important guardrails [#important-guardrails] * **Radiology** reports are blocked from this conversion path. * **Operation-view** sessions without privilege are blocked. * Permission is controlled through **`allow_report_file_conversion`** (and doctor access is allowed in the conversion API; see backend references). Step 1: User opens the upload modal [#step-1-user-opens-the-upload-modal] Primary UI entry points: * [`UploadFileButton.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/TestWaitingList/UploadFileButton.tsx) * [`UploadFileTypeReportModal.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/TestWaitingList/UploadFileTypeReportModal.tsx) Modal capabilities [#modal-capabilities] | Capability | Behavior | | :------------------------------------------ | :------------------------------------------------------------------------ | | Choose file | Accepts `.pdf`, `.png`, `.jpg`, `.jpeg` | | Apply header and footer | Routes to the **legacy server-side upload** path (`uploadFileWithHeader`) | | Mark all other reports in this bill as Done | Calls `mark_report_as_done_api` after successful upload | | Patient context | Uses selected `labReportDetails` from modal state | Engineering nuance [#engineering-nuance] Business language often says “PDF lab reports”, but **code supports images** (`png`, `jpg`, `jpeg`). Downstream print/render paths contain **image-specific** handling (see backend rendering notes). Step 2: Frontend validates the selected file [#step-2-frontend-validates-the-selected-file] [`src/utils/helpers.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/utils/helpers.ts) - client-side **encrypted-PDF** check before proceeding. | Check | Outcome | | :--------------------- | :------------------------ | | File missing | Upload does not proceed | | Encrypted PDF | User-facing failure alert | | Session missing lab id | Upload rejected | **Nuance:** the frontend still reads the file into a **data URL / base64** representation before direct upload. That means the file **no longer traverses the application backend** on the happy path, but the **browser** still performs an in-memory conversion before `uploadPresigned`. This is a major improvement over backend transport; it is **not** a zero-copy browser path. Step 3: Frontend requests a presigned upload contract [#step-3-frontend-requests-a-presigned-upload-contract] Temporary path shape (representative): ```text FileTypeReports///// ``` Example: ```text FileTypeReports/IN/28/123456/ab12cd/report.pdf ``` Payload sent by frontend (conceptual) [#payload-sent-by-frontend-conceptual] | Field | Meaning | | :-------------------- | :-------------------------- | | `filename` | Original filename | | `file_category` | `FileTypeReport` | | `cloud_path` | Requested temporary S3 path | | `patient_id` | Optional patient id | | `request_host_origin` | Current browser origin | Storage Manager work during this step (PY-2) [#storage-manager-work-during-this-step-py-2] * validates filename and extension * selects correct S3 account * generates storage metadata * creates a **Storage Manager log** record * returns presigned POST fields + URL + file path Step 4: Browser uploads directly to S3 [#step-4-browser-uploads-directly-to-s3] [`PromotionActions.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/redux/actions/PromotionActions.ts) - `uploadFileOnPresigned(...)`: * rebuild `Blob` / `File` from data URL * submit **multipart/form-data** to presigned URL * **skip** app-server binary transport on the happy path Step 5: Frontend reconciles the upload [#step-5-frontend-reconciles-the-upload] After direct S3 upload succeeds, the frontend waits briefly and calls **`POST /sm_reconcile_presigned_url/`** (helper in [`MultiAttachments/helper.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/MultiAttachments/helper.ts)). Reconcile confirms: * object exists in S3 * Storage Manager can fetch metadata * file size can be written to the log * presigned upload lifecycle is **complete** from Storage Manager’s perspective Step 6: Frontend finalizes with metadata only [#step-6-frontend-finalizes-with-metadata-only] `POST /uploadFileTypeReport/` with fields such as: | Field | Meaning | | :---------------- | :----------------------------------- | | `fileName` | Original file name | | `extension` | File extension | | `initialFilePath` | Temporary / already-uploaded S3 path | | `labReportId` | Target report | At this stage the frontend is **not** sending the raw file body on the main happy path. Step 7: PY-2 finalizes the file report [#step-7-py-2-finalizes-the-file-report] [`uploadFileTypeReportAPI`](https://github.com/CrelioHealth/livehealthapp/blob/develop/apps/livehealthapp/reports/views.py) - two internal branches: | Branch | Trigger | Behavior | | :------------------------- | :----------------------------------------- | :---------------------------------------------------------------------------------------------------- | | New presigned-aware branch | `initialFilePath` present and contains `/` | Assumes file already in storage; skips backend binary handling; updates `ReportValue` + report status | | Legacy branch | `fileData` present | Accepts base64 payload, validates/restructures, stores via `SmStore`, then updates report value | In the new branch PY-2: fetches `labReportRelation`, blocks radiology, writes path to `ReportValue`, updates status fields, triggers Pusher, re-indexes ES, writes activity log. Happy-path sequence diagram (observed stack) [#happy-path-sequence-diagram-observed-stack] Validation and branching [#validation-and-branching] Report state machine (conceptual) [#report-state-machine-conceptual] Basic local verification checklist [#basic-local-verification-checklist] 1. Sign in as a user with `allow_report_file_conversion`. 2. Pick a **non-radiology** incomplete report. 3. Convert it to file report mode. 4. Open the upload modal. 5. Upload a small **non-encrypted** PDF. 6. Confirm UI success. 7. Verify report shows **completed** state. 8. Verify `ReportValue.value` holds a **file path**, not typed parameter results. 9. Verify Storage Manager log is **reconciled** with file size. 10. Verify waiting list / search reflects the update (Pusher + ES). Edge scenarios [#edge-scenarios] | Scenario | Expected outcome | | :--------------------------- | :------------------------------------------------ | | Encrypted PDF | Rejected on frontend | | Unsupported extension | Presigned generation fails | | Radiology report | Conversion / upload blocked | | Header/footer enabled | Legacy backend upload path | | Mark-as-done enabled | Sibling reports may complete after upload | | Storage object deleted later | cleanup may rewrite path to `file_not_exists.pdf` | Failure modes and troubleshooting [#failure-modes-and-troubleshooting] | Symptom | Likely layer | What to check | | :--------------------------------------- | :----------------------- | :----------------------------------------------------------------- | | “The PDF is encrypted…” | Frontend | File is encrypted; rejected before network | | “Session might be incorrect” | Frontend / session | Missing `labId` / `docLabId` in session | | Presigned generation fails | Storage Manager / config | S3 account mapping, extension allowlist, filename validation | | Reconcile fails | Storage Manager / S3 | eventual consistency, wrong path, bucket permissions, log mismatch | | S3 OK but report not complete | PY-2 finalization | `uploadFileTypeReportAPI` errors, radiology block | | Header/footer struggles with large files | Legacy path | still posts base64 through backend | | UI stale after “success” | Pusher / ES | realtime or indexing pipeline | Debugging by stage [#debugging-by-stage] | Stage | Best inspection point | | :--------------------- | :------------------------------------- | | Convert to file report | report row state + activity log | | Generate presigned | Storage Manager log entry | | Direct upload | S3 object existence | | Reconcile | log status + file size | | Finalization | `ReportValue.value` + `completedTests` | | UI refresh | Pusher + patient report ES document | End-to-end narrative [#end-to-end-narrative] * A report can be switched into **file-report mode**: the result becomes a **path**, not a matrix of parameters. * The old implementation pushed the whole PDF through the **app backend** - poor fit for large files. * The revamp’s hot path: frontend requests a **presigned contract**, uploads **directly to S3**, **reconciles**, then calls **`uploadFileTypeReport`** with **`initialFilePath`** so PY-2 mutates domain state, Pusher, and ES **without** carrying the binary in the request body. * PY-3 adds a more decoupled shape where Lambda could validate a temp upload, compute a **canonical** path, and update report value - **Lambda code is external** to the inspected snapshot. * The accurate headline is not “backend disappears”; it is **“backend stops carrying the binary payload but still owns report state and indexing.”** # Design Decisions Design Decisions & Architecture [#design-decisions--architecture] *** Key Design Decisions & Constraints [#key-design-decisions--constraints] Pre-Aggregation Over Real-Time Queries [#pre-aggregation-over-real-time-queries] **Decision:** Store pre-computed aggregates in `operation_historical_summary` rather than querying `billing` and `labReportRelation` at request time. **Why:** The `billing` table can have tens of millions of rows for a large lab. Doing a `GROUP BY orgId, testId, DATE(billTime)` across a 6-month range on request would be unacceptably slow, taking seconds to minutes. By pre-aggregating overnight, the frontend query hits a small, well-indexed table and responds in milliseconds. **Trade-off:** We accept eventual consistency. Today's data won't appear in Historical View until tomorrow's nightly job runs. The frontend displays a notification: *"All activities related to orders & samples performed today will appear in the Historical View on the next day."* Trigger-Based Corrections vs. Nightly Idempotency [#trigger-based-corrections-vs-nightly-idempotency] **Decision:** Two MySQL triggers (`trg_lab_report_relation_after_update` and `trg_billing_after_update`) perform real-time delta corrections to the summary table. **Why:** The nightly job handles aggregation for yesterday's data. But what about today's corrections to historical records? If a lab cancels a bill from last Tuesday, the nightly job won't touch it again. The triggers catch these corrections and apply `+1` or `-1` deltas immediately. **Constraint:** Triggers only fire for `UPDATE` operations (not `INSERT`). New records created during the day are handled by the next nightly run. This is intentional since the insert volume is too high for trigger-based aggregation. Dimension A / Dimension B Generic Model [#dimension-a--dimension-b-generic-model] **Decision:** Use generic `dimension_a_id` and `dimension_b_id` columns instead of explicit `organization_id`, `test_id`, etc. **Why:** The summary table needs to serve 10 event types across 4 primary dimensions and 2 secondary dimensions. With explicit FK columns, you'd need 12 separate columns (most of which would be NULL for any given row) or 10 separate tables. The generic dimension model collapses all of this into one table with one set of indexes. **Trade-off:** You lose referential integrity at the DB level. `dimension_a_id = 42` could refer to an organization, a referral, or a branch, depending on the `event_type`. The application layer must ensure correctness. This is acceptable because the data is derived (not user-entered) and fully controlled by the trigger/procedure code. Year-Based Partitioning [#year-based-partitioning] **Decision:** The table is partitioned by `RANGE COLUMNS (ordered_at)` with yearly boundaries. **Why:** * **Nightly job performance** - each run only inserts into yesterday's partition. Other partitions are completely untouched. * **Historical data pruning** - if you need to drop 2021 data, `ALTER TABLE DROP PARTITION summary_table_2021` is near-instant vs. a `DELETE FROM` that locks the whole table. * **Query performance** - a weekly query for Jan 2026 only scans `summary_table_2026` partition. **Constraint:** A new partition needs to be added proactively each year. The `summary_table_future` MAXVALUE partition acts as a safety net, but data landing there would miss out on partition pruning optimizations. Timezone-Aware Date Bucketing [#timezone-aware-date-bucketing] **Decision:** `ordered_at` stores the date in the **lab's local timezone**, not UTC. **Why:** Labs think in local time. When a lab in IST (UTC+05:30) processes a bill at 23:00 IST (17:30 UTC), they expect it counted as "today" (the IST date), not "tomorrow" (which is what the UTC date would show after midnight IST). Bill-time bucketing must match the lab's business-day concept. **Implementation:** Every path that writes to `ordered_at` does `DATE(CONVERT_TZ(billTime, 'UTC', labTimeZone))`. The nightly scheduler groups labs by `labTimeZone` and computes each timezone's "yesterday" independently. Access Status as Separate Event Types [#access-status-as-separate-event-types] **Decision:** Rather than adding an `access_status` column to the summary table, accessed and not-accessed counts are stored as separate event types (e.g., `organization_sample_accessed`, `organization_sample_not_accessed`). **Why:** * **Index reuse** - the same composite index `(lab_id, event_type, ordered_month, ...)` serves all queries. Adding a column would require a new index or a wider covering index. * **Trigger simplicity** - when a sample's access status changes, the trigger just does `event_value -= 1` on one event type and `event_value += 1` on another. No conditional column updates needed. * **Query simplicity** - to get "total accessed samples for org X", just query `WHERE event_type = 'organization_sample_accessed'`. No `CASE WHEN` needed. **Trade-off:** The `operation_historical_summary` table has more rows (roughly 2× for sample events, since each sample counts toward both `*_sample` and `*_sample_accessed` or `*_sample_not_accessed`). Correlated Subqueries for Access Filtering [#correlated-subqueries-for-access-filtering] **Decision:** When the user applies an "Accessed" or "Not Accessed" filter, the backend swaps `SUM(event_value)` with a correlated subquery that counts directly from `labReportRelation`. **Why:** The pre-aggregated `event_value` includes *all* records regardless of access status. To filter by access status while keeping the primary dimension grouping (e.g., org-wise), we need to re-count from the source table with the access condition applied. **Trade-off:** Access-filtered queries are slower than unfiltered ones because they hit the transactional tables via subqueries. This is acceptable because: 1. The outer query still uses the indexed summary table for scoping (lab, event\_type, date range). 2. The filter is optional and used less frequently than the default view. Bucket Filling on the Backend [#bucket-filling-on-the-backend] **Decision:** The backend fills in zero-value rows for missing date buckets rather than having the frontend handle gaps. **Why:** AG Grid's pivot table rendering assumes a complete matrix. If "Apollo Hospitals" had tests on Mon and Wed but not Tue, the frontend would either show a missing column or misalign subsequent columns. By filling zeros server-side, every entity has a row for every date bucket, and the grid renders correctly. *** Architectural Rationale [#architectural-rationale] Why a Hybrid System (Triggers + Nightly Job)? [#why-a-hybrid-system-triggers--nightly-job] Each component handles a different temporal concern: | Component | When it runs | What it handles | | ----------------------- | ------------------------ | --------------------------------------------------------- | | **Nightly Job** | Daily at 04:00 UTC | Yesterday's full data, the bulk insert path | | **Triggers** | On every relevant UPDATE | Today's corrections to historical data, delta patches | | **Migration Procedure** | On feature activation | Historical backfill, months/years of data on first enable | No single approach handles all three. The nightly job can't handle retroactive changes (triggers). The triggers would create too much overhead for bulk inserts (nightly job). And neither handles the initial data load for a new lab (migration procedure). Why Raw SQL Instead of Django ORM? [#why-raw-sql-instead-of-django-orm] The query builder (`_prepare_fetch_sql_query`) uses raw SQL throughout. This is deliberate: 1. **Generated columns** - `ordered_month` isn't in the Django model and can't be queried via ORM. 2. **Dynamic JOINs** - The JOIN target changes based on `event_type` (organization, doctors, branch, etc.). The ORM's `select_related` / `prefetch_related` can't do conditional JOINs. 3. **Granularity expressions** - The bucketing SQL uses `DATE_ADD` with `FLOOR(TIMESTAMPDIFF(...))`, which is MySQL-specific and not expressible via ORM. 4. **Correlated subqueries** - Access-filtered queries use `(SELECT COUNT(...) FROM ... WHERE ... = outer.column)`, not supported via ORM. This is consistent with the pattern used elsewhere in the Operations Dashboard, which is primarily a reporting/analytics feature. *** Extensibility Guide [#extensibility-guide] | What you want to do | How to do it | | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Add a new primary dimension** (e.g., "patient\_type") | 1. Add event types to the `EVENT_TYPES` tuple in `OperationSummary`
2. Add mapping cases in `insert_operation_summary_for_event` SQL procedure
3. Add join/select mappings in `_prepare_join()` / `_prepare_select_columns()`
4. Add an `entityTypeTabs` entry in the frontend constants | | **Add a new secondary dimension** (e.g., "profile") | 1. Add event types
2. Update the SQL procedure's dimension mapping
3. Add join mapping in the backend query builder | | **Change the nightly schedule** | Modify the `STARTS` clause in the `CREATE EVENT` statement | | **Add a new access status variant** (e.g., "partially\_accessed") | 1. Add event types
2. Add trigger logic for the new transition
3. Add SQL procedure mapping
4. Add frontend `ACCESS_FILTER_OPTIONS` entry | | **Add a new granularity** (e.g., "quarterly") | 1. Add to `validate_payload()`
2. Add SQL expression in `_get_granularity_sql()`
3. Add bucket generation in `_generate_date_buckets()`
4. Add frontend dropdown option | *** Known Constraints & Gotchas [#known-constraints--gotchas] | Constraint | Impact | Mitigation | | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | **Triggers don't fire on INSERT** | New `labReportRelation` or `billing` rows created during the day are NOT reflected in the summary until the nightly job | This is by design since insert-triggered aggregation would be too expensive | | **Today's data is always missing** | Users may report "the numbers are wrong for today" | The frontend shows a warning notification | | **No automatic partition creation** | If the `summary_table_future` partition is hit, queries lose partition pruning | Add a yearly migration/runbook item to create next year's partition | | **Generic dimension IDs prevent FK constraints** | `dimension_a_id = 42` could be orphaned if the source entity is deleted | The source entities (orgs, referrals, etc.) use soft-deletes, so this is rare | | **Test Wise tab overrides event key** | When the frontend sends "Test Wise" as the group, `getEventKey()` maps it to `"organization"` to avoid producing `test_test` as an event type | This is documented in the frontend constants, see the `groupMap` in `helpers.ts` | # Overview Historical Summary [#historical-summary] Historical Summary is a feature within the Operations Dashboard that gives labs a longitudinal view of their test, sample, and bill volumes over time. Instead of looking at a single-day snapshot ("today you processed 280 tests"), it lets ops managers ask trend-oriented questions like "How did our referral-wise test volumes shift week-over-week for the last quarter?" and get answers in a pivot-grid that they can slice by organization, referral, branch, department, or individual test. Related JIRA / Feature References [#related-jira--feature-references] * Feature flag: `enable_operation_summary` on the `labFeatures` table * MySQL event scheduler: `operation_summary_nightly_event` * Support Dashboard: toggled under **Workflow Configurations → Enable Workflows/Features** Prerequisites [#prerequisites] | Requirement | Details | | ------------------------- | --------------------------------------------------------------------------------------------------------------- | | **Feature flag** | `enable_operation_summary = 1` in `labFeatures` for the lab | | **MySQL event scheduler** | Must be `ON`. The nightly aggregation job runs as a MySQL `EVENT` | | **billTime availability** | Lab must have historical billing data; the first `billTime` in `billing` determines the start of available data | | **Timezone config** | `labTimeZone` on the `labs` table must be set correctly. All date bucketing is timezone-aware | What is it For? [#what-is-it-for] Traditional operations dashboards show only the current day's activity. This works for real-time ops but fails when managers want to: * **Spot trends** - Are referrals from "Apollo Hospitals" growing or declining month-over-month? * **Benchmark branches** - Which branch processes the highest test volume every week? * **Audit sample access** - How many samples were accessed vs. not-accessed in a given period? * **Track billing patterns** - Department-wise bill counts per week, for capacity planning. Historical Summary answers all of these by pre-aggregating operational data into a star-schema-like table (`operation_historical_summary`) and exposing it through a pivot grid on the frontend. Core Concepts [#core-concepts] Dimensions [#dimensions] The system uses a two-dimension model to categorize every metric: | Dimension | Role | Examples | | ---------------------------- | ---------------------------------------------- | ------------------------------------------ | | **dimension\_a** (Primary) | The main grouping entity | Organization, Referral, Branch, Department | | **dimension\_b** (Secondary) | The drill-down entity within the primary group | Test ID, Sample ID | Event Types [#event-types] Each combination of primary × secondary dimension produces an `event_type`. Here is the full matrix: | Primary Dimension | Test Count | Sample Count | Bill Count | Sample Accessed | Sample Not Accessed | | ----------------- | ------------------- | --------------------- | ------------------- | ------------------------------ | ---------------------------------- | | Organization | `organization_test` | `organization_sample` | `organization_bill` | `organization_sample_accessed` | `organization_sample_not_accessed` | | Referral | `referral_test` | `referral_sample` | `referral_bill` | `referral_sample_accessed` | `referral_sample_not_accessed` | | Branch | `branch_test` | `branch_sample` | `branch_bill` | `branch_sample_accessed` | `branch_sample_not_accessed` | | Department | `department_test` | N/A | N/A | N/A | N/A | Granularity [#granularity] Users can view data at three time resolutions: | Granularity | Max Date Range | Bucketing Logic | | ----------- | -------------------- | --------------------------------------------------------------- | | **Daily** | 14 days | Each day = one bucket | | **Weekly** | 62 days (\~2 months) | 7-day intervals anchored to `start_date` | | **Monthly** | Unlimited | First-of-month buckets (uses the stored `ordered_month` column) | UI Overview [#ui-overview] The Historical View tab lives inside the Operations Dashboard, alongside the existing Summary View and Detailed View. Test Count — Organization Wise (Weekly) [#test-count--organization-wise-weekly] Historical View showing weekly organization-wise test counts with Grand Total and Grand Average rows Sample Count — Organization Wise with Sample Breakdown (Weekly) [#sample-count--organization-wise-with-sample-breakdown-weekly] Historical View showing weekly sample counts with Sample Wise breakdown, expandable organization groups Bill Count — Organization Wise (Monthly) [#bill-count--organization-wise-monthly] Historical View showing monthly organization-wise bill counts **Key elements visible in the interface:** * **View tabs** — Summary View / Detailed View / Historical View (Beta) * **Controls** — Filter By Sample Status, granularity dropdown (Weekly / Monthly), date range picker, Export button * **Count tabs** — Test Count, Sample Count, Bill Count (some hidden depending on entity) * **Entity tabs** — Organization Wise, Referral Wise, Department Wise, Test Wise, Branch Wise (varies by count tab) * **Breakdown selector** — "Select Breakdown" / "Sample Wise" for secondary grouping * **Pivot grid** — Entity names in the first column, date buckets as dynamic columns, with **Grand Total** and **Grand Average** pinned to the bottom * **Notification banner** — *"All activities related to orders & samples performed today will appear in the Historical View on the next day. All data is calculated based on the `{timezone}` timezone."* Feature Components [#feature-components] How to enable via Support Dashboard, what happens behind the scenes, and what to expect Database schema, triggers, nightly event scheduler, and migration procedures Django ORM model, SQL query builder internals, and date bucket filling logic REST API endpoint, request parameters, response format, and URL configuration React component tree, AG Grid integration, tab system, and service layer Architectural rationale, trade-offs, and extensibility guide How to trigger data rebuilds, monitor migration progress, and run ad-hoc repairs # Workflow Guide Workflow Guide [#workflow-guide] This page covers how to turn Historical Summary on for a lab, what the system does behind the scenes once enabled, and what the lab user will see in the product. *** How to Enable [#how-to-enable] Historical Summary is enabled per-lab through the **Support Dashboard**. Steps [#steps] 1. Navigate to the **Support Dashboard** for the target lab. 2. Go to **Workflow Configurations → Enable Workflows/Features**. 3. Under the **Features** section, toggle **"Historical View in Operations Dashboard"** to ON. 4. Save the configuration. Support Dashboard — enabling Historical View in Operations Dashboard What Happens Under the Hood [#what-happens-under-the-hood] When the toggle is saved, the following sequence runs in [`support_dashboard_settings_view.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/support_dashboard_settings_view.py): 1. **Feature flag write** — `labFeatures.enable_operation_summary` is set to `True` for the lab. 2. **Migration scheduling** — Any existing `crelio_data_migrations` record for this lab (type `HISTORICAL_SUMMARY`) is deleted. 3. **New migration record** — A new `CrelioDataMigrations` row is inserted with `is_scheduled=True` and `migration_type='HISTORICAL_SUMMARY'`. ```python # Simplified from support_dashboard_settings_view.py if "enable_operation_summary" in diff: CrelioDataMigrations.objects.filter( lab_id=lab_id, migration_type="HISTORICAL_SUMMARY" ).delete() value and CrelioDataMigrations.objects.create( lab_id=lab_id, is_scheduled=True, initiated_by_id=account_manager_id, migration_type="HISTORICAL_SUMMARY", ) ``` This record is what the nightly migration event uses to know which labs need historical data backfill. *** What to Expect After Enabling [#what-to-expect-after-enabling] Timeline [#timeline] | Step | When | What Happens | | ------------------------------------------ | -------------------------------------- | ---------------------------------------------------------------------------- | | **1. Flag saved** | Immediately | `enable_operation_summary = True` persisted; migration record created | | **2. Migration event runs** | Next occurrence of **03:30 UTC** daily | `operation_summary_migration` procedure picks up labs with `is_scheduled=1` | | **3. Historical backfill** | During migration run | Full recompute from the lab's earliest `billTime` to yesterday | | **4. Nightly job starts covering the lab** | Next occurrence of **04:00 UTC** daily | `operation_summary_nightly_event` includes this lab in its daily aggregation | > **Note:** If the toggle is saved before 03:30 UTC, the backfill will run the same day. If saved after, it runs the next day. Backfill Duration [#backfill-duration] The backfill processes the lab's entire billing history. Duration depends on data volume: * Small labs (\< 1 year of data): seconds to a few minutes * Large labs (3+ years, millions of bills): can take 10–30 minutes The actual execution time is recorded in `crelio_data_migrations.time_taken` (in seconds). What the Lab User Sees [#what-the-lab-user-sees] Once the backfill completes and the user refreshes the Operations Dashboard: 1. A new **"Historical View"** tab appears alongside "Summary View" and "Detailed View." 2. Selecting it shows the pivot grid with entity tabs, count tabs, and date columns. 3. A notification banner reads: *"All activities related to orders & samples performed today will appear in the Historical View on the next day. All data is calculated based on the `{timezone}` timezone."* Historical View — the pivot grid showing weekly organization-wise test counts Data Availability Rules [#data-availability-rules] | Scenario | Available in Historical View? | | ---------------------------------------------------------- | -------------------------------------- | | Bills billed yesterday or earlier | ✅ Yes (after nightly job runs) | | Bills billed today | ❌ No (will appear tomorrow) | | Corrections to historical bills (cancel, org change, etc.) | ✅ Yes, instantly (handled by triggers) | *** How to Disable [#how-to-disable] 1. Go back to **Workflow Configurations → Enable Workflows/Features**. 2. Toggle **"Historical View in Operations Dashboard"** to OFF. 3. Save. What Happens on Disable [#what-happens-on-disable] * The `enable_operation_summary` flag is set to `False`. * Any pending `crelio_data_migrations` record for this lab is **deleted** (no new migration will be scheduled). * The "Historical View" tab **disappears** from the frontend on next page load. * **Existing data in `operation_historical_summary` is NOT deleted.** The summary table retains the lab's data. If the feature is re-enabled later, the migration procedure will delete and recompute (clean-slate rebuild). *** Verifying the Enable Status [#verifying-the-enable-status] Check Feature Flag [#check-feature-flag] ```sql SELECT enable_operation_summary FROM labFeatures WHERE labForId_id = ; ``` Check Migration Status [#check-migration-status] ```sql SELECT lab_id, is_scheduled, job_status, time_taken, start_date, end_date, created_at FROM crelio_data_migrations WHERE lab_id = AND migration_type = 'HISTORICAL_SUMMARY'; ``` | `job_status` | Meaning | | ------------ | --------------------------------------------- | | `Pending` | Waiting for the migration event to pick it up | | `Completed` | Backfill finished successfully | Check Nightly Job Logs [#check-nightly-job-logs] ```sql SELECT * FROM operation_event_log ORDER BY created_at DESC LIMIT 10; ``` This shows the last few STARTED / COMPLETED entries for the nightly aggregation job. # Design Decisions Design Decisions [#design-decisions] These decisions prioritize **decoupling**, **observability**, and **resilience** — ensuring that integration failures never affect core product workflows and that every execution is fully traceable. *** 1. Event-Driven Configuration via Action IDs [#1-event-driven-configuration-via-action-ids] The Problem [#the-problem] Hardcoding integration logic (URLs, headers, auth) inside business controllers (e.g., `billing.save`) makes each controller brittle. Adding a new vendor requires modifying core business logic. The Decision [#the-decision] Use stable, business-event-based identifiers — **Action IDs** (e.g., `57` for Patient Registration, `1` for Bill Generation) — defined in `crm.integrations.constants`. Each `labIntegration` row maps an Action ID to a specific endpoint configuration. **Why:** * **Dynamic mapping** — a single business event can fan out to multiple integrations based purely on database configuration. No code change needed to add a vendor. * **Separation of concerns** — the business controller only declares *that an event occurred*. It does not know or care *what happens next*. * **Standardized context** — metadata (lab ID, bill ID, patient ID) is consistently extracted using the same mechanism for every action type. *** 2. Centralized Trigger Functions (Common Functions) [#2-centralized-trigger-functions-common-functions] The Decision [#the-decision-1] All integration triggers are routed through named "common functions" (e.g., `commomFunctionForIntegrationBillCategory`, `common_integration_sample_receive`) rather than calling integration APIs directly. **Why:** * **Consistency** — whether a record is created via UI, bulk upload, or API, the same integration logic runs. No parallel code paths. * **Code reuse** — payload preparation, `labIntegration` lookup, and Fusion queueing all live in one place per action family. * **Retryability** — the `_prepare_kwargs_*` pattern can map any failed log back to the same function, making retry reconstruction predictable. *** 3. Async Dispatch through Fusion [#3-async-dispatch-through-fusion] The Decision [#the-decision-2] All outbound integrations are queued through **Fusion** async workers rather than being dispatched inline. **Why:** * **Latency isolation** — user actions (signing a report, finalizing a bill) complete immediately. Third-party network delays have zero impact on the product experience. * **Reliability** — Fusion provides built-in queuing and isolation from transient failures. * **Throughput control** — high-volume report events don't overwhelm downstream partners; workers consume the queue at a controlled rate. *** 4. Decorator-Based Logging (IntegrationDirectoryLogger) [#4-decorator-based-logging-integrationdirectorylogger] The Decision [#the-decision-3] Logging is implemented as **Python decorator classes** applied to specific API views and webhook handlers, not as imperative logging calls inside business logic. **Why:** * **Non-invasive** — the business function doesn't know logging is happening. Adding observability requires one line (`@IntegrationDirectoryLogger()`), not changes to the function body. * **Guaranteed capture** — the decorator always runs after the function, even if the function raises. The log reflects what actually happened. * **Lifecycle tracking** — inbound decorators capture the full request → execute → response cycle. Outbound decorators update a pre-created `QUEUED` log to `SUCCESS` or `FAIL`. * **Fault tolerance** — if the active DB transaction is in a broken state (`needs_rollback=True`), the decorator falls back to a Fusion webhook so the log record is never silently lost. *** 5. DocumentDB for Integration Logs [#5-documentdb-for-integration-logs] The Decision [#the-decision-4] `IntegrationDirectory` and `IntegrationRetryLog` are stored in **DocumentDB** (MongoDB-compatible), not relational tables. **Why:** * **Schema flexibility** — every vendor returns a different JSON shape. A fixed relational schema would require migrations for every new integration or response structure change. * **Append-only retry history** — `IntegrationRetryLog` grows without modifying the parent record. Relational normalization would add unnecessary join complexity for an operational read pattern. * **Payload size** — webhook payloads (base64 PDFs, HL7 messages, large JSON bodies) don't belong as RDBMS columns. * **Read pattern fit** — dashboard queries are operational log reads and aggregations, not transactional joins. DocumentDB handles this efficiently. *** 6. Centralized Internal Dispatcher (integrationWebhookAPI) [#6-centralized-internal-dispatcher-integrationwebhookapi] The Decision [#the-decision-5] All Fusion-queued integration tasks are routed back through a centralized internal API (`integrationWebhookAPI` in `labs/pyhtonRQAPI.py`) rather than calling third-party endpoints directly from trigger functions. **Why:** * **Just-in-time payload enrichment** — binary content (base64 PDFs of reports or bills) is injected at dispatch time, not at trigger time. This avoids holding large payloads in the Fusion queue. * **Unified auth handling** — token refreshes and auth header construction happen in one place. * **Callback coordination** — the dispatcher manages the `integration_status_callback` lifecycle, keeping status update logic out of individual trigger functions. *** 7. Intent Reconstruction for Retries (_prepare_kwargs_*) [#7-intent-reconstruction-for-retries-_prepare_kwargs_] The Decision [#the-decision-6] Retries do not replay the original serialized HTTP request body. Instead, the system re-fetches live domain objects from the database and calls the original trigger function with fresh arguments. **Why:** * **Stateless workers** — background workers cannot safely reuse stale Python objects from a prior execution context. Only DB IDs and primitive values are safely serializable into a retry context. * **Data accuracy** — if a patient record, bill, or report was corrected between the first failure and the retry, the retry should use the corrected data. Replaying a stale payload would re-dispatch incorrect information. *** Summary [#summary] | Decision | Rationale | | ------------------------- | ------------------------------------------------------------------------------------ | | **Action IDs** | Decouples business events from integration handlers — no code change to add a vendor | | **Common functions** | Ensures consistent trigger logic regardless of how an event was initiated | | **Fusion workers** | Isolates user workflows from third-party network instability | | **Decorator logging** | Keeps business code clean, guarantees capture, supports fault-tolerant fallback | | **DocumentDB** | Handles schema-variable payloads and append-only retry history at scale | | **Central dispatcher** | Handles just-in-time enrichment, unified auth, and callback coordination | | **Intent reconstruction** | Ensures retries use fresh, accurate domain data — not stale serialized state | # Frontend Integration Dashboard — Frontend [#integration-dashboard--frontend] The Integration Dashboard frontend is an operational console for monitoring and managing third-party integrations in real time. It provides visibility into inbound API requests and outbound webhook dispatches, with the ability to manage configurations and initiate retries directly. State Management [#state-management] The dashboard uses a centralized `integrationDashBoardGenericState` slice to prevent UI inconsistencies between tabs, modals, and row selections. | State Key | Purpose | | ------------------------ | ------------------------------------------------------------------ | | `viewTab` | Active tab — `1`: API, `2`: Webhook, `3`: Errors | | `selectedRowData` | Log record currently being inspected or retried | | `manageModalData` | Active integration configuration being edited (URL, headers, etc.) | | `currentTabLogsDataList` | Filtered log rows rendered in the current tab view | *** Component Hierarchy [#component-hierarchy] IntegrationDashBoardHeader [#integrationdashboardheader] Displays summary metrics — total success vs. failure counts, a `DateRangePicker` for filtering, and access to the **Manage Integrations** modal for configuration changes. HitMapGridView [#hitmapgridview] A visual heatmap aggregating `SUCCESS` / `FAIL` counts by endpoint and time window. Useful for spotting sudden failure spikes for a specific vendor or action type. GridView (Primary Table) [#gridview-primary-table] Lists individual `IntegrationDirectory` log records with response code, timestamp, and action buttons. Each row exposes: * **View Payload** — opens the full request/response payload. * **Retry** — triggers the retry flow appropriate for that row type. *** Retry Flows [#retry-flows] Different row types open different retry modals: | Log Type | Modal | Behavior | | -------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------- | | Successful webhook re-send | `IntegrationResendModal.tsx` | Re-queues the same payload for manual re-delivery | | Failed webhook | `ActionCategoryModal.tsx` | Supports batch retry across affected records | | Failed inbound API | `ApiRetryModal` | Renders JSON editor (via `@uiw/react-json-view`) so the payload can be corrected before replaying | *** Real-Time Updates (Pusher) [#real-time-updates-pusher] The dashboard stays live using `linkIntegrationDashboardLogsToPusher(...)`. When a backend Fusion worker updates an `IntegrationDirectory` document (e.g., from `QUEUED` to `SUCCESS`), a Pusher event is broadcast and the dashboard updates the affected row and heatmap without a page refresh. *** Source File Map [#source-file-map] | Component | Path | | ------------------ | ----------------------------- | | Mount point | `Container/index.tsx` | | Grid rendering | `Components/GridView.tsx` | | Retry modals | `Components/Modals/` | | Pusher integration | `utils/pusher_integration.ts` | # Overview Integration Dashboard [#integration-dashboard] The Integration Dashboard is the central observability layer for all data exchanges between LiveHealth/Crelio and external third-party systems. It does not execute integration logic itself — it records what happened, when it happened, and whether it succeeded. The actual execution pipeline is event-driven: a core business action (e.g., bill generation) triggers a domain function, which evaluates the `labIntegration` config, constructs a payload, queues an async Fusion task, and dispatches it to a third-party endpoint. The dashboard's role is to produce a structured, queryable log of every such event — both inbound API calls (captured via decorator) and outbound webhook dispatches (captured via decorator on the internal handler). `IntegrationDirectoryLogger` and `WebhookIntegrationDirectoryLogger` are **decorators** — passive logging wrappers, not orchestrators. They do not route requests, queue tasks, or make integration decisions. They only observe the function they wrap and persist a structured record of what occurred. *** Core Entities [#core-entities] Event Definitions [#event-definitions] * `actionCategory` — the broader business category (e.g., Billing, Sample Lifecycle). * `actionCategoryList` — the concrete event (e.g., **Bill Generation**, **Report Signed**, **Sample Receive**, **New Patient Registered**). Runtime Configuration [#runtime-configuration] * `developerAuthentication` — the integration root record. Stores the auth key, developer details, and feature flags. * `labIntegration` — the action-level runtime config. Stores the endpoint URL, request type, headers, and flags like `is_batch` and `is_auto_retry`. Observability Models [#observability-models] * `IntegrationDirectory` — the structured log document for every API call or webhook execution. Stores request/response data, status, execution metadata, and business references (lab, bill, patient, LRR, report). * `IntegrationRetryLog` — one document per retry attempt, linked back to `IntegrationDirectory` via `integration_directory_id`. *** Business Flows Covered [#business-flows-covered] Integrations are organized around business-event families, not vendor-specific paths: | Category | Example Events | | ---------------------------------- | ------------------------------------------------------------- | | **Billing & Orders** | Bill generation, cancellation, settlement, outsource billing | | **Sample Lifecycle** | Sample receive, redraw, dismiss, uncollect, collect | | **Reporting & Results** | Report signed, report submitted, fax delivery, patient portal | | **Registration** | Patient profile create, demographic update | | **Appointments & Home Collection** | Booking, confirmation, dismissal, HC scheduling | | **Mirth / HL7 Handoff** | HL7 data push via TCP socket through Mirth | *** Sections [#sections] Step-by-step execution sequences for configuration, outbound triggers, bulk dispatch, inbound capture, retries, and Mirth handoff Component hierarchy, state management, Pusher real-time sync, and the retry modal flows Trigger functions, logging decorators, Fusion queueing, Mirth handoff, and retry mechanics Why Action IDs, why common functions, why DocumentDB, why decorator-based logging, why Fusion workers # Workflow Guide Integration Dashboard — Workflow Guide [#integration-dashboard--workflow-guide] import Image from 'next/image' import dashboardapitab from '@/images/integration-dashboard/dashboard-api-tab.png' import dashboarderrorstab from '@/images/integration-dashboard/dashboard-errors-tab.png' import dashboardwebhooktab from '@/images/integration-dashboard/dashboard-webhook-tab.png' import manageIntegrationsui from '@/images/integration-dashboard/manage-integrations-ui.png' This guide walks through the full lifecycle of an integration from a user's perspective: configuring the trigger, watching logs appear on the dashboard, reading outcomes, and retrying failures. For the underlying backend execution sequences, see the [Backend Workflow Guide](/docs/product-engineering/features/integration-dashboard/backend/workflow-guide). *** Step 1: Adding an Integration Trigger via manageIntegrations [#step-1-adding-an-integration-trigger-via-manageintegrations] The starting point for all integrations is the **Manage Integrations** tool inside the CrelioHealth Support Dashboard CRM. This is where the trigger configuration (which lab, which action, which endpoint) is created or updated. **URL:** `https://livehealth.solutions/supportdashboard/manageIntegrations/` How to configure an integration [#how-to-configure-an-integration] Manage Integrations Page **1. Search for the Lab** Open the `manageIntegrations` page. Use the **Search by Lab Name, Id** dropdown (powered by `AsynchronousSelect`) to find the lab you want to configure. Once selected, the tool auto-loads all existing `DeveloperAuthentication` entries for that lab. **2. Select a Developer** From the **Select Developer Name** dropdown, pick the developer/integration partner whose authentication key (`authKey`) the integration will use. Selecting a developer loads all existing `labIntegration` rows associated with that developer context. **3. Select or Create an Integration ID** From **Select Integration Id**, pick an existing `labIntegration` row to edit — or leave blank to create a new one. **4. Fill in the Configuration Panels** Two accordion panels appear: **Developer Authentication panel** — captures the core integration identity: | Field | Purpose | | -------------------- | ----------------------------------------------------------------------------------------------------- | | Developer Name | Display name for the integration partner | | Auth Key | Token that third-party systems present for inbound calls | | URL / Endpoint | **The third-party endpoint** payloads are delivered to (e.g., `https://fierce-raven-45.webhook.cool`) | | Integration Type | The integration platform/service (e.g., Webhook, API) | | Action Category List | **The trigger** — maps a LiveHealth business event (e.g., "Bill Generated") to this integration | | Request Type | `GET` or `POST` (or the enum value `0`/`1` for outbound webhooks) | | Payload Type | HL7 message type if applicable (e.g., `ORM`, `ADT`, `ORUR01`) | | Template Text | Optional payload template for structured message formats | | Extra Details | JSON blob for extended config — Mirth `host`/`port`, `vendor`, `x-api-key`, etc. | | Organisation | Visible only when `isOrg=1`; scopes the trigger to a specific org rather than the whole lab | **Developer Authentication Flags panel** — boolean toggles for advanced behaviours: | Flag | Meaning | | --------------------- | ---------------------------------------------------------------- | | `isOrg` | Scope trigger to an organisation instead of the full lab | | `isIntegrationEnable` | Master enable/disable switch for this auth entry | | `isDisable` | Soft-disable a specific `labIntegration` row without deleting it | | (others) | Additional feature flags as needed | **5. Save** Click **Add/Update Integration**. The button calls `addUpdateIntegration(...)` which writes: * A `developerAuthentication` row (resolved by `authKey`) * One or more `labIntegration` rows keyed by `actionCategoryListId` On success, the response returns the generated `authKey` and a `200` toast. Once the `labIntegration` row is active (`isIntegrationEnable=1`, `isDisable=0`), the next time the mapped business event fires for that lab, the trigger functions (e.g., `commomFunctionForIntegrationBillCategory`) will detect this row and begin dispatching payloads and creating logs automatically. No code change is required. *** Local Development Setup [#local-development-setup] On local environments you don’t need to use `manageIntegrations`. Instead: 1. Open the `labIntegrations` table in your database. 2. Add (or update) a row with the required `actionCategoryListId_id` that maps to the business event you want to test. 3. Set the `url` / endpoint field to your test webhook receiver (e.g., a [webhook.cool](https://webhook.cool) or similar tool). 4. If the integration type is **Mirth**, also populate `integrationExtraDetails` with the host and port JSON: ```json { "host": "127.0.0.1", "port": 6661 } ``` 5. Ensure `isDisable=0` and `isIntegrationEnable=1` on the related `developerAuthentication` row. 6. The next business event matching that `actionCategoryListId` will automatically trigger the integration. No server restart is needed — the trigger functions query the database on every event. *** Step 2: How a Log Gets Created [#step-2-how-a-log-gets-created] After an integration is configured and a business event fires, the log lifecycle proceeds automatically: **Key observations:** * The log is **pre-created as `QUEUED`** before the HTTP call is made. This ensures the record is never lost even if the worker times out. * The `log_id` (DocumentDB `_id`) travels through the entire chain — from `get_integration_payload()` → Fusion task payload → handler → decorator → DocumentDB update. * Logging is **always a passive side-effect**. It cannot break the business event that triggered it. *** Step 3: Reading Integration Logs on the Dashboard [#step-3-reading-integration-logs-on-the-dashboard] Once events are firing, operators and support staff can observe the outcomes on the Integration Dashboard. **API tab** — shows inbound third-party API calls captured by `IntegrationDirectoryLogger`: Integration Dashboard — API tab showing LHRegisterBillAPI POST requests with SUCCESS and FAILED statuses **Webhook tab** — shows outbound webhook dispatch events, one row per trigger per integration: Integration Dashboard — Webhook tab showing 486 rows with QUEUED, SUCCESS, and FAILED statuses across Bill Generation HL7, Sample Receive, and Report Submit operations, with Retry and Resend action buttons **Errors tab** — filtered view showing only FAIL/QUEUED records across both API and Webhook types: Integration Dashboard — Errors tab with 17 errors, showing all Webhook rows as FAILED with Retry buttons Log status values [#log-status-values] | Status | Meaning | | --------- | ---------------------------------------------------------------------------------------------------- | | `QUEUED` | Pre-created — the HTTP call hasn't returned yet, or is pending for an async vendor (e.g., HumbleFax) | | `SUCCESS` | HTTP 2xx received and (for Mirth) `sent_to_mirth=True` confirmed | | `FAIL` | Non-2xx response or exception during dispatch | | *(blank)* | Suppressed — response code `209` or `#NOTFORDASHBOARD` in body; intentionally hidden | Key fields to inspect [#key-fields-to-inspect] | Field | What to look at | | ---------------------- | ------------------------------------------------------------------------------------------- | | `error_message` | First error description — often the third-party's error body | | `response_body` | Full JSON response from the third-party endpoint | | `response_status_code` | Raw HTTP status code returned | | `response_time` | Round-trip time in milliseconds | | `fusion_log` | Mirth-specific: `"Sent to Mirth"` or `"Failed at Livehealthapp"` / `"Failed at Crelio-app"` | | `retry_count` | Number of manual retry attempts made | | `auto_retry_count` | Number of auto-retry attempts made by the scheduler | | `triggered_by_py3` | `true` if the log was created from crelio-app (py3); affects retry routing | | `inserted_at` | UTC timestamp of when the log was first pre-created | | `activity_date` | Business date of the event that triggered the integration | *** Step 4: Retrying a Failed Integration [#step-4-retrying-a-failed-integration] Failed integrations can be retried manually from the dashboard or automatically by the scheduler. Manual retry flow [#manual-retry-flow] 1. **Locate the failed log** — find the `IntegrationDirectory` document with `status=FAIL` and note the `log_id` (`_id`). 2. **Trigger the retry** — click the **Retry** action in the dashboard UI. This calls `RetryIntegrationView` with the `log_id`, `retry_id`, and `lab_id`. 3. **System re-fetches live domain context** — the system reads `retry_context` (serialized at first execution) and calls the appropriate `_prepare_kwargs_*` function to **re-fetch live objects** from the database, ensuring the retry uses fresh data, not a stale snapshot. 4. **Integration re-dispatched** — the original trigger function is called with the freshly rebuilt arguments. The outcome is written to an `IntegrationRetryLog` record. 5. **Dashboard updated in real time** — a Pusher event (`UpdateRetryIntegrationStatus`) fires immediately, updating the retry count and status on the dashboard without a page refresh. Auto-retry [#auto-retry] The auto-retry scheduler runs periodically and checks for logs that: * Are in `FAIL` or `QUEUED` status * Were created within the past 24 hours (or within a configured date range) * Have an `auto_retry_count` below the allowed maximum * Match one of the auto-retry error patterns (e.g., `"No response"`, socket timeout errors, or Mirth `QUEUED` logs older than 10 minutes) When `triggered_by_py3=True` on the log, the auto-retry scheduler routes the retry call to the py3 endpoint (`/api-v3/integration/retry-integration/...`) instead of the py2 endpoint. This ensures Action IDs 60 and 61 (crelio-app) always retry through the correct service. *** Step 5: Troubleshooting a Specific Log [#step-5-troubleshooting-a-specific-log] Use the following decision flow when a log is in an unexpected state: Common failure patterns [#common-failure-patterns] | Symptom | Likely cause | Resolution | | ----------------------------------------- | --------------------------------------------------- | --------------------------------------------------------------- | | `No response` in `error_message` | Fusion task timed out before the endpoint responded | Retry — eligible for auto-retry | | `Session.connect: SocketTimeoutException` | Mirth TCP socket timed out | Auto-retry eligible; check Mirth host/port config | | `QUEUED` for > 10 min (non-Mirth) | Integration callback URL never hit | Check Fusion job logs; verify callback URL is reachable | | `status` is blank | Response suppressed (`209` or `#NOTFORDASHBOARD`) | Expected — log is for internal tracking only | | `retry_count` not incrementing | `retry_context` missing or corrupted | Check if original trigger function serialized context correctly | *** Configuration Reference [#configuration-reference] The fields set in `manageIntegrations` map directly to the database models: ``` developerAuthentication ├── authKey → token used by inbound API calls ├── labId → scopes the auth to a lab ├── orgId → (if isOrg=1) scopes to an org ├── developerId → the integration partner └── isIntegrationEnable labIntegration (one row per action trigger) ├── actionCategoryListId → the business event trigger (Action ID) ├── url → third-party endpoint ├── requestType → 0 = GET query-string, 1 = POST JSON body ├── integrationExtraDetails → JSON: Mirth host/port, vendor, x-api-key, etc. ├── authId → FK to developerAuthentication └── isDisable → soft-disable without deleting ``` *** Workflow Example: Bill Generation Webhook Trigger [#workflow-example-bill-generation-webhook-trigger] The following is the simplest end-to-end test you can run to verify that integration logging is working correctly. **Pre-requisite:** A `labIntegration` row exists with `actionCategoryListId_id` set to the Bill Generation action ID, and `isDisable=0`. **1. Create a bill** Generate a bill for any patient on the lab. This fires the domain event that calls `commomFunctionForIntegrationBillCategory`, which builds the payload and enqueues a Fusion task. **2. Run the Fusion scheduler** Fusion processes queued tasks asynchronously. On local, tasks won't execute until the scheduler is running. Start it with: ```bash python run_scheduler.py ``` On UAT/production, the Fusion scheduler runs continuously — no manual step needed. **3. Check the Integration Dashboard** Navigate to the **Integration Dashboard** for the lab. Switch to the **Webhook** tab and filter by the Bill Generation action category. You should see a new row with: * **Operation:** Bill Generation * **Status:** `SUCCESS` or `FAIL` (depending on whether the endpoint responded correctly) * **Date & Time:** the timestamp of the bill creation event Integration Dashboard — Webhook tab showing 486 rows with QUEUED, SUCCESS, and FAILED statuses across Bill Generation HL7, Sample Receive, and Report Submit operations, with Retry and Resend action buttons If the log is stuck in `QUEUED`, the Fusion scheduler is likely not running or the task is still in the queue. Run `python run_scheduler.py` and refresh the dashboard. # Design Decisions Design Decisions & Architecture [#design-decisions--architecture] *** Key Design Decisions & Constraints [#key-design-decisions--constraints] Preset System [#preset-system] Presets are "global template" processes with `lab_id=NULL` and `process_type=Preset`. They serve as starting points that can be cloned into lab-specific configs. Creation is restricted to a hardcoded list of allowed account manager emails. Communication Subprocess (Consent Only) [#communication-subprocess-consent-only] A special subprocess type (`for_communication=True`) is only valid on `consent` forms. Questions within it must use reserved question codes. When values are captured, the system automatically updates `PatientConsent` flags (email, SMS, WhatsApp, fax, mail) with TTL-based expiry. TTL / Expiry (Consent Only) [#ttl--expiry-consent-only] Only `Patient` process type on `consent` forms supports TTL. Expiry is calculated from `created_at` using `ttl` value and `ttl_mode` (days/months/years). Expired patient-level processes are excluded when creating consent for subsequent bills. Iteration Support [#iteration-support] Processes with `requires_pdf_iterations=True` support multiple iterations of responses. Each `QuestionValue` carries an `iteration` field. PDF generation can produce per-iteration documents. Offline/Mobile Sync [#offlinemobile-sync] The system supports offline mobile data collection with: * `BulkQuestionValueView` for batch sync * `BulkQuestionValueSyncFailureView` for logging sync failures to Elasticsearch * `GuestBulkQuestionValue` for unauthenticated (patient portal) submissions Legacy instance_id on Process [#legacy-instance_id-on-process] The `Process.instance_id` column is being deprecated in favor of the `ProcessLinkedInstances` table. A TODO comment in the codebase marks this for removal. Branch Scoping [#branch-scoping] For multi-branch labs: * Lab form history is filtered by `bill.branch_id` * MIS export uses Elasticsearch to resolve patient → branch mapping * Bill-level form details check branch authorization Email Notifications (Consent) [#email-notifications-consent] Patient consent emails are sent on: * `created` — new consent form * `resend` — manual resend * `linked_process` — new processes added * `revoked` — consent revoked Emails include a QR code (uploaded to S3) linking to the patient portal consent form. Email sending requires both `consent_management` feature flag and `patient_consent_email_communication` lab setting to be enabled. Additional Patient Info — Patient-Scoped (Not Bill-Scoped) [#additional-patient-info--patient-scoped-not-bill-scoped] Unlike `aoe` and `consent` which store captured data in `QuestionValue` (keyed by `bill_id`), `additional_patient_info` stores data in its own `AdditionalPatientInfo` table keyed by `patient_id`. This is because patient demographic/clinical info (gender identity, sexual orientation, etc.) persists across bills and should be captured once per patient, not once per billing event. Consequences: * **No `LabForm` runtime entity** — `additional_patient_info` does not create `LabForm` or `LabFormLinkedProcesses` records. Data flows directly from the UI to `AdditionalPatientInfo` via `UserDetails.handle_additional_patient_info()`. * **No iteration support** — since data is patient-level, there is no concept of multiple iterations. * **No form\_status** — there is no pending/completed/revoked lifecycle. Values are simply created or updated. * **Triggered on patient save** — data capture is integrated into the patient registration/update flow (via `UserDetails.after_save()`), not triggered by a separate form submission event. Additional Patient Info — Single Subprocess Constraint [#additional-patient-info--single-subprocess-constraint] Processes with `form_type="additional_patient_info"` are limited to **exactly one subprocess**. This is enforced in `Process.after_save()` — the `process_type` is also auto-set to `"Patient"` during validation, regardless of what the caller provides. Additional Patient Info — Page Settings & Defaults [#additional-patient-info--page-settings--defaults] `AdditionalPatientInfoSettings` provides a mapping layer between form processes and UI pages. Each page type (`appointment`, `registration`, `home_collection`, `cc_registration`) can have multiple processes assigned, but only **one default** per `(lab, page, custom_page)` combination. The default process cannot be disabled. Custom registration pages are supported via the `custom_page` FK, gated by the `allow_registration_custom_pages` session flag. Additional Patient Info — Access Control [#additional-patient-info--access-control] * **Feature flag:** `lab_forms_management` in session — required for all patient-facing and settings APIs * **User flag:** `user_additional_patient_info_history_flag` on `LabUser` — controls access to view history (mapped to `LAB_USER_REGISTRATION_ACCESS` permission group) * **Custom page access:** `allow_registration_custom_pages` session flag — required when `custom_page_id` is specified in settings *** Architectural Rationale [#architectural-rationale] One Engine, Infinite Form Types [#one-engine-infinite-form-types] The core motivation behind the Lab Forms architecture is **building a single, generalised form engine rather than coding each form type as a separate feature**. Consent, AOE and Additional Patient Info all look very different to end users, but structurally they are the same thing: a **hierarchy of Processes → SubProcesses → Questions → Attributes** with answers captured as **QuestionValues**. By abstracting this structure once, every new form type the business needs in the future is just a new `form_type` string — no new tables, no new views, no new serializers. The entire config + capture + history + export + print pipeline works automatically. Configuration Over Code [#configuration-over-code] The system follows a **configuration-driven** philosophy: * **Lab admins** define forms entirely through the UI — no developer involvement needed to add a new question, change field order, toggle mandatory/hidden, or set up skip logic. * **Question Attributes** (validators, prefilling, skip conditions, options, date constraints) are stored as key-value metadata, not as columns. This means new attribute types can be introduced without schema migrations. * **Presets** provide global templates so CrelioHealth can ship best-practice form configurations out of the box while still letting each lab customise. This dramatically reduces the cost of supporting 1000+ labs, each with different compliance and workflow requirements. Entity-Agnostic Linking [#entity-agnostic-linking] The `ProcessLinkedInstances` + `PROCESS_TYPE_MODEL_META_MAPPER` pattern decouples form configuration from the specific entities it applies to. A single Process doesn't "know" whether it's attached to a Test, Profile, Sample, Promotion, Store, or Bill — the mapper resolves the correct Django model, lab-ID key, and validation rules at runtime. This means: * New linkable entity types can be added by adding one entry to the mapper dict. * The same validation, caching, and serialization code path handles every entity type. * Business rules (e.g., "many processes per test in AOE") are expressed as simple conditionals, not duplicated codepaths. Separation of Config-Time and Runtime [#separation-of-config-time-and-runtime] The architecture cleanly separates two concerns: | Concern | Entities | Who uses it | | --------------------------------------------------- | ------------------------------------------------------------------------------------------ | ---------------------------------------- | | **Config-time** ("what should the form look like?") | Process, SubProcess, Question, QuestionAttribute, LinkedSubProcess, ProcessLinkedInstances | Lab admins, account managers | | **Runtime** ("what did the user actually fill in?") | LabForm, LabFormLinkedProcesses, QuestionValue, PatientConsent | Phlebotomists, patients, lab technicians | This separation means config changes don't corrupt existing captured data, and the same form definition can generate responses across thousands of bills without duplication of structure. Robustness Highlights [#robustness-highlights] * **Transactional integrity** — Form config creation/updates are wrapped in `@transaction.atomic` so a failure mid-way through subprocess or question creation rolls back cleanly, preventing orphaned records. * **Soft-delete everywhere** — Processes, SubProcesses, and Questions use `is_disabled` rather than hard deletion, preserving referential integrity with historical QuestionValues. * **Duplicate guards** — Every level (Process, SubProcess, Question) enforces uniqueness checks scoped appropriately (per lab, per form type, per subprocess) to prevent configuration errors. * **Skip logic & conditional flows** — The attribute-based skip system supports complex branching (jump to another process, abort, restart with value retention) without hard-coding any workflow paths. * **Offline-first design** — Bulk sync endpoints, failure logging to Elasticsearch, and iteration support ensure mobile data collection works reliably even with intermittent connectivity. * **Branch-aware multi-tenancy** — The same lab form infrastructure supports both single-location and multi-branch labs with proper data isolation via branch scoping. *** Reusability & Extensibility [#reusability--extensibility] | What you can do | How | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Add a completely new form type | Add a string to `lab_form_types` tuple — all APIs, caching, MIS export, and history views work immediately | | Add a new question field type | Add to `field_types` tuple + optionally add a renderer mapping in `FIELD_TYPE_MAPPER` and a validation function in `QuestionAttribute` | | Add a new linkable entity | Add one entry to `PROCESS_TYPE_MODEL_META_MAPPER` and optionally to `instance_id_dependent_process_types` | | Add a new question attribute | Add to the attribute type lists and optionally implement a field-specific validator | | Add new skip-to behaviours | Add to `ALLOWED_SKIP_TO_TYPES` list and handle in the renderer | | Add a new prefilling source | Add to `VALID_PREFILLING_SOURCES` and implement the field resolver | | Ship a default form for all labs | Create a Preset process (lab\_id=NULL, process\_type=Preset) | *** Why Not a Generic Form Builder Library? [#why-not-a-generic-form-builder-library] While third-party form builders exist, this custom implementation was chosen because: 1. **Deep integration with domain entities** — Forms need to link to tests, samples, profiles, bills, promotions, and home collections with lab-specific validation. No generic library handles this. 2. **LIMS-specific workflows** — Skip logic that aborts a process and moves to the next sample, or restarts with retained values, is domain-specific branching that generic form engines don't support. 3. **Communication consent lifecycle** — The tight coupling between consent form answers and `PatientConsent` flags with TTL-based expiry is a compliance requirement unique to healthcare. 4. **Offline-first mobile sync** — The bulk value capture with iteration support, failure recovery, and per-subprocess sequencing was designed specifically for phlebotomists working in the field. 5. **Performance at scale** — Redis-cached config, raw SQL for MIS export, and per-process hash caching are tuned for the platform's scale requirements — something a generic solution would require heavy customisation to achieve. 6. **Full audit trail** — Every config change and value capture is activity-logged with category-level granularity, which is critical for lab accreditation and compliance. # Overview Lab Forms [#lab-forms] Lab Forms are configurable forms that labs can customize to capture additional details during various workflows like registration, billing, bill update, and more. Related JIRA Tickets [#related-jira-tickets] * [EN-6270](https://crelio.atlassian.net/browse/EN-6270) (Consent Management) * [EN-6879](https://crelio.atlassian.net/browse/EN-6879) (AOE) * [EN-7491](https://crelio.atlassian.net/browse/EN-7491) (Additional Patient Info) Prerequisites [#prerequisites] The feature requires the `lab_forms_management` feature flag enabled for all lab form types except consent forms, which use the separate `consent_management` feature flag. What is it for? [#what-is-it-for] Lab Forms allow labs to configure custom forms for capturing extra information at different stages of their workflow: * **Registration** — Capture additional patient demographics or clinical information * **Billing** — Capture details related to the bill, specific tests, or the patient at the point of order entry * **Bill updates** — Update captured information when modifying existing bills * **Accessioning** — View and edit captured form values for a particular report as samples are received and accessioned * **Report entry** — View and edit captured form values for a particular report while entering or reviewing results * **Home collection** — Capture field-level details (collection notes, phlebotomist observations) during patient visits * **Patient consent** — Collect legally compliant consent from patients for audit and compliance purposes Form Types [#form-types] There are currently **3 types** of lab forms, each serving a different purpose: 1. AOE (Ask at Order Entry) [#1-aoe-ask-at-order-entry] * **Purpose:** Captures extra details during billing related to the bill, tests in the bill, or the patient * **Rendered:** On the Billing Modal as a "Complete AOE" button * **Behavior:** * Appears beside "Confirm & Bill" button during billing * Billing cannot be completed until all mandatory AOE questions are answered * For **Test-level AOE**: Related forms are added as tests are selected, with an "AOE Required" tag beside the test name * For **Profile-level AOE**: AOE can be captured once for the profile, but values are saved per test report; if only some tests in the profile have AOE, values are saved only for those tests * For **Promotion-level AOE**: Applicable AOE is resolved from mapped tests and values are saved against each applicable test report * For **Bill-level AOE**: A single form applies to the entire bill * **Use cases:** Clinical questions, sample collection details, test-specific requirements 2. Consent (Patient Consent) [#2-consent-patient-consent] * **Purpose:** Captures patient consent for legal compliance and audit purposes * **Rendered:** In the patient portal after billing * **Behavior:** * Sent to patients via email with a QR code link * Tracks communication preferences (email, SMS, WhatsApp, fax, mail) * If the lab has configured a communication section, all outbound communication (bill confirmation, report ready, etc.) is hard-stopped for that patient until consent is explicitly provided * Supports expiry duration for consent validity * Can be revoked by the lab * **Use cases:** HIPAA consent, report delivery preferences, communications opt-in/out 3. Additional Patient Info [#3-additional-patient-info] * **Purpose:** Captures additional patient details during registration * **Rendered:** Beside the registration form (in place of COVID history section) * **Behavior:** * Stored at patient level (not bill level) — persists across all patient bills * Can be configured per page type (appointment, registration, home collection, CC registration) * **Use cases:** Gender identity, sexual orientation, patient demographics for lab reporting (e.g., Labcorp integration) Form Structure [#form-structure] A lab form follows a simple hierarchical structure: * **Form** — The main container (e.g., "Patient History", "Sample Collection Details") * **Sections** — Subdivisions within a form (e.g., "Personal Details", "Medical History") * **Questions** — Individual fields within each section (e.g., "Date of Birth", "Preferred Contact Method") This mirrors traditional forms where you have a main form divided into logical sections, each containing specific questions. Key Features [#key-features] * **Fully configurable** — No code changes needed to add/modify forms * **20+ field types** — Text, number, date, select, checkbox, radio, signature, file upload, camera, barcode, address, and more * **Skip logic** — Conditional branching based on answers * **Prefilling** — Auto-populate from existing patient/bill/test data * **Validation** — Min/max, required, format validation per field type * **File uploads** — Signature, images, documents, camera capture * **Multi-level forms** — Bill-level, test-level, profile-level, patient-level * **MIS export** — Export captured data for reporting * **Audit trail** — Full activity logging for config changes and value updates # Workflow Guide Workflow Guide [#workflow-guide] This page walks through the end-to-end workflow for Lab Forms — from configuring a form, to capturing responses, to viewing the data. *** Configuring a Form [#configuring-a-form] Navigate to **Admin → Lab Forms Management** From the configuration list header, you can use **Add Via Template** to either start with an existing template or create a form from scratch. 1. Create the Form [#1-create-the-form] * Select the **form type**: AOE, Consent, or Additional Patient Info * Select what the form **applies to**: the entire bill, specific tests, specific profiles, or the patient * Allowed options vary by form type * Add basic details: name, icon, description * For test/profile/promotion level forms: link to the specific tests or profiles this form should apply to * For Promotion & Store can also map from CRM * For **Promotion** mappings: forms can be mapped/unmapped from **CRM → Promotions → \ → Configuration**, and mapping is not restricted to only Promotion-type forms * For **Store** mappings: manage from **CRM → Store → Configurations → Checkout**; only Bill and Store type forms are allowed * At booking time, mapped forms for the selected Promotion/Store are rendered in the flow 2. Add Sections [#2-add-sections] * Create sections to organize questions into logical groups (e.g., "Personal Details", "Medical History") * Configure each section: * Name and icon * Mandatory, optional, or hidden * Order sections by drag or sequence number The **Communication** section is available only for the **Consent** form type. It includes predefined questions for communication preferences such as **SMS**, **WhatsApp**, **email**, **fax**, and other supported channels. If a lab configures any consent form with a Communication section, patient communication hard stops unless the patient has explicitly provided consent for the specific communication type. 3. Add Questions [#3-add-questions] Choose from **20+ field types** for each question: | Category | Field Types | | --------- | ------------------------------------------------------------- | | Text | text, textarea, email, phone number | | Number | number, decimal | | Date/Time | date, time, date+time | | Choices | dropdown, radio button, radio group, checkbox, checkbox group | | Files | signature, image, file upload, camera capture | | Special | barcode, address | Customize each question with: * **Options** — for dropdown, radio, and checkbox types * **Validation** — date format, min/max value, required/optional * **Skip logic** — show/hide subsequent questions or sections based on the answer * **Prefilling** — auto-populate from existing patient, bill, or test data * **Visibility** — mandatory, optional, or hidden 4. Save and Enable [#4-save-and-enable] Once saved, the form becomes active and available in the relevant workflow. Bulk Mapping for AOE and Consent [#bulk-mapping-for-aoe-and-consent] From the configuration list, use the **Map Instance(s)** button to map multiple instances to multiple forms in one action. This is supported for **AOE** and **Consent** forms. It is useful when the same set of instances should be linked to several forms, and avoids opening each form configuration one by one. Example: * If **CBC** and **Ammonia** tests need to be mapped to **4 AOE forms**, this can be done together from **Map Instance(s)** instead of opening each AOE configuration separately and mapping them one at a time. *** Using the Form [#using-the-form] AOE — During Billing [#aoe--during-billing] 1. Add tests/services to the bill 2. If AOE forms are configured for the bill or selected tests: * A **"Complete AOE"** button appears beside "Confirm & Bill" * Test-level AOE shows an **"AOE Required"** tag beside the test name 3. Click "Complete AOE" to open and fill the form 4. All mandatory questions must be answered before proceeding 5. Once complete, the "Confirm & Bill" button becomes enabled 6. Answers are saved upon billing confirmation Consent — After Billing [#consent--after-billing] 1. Billing is completed — the **Bill Confirmation modal** appears 2. The modal footer shows a **"Show Patient Consent"** button 3. Clicking it opens the **Consent modal**, which contains: * A **QR code** linking the patient directly to their consent form in the patient portal * The ability to **map or unmap consent forms** to this bill/patient * Then we can click on view form and it redirects to the consent form for filling. 4. An email with the QR code link is also sent to the patient (if patient email exists) 5. The patient scans the QR code or opens the link and completes the consent form in the patient portal 6. Answers are saved and communication preferences (email, SMS, WhatsApp, fax, mail) are updated automatically Additional Patient Info — During Registration [#additional-patient-info--during-registration] Additional Patient Info forms are mapped to specific registration pages. The mapping is configured in **Registration Settings → Additional Patient Info** tab. **Supported page types:** | Page Type | Where it appears | | ----------------- | ---------------------------------------- | | `registration` | Standard patient registration | | `appointment` | Appointment booking flow | | `home_collection` | Home collection patient form | | `cc_registration` | CC (custom/corporate) registration pages | Each page type can have multiple forms assigned, with one marked as the **default**. The default form cannot be disabled. **Capturing the form:** 1. Open patient registration (or appointment / home collection / CC registration) 2. The mapped Additional Patient Info form appears beside the standard registration fields 3. Fill the form along with standard patient details 4. On save, answers are stored at the **patient level** — they persist across all bills for that patient *** Viewing Captured Data [#viewing-captured-data] Lab Form History [#lab-form-history] Navigate to **Registration → Lab Form History**: * Filter by date range and form type * View complete form responses with all answered questions Other Locations [#other-locations] | Page | What's Visible | | ------------------ | ------------------------------------------------------------ | | **Order Update** | AOE values for the bill | | **Accession** | AOE and Consent values while accessioning samples | | **Waiting List** | AOE and Consent values for pending orders | | **Report Entry** | Test-level AOE values and Consent values during result entry | | **Patient Update** | Additional Patient Info values | # Backend Overview [#overview] Machine Flags provide a mechanism to store and display device-generated flags received from interfacing analyzers inside LIMS workflows. These flags are primarily used in: * Report Entry * Overview Reports * Device Results Validation Machine Flags are not rendered in PDF preview or printable report output. *** 1. Configuring Machine Flags Feature [#1-configuring-machine-flags-feature] Overview [#overview-1] Machine Flags support is configured at the Test/Report level. When the user enables or disables the `Enable Device Flags (Store results in MongoDB)` checkbox from the frontend and saves the test configuration, the following API is triggered. *** API [#api] `POST reporting/report-format//update/` URL Pattern [#url-pattern] `url(r"(?P\d+)/update/$", ReportFormatView.as_view(is_new=False))` *** Important Payload Field [#important-payload-field] The frontend sends the following field inside `testData.test.enable_device_flags` Example: ```json { "testData": { "test": { "enable_device_flags": true } } } ``` *** Backend Flow [#backend-flow] Inside both: * `LabTest.create` * `LabTest.update` the following logic is executed: ```python enable_device_flags = test_info.pop("enable_device_flags", False) or False ``` If the test type is `Normal` then: ```python instance.document_report = int(bool(enable_device_flags)) ``` is set before saving the test. *** Important Note [#important-note] `enable_device_flags` is **not stored directly** in any database column. Instead, it is only used to decide whether the report should behave as a `Document Report` which enables storage of: * Machine Flags * Parameter Flags * Device Extra Data inside DocumentDB-backed report values. *** Request Payload Structure [#request-payload-structure] The payload contains multiple sections: ```json { "testData": {}, "reportData": [], "ageRangeData": [], "calculations": [], "addTestToChildLabsData": {} } ``` *** Relevant Request Example [#relevant-request-example] ```json { "testData": { "addFlag": 0, "testID": 10268009, "test": { "testName": "CBC", "test_type": "Normal", "enable_device_flags": true, "show_device_extra_data_in_pdf": false } } } ``` *** 2. Sending Data from Interfacing Device to LIMS [#2-sending-data-from-interfacing-device-to-lims] Overview [#overview-2] When a device sends raw analyzer output, the interfacing layer parses the raw string into a structured payload and sends it to the LIMS backend. Each mapped test inside LIMS is associated with a dedicated processing function responsible for filling report values. Machine Flags are currently supported only in `PartialFill` mapping function. *** API [#api-1] ```http POST dataPartialFromDevice/ ``` URL Pattern [#url-pattern-1] ```python url(r"^dataPartialFromDevice/$", getDeviceDataForPending) ``` *** Request Payload Structure [#request-payload-structure-1] ```json { "labId": 11957, "sampleId": "000114626", "deviceAuth": "f94032be-5e1f-47fd-96a1-92a66c6014c3", "test_flags": [ "DUMMY TEST FLAG1", "FLAG2" ], "data": { "values": [ { "testName": "hmg", "value": 110.12, "param_flags": [ "P1", "H~N" ] } ] } } ``` *** Payload Fields [#payload-fields] Root Level Fields [#root-level-fields] | Field | Description | | ------------- | --------------------------------------- | | `labId` | Lab identifier | | `sampleId` | Sample accession ID | | `deviceAuth` | Device authentication token | | `test_flags` | Report-level machine flags | | `data.values` | Parameter values received from analyzer | *** Parameter Structure [#parameter-structure] Each parameter inside `data.values` may contain: ```json { "testName": "hmg", "value": 110.12, "param_flags": [ "P1", "H~N" ] } ``` *** Backend Processing Flow [#backend-processing-flow] Step 1 - Validate Device [#step-1---validate-device] The API validates: * Device authentication * Lab ID * Sample ID * Payload structure *** Step 2 - Find Device Mapping [#step-2---find-device-mapping] Mapped tests are fetched using `deviceTestMapping.objects.filter(...)` Each mapping contains a processing function name. Example `PartialFill` *** Step 3 - Invoke Mapping Function [#step-3---invoke-mapping-function] The mapped function is dynamically executed: ```python methodCall = getattr(devices.functions, test.functionName, testValidateFlag) ``` *** PartialFill Function [#partialfill-function] Machine Flags are processed inside `def PartialFill(...)` *** Machine Flag Handling [#machine-flag-handling] Report-Level Flags [#report-level-flags] Report-level flags are received from `test_flags` and stored in report values: ```python update_values["test_flags"] = test_flags ``` *** Parameter-Level Flags [#parameter-level-flags] Parameter flags are received from `param_flags` and stored as: ```python update_values["param_flags"] = param_flags ``` *** Example Stored Structure [#example-stored-structure] ```json { "value": 110.12, "test_flags": [ "FLAG1", "FLAG2" ], "param_flags": [ "P1", "H~N" ] } ``` *** 3. Fetching Data for Report Entry Modal [#3-fetching-data-for-report-entry-modal] Overview [#overview-3] When the user opens a report from: `Pending Reports` the frontend fetches report values and metadata using: ```http GET getReportFormatForTestEntry/ ``` This response also contains: * Machine Flags * Parameter Flags * Device Name for each report value. *** API [#api-2] ```http GET getReportFormatForTestEntry/ ``` URL Pattern [#url-pattern-2] ```python url(r"^getReportFormatForTestEntry/$", getReportFormatForTestEntry) ``` *** Request Payload [#request-payload] ```json { "labReportId": 123, "isSigned": 0 } ``` *** Machine Flag Structure in Response [#machine-flag-structure-in-response] Inside `value[]` each parameter may contain: ```json { "index": 4, "value": 110.12, "device_name": "Alinity", "test_flags": [ "DUMMY TEST FLAG1", "FLAG2" ], "param_flags": [ "P1", "H~N" ] } ``` *** Important Response Fields [#important-response-fields] | Field | Description | | -------------- | -------------------------- | | `value` | Parameter values | | `test_flags` | Report-level machine flags | | `param_flags` | Parameter-level flags | | `device_name` | Analyzer name | | `reportFormat` | Report format metadata | *** Response Structure [#response-structure] ```json { "value": [], "reportFormat": [], "calculationList": [], "qcArray": [], "historicalValuesByIndex": {}, "labReportSigningMeta": {} } ``` *** 4. Fetching Data for Overview Section [#4-fetching-data-for-overview-section] Overview [#overview-4] When the user opens the `Patient Overview` screen, the frontend fetches all patient reports along with associated report values. Machine Flags are included in the same API response. *** API [#api-3] ```http GET api-v3/patient//all-reports ``` Backend Entry Point [#backend-entry-point] ```python PatientOverview.fetch_patient_all_reports() ``` *** DocumentDB Report Values [#documentdb-report-values] For Document Reports, report values are fetched from: ```python get_docdb_report_values() ``` *** Machine Flags Projection [#machine-flags-projection] The aggregation pipeline includes: ```python "test_flags": 1, "param_flags": 1, ``` *** Returned Structure [#returned-structure] Machine flags are returned inside `report_result_mapper` Example: ```json { "index": 4, "value": 110.12, "test_flags": [ "FLAG1", "FLAG2" ], "param_flags": [ "P1", "H~N" ] } ``` *** Main Response Structure [#main-response-structure] ```json { "reports": [], "report_result_mapper": {}, "report_format_mapper": {}, "calculations_mapper": {}, "qc_values": {}, "patient": {} } ``` *** 5. Fetching Data for Device Results Validation [#5-fetching-data-for-device-results-validation] Overview [#overview-5] Machine Flags are also displayed inside `Device Results Validation` for pathology devices. This screen displays un-reviewed interfaced device results along with associated report information. *** API [#api-4] ```http GET /api-v3/interfacing/device-results-validation/ ``` Query Parameters [#query-parameters] | Parameter | Description | | ----------- | ----------------- | | `startDate` | Start datetime | | `endDate` | End datetime | | `deviceId` | Device identifier | Example: ```http /api-v3/interfacing/device-results-validation/3?startDate=2026-05-25T18:30:00.000Z&endDate=2026-05-26T18:29:59.000Z ``` *** Backend Entry Point [#backend-entry-point-1] ```python FetchDeviceResultsForValidation.get() ``` *** Main Processing Method [#main-processing-method] ```python DeviceResultsValidation.get_device_path_results() ``` *** Machine Flag Formatting [#machine-flag-formatting] During response preparation: ```python device_result["value"] = [ { "value": value.get("value"), "test_flags": value.get("test_flags", []), "param_flags": value.get("param_flags", []), } ] ``` *** Example Response Value [#example-response-value] ```json { "value": [ { "value": 110.12, "test_flags": [ "FLAG1", "FLAG2" ], "param_flags": [ "P1", "H~N" ] } ] } ``` *** Related Report Information [#related-report-information] Each result also contains: ```json { "related_reports": [ { "report_name": "CBC", "parameter_index": 4, "linearity_ranges": {}, "parameter_historical_values": [] } ] } ``` *** Final Response Structure [#final-response-structure] ```json { "results": [], "device_id": 3, "device_name": "Alinity", "bill_and_user_details_by_sample": {} } ``` *** Summary [#summary] Machine Flags Lifecycle [#machine-flags-lifecycle] Configuration [#configuration] `enable_device_flags` determines whether the report behaves as a Document Report. *** Ingestion [#ingestion] Interfacing devices send: * `test_flags` * `param_flags` through `dataPartialFromDevice/` *** Storage [#storage] Flags are stored inside `DocumentDB Report Values` through the `PartialFill` function. *** Retrieval APIs [#retrieval-apis] Machine Flags are available in: | Module | API | | ----------------- | --------------------------------------------------------- | | Report Entry | `getReportFormatForTestEntry/` | | Overview | `api-v3/patient//all-reports` | | Device Validation | `api-v3/interfacing/device-results-validation/` | *** PDF Behaviour [#pdf-behaviour] Machine Flags are intentionally excluded from: * PDF Preview * Printable Reports ``` ``` # Design Decisions Design Decisions [#design-decisions] The Machine Flags feature was designed to support device-generated metadata alongside pathology parameter results across the complete interfacing pipeline. The implementation spans: * Interfacing parsers * LIMS APIs * MongoDB-backed report storage * Report rendering * Device validation workflows The architecture was intentionally designed to support highly variable device payloads while keeping frontend rendering reusable and scalable. *** Device-Agnostic Parser Architecture [#device-agnostic-parser-architecture] Machine Flags are parsed inside device-specific parser files located in the interfacing application. Each device has its own parser implementation. Examples: * Alinity * Siemens * Sysmex * Erba *** Why [#why] Different lab devices send: * Different protocols * Different HL7 structures * Different delimiters * Different flag formats * Different OBX segment mappings A generic parser would become extremely complex and error-prone. *** Design Choice [#design-choice] Use `One Device → One Dedicated Parser` Each parser converts raw device output into a normalized payload structure. *** Benefit [#benefit] * Easy onboarding of new devices * Isolated device-specific logic * Easier debugging * Independent parser evolution *** Trade-off [#trade-off] * More parser files to maintain * Device-specific testing required *** Normalized Payload Structure [#normalized-payload-structure] All device parsers convert raw device output into a standardized payload. *** Standard Payload [#standard-payload] ```json { "test_flags": [], "data": { "values": [ { "testName": "", "value": "", "param_flags": [] } ] } } ``` *** Why [#why-1] Lab devices generate highly inconsistent outputs. The normalized structure provides: * Stable backend contracts * Reusable frontend rendering * Device-independent APIs *** Separation of Test Flags and Parameter Flags [#separation-of-test-flags-and-parameter-flags] Machine Flags were intentionally divided into two independent categories. *** Test-Level Flags [#test-level-flags] Stored in `test_flags` Examples: * Hemolyzed * Review Needed * Clotted Sample These represent sample-level or report-level observations. *** Parameter-Level Flags [#parameter-level-flags] Stored in `param_flags` Examples: * H * L * A++ * Critical These represent parameter-specific abnormalities. *** Why [#why-2] Some devices send: * Only test flags * Only parameter flags * Partial parameter flags * Both types simultaneously A unified structure would create ambiguity during rendering. *** Benefit [#benefit-1] * Cleaner rendering logic * Flexible device compatibility * Better UI separation *** Partial Flag Support [#partial-flag-support] The system was designed to support incomplete Machine Flag payloads. *** Supported Scenarios [#supported-scenarios] | Scenario | Supported | | ----------------------- | --------- | | Only test flags | ✅ | | Only parameter flags | ✅ | | Partial parameter flags | ✅ | | Mixed flags | ✅ | | No flags | ✅ | *** Why [#why-3] Different devices behave differently. Some devices: * Do not send flags at all * Send flags only for abnormal parameters * Send sample-level warnings only The architecture needed to tolerate incomplete payloads. *** MongoDB-Based Report Storage [#mongodb-based-report-storage] Machine Flag-enabled reports are stored in MongoDB-backed report documents. *** Controlled By [#controlled-by] `enableDeviceFlags` configuration. *** Why MongoDB? [#why-mongodb] Traditional relational storage becomes inefficient for: * Highly dynamic parameter structures * Large interfaced reports * Device metadata * Nested flag arrays MongoDB provides: * Flexible schemas * Easier nested storage * Faster document retrieval *** Design Choice [#design-choice-1] Each parameter value is stored as an independent document-like object. Example: ```json { "value": 110.12, "param_flags": ["P1", "H~N"], "test_flags": ["Review Needed"] } ``` *** Benefit [#benefit-2] * Flexible report structure * Easier interfacing support * Simplified frontend rendering * Better extensibility *** Trade-off [#trade-off-1] * Additional MongoDB dependency * Dual storage architecture *** Repeated Test Flags Across Parameters [#repeated-test-flags-across-parameters] `test_flags` are repeated across report parameter objects. Example: ```json { "test_flags": ["Review Needed"] } ``` exists on multiple parameter entries. *** Why [#why-4] This avoids: * Separate lookup structures * Additional joins * Complex frontend state synchronization Frontend components can retrieve flags directly from any report value. *** Trade-off [#trade-off-2] * Small amount of duplicated data *** Benefit [#benefit-3] * Simpler rendering logic * Faster UI access * Reduced transformation overhead *** Reusable Frontend Rendering Pipeline [#reusable-frontend-rendering-pipeline] Machine Flags were integrated into the existing report rendering system instead of creating separate rendering modules. *** Why [#why-5] The report rendering system already handled: * Parameter rendering * PDF rendering * Report editing * Read-only preview Adding Machine Flags into the same pipeline minimized duplication. *** Design Choice [#design-choice-2] Reuse existing components: * EditReportView * ReadReportView * TestNameColumn * ReportRangeInput *** Benefit [#benefit-4] * Shared UI logic * Consistent rendering * Easier maintenance * Faster implementation *** Shared Rendering Across Multiple Screens [#shared-rendering-across-multiple-screens] Machine Flags use the same rendering architecture in: | Screen | Shared Components | | ----------------- | ------------------------------- | | Report Entry | EditReportView / ReadReportView | | Overview | EditReportView / ReadReportView | | PDF Preview | ReadReportView | | Device Validation | Shared renderers | *** Why [#why-6] Users should see identical Machine Flag behaviour across all report workflows. *** PDF Rendering Strategy [#pdf-rendering-strategy] Machine Flags are rendered only inside interactive LIMS UI workflows. Examples: * Report Entry * Overview Reports * Device Results Validation *** PDF Preview Behaviour [#pdf-preview-behaviour] Machine Flags are intentionally hidden when the user switches to `PDF Preview Mode` *** Why [#why-7] Machine Flags are considered operational review metadata intended primarily for internal validation workflows. The PDF preview is treated as the final printable clinical report format. To keep the printable report cleaner and clinically focused: * Machine Flags are excluded from PDF rendering * Internal validation metadata is hidden * Only finalized report content is displayed *** Why [#why-8] The final printable clinical report is treated as the authoritative reviewable format. Showing flags in PDF mode ensures: * Clinical visibility * Print consistency * Review accuracy *** Design Choice [#design-choice-3] Machine Flags are rendered only in frontend interactive workflows where users actively review interfaced results. They are not included in: * PDF preview rendering * Printable report output *** Benefit [#benefit-5] * Cleaner printable reports * Reduced PDF clutter * Better patient-facing readability * Separation between operational metadata and final report content *** Trade-off [#trade-off-3] Users cannot view Machine Flags inside PDF preview mode. To review Machine Flags, users must use: * Report Entry * Overview Reports * Device Results Validation *** Device Result Validation Integration [#device-result-validation-integration] Machine Flags were integrated into the Device Results Validation screen. *** Why [#why-9] Users validating interfaced results need visibility into: * Sample abnormalities * Parameter warnings * Device-generated alerts before report approval. *** Design Choice [#design-choice-4] Render: * Test flags in grouped rows * Parameter flags beside parameter names *** Dynamic Rendering Instead of Hardcoded Flags [#dynamic-rendering-instead-of-hardcoded-flags] Frontend rendering treats Machine Flags as dynamic strings. Example: ```json ["H", "A++", "Review Needed"] ``` *** Why [#why-10] Different devices generate different flag formats. Hardcoding `if flag === "H"` would make onboarding new devices difficult. *** Benefit [#benefit-6] * Fully device-agnostic rendering * No frontend dependency on device types * Future-proof implementation *** Automated Result Awareness [#automated-result-awareness] Machine Flag-enabled results include `automatedValue = 1` *** Why [#why-11] Frontend workflows need to distinguish: * Device-generated results * Manual entries This supports: * Validation workflows * Review flows * Approval logic *** MongoDB Retrieval Strategy [#mongodb-retrieval-strategy] Machine Flags are fetched together with report values. *** Why [#why-12] Avoid: * Separate flag APIs * Multiple frontend fetches * Additional synchronization logic *** Benefit [#benefit-7] * Single-source report rendering * Reduced API calls * Faster report loading *** Interfacing-Centric Architecture [#interfacing-centric-architecture] Machine Flags processing primarily happens in the interfacing layer. *** Why [#why-13] The LIMS backend should remain device-agnostic. The interfacing application acts as the translation layer between: ```text Device Protocols ↓ Normalized Payload ↓ LIMS APIs ``` *** Benefit [#benefit-8] * Cleaner backend APIs * Reduced backend parser complexity * Easier device onboarding *** Extensibility Considerations [#extensibility-considerations] The architecture was designed to support future additions such as: * Device extra metadata * QC markers * Delta-check warnings * AI-generated interpretations * Instrument comments * Device images/graphs without major schema redesign. *** Failure-Tolerant Rendering [#failure-tolerant-rendering] Frontend rendering was intentionally built to tolerate missing data. *** Examples [#examples] Supported safely: ```json "param_flags": [] ``` ```json "test_flags": null ``` ```json "param_flags": undefined ``` *** Why [#why-14] Device payloads are often inconsistent across vendors. Strict rendering assumptions would cause runtime failures. *** Workflow Architecture [#workflow-architecture] ```text Patient Sample Collected │ ▼ Sample Processed in Device │ ▼ Device Generates Raw Output String │ ▼ Interfacing Application Receives Raw Data │ ▼ Device-Specific Parser Selected │ ▼ Parser Extracts: ├── Parameter Values ├── Test Flags └── Parameter Flags │ ▼ Normalized Payload Generated │ ▼ Payload Sent to LIMS API │ ▼ Data Partial API Processes Payload │ ▼ Results + Flags Stored in MongoDB │ ▼ Report Values Retrieved by APIs ├── getReportFormatForTestEntry() ├── getPatientAllReports() └── fetchDeviceResultsAPI() │ ▼ Frontend Rendering Pipeline ├── Report Entry ├── Overview ├── PDF Preview └── Device Validation │ ▼ Machine Flags Displayed to User ``` *** Frontend Rendering Flow [#frontend-rendering-flow] ```text MongoDB Report Values │ ▼ API Response │ ▼ Report Value Objects ├── test_flags └── param_flags │ ▼ Frontend Renderers ├── testFlagsRenderer ├── TestNameColumn ├── groupRowRenderer └── parameterNameCellRenderer │ ▼ Machine Flags Visible in UI ``` *** Summary [#summary] The Machine Flags architecture was designed around: * Device variability * Flexible payload handling * MongoDB-backed report storage * Reusable frontend rendering * Device-agnostic parsing * Failure-tolerant rendering * Unified report workflows The implementation prioritizes: * Scalability * Extensibility * Device compatibility * Reusable rendering * Clinical visibility * Simplified frontend integration # Frontend Frontend Architecture [#frontend-architecture] The frontend implementation for Machine Flags was divided into two major areas: 1. Enabling Machine Flags during test configuration 2. Viewing Machine Flags in report and validation screens The feature was implemented across multiple React component layers inside the LIMS frontend application. *** Frontend Areas Covered [#frontend-areas-covered] | Area | Purpose | | ------------------------- | ----------------------------------------------------- | | Report Configuration | Enable Machine Flags for pathology tests | | Report Entry Modal | Display test and parameter flags during report review | | Overview Section | Show Machine Flags in patient overview reports | | Device Results Validation | Show flags during device validation workflow | *** 1. Enabling Machine Flags [#1-enabling-machine-flags] Machine Flags are enabled during pathology test configuration. *** Route: /admin/test-list/ [#route-admintest-listtestid] File Reference: src/components/LabAdmin/AddEditReport/components/ReportConfigurations/index.tsx [#file-reference-srccomponentslabadminaddeditreportcomponentsreportconfigurationsindextsx] Component Hierarchy: [#component-hierarchy] ```text ReportConfigurations ↓ TestInformation ↓ ConfigCheckBox ``` *** ReportConfigurations Component [#reportconfigurations-component] Responsibility [#responsibility] Acts as the root container for: * Test configuration * Report parameter configuration * Report templating * Parent mapping * Feature flags This component initializes: * Redux form state * Generic state * Report settings * Test configuration tabs *** Important Responsibility for Machine Flags [#important-responsibility-for-machine-flags] The component loads and manages `testInformationForm` state which contains `enableDeviceFlags` configuration. *** TestInformation Component [#testinformation-component] File Reference: [#file-reference] `src/components/LabAdmin/AddEditReport/components/ReportConfigurations/TestInformation/index.tsx` Responsibility [#responsibility-1] The `TestInformation` component handles: * Test metadata * Test configuration * Pricing * Sample settings * Configuration checkboxes This component renders the reusable `` component. *** Machine Flags Integration [#machine-flags-integration] The Machine Flags feature is exposed through `enableDeviceFlags` checkbox configuration. The checkbox is rendered only for eligible pathology tests. *** ConfigCheckBox Component [#configcheckbox-component] File Reference: [#file-reference-1] `src/components/LabAdmin/AddEditReport/reusable/ConfigCheckBox/index.tsx` Responsibility [#responsibility-2] The `ConfigCheckBox` component renders all configurable test-level feature flags. Examples: * Auto Approval * Auto Dispatch * Outsourced Test * Cell Counter * Enable Device Flags *** Machine Flags Checkbox [#machine-flags-checkbox] The feature was added as: ```ts { label: i18n.t("Enable Device Flags (Store results in MongoDB)"), id: "enableDeviceFlags", isChecked: 0, isDisabled: false, } ``` *** Visibility Logic [#visibility-logic] The frontend only shows the checkbox when: `document_db_enabled === true` AND `test_type === "normal"` *** Filtering Logic [#filtering-logic] ```ts if (!document_db_enabled || testInformationForm?.test_type?.toLowerCase() != "normal") { newList = newList.filter((item: JsonObject) => item.id !== "enableDeviceFlags"); } ``` *** Automatic State Handling [#automatic-state-handling] If the test is already a document report `isDocumentReport === true` then frontend automatically enables `enableDeviceFlags = 1` *** Logic [#logic] ```ts if ( ins?.id == "enableDeviceFlags" && !("enableDeviceFlags" in testInformationForm) && !!isDocumentReport ) { updateTestInformationState("enableDeviceFlags", 1); } ``` *** Save Flow [#save-flow] During report save/update `addNewReport()` includes `enableDeviceFlags` inside the payload sent to backend APIs. This persists the Machine Flag configuration for the test. *** 2. Viewing Machine Flags [#2-viewing-machine-flags] *** A. Report Entry Modal [#a-report-entry-modal] Machine Flags are displayed inside the Report Entry workflow while reviewing or editing pathology reports. This is the primary screen where users interact with interfaced report values and validate machine-generated results before approval or submission. *** Route [#route] `/operation/patients-waiting-list/all-tests/` Component Hierarchy [#component-hierarchy-1] ```text PatientWiseContainer ↓ PatientWiseReportListView ↓ ParticularPatientsTestListView ↓ PatientReport ``` *** File Reference: [#file-reference-2] `src/components/reusable/Modals/Report/index.tsx` *** Data Fetching Flow [#data-fetching-flow] The Machine Flags data is fetched during component initialization. Inside `PatientReport`: ```ts useEffect(() => { fetchData(); }, [selectedReport]); ``` The `fetchData()` method internally calls `fetchReportsDataAPI()` which then calls `fetchReportsData()` *** getReportFormatForTestEntry API [#getreportformatfortestentry-api] This API is the primary source for Machine Flags data in Report Entry. It returns: * Report values * Report formats * Historical values * Calculations * QC values * Test-level flags * Parameter-level flags *** API Signature [#api-signature] ```ts const getReportFormatForTestEntry = async ( labReportId: number, isSigned: number, docLogin: number = 1, formatId: number = 0 ) ``` *** API Response Structure [#api-response-structure] The API response contains a `value: []` array. Each object inside this array represents a single report parameter value. *** Machine Flags Data Structure [#machine-flags-data-structure] Test-Level Flags [#test-level-flags] Sample or report-level Machine Flags are stored in `test_flags` Example: ```json "test_flags": [ "DUMMY TEST FLAG1", "FLAG2" ] ``` These flags are repeated across report values for easier frontend rendering. *** Parameter-Level Flags [#parameter-level-flags] Parameter-specific flags are stored in `param_flags` Example: ```json "param_flags": [ "P1", "H~N" ] ``` *** Example Parameter Value Object [#example-parameter-value-object] ```json { "index": 4, "value": 110.12, "device_name": "Alinity", "param_flags": ["P1", "H~N"], "test_flags": ["DUMMY TEST FLAG1", "FLAG2"], "automatedValue": 1 } ``` *** Edit Mode Rendering [#edit-mode-rendering] Machine Flags are rendered while editing reports. Test Flags Rendering Flow [#test-flags-rendering-flow] ```text EditReportView ↓ testFlagsRenderer ``` *** Parameter Flags Rendering Flow [#parameter-flags-rendering-flow] ```text EditReportView ↓ renderReportCell ↓ EditReportCell ↓ ReportRangeInput ↓ TestNameColumn ``` *** EditReportView Component [#editreportview-component] Responsibility [#responsibility-3] The `EditReportView` component handles editable pathology report rendering. It processes: * Report values * Historical values * QC values * Machine Flags * Parameter rendering *** File Reference [#file-reference-3] `src/components/reusable/Modals/Report/EditReportView/index.tsx` *** Test Flags Renderer [#test-flags-renderer] Test-level Machine Flags are rendered through `testFlagsRenderer()` These flags are displayed near the report header section. Examples: * Hemolyzed * Clotted Sample * Recheck Required *** Parameter Flags Rendering [#parameter-flags-rendering] Parameter-level flags are rendered beside parameter names or values. Examples: * H * L * P1 * A++ * Critical The rendering pipeline flows through reusable report cell components. *** Read Mode Rendering [#read-mode-rendering] Machine Flags are also rendered in read-only report preview mode. Test Flags Rendering Flow [#test-flags-rendering-flow-1] ```text ReadReportView ↓ testFlagsRenderer ``` *** Parameter Flags Rendering Flow [#parameter-flags-rendering-flow-1] ```text ReadReportView ↓ renderPathReport ↓ renderTopView ``` *** Rendering Architecture Summary [#rendering-architecture-summary] The Report Entry workflow uses: | Layer | Responsibility | | --------------------------- | --------------------------- | | PatientReport | Fetch report data | | getReportFormatForTestEntry | Fetch report values + flags | | EditReportView | Editable rendering | | ReadReportView | Read-only rendering | | testFlagsRenderer | Test-level flags | | TestNameColumn | Parameter-level flags | *** B. Overview Section [#b-overview-section] Machine Flags are also displayed in the patient Overview section. The Overview screen shows all patient reports in a consolidated view. Component Hierarchy [#component-hierarchy-2] ```text PatientWiseContainer ↓ PatientWiseReportListView ↓ ParticularPatientsTestListView ↓ PatientWiseAllReports ↓ RenderReports ``` *** Data Fetching Flow [#data-fetching-flow-1] Inside `PatientWiseAllReports` component `getInitialData()` calls `getPatientAllReports()` This API fetches all report-related data for a patient. *** API Response Structure [#api-response-structure-1] The API response contains `report_result_mapper` which stores all report values grouped by `labReportId` *** Machine Flags Storage [#machine-flags-storage] Machine Flags are included directly inside each report value object. *** Test-Level Flags [#test-level-flags-1] Stored in `test_flags` Example: ```json "test_flags": [ "DUMMY TEST FLAG1", "FLAG2" ] ``` *** Parameter-Level Flags [#parameter-level-flags-1] Stored in `param_flags` Example: ```json "param_flags": [ "A++" ] ``` *** Example Overview Report Value [#example-overview-report-value] ```json { "index": 8, "value": 220000, "param_flags": ["A++"], "test_flags": ["DUMMY TEST FLAG1", "FLAG2"] } ``` *** Overview Rendering Architecture [#overview-rendering-architecture] The Overview section reuses the same report rendering pipeline used in Report Entry. This ensures: * Shared Machine Flag rendering * Consistent UI * Reusable rendering logic *** Edit Mode Rendering [#edit-mode-rendering-1] Test Flags [#test-flags] ```text EditReportView ↓ testFlagsRenderer ``` *** Parameter Flags [#parameter-flags] ```text EditReportView ↓ renderReportCell ↓ EditReportCell ↓ ReportRangeInput ↓ TestNameColumn ``` *** Read Mode Rendering [#read-mode-rendering-1] Test Flags [#test-flags-1] ```text ReadReportView ↓ testFlagsRenderer ``` *** Parameter Flags [#parameter-flags-1] ```text ReadReportView ↓ renderPathReport ↓ renderTopView ``` *** C. Device Results Validation [#c-device-results-validation] Machine Flags are also displayed inside the Device Results Validation workflow. This screen helps users validate interfaced device results before release or approval. Route [#route-1] `/operation/device-results-validation/` Component Hierarchy [#component-hierarchy-3] ```text DeviceResultsForValidation ↓ Results ``` *** Data Fetching Flow [#data-fetching-flow-2] Inside `Results` component `getDeviceResults()` fetches all device validation result data. *** API Flow [#api-flow] ```text Results ↓ getDeviceResults() ↓ fetchDeviceResultsAPI() ``` *** getDeviceResults Responsibility [#getdeviceresults-responsibility] This method processes: * Device results * Related reports * Patient details * Report mappings * Validation state * Machine Flags *** Machine Flags Data [#machine-flags-data] The fetched device result objects contain: | Field | Purpose | | ---------- | ----------------------- | | testFlags | Sample/Test-level flags | | paramFlags | Parameter-level flags | These values originate from MongoDB-backed report values saved during interfacing. *** Rendering Architecture [#rendering-architecture] Test Flags [#test-flags-2] Rendered through `groupRowRenderer` This renderer displays sample-level flags for grouped report rows. *** Parameter Flags [#parameter-flags-2] Rendered through `parameterNameCellRenderer` This renderer displays parameter-level flags beside parameter names. *** Shared Machine Flag Architecture [#shared-machine-flag-architecture] All frontend screens share the same Machine Flag design principles: | Concept | Implementation | | ---------------- | ----------------------------- | | Test Flags | `test_flags` | | Parameter Flags | `param_flags` | | MongoDB Storage | `store_values_to_document_db` | | Shared Rendering | Reusable report components | | Device Data | Automated interfaced values | *** Frontend Design Summary [#frontend-design-summary] The frontend Machine Flags implementation was designed using reusable rendering pipelines across: * Report Entry * Overview Reports * Device Results Validation * PDF Preview This architecture ensures: * Consistent Machine Flag rendering * Shared UI behaviour * Reusable React components * Unified MongoDB-backed report rendering * Consistent interfaced result handling # Overview Machine Flags [#machine-flags] Machine Flags are additional indicators sent by lab devices along with parameter results when a sample is processed. These flags help lab staff identify abnormal conditions, quality issues, critical findings, or device-generated warnings before report approval. The flags originate directly from the lab device and are transmitted through the Interfacing application into the LIMS platform. Related Concepts [#related-concepts] * Device Interfacing * Device Parser * HL7 / ASTM Parsing * Parameter Mapping *** What are Machine Flags? [#what-are-machine-flags] When a lab device processes a patient sample, it does not only generate parameter values like: * Hemoglobin * Hematocrit * RBC * WBC * Platelets It may also generate additional diagnostic or operational indicators such as: | Type | Example | | ------------------------- | ---------------------------------------- | | **Sample-Level Flags** | Hemolyzed, Review Needed, Clotted Sample | | **Parameter-Level Flags** | H, L, A++, Critical, Warning | | **Analyzer Warnings** | Suspect Result, Recheck Required | | **Quality Indicators** | Invalid Count, Instrument Alert | These flags are collectively referred to as **Machine Flags**. *** High Level Workflow [#high-level-workflow] ```text Patient Billing ↓ Sample Collection ↓ Sample Processed in Device ↓ Device Generates Raw Result String ↓ Interfacing Application Receives Raw Data ↓ Device-Specific Parser Extracts Results + Flags ↓ Structured Payload Sent to LIMS API ↓ LIMS Stores Results & Flags ↓ Flags Displayed on Report UI ``` *** Core Architecture [#core-architecture] The Machine Flags flow involves three major systems: | Component | Responsibility | | --------------------------- | ------------------------------------------ | | **Lab Device** | Processes sample and generates raw output | | **Interfacing Application** | Parses device-specific raw data | | **LIMS Backend** | Stores and displays parsed results & flags | *** Test and Parameter Mapping [#test-and-parameter-mapping] Before a device can send results into the system, mapping must exist between: * Test ↔ Device * Test Parameter ↔ Device Example: | Test | Device | Parameters | | ---- | ------- | ------------------------------------------- | | CBC | Alinity | Hemoglobin, Hematocrit, RBC, WBC, Platelets | This mapping ensures that incoming parsed values can be correctly associated with the corresponding LIMS parameters. *** Real World Example [#real-world-example] Suppose a patient is billed for: * **CBC Test** The patient's blood sample is processed in: * **Alinity Device** The device evaluates all mapped CBC parameters and generates a raw output string. *** Raw Device Response [#raw-device-response] Devices usually send results in proprietary formats such as: * HL7 * ASTM * Custom protocols Example raw payload: ```text MSH|^~&|H560|Erba|||20240330152429||ORU^R01|20240330_150354_204|P|2.3.1 OBX|25|NM|718-7^HGB^LN||11.3|g/dL|11.5-17.5|L~A|||F OBX|26|NM|4544-3^HCT^LN||34.4|%|35.0-50.0|L~A|||F OBX|32|NM|777-3^PLT^LN||280|10*3/uL|125-350|~N|||F ``` The LIMS application cannot directly understand this raw device format. *** Interfacing Layer [#interfacing-layer] To solve this problem, an intermediate application called the **Interfacing Application** is used. Responsibilities of Interfacing [#responsibilities-of-interfacing] * Receive raw device strings * Identify the correct parser * Parse the raw payload * Generate structured JSON * Push data into LIMS APIs *** Device Parsers [#device-parsers] Each lab device has its own parser. A parser is generally a JavaScript file responsible for: * Reading raw device data * Extracting parameter values * Extracting flags * Converting output into a standard structure Parser Input [#parser-input] ```js parse(rawDeviceString) ``` Parser Output [#parser-output] ```json { "labId": 1, "sampleId": "000213826", "test_flags": ["Hemolyzed"], "data": { "values": [ { "testName": "hmg", "value": 11.3, "param_flags": ["L", "A"] } ] } } ``` *** Types of Machine Flags [#types-of-machine-flags] Machine Flags are divided into two major categories. 1. Test Level Flags [#1-test-level-flags] These flags apply to the entire sample or test. Examples: * Hemolyzed * Review Needed * Clotted Sample * Recollect Sample Example: ```json { "test_flags": [ "Hemolyzed", "Review Needed" ] } ``` *** 2. Parameter Level Flags [#2-parameter-level-flags] These flags apply to specific parameters only. Examples: | Parameter | Flags | | ---------- | ------------------ | | Hemoglobin | L, Critical Low | | Platelets | A++, Review Needed | | WBC | H | Example: ```json { "testName": "hmg", "value": 11.3, "param_flags": ["L", "A"] } ``` *** Important Behaviour [#important-behaviour] A device may send: | Scenario | Supported | | --------------------------- | --------- | | Only Test Flags | ✅ | | Only Parameter Flags | ✅ | | Both Test + Parameter Flags | ✅ | | No Flags | ✅ | | Partial Parameter Flags | ✅ | Not every parameter is guaranteed to contain flags. *** Standard LIMS Payload [#standard-lims-payload] After parsing, the interfacing application sends structured data to the Data Partial API. Example payload: ```json { "labId": 9, "sampleId": "000213826", "deviceAuth": "a3c730d7-ba96-449d-89e7-a9dd14a3b03f", "test_flags": [ "DUMMY TEST FLAG1", "FLAG2" ], "data": { "values": [ { "testName": "hmg", "value": 110.12, "param_flags": [ "P1", "H~N" ] }, { "testName": "plats", "value": 220000, "param_flags": [ "A++" ] } ] } } ``` *** Flag Storage in LIMS [#flag-storage-in-lims] After API consumption: * Test-level flags are stored against the sample/test * Parameter-level flags are stored against individual parameter results These flags become part of the permanent report metadata. *** UI Representation [#ui-representation] Machine Flags are displayed visually in the report approval screen. Sample-Level Flag Display [#sample-level-flag-display] Displayed near the test header. Examples: * Hemolyzed * Review Needed Parameter-Level Flag Display [#parameter-level-flag-display] Displayed directly below the affected parameter. Examples: * H * L * A++ * Warning *** Why Machine Flags Matter [#why-machine-flags-matter] Machine Flags are critical for: | Purpose | Description | | --------------------- | ------------------------------------- | | **Clinical Safety** | Identify abnormal or critical results | | **Quality Control** | Detect sample quality issues | | **Manual Review** | Notify staff for rechecking | | **Report Validation** | Help pathologists during approval | | **Automation Safety** | Prevent blind auto-approval | *** Common Examples [#common-examples] | Flag | Meaning | | ------------- | ------------------------------- | | H | High | | L | Low | | A | Abnormal | | HH | Critical High | | LL | Critical Low | | Hemolyzed | Sample damaged due to hemolysis | | Review Needed | Manual review required | | Clotted | Sample clot detected | *** Design Considerations [#design-considerations] Device Flexibility [#device-flexibility] Different devices send flags in completely different formats. The parser layer standardizes all of them into a common structure. *** Backward Compatibility [#backward-compatibility] Some old devices: * Do not send flags * Send flags in unusual delimiters * Send combined values like `L~A` Parsers must normalize such variations safely. *** Non-Blocking Behaviour [#non-blocking-behaviour] Flags should never block result ingestion. Even if flags are malformed: * Parameter values should still be processed * Parser should gracefully fallback > \[!NOTE] > Machine Flags are treated as supplementary metadata and should not prevent successful result import unless explicitly configured. *** Summary [#summary] Machine Flags are device-generated indicators received alongside parameter results during sample processing. The complete lifecycle involves: 1. Device processing 2. Raw payload generation 3. Parser extraction 4. API payload transformation 5. LIMS storage 6. UI visualization They play a critical role in ensuring report quality, patient safety, and efficient lab operations. # Workflow Guide Machine Flags Workflow Guide [#machine-flags-workflow-guide] This guide explains the complete workflow required to configure, process, and view Machine Flags inside the LIMS platform. The workflow includes: 1. Enabling Machine Flags for a pathology test 2. Mapping the test to a device 3. Sending Machine Flags from the interfacing application 4. Processing and storing flags in LIMS 5. Viewing Machine Flags in the UI High Level Workflow [#high-level-workflow] ```text Enable Machine Flags for Test ↓ Map Test with Device ↓ Bill Patient with Configured Test ↓ Device Processes Sample ↓ Interfacing Parses Raw String ↓ Machine Flags Sent to LIMS API ↓ LIMS Stores Flags in MongoDB ↓ Flags Visible in Report UI ``` *** Step 1 - Enable Machine Flags for a Test [#step-1---enable-machine-flags-for-a-test] Machine Flags can only be configured for **Pathology Tests**. Configuration Steps [#configuration-steps] 1. Open Profile & Report Management [#1-open-profile--report-management] Go to `Admin Module → Profile & Report Management` 2. Select the Required Pathology Test [#2-select-the-required-pathology-test] Open the pathology report/test for which Machine Flags should be enabled. Example: * CBC * Lipid Profile * Total Cholesterol 3. Open Test Information Config Section [#3-open-test-information-config-section] Inside the selected test `Test Information → Config` 4. Enable Device Flags Option [#4-enable-device-flags-option] Enable the checkbox `Enable Device Flags (Store results in MongoDB)` Machine Flags Configuration 5. Save Changes [#5-save-changes] Click Save to apply the configuration. > \[!NOTE] > The **Enable Device Flags** checkbox is only visible for **Pathology Tests**. > \[!IMPORTANT] > Enabling this option converts the test into a **document test**, meaning parameter values and Machine Flags are stored inside MongoDB. *** Step 2 - Configure Device Test Mapping [#step-2---configure-device-test-mapping] Before Machine Flags can be received from interfacing, the test must be mapped to a lab device and its parameters. Why Mapping Is Required [#why-mapping-is-required] The interfacing application sends results using device-specific parameter identifiers such as: * HGB * HCT * PLT LIMS must know which report parameter each identifier corresponds to. Example Mapping [#example-mapping] | LIMS Parameter | Device Parameter | | -------------- | ---------------- | | Hemoglobin | HGB | | Hematocrit | HCT | | Platelets | PLT | Configuration [#configuration] Navigate to `Operation Module → Device / Instrument Management` 1. Open the required device. 2. Add the pathology test. 3. Configure parameter mappings. 4. Save the mapping. Once completed, incoming device results can be matched with the correct report parameters. Machine Flags Device Mapping Config *** Step 3 - Bill Patient with Configured Test [#step-3---bill-patient-with-configured-test] After enabling Machine Flags: 1. Create a patient bill 2. Add the Machine Flag enabled pathology test 3. Complete sample collection Example: `Patient Bill → CBC Test Added → Sample Collected` *** Step 4 - Device Generates Results [#step-4---device-generates-results] After a patient sample is processed, the lab device generates a raw result payload containing the test results and any associated machine-generated indicators. The format of this payload varies between devices and may use protocols such as HL7, ASTM, or vendor-specific formats. In addition to parameter values, the payload can also contain: * Parameter values * Test-level flags * Parameter-level flags Example: ```text OBX|25|NM|718-7^HGB^LN||11.3|g/dL|11.5-17.5|L~A|||F ``` *** Step 5 - Interfacing Parses Machine Flags [#step-5---interfacing-parses-machine-flags] The interfacing application receives the raw device string and passes it to the device parser. The parser: * Extracts parameter values * Extracts sample-level flags * Extracts parameter-level flags * Generates structured JSON payload Parsed Payload Example [#parsed-payload-example] ```json { "labId": 11957, "sampleId": "000114626", "deviceAuth": "f94032be-5e1f-47fd-96a1-92a66c6014c3", "test_flags": [ "DUMMY TEST FLAG1", "FLAG2" ], "data": { "values": [ { "testName": "hmg", "value": 110.12, "param_flags": [ "P1", "H~N" ] }, { "testName": "hct", "value": 123.12 }, { "testName": "rbc", "value": 11.1 }, { "testName": "wbc", "value": 45.34 }, { "testName": "plats", "value": 220000, "param_flags": [ "A++" ] }, { "testName": "mcv", "value": 19 }, { "testName": "mch", "value": 22 } ] } } ``` Payload Structure [#payload-structure] Sample level flags are Stored at root level `"test_flags": []` These flags apply to the entire test/sample. Parameter level flags are stored inside parameter object `"param_flags": []` These flags apply only to that specific parameter. Important Behaviour [#important-behaviour] Device behaviour may vary. A device can send: | Scenario | Supported | | --------------------------- | --------- | | Only Test Flags | ✅ | | Only Parameter Flags | ✅ | | Both Test + Parameter Flags | ✅ | | Partial Parameter Flags | ✅ | | No Flags | ✅ | > \[!NOTE] > It is completely valid for some parameters to contain flags while others do not. *** Step 6 - LIMS API Processing [#step-6---lims-api-processing] After payload generation, the interfacing application sends the payload to the LIMS Data Partial API. 1. Validates device authentication 2. Matches sample ID 3. Maps device parameters 4. Processes parameter values 5. Stores Machine Flags 6. Saves report values into MongoDB *** Step 7 - Viewing Machine Flags in LIMS [#step-7---viewing-machine-flags-in-lims] Machine Flags become visible after report values are processed successfully. Pending Report Screen [#pending-report-screen] To view Machine Flags `Operation Module → Pending Report` Open the report for which Machine Flags were received. Machine Flags Report Entry Modal View Image Overview Tab [#overview-tab] Machine Flags are also visible in the report Overview tab. Machine Flags Overview View Image *** Device Results Validation Screen [#device-results-validation-screen] Machine Flags are also shown in `Device Results Validation` screen. Machine Flags Device Results Validation Image *** PDF Preview Behaviour [#pdf-preview-behaviour] Machine Flags are not visible when the report is opened in `PDF Preview Mode` > \[!IMPORTANT] > Machine Flags are not displayed in normal PDF report preview mode. *** End-to-End Example [#end-to-end-example] ```text CBC Test Configured ↓ CBC Mapped with Alinity ↓ Patient Sample Processed ↓ Alinity Sends Raw Result String ↓ Interfacing Parser Extracts Flags ↓ Parsed Payload Sent to LIMS API ↓ LIMS Stores Flags in MongoDB ↓ Flags Visible in Report UI ``` *** Summary [#summary] Machine Flags workflow involves: 1. Configuring pathology tests 2. Mapping tests with devices 3. Parsing raw device data 4. Sending structured payloads 5. Storing flags in MongoDB 6. Displaying flags in report UI This workflow enables laboratories to preserve and display device-generated abnormalities, warnings, and review indicators directly inside the LIMS ecosystem. # Backend Backend [#backend] What Backend Owns [#what-backend-owns] | Concern | Backend responsibility | | :---------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------- | | Master data persistence | `Organism`, `Antibiotic`, `OrganismAntibiotics`, `MolecularOrganismAntibiotics` models, validation, and lifecycle hooks | | Caching | Organism and antibiotic records are cached individually and in collections using `ReportingCacheBase` to avoid repeated DB queries | | Validation | Mandatory field checks, duplicate prevention, RIS range pattern validation, and logic purity for all master data operations | | API endpoints | CRUD for organisms and antibiotics, enable/disable flows, and linked-entity queries (which organisms use an antibiotic and vice versa) | | Report submission | Microbiology report data is persisted as part of the generic report-entry submission flow; microbiology-specific parsing handled by `MicrobiologyComponent` | | Authorization | `access_admin` decorator on all write endpoints; `reporting_read` decorator on all read endpoints | Backend Repos [#backend-repos] * **`livehealthapp`**: Contains model definitions, serializers, views, and URL configurations for `Organism`, `Antibiotic`, `OrganismAntibiotics`, and `MolecularOrganismAntibiotics`. * **`crelio-app`**: Shares the core logic for report submission, organism/antibiotic caching, and microbiology report component processing for multi-centre use cases. Core Backend Models [#core-backend-models] Organism [#organism] Table: `Organism` Source: [`livehealthapp/reporting/models/organism.py`](https://github.com/CrelioHealth/livehealthapp/tree/main/reporting/models/organism.py) | Column | Type | Description | | :------------ | :------------- | :------------------------------------------------------ | | `id` | AutoField | Primary key | | `name` | CharField(150) | Organism name; unique per lab | | `lab` | FK → `labs` | The lab owning the organism; `null` for system defaults | | `category` | CharField(150) | Free-text grouping (e.g. Bacteria, Fungi) | | `code` | CharField(25) | Short identifier | | `is_disabled` | BooleanField | Controls visibility | | `sample_type` | CharField(150) | Associated sample type | **After save:** Clears caches for the specific organism and the organism list. **Model hooks:** `save_molecular_antibiotic_mappings` handles the ordered Molecular tabs list on organism save. Antibiotic [#antibiotic] Table: `Antibiotic` Source: [`livehealthapp/reporting/models/antibiotic.py`](https://github.com/CrelioHealth/livehealthapp/tree/main/reporting/models/antibiotic.py) | Column | Type | Description | | :------------ | :------------- | :---------------------------- | | `id` | AutoField | Primary key | | `name` | CharField(150) | Antibiotic name | | `lab` | FK → `labs` | The lab owning the antibiotic | | `category` | CharField(150) | Free-text grouping | | `code` | CharField(25) | Short identifier | | `is_disabled` | BooleanField | Controls visibility | | `sample_type` | CharField(150) | Associated sample type | **After save:** Clears caches for the specific antibiotic and the list. OrganismAntibiotics [#organismantibiotics] Table: `OrganismAntibiotics` Source: [`livehealthapp/reporting/models/organism_antibiotics.py`](https://github.com/CrelioHealth/livehealthapp/tree/main/reporting/models/organism_antibiotics.py) | Column | Type | Description | | :--------------------------------------- | :---------------- | :------------------------------------- | | `id` | AutoField | Primary key | | `organism` | FK → `Organism` | The parent organism | | `antibiotic` | FK → `Antibiotic` | The mapped antibiotic | | `resistance_diameter_upper` / `_lower` | FloatField | Disk diffusion ranges for Resistant | | `intermediate_diameter_upper` / `_lower` | FloatField | Disk diffusion ranges for Intermediate | | `sensitive_diameter_upper` / `_lower` | FloatField | Disk diffusion ranges for Sensitive | | `resistance_mic_upper` / `_lower` | FloatField | MIC ranges for Resistant | | `intermediate_mic_upper` / `_lower` | FloatField | MIC ranges for Intermediate | | `sensitive_mic_upper` / `_lower` | FloatField | MIC ranges for Sensitive | **Bulk create (`OrganismAntibiotics.bulk_create`):** * Deletes all existing mappings matching the `filters` dict (e.g. `\{"organism_id": self.pk\}`). * Re-creates all passed instances; validates each before save. * Used during `Organism.after_save(...)` to replace the full antibiogram in a single atomic block. MolecularOrganismAntibiotics [#molecularorganismantibiotics] Table: `MolecularOrganismAntibiotics` Source: [`livehealthapp/reporting/models/molecular_organism_antibiotics.py`](https://github.com/CrelioHealth/livehealthapp/tree/main/reporting/models/molecular_organism_antibiotics.py) | Column | Type | Description | | :----------- | :---------------- | :---------------------------------------- | | `id` | AutoField | Primary key | | `organism` | FK → `Organism` | The parent organism | | `antibiotic` | FK → `Antibiotic` | The mapped antibiotic | | `sequence` | IntegerField | Determines display order in the grid | | `is_active` | BooleanField | Whether it is actively used in the report | Payload Reference [#payload-reference] Organism Create/Update Payload [#organism-createupdate-payload] When creating or updating an organism, the frontend sends the mapping lists inside the organism payload: ```json { "name": "Escherichia coli", "category": "Bacteria", "code": "ECOLI", "sample_type": "blood", "antibiotic_mappings": [ { "antibiotic_id": 42, "antibiotic_name": "Ciprofloxacin", "resistance_diameter_upper": 12, "resistance_diameter_lower": 0, "intermediate_diameter_upper": 19, "intermediate_diameter_lower": 13, "sensitive_diameter_upper": 40, "sensitive_diameter_lower": 20, "resistance_mic_upper": 8, "resistance_mic_lower": 0, "intermediate_mic_upper": 4, "intermediate_mic_lower": 2, "sensitive_mic_upper": 1, "sensitive_mic_lower": 0 } ], "molecular_organism_antibiotics": [ { "antibiotic_id": 42, "sequence": 1, "is_active": true } ] } ``` API Endpoints [#api-endpoints] Organism Endpoints [#organism-endpoints] | Method | Endpoint | Description | | :----- | :-------------------------------------------------- | :------------------------------------------------------------------------- | | `GET` | `/reporting/organisms/` | List all organisms; supports `is_disabled`, `name`, `lab_ids` query params | | `GET` | `/reporting/organisms/\{organism_id\}` | Fetch a single organism by ID | | `POST` | `/reporting/organisms/new/` | Create a new organism | | `POST` | `/reporting/organisms/\{organism_id\}/update/` | Update an existing organism | | `POST` | `/reporting/organisms/\{instance_id\}/disable/` | Disable an organism | | `POST` | `/reporting/organisms/\{instance_id\}/enable/` | Re-enable a disabled organism | | `GET` | `/reporting/organisms/\{instance_id\}/antibiotics/` | List antibiotics mapped to an organism | | `GET` | `/reporting/antibiotics/\{instance_id\}/organisms/` | List organisms mapped to an antibiotic | Antibiotic Endpoints [#antibiotic-endpoints] | Method | Endpoint | Description | | :----- | :------------------------------------------------- | :--------------------------------------------------------------------------- | | `GET` | `/reporting/antibiotics/` | List all antibiotics; supports `is_disabled`, `name`, `lab_ids` query params | | `GET` | `/reporting/antibiotics/\{antibiotic_id\}` | Fetch a single antibiotic by ID | | `POST` | `/reporting/antibiotics/new/` | Create a new antibiotic | | `POST` | `/reporting/antibiotics/\{antibiotic_id\}/update/` | Update an existing antibiotic | | `POST` | `/reporting/antibiotics/\{instance_id\}/disable/` | Disable an antibiotic | | `POST` | `/reporting/antibiotics/\{instance_id\}/enable/` | Re-enable a disabled antibiotic | Backend Data Flow for Microbiology Report Entry [#backend-data-flow-for-microbiology-report-entry] 1. Report entry values are submitted via the standard report submission API. 2. The payload contains an array of distinct objects for each organism and its matched antibiotics under the `MICROBIOLOGY` parameter type. 3. The `MicrobiologyComponent` proxy evaluates these records, builds the result, and stores it in the `ReportData` payload using standard structure. 4. The frontend performs all the intermediate/sensitive/resistant text derivation on the client side, then submits the final calculated values in the text fields (`result_2`). Source File Reference [#source-file-reference] | Area | File | Description | | :--------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------- | | Organism model | [`reporting/models/organism.py`](https://github.com/CrelioHealth/livehealthapp/tree/main/reporting/models/organism.py) | Core organism entity, RIS mapping management, caching, activity logging | | OrganismAntibiotics model | [`reporting/models/organism_antibiotics.py`](https://github.com/CrelioHealth/livehealthapp/tree/main/reporting/models/organism_antibiotics.py) | Through-table for organism ↔ antibiotic RIS breakpoints | | MolecularOrganismAntibiotics model | [`reporting/models/molecular_organism_antibiotics.py`](https://github.com/CrelioHealth/livehealthapp/tree/main/reporting/models/molecular_organism_antibiotics.py) | Through-table for Molecular mapping tab | | Organism view | [`reporting/views/organism.py`](https://github.com/CrelioHealth/livehealthapp/tree/main/reporting/views/organism.py) | HTTP handlers for organism CRUD | | Organism serializer | [`reporting/serializers/organism.py`](https://github.com/CrelioHealth/livehealthapp/tree/main/reporting/serializers/organism.py) | Serializes organism data including nested mappings | | Organism component proxy | [`reporting/proxies/report_format/organism_component.py`](https://github.com/CrelioHealth/livehealthapp/tree/main/reporting/proxies/report_format/organism_component.py) | Report-format proxy for the ORGANISM component type | # Design Decisions Design Decisions [#design-decisions] Separate Through-Table for RIS Breakpoints vs Molecular Mapping [#separate-through-table-for-ris-breakpoints-vs-molecular-mapping] **Decision:** The `OrganismAntibiotics` through-table (Microbiology Ranges) is separate from `MolecularOrganismAntibiotics` (Molecular Mapping). Both map organism ↔ antibiotic, but they carry different data and serve different workflows. **Rationale:** * `OrganismAntibiotics` carries 12 float columns of RIS breakpoints (diameter and MIC for R, I, S) and has strict range validation. Its primary consumer is the microbiology susceptibility workflow. * `MolecularOrganismAntibiotics` carries sequence order and an active flag. Its primary consumer is the molecular Antibiotic Resistance summary component. It has no RIS data. * Mixing both concerns into a single table would create nullable columns on both sides and make the validation logic significantly more complex. * Two tables also allow independent cache invalidation: a molecular priority change does not need to invalidate RIS breakpoint data and vice versa. **Tradeoff:** Labs using both workflows must set up the antibiotic relationship in two places (the `Microbiology Ranges` sub-tab and the `Molecular Mapping` sub-tab). The UI exposes both as sub-tabs inside the same `Antibiotic Mapping` tab to reduce navigation friction. Bulk Replace Strategy for OrganismAntibiotics [#bulk-replace-strategy-for-organismantibiotics] **Decision:** When saving an organism, all existing `OrganismAntibiotics` records for that organism are deleted and re-created from the incoming payload (`OrganismAntibiotics.bulk_create(...)`). **Rationale:** * The microbiology ranges grid in the UI sends the complete intended mapping state, not a delta. * A bulk-replace pattern avoids the complexity of tracking additions, deletions, and edits separately in the API layer. * Since the organism's antibiotic mappings are always re-validated before creation, data consistency is maintained even on a full replace. **Tradeoff:** If a save fails mid-operation, there is a risk of transient data loss. The `transaction.atomic` decorator on the POST view mitigates this by wrapping the full save in a single database transaction. Interpretation Only Method Type Reduces Displayed Columns [#interpretation-only-method-type-reduces-displayed-columns] **Decision:** When the method type is set to `Interpretation Only`, the `result_1`, `result_r`, `result_i`, and `result_s` columns are hidden from the microbiology antibiotic grid at both configuration time and report entry. **Rationale:** * `Interpretation Only` is intended for scenarios where labs manually categorize susceptibility without entering a numeric result (e.g. direct S/I/R assignment from clinical assessment or external source). * Showing numeric input and RIS reference range columns in this mode would be visually misleading. * The `updateMicroEditables(cols, type, isNewReport)` function handles the hiding logic: * For `Interpretation Only` on an existing report: hides result/RIS columns in-place. * For `Interpretation Only` on a new report: reduces the column set to only `name` and `result_2` (renamed to `Manual Interpretation`) with `INTERPRETATION_OPTIONS` dropdown. **Tradeoff:** Labs that later switch from `Interpretation Only` to `Detection Window` or `MIC` must revisit column configuration as the hidden columns must be manually re-added. max_allowed_microorganism Enforcement Is Frontend-Only at Report Entry [#max_allowed_microorganism-enforcement-is-frontend-only-at-report-entry] **Decision:** The maximum organism count check (capping the `Add Organism` action at report entry) is performed on the frontend using the `max_allowed_microorganism` value from the component's meta object. **Rationale:** * Report entry is a high-frequency, real-time UI interaction. Enforcing the cap via a round-trip API call per organism add would introduce unnecessary latency and server load. * The `max_allowed_microorganism` value is already loaded with the report format and is available in Redux state. * Backend validation for this constraint is not the bottleneck; what matters is the user-facing experience of being immediately informed when the cap is reached. **Tradeoff:** A motivated user who manipulates client-side state could bypass the cap at the browser level. Backend validation on report submission could be added if this becomes a compliance requirement. max_allowed_microorganism = 1 Disables the Pivot PDF Checkbox [#max_allowed_microorganism--1-disables-the-pivot-pdf-checkbox] **Decision:** When `Max Number Of Micro Organisms` is `1`, the `Render Pivot Table on PDF` checkbox is automatically disabled and `meta.should_pivot` is set to `false`. **Rationale:** * A pivot table view is only meaningful when multiple organisms are expected; pivoting a single-organism antibiogram gives no added value. * Disabling the checkbox prevents a misleading configuration that would produce an empty pivot with one column. Organism Count is Computed from group_by_id Groups, Not Row Count [#organism-count-is-computed-from-group_by_id-groups-not-row-count] **Decision:** At report entry, the number of organisms currently added is computed as `Object.keys(groupBy(value?.value, "group_by_id")).length` (distinct organism groups), not as `value?.value.length` (total antibiotic rows). **Rationale:** * Each organism in the microbiology grid is represented by multiple rows — one per antibiotic added to that organism. * If the count was computed using the raw row count, adding antibiotics to an organism would inadvertently consume the organism limit. * Using the distinct `group_by_id` count correctly measures how many organisms are in the report section, regardless of how many antibiotics each organism has. RIS Range Fields Are All Nullable But Validation Requires Consistency [#ris-range-fields-are-all-nullable-but-validation-requires-consistency] **Decision:** All 12 RIS breakpoint columns (`resistance_diameter_upper`, `sensitive_mic_lower`, etc.) in `OrganismAntibiotics` are nullable at the database level, but the `before_save(...)` hook enforces that either all diameter fields or all MIC fields must be populated (not a mix). **Rationale:** * Nullable fields allow the through-table to be created initially empty before any breakpoints are assigned. * Consistency enforcement is deferred to `before_save(...)` rather than database constraints because the mix-validation rule (all diameters or all MICs) cannot be expressed cleanly as a database CHECK constraint across 12 columns. * The `has_valid_patterns` check ensures upper ≥ lower before persisting any range. **Error messages raised:** | Condition | Error message | | :------------------------------------------- | :------------------------------------------------------------------------------- | | Missing any R, I, or S diameter or MIC group | "Mandatory intermediate/resistance/sensitive diameter or mic values are absent!" | | Mixing diameter and MIC fields | "Sensitivity pattern can either be diameters or mics!" | | Upper \< lower for any range | "Pattern ranges are not valid!" | Duplicate Antibiotic Mapping Prevention Uses Python-Level Check [#duplicate-antibiotic-mapping-prevention-uses-python-level-check] **Decision:** Duplicate `antibiotic_id` values in the `antibiotic_mappings` payload are rejected by checking `len(set(antibiotic_ids)) != len(antibiotic_ids)` in `Organism.after_save(...)`. **Rationale:** * The `OrganismAntibiotics` table has no database-level unique constraint on `(organism_id, antibiotic_id)`. * Checking at the Python level allows a descriptive error to be returned before any DB operation begins. **Known gotcha:** Because the table has no unique constraint, a race condition between two concurrent save requests could theoretically create duplicate mappings. This is mitigated by `transaction.atomic` on the POST view. Caching Organism and Antibiotic Data for Report Entry Performance [#caching-organism-and-antibiotic-data-for-report-entry-performance] **Decision:** The full organism catalog (including nested `antibiotic_mappings` with RIS breakpoints) is loaded once into Redux `MODEL.allOrganisms` and used for all downstream operations including antibiotic dropdown resolution and range display at report entry. **Rationale:** * Loading per-organism antibiotic lists on demand at report entry would require N+1 API calls. * Antibiotic RIS breakpoints are relatively static master data; loading them upfront does not cause stale-data concerns in practice. * The backend also caches these records via `ReportingCacheBase` to avoid DB round-trips per request. **Tradeoff:** If an organism's RIS breakpoints are updated while a user has the report entry modal open, the user will see the old breakpoints until they reload the page or refetch the organism catalog. This is acceptable for master data that changes infrequently. RIS Interpretation Precedence: Last-Matched Wins [#ris-interpretation-precedence-last-matched-wins] **Decision:** The `calculateValueForMicro` function evaluates all three conditions (S, I, R) sequentially without `else if`. The last match overrides the previous one. **Current behavior:** ```ts if (...sensitive check...) result = t("Sensitive"); if (...intermediate check...) result = t("Intermediate"); if (...resistant check...) result = t("Resistant"); ``` **Implication:** If breakpoints are configured so that a value falls into multiple categories (overlapping ranges), the interpretation displayed will be **Resistant** (the last evaluated category). This is consistent with clinical convention where resistant interpretation takes the highest clinical priority. **Known gotcha:** If ranges are non-overlapping and disjoint, a value that falls outside all three ranges returns an empty string for `result_2`. The UI will display no interpretation rather than an error. Microbiology vs Molecular Parameter Separation [#microbiology-vs-molecular-parameter-separation] **Decision:** The Microbiology Parameter component (`type: 20`) is distinct from Molecular components (`GENE: 23`, `ORGANISM: 22`, `ANTIBIOTIC_RESISTANCE: 24`). Each has its own form, column options, and validation path. **Rationale:** * Microbiology susceptibility workflows (culture + MIC/disk diffusion) and Molecular workflows (PCR gene detection + antibiotic resistance pivots) have different data models, display requirements, and interpretation logic. * Separating the components allows each to evolve independently. For example, the `Interpretation Only` mode in Microbiology has no equivalent in Molecular. * Validation for Microbiology (`component_type === MICROBIOLOGY && type === 20`) is a separate block in `reportParamterValidation(...)` and covers `linked_model`, `method_type`, and `max_allowed_microorganism`. # Frontend Frontend [#frontend] Core Repositories [#core-repositories] The Microbiology frontend code resides primarily in **`livehealth-frontend`**. This handles the master data UI in `Drug Master / Panel Master` and the report builder / report entry UI in `Profile & Report Management`. Master Data Components [#master-data-components] Antibiotic Master [#antibiotic-master] Source: [`livehealth-frontend/src/components/reusable/Antibiotic/container/index.tsx`](https://github.com/CrelioHealth/livehealth-frontend/tree/main/src/components/reusable/Antibiotic/container/index.tsx) The Antibiotic Master component provides a list view and a creation/edit modal. It interacts with the `/reporting/antibiotics/` APIs. * Fetches and displays the list of antibiotics. * Provides standard table actions (Copy, Edit, Disable, Enable). * When disabling an antibiotic, it fetches related organisms from `/reporting/antibiotics/\{id\}/organisms` and displays them in a warning modal. Organism Master & Mapping [#organism-master--mapping] Source: [`livehealth-frontend/src/components/reusable/Organism/container/index.tsx`](https://github.com/CrelioHealth/livehealth-frontend/tree/main/src/components/reusable/Organism/container/index.tsx) The Organism Master provides the main CRUD operations for organisms. * Contains a `MappingTab` which acts as the host for the **Microbiology Ranges** and **Molecular Mapping** sub-tabs. * The `Microbiology Ranges` tab enables searching for an antibiotic and configuring the R, I, and S breakpoints for Disk Diffusion (Diameter) and MIC. * When disabling an organism, it fetches related antibiotics from `/reporting/organisms/\{id\}/antibiotics`. RIS Mapping Helper (mappingListParsing) [#ris-mapping-helper-mappinglistparsing] Source: [`livehealth-frontend/src/components/reusable/Antibiotic/utils/helpers.ts`](https://github.com/CrelioHealth/livehealth-frontend/tree/main/src/components/reusable/Antibiotic/utils/helpers.ts) This helper translates the UI representation of RIS grids into the flat payload structure required by the backend API: ```javascript // Each antibiotic is transformed to: { antibiotic_id: 42, antibiotic_name: "Ciprofloxacin", resistance_diameter_upper: 12, resistance_diameter_lower: 0, // ... (all 12 fields) } ``` Report Configuration Components [#report-configuration-components] These components are used when configuring a `Microbiology` test under `Test List > Add New Parameter`. MicrobiologyForm [#microbiologyform] Source: [`livehealth-frontend/src/components/LabAdmin/AddEditReport/components/MicrobiologyForm/index.tsx`](https://github.com/CrelioHealth/livehealth-frontend/tree/main/src/components/LabAdmin/AddEditReport/components/MicrobiologyForm/index.tsx) * Mounts the main parameter settings. * Forces `Render Pivot Table on PDF` (`should_pivot`) to `false` if `max_allowed_microorganism` is set to `1`. * Contains tabs for Configuration and Meta. updateMicroEditables [#updatemicroeditables] Source: [`livehealth-frontend/src/components/LabAdmin/Utils/helpers.ts`](https://github.com/CrelioHealth/livehealth-frontend/tree/main/src/components/LabAdmin/Utils/helpers.ts) When `Method Type` changes between `Detection Window`, `MIC`, and `Interpretation Only`, this helper manages visibility of the result grid columns (`result_1`, `result_2`, `result_r`, etc.). For `Interpretation Only`, it hides all columns except `name` and `result_2`. Report Entry & AgGrid [#report-entry--aggrid] At report entry, the Microbiology component builds a dynamic grid using `ag-grid-react`. Rendering the Component (renderComponent) [#rendering-the-component-rendercomponent] Source: [`livehealth-frontend/src/components/reusable/Modals/Report/EditReportView/index.tsx`](https://github.com/CrelioHealth/livehealth-frontend/tree/main/src/components/reusable/Modals/Report/EditReportView/index.tsx) The standard `EditReportView` handles rendering based on `component_type`. For `MICROBIOLOGY`: * It renders a multi-select dropdown for adding organisms up to `max_allowed_microorganism`. * Below each selected organism, it renders an AgGrid of mapped antibiotics. Maximum Organism Enforcement [#maximum-organism-enforcement] Before adding a new organism, the system validates the current count: ```javascript if (Object.keys(microbiologyOrganisms).length >= max_allowed_microorganism) { message: `${t("Maximum")} ${max_allowed_microorganism} ${t("organism")} ${t("allowed")}`; // Prevents addition } ``` Auto-interpretation (calculateValueForMicro) [#auto-interpretation-calculatevalueformicro] Source: [`livehealth-frontend/src/components/reusable/Modals/Report/helpers.ts`](https://github.com/CrelioHealth/livehealth-frontend/tree/main/src/components/reusable/Modals/Report/helpers.ts) When the user types a value in the `result_1` cell, `calculateValueForMicro` runs: 1. Determines if the method type is `MIC` or `Detection Window`. 2. Checks the entered value against `resistance_upper`/`_lower`, `intermediate_upper`/`_lower`, and `sensitive_upper`/`_lower`. 3. Populates `result_2` with the corresponding interpretation text (e.g. `Sensitive`). Reference Range Formatting [#reference-range-formatting] Source: [`livehealth-frontend/src/components/reusable/Modals/Report/helpers.ts`](https://github.com/CrelioHealth/livehealth-frontend/tree/main/src/components/reusable/Modals/Report/helpers.ts) The read-only reference columns (`result_r`, `result_i`, `result_s`) are formatted as strings: `{lower} - {upper}`. If no range is configured, they display `- - -`. # Overview Overview [#overview] Microbiology in the medical laboratory involves the identification of microorganisms such as bacteria, fungi, and other pathogens in clinical samples through culture and susceptibility testing. The Microbiology feature supports labs in configuring antibiotic susceptibility reports by linking organisms to antibiotics, defining RIS (Resistant / Intermediate / Sensitive) ranges, and generating structured antimicrobial susceptibility grids at report entry. Microbiology [#microbiology] Microbiology provides the product foundation for managing organism-antibiotic susceptibility configuration and report entry behavior. Before a microbiology workflow can be used in billing or report entry, labs must define the underlying organism catalog and map antibiotics to each organism with their associated RIS breakpoint ranges (Detection Window or MIC). This setup ensures that microbiology tests use consistent susceptibility patterns and interpretations across the workflow. After prerequisite master data is ready, a microbiology report/test is created from `Profile & Report Management > Test List`. When the test type is selected as `Microbiology`, the report parameter builder exposes a `Microbiology Parameter` component. Prerequisites [#prerequisites] | Requirement | Why it matters | Where it is enforced | | :--------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------- | | Organism master data must exist | The Microbiology component lets report-entry users add organisms from the organism catalog | Organism Master module in `livehealth-frontend`; backend persistence in `livehealthapp` and `crelio-app` | | Antibiotic master data must exist | An antibiotic record represents a drug used to determine susceptibility patterns | Antibiotic Master UI and related backend APIs | | Organism must be mapped to antibiotics with RIS ranges | RIS breakpoints (Diameter / MIC) on each organism-antibiotic mapping drive the automatic interpretation shown in the report entry grid | Organism Master → `Antibiotic Mapping` tab → `Microbiology Ranges` sub-tab | | User must have access to Drug Master / Panel Master screens | The master setup for organisms and antibiotics is managed from the Drug Master / Panel Master area in the application sidebar | Frontend route/sidebar permissions and backend authorization | | Microbiology test/report must be created with test type `Microbiology` | The Microbiology report component is available only after selecting the Microbiology test type | Test List / Add New Test flow in `livehealth-frontend` | What Is It For [#what-is-it-for] Frontend perspective [#frontend-perspective] * Provide Organism Master and Antibiotic Master screens under `Drug Master / Panel Master`. * Let users create, update, disable, download, and bulk-manage organism and antibiotic records. * Let users define organism-to-antibiotic mappings with RIS breakpoint ranges under the `Microbiology Ranges` sub-tab inside the organism creation flow. * Show system default, custom, disabled, and all-record views where applicable. * Create a microbiology report/test by selecting test type `Microbiology`. * Add the `Microbiology Parameter` report component from the report parameter menu. * Configure max number of organisms, type of method (Detection Window / MIC / Interpretation Only), referring list (Organisms), and display field columns. * At report entry, enforce the maximum organism count and auto-calculate interpretation from entered results against the configured RIS ranges. Backend perspective [#backend-perspective] * Persist organism and antibiotic master data across `livehealthapp` and `crelio-app`. * Validate required fields such as organism name/category/code and antibiotic name/category/code/unit. * Maintain relationships between organisms and antibiotics through the `OrganismAntibiotics` table (standard antibiogram) with RIS diameter and MIC breakpoints. * Maintain molecular-specific organism-to-antibiotic ordering and active state through the `MolecularOrganismAntibiotics` table. * Persist microbiology report parameter configuration and component metadata for report entry and billing workflows. Types / Modes [#types--modes] | Type | Example | Runtime behavior | Notes | | :------------------------------- | :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------- | | Organism | E. coli, S. aureus, K. pneumoniae | Defines microorganisms with name, category, code, cut off, sample type, and description; linked to antibiotics through `OrganismAntibiotics` | Base entity for microbiology report entry | | Antibiotic | Ciprofloxacin, Cefotaxime, Amikacin | Defines antimicrobial drugs with name, category, code, method, unit, sample type, device name, and dosage | Shared between Organism master and Molecular master contexts | | Microbiology Parameter component | Microbiology | The single report component for microbiology test type | Configured with Method Type, Max Number Of Micro Organisms, Referring List, and column fields | | Detection Window method | Disk diffusion susceptibility | Interpretation computed using diameter breakpoints (lower–upper range per RIS category) | `result_r`, `result_i`, `result_s` columns show configured breakpoint ranges | | MIC method | Minimum Inhibitory Concentration | Interpretation computed using MIC breakpoints (lower–upper range per RIS category) | Uses MIC lower/upper per RIS category | | Interpretation Only method | Text-only interpretation | Hides result columns; shows only organism name and manual interpretation | No automatic RIS calculation | Structure Of Microbiology [#structure-of-microbiology] | Layer | What it stores or owns | Table / state / file | Why it exists | | :------------------------ | :----------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------- | | Antibiotic master layer | Antimicrobial drug definitions | `Antibiotic` | Provides the base antibiotic catalog shared by organism susceptibility workflows | | Organism master layer | Microorganism definitions with antibiogram mappings | `Organism` with `OrganismAntibiotics` and `MolecularOrganismAntibiotics` | Represents detected organisms and their RIS susceptibility patterns | | RIS range layer | Per-organism, per-antibiotic breakpoints | `OrganismAntibiotics` (resistance/intermediate/sensitive diameter and MIC fields) | Drives automatic interpretation at report entry based on entered result | | Microbiology report layer | Microbiology report/test configuration and component | `Profile & Report Management > Test List > Add New Test > Test Type: Microbiology > Report Parameters` | Defines report-entry behavior for organism identification and antibiotic susceptibility grids | | Frontend UI layer | Lists, modals, filters, bulk actions, add/update flows | `Drug Master / Panel Master` screens and `Test List > Report Parameters` builder | Lets users manage prerequisite master data and configure microbiology report components | | Backend service layer | APIs, validation, persistence, permissions | `Organism`, `Antibiotic`, `OrganismAntibiotics`, `MolecularOrganismAntibiotics` plus microbiology handling inside generic report-submit flows | Owns source-of-truth behavior for master data, billing defaults, and report submission | Master Data Model Notes [#master-data-model-notes] Organisms maintain two relationships to antibiotics: * `OrganismAntibiotics`: standard antibiogram mappings that carry RIS breakpoint values (diameter and MIC lower/upper per R/I/S category). This is what the `Microbiology Ranges` tab inside Organism Master populates. * `MolecularOrganismAntibiotics`: molecular-specific ordered and activatable mappings with `sequence` and `is_active` fields. This is managed under the `Molecular Mapping` sub-tab and is used by the Molecular feature's Antibiotic Resistance component. Key `OrganismAntibiotics` breakpoint fields populated by `Microbiology Ranges`: | Field | Description | | :------------------------------------------------------------ | :------------------------------------------------- | | `resistance_diameter_lower` / `resistance_diameter_upper` | Disk diffusion diameter range for **Resistant** | | `intermediate_diameter_lower` / `intermediate_diameter_upper` | Disk diffusion diameter range for **Intermediate** | | `sensitive_diameter_lower` / `sensitive_diameter_upper` | Disk diffusion diameter range for **Sensitive** | | `resistance_mic_lower` / `resistance_mic_upper` | MIC range for **Resistant** | | `intermediate_mic_lower` / `intermediate_mic_upper` | MIC range for **Intermediate** | | `sensitive_mic_lower` / `sensitive_mic_upper` | MIC range for **Sensitive** | Key Features [#key-features] * Organism Master list with organism name, sample type, organism category, organism code, and status. * Antibiotic Master list with antibiotic name, sample type, antibiotic category, antibiotic code, and status. * Organism `Antibiotic Mapping` tab with two sub-tabs: `Molecular Mapping` (for molecular workflows) and `Microbiology Ranges` (for microbiology susceptibility breakpoints). * `Microbiology Ranges` sub-tab shows an editable grid per antibiotic with R, I, S rows, each configurable with Diameter Upper, Diameter Lower, MIC Upper, and MIC Lower fields. * Microbiology report parameter component configured with Referring List, Max Number Of Micro Organisms, Type Of Method, and column Configuration. * At report entry: the `Add Organism` dropdown is capped by the configured `max_allowed_microorganism`; adding beyond the limit shows an alert and blocks addition. * Once an organism is added at report entry, the user can add antibiotics (drawn from the organism's `OrganismAntibiotics` mappings) to each organism section. * Entering a result value in `result_1` triggers automatic interpretation (`result_2`) based on the configured method type and RIS breakpoints. * `result_r`, `result_i`, `result_s` columns are read-only and display the configured breakpoint ranges for reference. * `Interpretation Only` method type hides result and RIS columns; only organism name and manual interpretation are shown. # Workflow Guide Workflow Guide [#workflow-guide] This section explains how the Microbiology feature is used in practice before diving into implementation details. import Image from "next/image"; import testTypeImg from "@/images/microbiology/test-type-microbiology.png"; import parameterConfigImg from "@/images/microbiology/microbiology-parameter-config.png"; import rangesImg from "@/images/microbiology/microbiology-ranges.png"; > **Important:** Document Db should be enabled for this feature to work. Prerequisite Master Data Setup [#prerequisite-master-data-setup] Before Microbiology configuration is used, the lab needs prerequisite master data for organisms and antibiotics. Where the user goes [#where-the-user-goes] 1. Open the application sidebar. 2. Expand `Drug Master / Panel Master`. 3. Open `Antibiotic Master` or `Organism Master` depending on the master data being configured. What the controls do [#what-the-controls-do] | Control | What it does | | :---------------------------------------- | :--------------------------------------------------- | | `Add Antibiotic` | Opens antibiotic creation flow | | `Add Organism` | Opens organism creation flow | | `Download List` | Downloads visible master data | | `Bulk Actions` | Applies supported operations to selected master rows | | `Request New ... For System Default List` | Requests a new system default antibiotic or organism | Antibiotic Master [#antibiotic-master] Antibiotic Master defines the antimicrobial drugs used in susceptibility testing. The list view shows: * antibiotic name, * sample type, * antibiotic category, * antibiotic code, * status/action controls. Antibiotics are mapped to organisms for standard antibiogram purposes. When disabling an antibiotic, the system checks for any linked organisms and displays them in a warning modal to prevent accidental disruption of antibiogram mappings. Organism Master [#organism-master] Organism Master defines the microorganisms that may be detected in a clinical sample. The list view shows: * organism name, * sample type, * organism category, * organism code, * status/action controls. When creating or updating an organism, the user can map antibiotics to the organism. The `Antibiotic Mapping` tab inside the organism creation/edit modal contains two sub-tabs: 1. **Molecular Mapping** — used by the Molecular feature's Antibiotic Resistance component. Antibiotics are added and prioritized with drag-and-drop sequence ordering. 2. **Microbiology Ranges** — used by the Microbiology feature's report-entry antibiogram grid. Antibiotics are added here and each mapping has an editable RIS range grid. Microbiology Ranges Tab [#microbiology-ranges-tab] Organism Master - Microbiology Ranges Tab After adding an antibiotic to the `Microbiology Ranges` sub-tab, the grid shows three rows per antibiotic — **R** (Resistant), **I** (Intermediate), and **S** (Sensitive) — each with four configurable fields: | Column | Description | | :--------------- | :------------------------------------------------------------------------------- | | `Diameter Upper` | Upper bound of disk diffusion diameter (mm) for this susceptibility category | | `Diameter Lower` | Lower bound of disk diffusion diameter (mm) for this susceptibility category | | `MIC Upper` | Upper bound of Minimum Inhibitory Concentration for this susceptibility category | | `MIC Lower` | Lower bound of Minimum Inhibitory Concentration for this susceptibility category | The system validates that lower values are less than upper values. These configured ranges are used at report entry to auto-interpret the entered result as Sensitive, Intermediate, or Resistant. **How RIS interpretation works at report entry:** * When **Detection Window** is selected as the method type, the entered `result_1` value is compared against the diameter breakpoints. * If the value falls within `sensitive_diameter_lower ≤ result_1 ≤ sensitive_diameter_upper`, the interpretation is **Sensitive**. * If the value falls within `intermediate_diameter_lower ≤ result_1 ≤ intermediate_diameter_upper`, the interpretation is **Intermediate**. * If the value falls within `resistance_diameter_lower ≤ result_1 ≤ resistance_diameter_upper`, the interpretation is **Resistant**. * When **MIC** is selected, the same logic applies using the corresponding `_mic_lower` and `_mic_upper` fields. * When **Interpretation Only** is selected, there is no automatic calculation; the user manually enters the interpretation text. > **Note:** The `result_r`, `result_i`, and `result_s` columns in the report entry grid are read-only and show the configured breakpoint ranges as reference strings in the format `lower - upper`. When disabling an organism, the system checks for linked antibiotics and displays them in a warning modal. Microbiology Report / Test Setup [#microbiology-report--test-setup] After master data is configured, create a microbiology report from the Test List. Where the user goes [#where-the-user-goes-1] 1. Open `Profile & Report Management`. 2. Open `Test List`. 3. Click `Add New Test`. 4. On `Test Information`, set `Test Type` to `Microbiology`. 5. Save the test/report after report parameters are configured. Selecting `Microbiology` as the test type (internally `isRadiology: 4`, `test_type: "Microbiology"`) enables the microbiology-specific report component in the `Report Parameters` tab. Test List - Add New Test - Microbiology Test Type Microbiology Report Component [#microbiology-report-component] The `Add New Parameter` menu exposes the `Microbiology` category with one sub-component: **Microbiology Parameter** (`component_type: "microbiology"`, `type: 20`). Microbiology Parameter Configuration Microbiology Form Fields [#microbiology-form-fields] The Microbiology Parameter form includes the following configuration fields: | Field | Purpose | | :-------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Component Name** | Display name for the microbiology section in the report | | **Referring List** | The master data source (must be set to `Organisms`) | | **Max Number Of Micro Organisms** | Maximum number of organisms a report-entry user can add (1–8). If the user tries to add more, the system shows an alert and blocks the addition | | **Type Of Method** | Determines how result interpretation is computed: `Detection Window` (disk diffusion diameter), `MIC` (Minimum Inhibitory Concentration), or `Interpretation Only` (manual text) | | **Configuration tab** | Lets the user select and arrange display columns for the antibiotic grid | | **Meta tab** | Controls `Render Pivot Table on PDF`, `Sort By`, and `Order By` for the pivot output | > **Note:** If `Max Number Of Micro Organisms` is set to `1`, the `Render Pivot Table on PDF` checkbox is automatically disabled. Max Number Of Micro Organisms Enforcement [#max-number-of-micro-organisms-enforcement] At report entry, when the user selects an organism from the `Add Organism` dropdown, the system counts the current number of distinct organisms already added. If the count equals or exceeds the configured `max_allowed_microorganism`, an alert is shown: > *"Maximum \{n} organism allowed"* and the organism is not added. Type Of Method Behavior [#type-of-method-behavior] | Method | Columns shown | Auto-interpretation | | :-------------------- | :-------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------- | | `Detection Window` | `result_1` (editable), `result_2` (auto-filled), `result_r` / `result_i` / `result_s` (read-only range display) | Yes — result is matched against diameter breakpoints | | `MIC` | Same column set as Detection Window | Yes — result is matched against MIC breakpoints | | `Interpretation Only` | `name` + `result_2` (manual interpretation) only; result and RIS columns are hidden | No — user types interpretation manually | When the Method Type is changed, `updateMicroEditables(...)` is called to hide or unhide the result/RIS columns accordingly. For `Interpretation Only` on a new report, the configuration is reduced to only `name` and `result_2`. Primary User Workflow [#primary-user-workflow] 1. User opens `Drug Master / Panel Master`. 2. User ensures required antibiotics are present in `Antibiotic Master`. 3. User creates organisms in `Organism Master`. 4. Inside each organism's `Antibiotic Mapping` tab, user opens the `Microbiology Ranges` sub-tab, searches and adds antibiotics, and configures the R/I/S diameter and MIC breakpoints. 5. User opens `Profile & Report Management > Test List`. 6. User creates a new test and selects `Microbiology` as the test type. 7. User adds the `Microbiology Parameter` component from the `Add New Parameter` menu. 8. User sets `Referring List` to `Organisms`, sets `Max Number Of Micro Organisms`, and selects `Type Of Method`. 9. User configures display columns in the `Configuration` tab. 10. Backend validates and persists the master data and report configuration. 11. At report entry, the user selects organisms from the `Add Organism` dropdown (limited by `max_allowed_microorganism`), and for each organism adds antibiotics from the organism's configured mapping. 12. When the user enters a result value in `result_1`, the system automatically fills in the `result_2` interpretation based on the configured RIS ranges and method type. Validation And Edge Cases [#validation-and-edge-cases] | Case | Expected behavior | Notes | | :---------------------------------------------------- | :--------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------- | | Adding more organisms than the max | Alert shown: "Maximum \{n} organism allowed"; organism not added | Enforced at report-entry level using `max_allowed_microorganism` from component meta | | Duplicate antibiotic mapping | Save blocked with "Antibiotic already exists" | Enforced during organism save on the `Microbiology Ranges` tab | | Disable antibiotic with linked organisms | Warning modal displays linked organisms | User must confirm to proceed; mappings are cleared upon disable | | Disable organism with linked antibiotics | Warning modal displays linked antibiotics | User must confirm to proceed; mappings are cleared upon disable | | Microbiology component without Microbiology test type | Component does not appear in the menu | Handled by frontend test type condition | | Interpretation Only method type | Result and RIS columns are hidden; only name and manual interpretation are shown | Controlled by `updateMicroEditables(...)` | | RIS ranges not configured | `result_r`, `result_i`, `result_s` display `- - -` (empty lower/upper) | No error; interpretation stays blank if no range matches | | Max organisms = 1 | `Render Pivot Table on PDF` checkbox is disabled and `should_pivot` is forced to `false` | Enforced in `MicrobiologyForm` onChange handler | | Missing Referring List or Method Type | Validation error shown; save blocked | Enforced by `reportParamterValidation` in `helpers.ts` | # Backend Backend [#backend] import Image from 'next/image' import activityLog from '@/images/activity-log.png' Architecture Overview [#architecture-overview] Missing Details spans `livehealth-frontend`, `crelio-app` (PY-3), and legacy `livehealthapp` (PY-2). Config management and list/patch APIs are in PY-3; some entity creation and legacy flows still run through PY-2 controllers. System Design Diagram [#system-design-diagram] Storage & Models [#storage--models] Primary tables: \-- `MissingDetailsTemplates` — master template catalog. \-- `LabMissingDetailsConfiguration` — per-lab active config rows (soft-disable semantics). \-- `LabMissingDetails` — runtime missing-detail incidents (is\_resolved flag, placeholder value). \-- `MissingDetailsEntityCreation` — trace rows for entity creation caused by config save. Key model locations: * [`missing_details_templates.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/models/missing_details_templates.py) * [`lab_missing_details_configuration.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/models/lab_missing_details_configuration.py) * [`lab_missing_details.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/models/lab_missing_details.py) * legacy mirrors in [`livehealthapp/labs/models.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/labs/models.py) Core backend responsibilities [#core-backend-responsibilities] | Concern | Backend responsibility | | ------------------ | ---------------------------------------------------------------------------- | | Source of truth | Persist templates, lab configs, runtime missing rows, entity creation traces | | Auto-resolution | Compare stored placeholder vs actual incoming value | | Scope enforcement | Lab/org restrictions on list endpoints | | Flag propagation | Attach `has_missing_fields` to report/patient/bill payloads | | Bootstrap entities | Queue or create domain entities for entity-backed defaults | | Activity logging | Record queueing, success, failure, and creation traces | Runtime engine & resolution [#runtime-engine--resolution] Auto-resolution flow: 1. Find unresolved `LabMissingDetails` rows for the patient/bill context. 2. Map the missing field into a lookup key via `MISSING_FIELDS_MODEL_FIELD_MAPPER`. 3. Check if that key exists in `values_map`. 4. Compare the incoming actual value with the stored placeholder value. 5. If the values differ, mark the row resolved and set `resolved_at` and resolver identity where available. Important helpers: * `build_value_mapper(...)` — flattens saved objects into comparable strings. * `create_or_update_missing_details_entries(...)` — main create/resolve engine. * `MISSING_FIELDS_MODEL_FIELD_MAPPER` — maps template names to object fields for resolution. Important Missing Details Snippets (Backend) [#important-missing-details-snippets-backend] These examples show the core backend resolution and manual-resolve patterns. Auto-resolution loop (PY-3) [#auto-resolution-loop-py-3] Source: [`utils.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/utils.py) ```py for field_id, config in config_map.items(): existing = existing_map.get(field_id) if not existing: continue search_key = lookup_key( getattr(existing.field, "module", ""), getattr(existing.field, "name", ""), ) if not search_key or search_key not in values_map: continue actual_value = values_map.get(search_key) if actual_value == existing.field_value: continue existing.is_resolved = True existing.resolved_at = now existing.resolved_by_doctor_id = doctor_id or None existing.resolved_by_lab_user_id = lab_user_id or None to_update.append(existing) ``` Why it matters: maps template → object key, compares incoming value vs placeholder, and marks rows resolved when they differ. Manual resolution (PATCH → model) [#manual-resolution-patch--model] Source: [`lab_missing_details_view.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/views/lab_missing_details_view.py) and [`lab_missing_details.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/models/lab_missing_details.py) ```py # View layer updated_count = LabMissingDetails.resolve( ids=ids, lab_id=lab_id, resolved=is_resolved, actor="doctor" if request.session.get("is_doctor", None) else "lab_user", ) # Model layer @classmethod def resolve(cls, *, ids: list[int], lab_id: int, resolved: bool, actor=None) -> int: qs = cls.objects.filter(id__in=ids, lab_id=lab_id) now = timezone.now() update_data = {"is_resolved": resolved, "updated_at": now} if resolved: update_data["resolved_at"] = now else: update_data["resolved_at"] = None update_data["resolved_by_doctor"] = None update_data["resolved_by_lab_user"] = None with transaction.atomic(): return qs.update(**update_data) ``` Why it matters: bulk resolution with lab-scope enforcement and consistent audit fields. APIs [#apis] | Method | Endpoint | Purpose | | :------ | :--------------------------------------------------------- | :-------------------------------- | | `GET` | `/api-v3/registration/patients/missing-details-templates` | Fetch master templates | | `GET` | `/api-v3/registration/patients/lab-missing-details-config` | Fetch active lab configs | | `POST` | `/api-v3/registration/patients/lab-missing-details-config` | Create/update/disable configs | | `GET` | `/api-v3/account/missing-details/lab-fields` | Fetch runtime missing-detail rows | | `PATCH` | `/api-v3/account/missing-details/lab-fields` | Mark rows resolved/unresolved | Runtime capture piggybacks on: * `POST /api-v3/registration/patients/new` — patient create/update path. * `POST /api-v3/finance/bill/{labBillId}/update` — billing update path. Entity creation and logging [#entity-creation-and-logging] Two tracks: 1. PY-3 local creation: `LabMissingDetailsConfiguration.process_field(...)` may create insurance/group entities locally and logs via `ActivityLog`. 2. Legacy PY-2 webhook: PY-3 queues a Fusion webhook; PY-2 controller executes creation and `MissingFieldsActivityLogger` writes `ActivityLog` and `MissingDetailsEntityCreation`. Key backend locations & helpers [#key-backend-locations--helpers] | Area | Function / class | Path | Role | | ---------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | PY-3 config API | `LabMissingDetailsConfigView` | [`missing_details_view.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/views/missing_details_view.py) | Read/write active lab config | | PY-3 config model | `LabMissingDetailsConfiguration` | [`lab_missing_details_configuration.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/models/lab_missing_details_configuration.py) | Config persistence and entity bootstrap | | PY-3 runtime engine | `create_or_update_missing_details_entries` | [`utils.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/utils.py) | Create/resolve runtime rows | | PY-3 value mapper | `build_value_mapper` | [`utils.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/utils.py) | Flatten patient data into comparable strings | | PY-3 mapper contract | `MISSING_FIELDS_MODEL_FIELD_MAPPER` | [`constants.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/constants.py) | Bridge between template names and object fields | | PY-3 list/patch API | `LabMissingDetailsView` | [`lab_missing_details_view.py`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/views/lab_missing_details_view.py) | Missing detail listing and resolution | | Legacy entity endpoint | `missing_field_controller` | [`registration/api.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/livehealth_4/registration/api.py) | Dispatch entity creation to the right py2 controller | | Legacy activity logger | `MissingFieldsActivityLogger` | [`utils.py`](https://github.com/CrelioHealth/livehealthapp/blob/develop/livehealth_4/utils/utils.py) | Log entity creation success/failure and create `MissingDetailsEntityCreation` | Example: Activity log for entity creation [#example-activity-log-for-entity-creation] This screenshot shows an activity log entry validating entity creation triggered by a Missing Details configuration. Activity log - entity creation Safe SQL / Debugging Cheatsheet [#safe-sql--debugging-cheatsheet] \-- Active configs for a lab ```sql SELECT id, field_name, field_value, field_module, field_category, is_disabled FROM LabMissingDetailsConfiguration WHERE lab_id = ? AND is_disabled = 0 ORDER BY field_name; ``` \-- Runtime unresolved missing details for a patient ```sql SELECT id, field_name, field_value, field_module, is_resolved, created_at, resolved_at FROM LabMissingDetails WHERE lab_id = ? AND user_details_id = ? AND is_resolved = 0 ORDER BY created_at DESC; ``` \-- Entity creation traces caused by config save ```sql SELECT id, config_id, entity_name, object_id, lab_id, org_id, created_at FROM MissingDetailsEntityCreation WHERE lab_id = ? ORDER BY created_at DESC; ``` # Design Decisions Missing Details Design Decisions [#missing-details-design-decisions] Key design constraints, architectural rationale, tradeoffs, and extensibility notes for the Missing Details feature across `livehealth-frontend`, `crelio-app`, and `livehealthapp`. Design Intent [#design-intent] Missing Details was not built as a "nice-to-have data entry shortcut". It was built as a controlled failure mode for registration and billing. The core problem is operational, not cosmetic: * labs cannot afford to hard-stop high-volume registration or billing every time one field is unavailable, * but they also cannot silently accept incomplete data and lose the fact that something was missing, * and downstream teams still need visibility so those gaps can be corrected before they become reporting, billing, or compliance issues. The design premise is simple: > let the workflow continue, but make incompleteness explicit, queryable, reversible, and eventually resolvable. That single premise explains most of the implementation. Decision Stack [#decision-stack] Key Design Constraints [#key-design-constraints] | Constraint | Why it is real | Architectural consequence | | :---------------------------------------------------------------------------- | :--------------------------------------------------------------------------- | :------------------------------------------------------------- | | Front-desk throughput matters more than perfect first-pass completeness | Registration and order creation are operational bottlenecks | Hard validation failure is not the primary strategy | | Missing data is not uniformly bad | Some fields can be safely deferred, others need downstream visibility | Feature is selective and config-driven, not blanket permissive | | Labs vary in tolerated defaults and allowed missing fields | Labs operate differently and across different geographies/workflows | Configuration is per-lab, not hardcoded globally | | Patient gaps and order gaps are not the same thing | A missing DOB and a missing order number do not belong to the same lifecycle | Runtime rows need explicit module semantics | | Existing workflows span py3 and py2 | A clean-sheet rewrite would have delayed usable rollout | The feature intentionally straddles both stacks | | Downstream screens mostly need a badge flag, not full missing-detail payloads | Waiting lists and dashboards must stay light | Boolean `has_missing_fields` flags are propagated separately | Architectural Rationale [#architectural-rationale] 1. Workflow continuity was preferred over hard blocking [#1-workflow-continuity-was-preferred-over-hard-blocking] The first design fork was straightforward but high stakes: Should the system reject registration/billing when a field is missing, or should it let the workflow continue under controlled conditions? We chose controlled continuation. Why this was preferred [#why-this-was-preferred] Hard blocking looks neat in theory and behaves badly at a live counter. In practice it causes: * queue buildup, * billing delays, * front-desk workarounds, * junk values being typed just to satisfy validation, * and a general loss of trust in the product. That is the wrong optimization target. The system should protect data quality without forcing operational deadlock. So the chosen tradeoff was: * allow continuation, * but only for a known set of fields, * only with lab-approved defaults, * and only while creating an explicit unresolved record. Missing Details is therefore not a generic "ignore validation" switch. It is a controlled bypass with traceability attached. Alternative we did not choose [#alternative-we-did-not-choose] `Alternative: make the fields mandatory and reject the save` Why it was not chosen: * operationally brittle, * encourages fake-value workarounds, * gives no structured way to revisit deferred data, * and converts a data-quality problem into a throughput problem. 2. Config-time and runtime concerns were deliberately separated [#2-config-time-and-runtime-concerns-were-deliberately-separated] One of the strongest design choices in this feature is the separation between setup state and business incidents. | Concern | Table | Question being answered | | :---------------- | :------------------------------- | :---------------------------------------------------------------- | | Template-time | `MissingDetailsTemplates` | Which fields are even eligible for this feature? | | Config-time | `LabMissingDetailsConfiguration` | Which of those fields does this lab allow, and with what default? | | Runtime | `LabMissingDetails` | Which patient/order actually used the missing-field path? | | Side-effect trace | `MissingDetailsEntityCreation` | Which config caused a domain entity to be created? | Why this split was preferred [#why-this-split-was-preferred] If config and runtime were fused into one table: * config edits could distort the meaning of old incidents, * disabled configs would become awkward lifecycle flags, * historical analysis would become muddier, * and runtime queries would drag around setup metadata they do not actually own. Separating the layers keeps the model sane: * templates define vocabulary, * config defines lab behavior, * runtime rows capture actual incompleteness, * trace rows capture config-side side effects. 3. A global template catalog was preferred over lab-defined free-form fields [#3-a-global-template-catalog-was-preferred-over-lab-defined-free-form-fields] It would have been possible to let every lab invent its own missing-field definitions from scratch and skip the master template layer. That sounds flexible. It is also how you end up with a support nightmare. Why the template layer exists [#why-the-template-layer-exists] The global `MissingDetailsTemplates` catalog gives the system a stable vocabulary: * field names stay canonical, * categories stay broadly aligned, * modules stay explicit, * resolution mapping can be written against known names, * frontend enums and backend logic can target the same identifiers. Without that layer: * each lab would effectively invent its own DSL, * resolution would degrade into string-matching chaos, * analytics would become inconsistent, * and cross-lab rollout of new supported fields would become manual and error-prone. Tradeoff we accepted [#tradeoff-we-accepted] The template layer reduces free-form flexibility. That is intentional. Missing Details is meant to be configurable, not anarchic. 4. Placeholder values are stored on the runtime row, not only in config [#4-placeholder-values-are-stored-on-the-runtime-row-not-only-in-config] At first glance, storing `LabMissingDetails.field_value` can look redundant because the config row already has a default value. It is not redundant. It is necessary. Why this was preferred [#why-this-was-preferred-1] The runtime row must capture the value that was actually used when the incident occurred. That matters because: * lab config defaults can change later, * unresolved old rows still need to compare against the original placeholder, * and resolution logic needs a deterministic baseline for `actual_value != stored_missing_value`. If runtime rows only referenced the current config value: * a config edit could silently rewrite the meaning of history, * old unresolved rows could begin comparing against the wrong placeholder, * and auto-resolution would become nondeterministic over time. This is a classic configuration-vs-event boundary: runtime incidents must preserve the effective value that existed at the time they were created. 5. Patient-module and order-module missing details were modeled separately [#5-patient-module-and-order-module-missing-details-were-modeled-separately] Missing data in this feature is not one homogeneous thing. Some gaps belong to the patient profile, others belong to a specific order/bill. That distinction is reflected in: * template `module`, * config `field_module`, * runtime `field_module`, * frontend tabs, * resolution scope, * and list filtering. Why this split was preferred [#why-this-split-was-preferred-1] If everything were modeled as patient-scoped: * order fields like test, ICDs, pre-auth number, and order number would have ambiguous ownership, * resolution would be too broad, * and UI semantics would get muddy. If everything were modeled as bill-scoped: * patient-level incompleteness would be duplicated per bill, * longitudinal patient gaps would fragment, * and downstream badge logic would get noisy. So the model uses a clean split: * `patient` module = persistent profile-like gaps, * `order` module = bill-specific incompleteness. 6. Configs are soft-disabled instead of deleted [#6-configs-are-soft-disabled-instead-of-deleted] Disabling a missing field in setup sets `is_disabled = 1`. It does not remove the config row. Why this was preferred [#why-this-was-preferred-2] Soft-disable preserves: * historical interpretation, * traceability from `MissingDetailsEntityCreation.config_id`, * audit continuity, * and rollback friendliness. Hard delete would be cheaper in the short term and more expensive in every debugging, support, and audit conversation after that. For a feature whose purpose is to track incomplete business data, historical continuity is more valuable than storage neatness. 7. Resolution is hybrid: implicit where possible, manual where necessary [#7-resolution-is-hybrid-implicit-where-possible-manual-where-necessary] The feature supports two resolution paths: 1. implicit resolution through normal business flows, 2. manual resolution through the Missing Data UI. Why implicit resolution was preferred [#why-implicit-resolution-was-preferred] If every resolved field required a second explicit user step: * many rows would stay unresolved forever, * the Missing Data list would become stale, * badge accuracy would drift, * and the system would accumulate long-dead missing incidents. The backend already has the updated patient/order values in hand during save flows. Not using that context to resolve existing rows would be wasted signal. Why manual resolution was retained [#why-manual-resolution-was-retained] Implicit resolution is not enough on its own. There are plenty of real-world cases where: * the correction happened through a side workflow, * the backend does not have a rich enough values map to compare safely, * or the operator knows the issue is operationally resolved even if the system cannot infer it cleanly. That is why the manual `PATCH /api-v3/account/missing-details/lab-fields` path remains necessary. The design is intentionally hybrid: * let the system self-heal when it can, * let humans close the loop when it cannot. 8. A mapper bridge was preferred over field-specific resolution branches [#8-a-mapper-bridge-was-preferred-over-field-specific-resolution-branches] `MISSING_FIELDS_MODEL_FIELD_MAPPER` is one of the key architecture choices in the feature. What problem it solves [#what-problem-it-solves] The product thinks in business labels: * `Patient Name` * `Order Number` * `Bill ICD codes` The backend objects think in model fields: * `fullName` * `order_number` * `bill_icd` The mapper is the translation layer between those two worlds. Why this was preferred [#why-this-was-preferred-3] Without a mapper, the resolution engine would degenerate into scattered special-case conditionals: * `if field_name == "Patient Name": use obj.fullName` * `if field_name == "Address": use obj.area` * and so on. That would spread field knowledge across multiple save flows and make extension ugly. With a mapper: * the engine stays generic, * adding supported fields becomes largely a mapping exercise, * and the business vocabulary remains stable even when object shapes differ. Tradeoff we accepted [#tradeoff-we-accepted-1] The mapper can drift. That is real. But centralized drift is still better than distributed branch logic drift. 9. Downstream systems get a badge flag, not full missing-detail payloads [#9-downstream-systems-get-a-badge-flag-not-full-missing-detail-payloads] Most downstream screens do not need the full `LabMissingDetails` row set. They only need to answer: > should I show a Missing Data badge here? Why the flag approach was preferred [#why-the-flag-approach-was-preferred] Pushing full missing-detail payloads into every waiting list, dashboard, accession, or report screen would: * inflate payload size, * duplicate data-fetch work, * complicate rendering logic, * and couple simple badge consumers to the full missing-detail model. The chosen design is intentionally cheap: * compute unresolved existence once, * attach `has_missing_fields`, * let the screen render a red badge. The detailed list still exists separately because the Missing Data module genuinely needs row-level detail. That separation keeps edge consumers light without taking away investigative depth. 10. Runtime rows are lab-owned first, org-scoped second [#10-runtime-rows-are-lab-owned-first-org-scoped-second] The feature is modeled as lab-owned first. `org_id` exists where needed, but `lab_id` remains the primary partition. Why this was preferred [#why-this-was-preferred-4] Labs are the operational owners of: * field configuration, * registration infrastructure, * billing workflows, * and activity traceability. Organization visibility matters, but it is not the root tenancy model. That means: * lab ownership stays clear, * org filtering can be layered in where needed, * and org/CC login behavior can remain access semantics rather than becoming the primary data model. This is subtle, but it is the correct bias for how the product actually operates. 11. Entity-backed defaults are bootstrapped at config time, not at registration/billing time [#11-entity-backed-defaults-are-bootstrapped-at-config-time-not-at-registrationbilling-time] Some missing fields are not plain scalars. They correspond to real entities such as referral, organization, test, ICD, insurance, or group. The chosen design is: * when config is created, optionally create or queue creation of the backing entity, * then let runtime workflows reference something that already exists. Why this was preferred [#why-this-was-preferred-5] Doing entity creation inside registration or billing would have been the wrong coupling: * save paths would become slower and more failure-prone, * hot operational workflows would inherit catalog-creation side effects, * and the feature would blur the line between "this data is missing" and "also create a new master record right now." By moving entity bootstrap closer to config-time: * the lab defines allowed missing-field behavior once, * expensive or legacy entity creation happens outside the hot path, * and runtime behavior becomes simpler and more predictable. This is also why the feature tolerates a py3/py2 split. Some entities already have usable py3 creation paths. Others still rely on mature py2 controllers. The design chose interoperability over waiting for a perfect migration boundary. 12. Activity logging is separate from business-state storage [#12-activity-logging-is-separate-from-business-state-storage] The feature writes to both business tables and activity logs, but it does not confuse those concerns. Why this was preferred [#why-this-was-preferred-6] Each store answers a different question: * `LabMissingDetails` = business state, * `ActivityLog` = operator/system trace, * `MissingDetailsEntityCreation` = config-side side-effect trace. Overloading one into the other would make both querying and debugging noisier. Keeping them separate makes it easier to answer: * "is this row unresolved?", * "who queued the entity create?", * "which config created this domain entity?" Those are different questions and deserve different storage shapes. Decision Boundaries [#decision-boundaries] Alternatives We Deliberately Did Not Choose [#alternatives-we-deliberately-did-not-choose] | Alternative | Why it looked attractive | Why we did not choose it | | :----------------------------------------------------------- | :------------------------ | :--------------------------------------------------------------------- | | Hard-block all saves when required fields are missing | Simple validation story | Breaks operational throughput and encourages fake values | | Let users type any placeholder ad hoc | Minimal engineering work | Destroys consistency, traceability, and implicit-resolution quality | | Store config and incidents in one table | Fewer tables | Mixes setup with runtime history and makes lifecycle handling messy | | Manual resolution only | Easy to reason about | Produces stale unresolved rows and too much operator overhead | | Implicit-resolution only, no manual override | Cleaner state model | Unrealistic for operational edge cases and side-channel corrections | | Create backing entities on every runtime use | Single-step behavior | Bloats hot paths and couples catalog creation to front-desk throughput | | Push full missing-detail payloads into every downstream list | Rich data everywhere | Payload bloat and unnecessary coupling for badge-only consumers | | Rewrite everything into py3 first | Architectural cleanliness | Delays feature value behind platform migration work | Robustness, Reusability, and Tradeoffs [#robustness-reusability-and-tradeoffs] The implementation is not tiny, but the tradeoffs are coherent. Robustness highlights [#robustness-highlights] | Decision | Why it matters | | :------------------------------------------------------ | :----------------------------------------------------------- | | Soft-disable config | Historical rows and config lineage survive changes | | Separate runtime incident row | Missing state is queryable independent of current form state | | Implicit resolution compares against stored placeholder | Deterministic enough to avoid magic guesswork | | Downstream badge flags | Keeps rendering cheap and scalable | | Hybrid implicit + manual resolution | Handles both clean and messy correction paths | | Entity creation trace table | Makes config-side side effects debuggable | Reusability and extensibility [#reusability-and-extensibility] The feature is reasonably extensible because it is built out of generic primitives: * a master field catalog, * a per-lab config layer, * a mapper bridge, * a generic runtime incident table, * a list/resolve API, * and reusable frontend checkbox + auto-fill mechanics. That makes these changes relatively straightforward: | Change type | What usually needs to happen | | :--------------------------------- | :--------------------------------------------------------------------------------- | | Add a new missing field | Add template, frontend enum/support, mapper entry if implicit resolution is needed | | Add a new value-only patient field | Mostly frontend injection + backend mapper work | | Add a new order field | Same, but order-module aware | | Add a new entity-backed field | Template + config handler + mapper + optional entity creation path | | Add a new list consumer | Reuse `has_missing_fields` badge pattern or call the Missing Data list API | Negative consequences we accepted [#negative-consequences-we-accepted] * The implementation spans more files than a single-module feature. * Engineers need both py3 and py2 mental models for full-stack debugging. * Mapper drift becomes a maintenance concern, especially for entity-backed fields. * Some fields will always be awkward because domain object shapes do not line up neatly. That is the cost of choosing operational fit and staged migration over architectural minimalism. Final Take [#final-take] The Missing Details design is opinionated in the right places. It does not try to make incomplete data disappear.\ It does not overreact by blocking the workflow completely.\ It does not treat setup and runtime as the same thing.\ It does not assume users will remember to clean everything up manually.\ It does not pretend the stack migration is more important than feature value. Instead, it chooses a very product-engineering middle path: * accept messy operational reality, * constrain it with configuration, * record it explicitly, * surface it downstream, * and let the system heal itself when real data catches up. That is the real design philosophy behind Missing Details. # Frontend Frontend [#frontend] What frontend owns [#what-frontend-owns] | Concern | Frontend responsibility | | :---------------- | :------------------------------------------------------------------------------------------ | | Config UX | Load templates/configs, let user add/edit/remove configs, validate obvious input issues | | Runtime capture | Show checkboxes, inject defaults, remember which config ids were used | | UI replay | Re-open unresolved missing details into the form using `fetchAndPopulateMissingFields(...)` | | Discoverability | Show `Missing Data` tags in grids and detail flows | | Manual resolution | Call patch APIs from modals or action screens | Frontend perspective (runtime) [#frontend-perspective-runtime] * Let staff keep the workflow moving instead of blocking registration or billing because one field is missing. * Show a `Missing` checkbox only for fields the lab has explicitly configured. * Auto-fill a known placeholder/default value when the operator marks a field as missing. * Keep enough generic state to know which config entry was used, what value was injected, and whether the UI should still show the checkbox or switch to a `Missing Data` tag. * Show unresolved missing details on the waiting list, dashboards, accession pages, and the `Missing Data` tab. * Give staff one-click routes into patient update or order update from the Missing Data tab. Core frontend state objects [#core-frontend-state-objects] | State key | What it stores | Why it exists | | :--------------------------------- | :--------------------------------------------------- | :--------------------------------------------------------------------- | | `missingFieldState` | Whether a field is currently marked as missing | Source of truth for checked/unchecked behavior | | `missingFieldUIVisible` | Whether the checkbox should still be shown vs hidden | Lets the UI swap between checkbox and `Missing Data` label | | `missingFieldConfigUsed` | Config row id keyed by field name | This is what the backend later receives as selected missing config ids | | `allowMissingFieldsInRegistration` | Gate for collection-centre visibility | Prevents unsupported CC flows from showing missing checkboxes | Additional: | State key | What it stores | Where it matters | | :--------------------------- | :------------------------------------------------------ | :------------------------------------------- | | `patientMissingFieldEntries` | Map of patient/bill id to unresolved field names | CC-specific filtering and prepopulation | | `currentPatientMissingId` | Which patient is currently being replayed into the form | Used when reopening existing missing records | Checkbox visibility rules [#checkbox-visibility-rules] The checkbox renders only if: * config has been loaded, * the specific field is configured for the lab, * the CC gate allows it, * and, in some CC replay cases, the field is part of `patientMissingFieldEntries[currentPatientMissingId]`. Source: [`MissingFieldsCheckBox/index.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/components/MissingFieldsCheckBox/index.tsx) Runtime UI behaviors [#runtime-ui-behaviors] * When operator toggles a Missing checkbox, `handleMissingToggle` injects the configured value and records the `config id` that was used. * When operator later types a value that differs from the injected placeholder, `checkAndResetMissingField(...)` clears the missing flags so the frontend does not claim the value is still intentionally missing. * On submit, the frontend includes the selected missing config ids in the registration/billing payload (e.g., `labMissingFieldIds`, `labMissingDetailsIds`). Important Missing Details Snippets (Frontend) [#important-missing-details-snippets-frontend] These snippets show the common frontend patterns for wiring fields to the Missing checkbox and for rehydrating unresolved missing fields back into the form. Field + Missing checkbox wiring [#field--missing-checkbox-wiring] Source: [`RegistrationForm.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/Container/RegistrationForm.tsx) and [`missingFields.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/utils/missingFields.ts) ```tsx { updateRegistrationState("nationality", val); updateGenericState(dispatch, "missingFieldUIVisible", { ...missingFieldUIVisible, [MissingField.NATIONALITY]: false, }); checkAndResetMissingField( MissingField.NATIONALITY, val?.value || "", dispatch ); }} otherProps={{ onFocus: () => { updateGenericState(dispatch, "missingFieldUIVisible", { ...missingFieldUIVisible, [MissingField.NATIONALITY]: true, }); }, }} /> handleMissingToggle(id, checked, dispatch) } /> ``` Why it matters: focus shows checkbox, typing hides it, typing a real value clears missing state, and the checkbox delegates auto-fill to `handleMissingToggle`. Rehydrate unresolved missing fields into the form [#rehydrate-unresolved-missing-fields-into-the-form] Source: [`missingFields.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/utils/missingFields.ts) ```ts export const fetchAndPopulateMissingFields = async ( searchId: number, isPatient: boolean = true, clearPreviousState: boolean = true, dispatch: ThunkDispatch ): Promise => { if (clearPreviousState) { dispatch( setGenericState({ missingFieldState: {}, missingFieldSavedValue: {}, missingFieldConfigUsed: {}, patientMissingFieldEntries: {}, currentPatientMissingId: null, }) ); } const response: JsonObject = await getLabMissingDetails( undefined, undefined, !isPatient ? searchId : undefined, isPatient ? searchId : undefined, false, false ); const unresolvedFields: string[] = (response?.results || []) .filter((f: JsonObject) => Number(f?.is_resolved) === 0 && f?.field_name) .map((f: JsonObject) => String(f?.field_name).toUpperCase()); dispatch( setGenericState({ patientMissingFieldEntries: { ...patientMissingFieldEntries, [searchId]: unresolvedFields, }, currentPatientMissingId: isPatient ? searchId : null, }) ); for (const field of response?.results) { if (!field?.field_name) continue; populateMissingFieldState(field.field_name, true, dispatch); updateSavedValue(field.field_name, field.field_value, null, dispatch); handleMissingToggle(field.field_name, true, dispatch); } }; ``` What it does: clears stale state, fetches unresolved rows, stores unresolved field names for the patient/bill, and replays each unresolved field into the UI. Manual Resolution Path from Missing Data UI [#manual-resolution-path-from-missing-data-ui] Key frontend locations & helpers [#key-frontend-locations--helpers] | Area | Function / component | Path | Role | | :------------------ | :------------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------- | | Field checkbox | `MissingFieldCustomCheckBox` | [`MissingFieldsCheckBox/index.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/components/MissingFieldsCheckBox/index.tsx) | Render/hide checkbox, update generic state | | Field enum | `MissingField` enum | [`constants.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/constants.ts) | Canonical field names used across UI helpers | | Runtime mutation | `handleMissingToggle` | [`missingFields.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/utils/missingFields.ts) | Inject default value into form and persist config-used mapping | | Replay helper | `fetchAndPopulateMissingFields` | [`missingFields.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/utils/missingFields.ts) | Reload unresolved records into UI state | | Reset helper | `checkAndResetMissingField` | [`missingFields.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/utils/missingFields.ts) | Clear missing state when user enters a real value | | List API helper | `getLabMissingDetails` | [`helpers.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/Registration/utils/helpers.ts) | Read unresolved/resolved rows | | Patch helper | `patchLabMissingDetailsResolution` | [`helpers.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/Registration/utils/helpers.ts) | Manual resolve call | | Config preload | `fetchLabMissingDetailsConfiguration` | [`helpers.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/utils/helpers.ts) | Load active lab config into model state | | Primary list screen | `MissingDetailsTab` | [`missingDetailsTab.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/Registration/Components/MissingDetails/missingDetailsTab.tsx) | Main Missing Data module | Frontend state lifecycle [#frontend-state-lifecycle] Summary: * Config loaded → Checkbox visible → User marks Missing → Default injected → Tag shown → On real-value input, missing state cleared → Submitted → Restored on later open. # Overview Overview [#overview] Continue patient registration and order creation even when some business-critical details are missing, while keeping a clean audit trail until those details are actually filled. Missing Details [#missing-details] Missing Details is a cross-stack fallback-and-tracking system. It lets the lab move forward when patient or order data is unavailable right now, but it does not pretend the data is complete. Instead, it stores which field was missing, what placeholder/default value was used, where that happened, who did it, and whether the record has been resolved later. In plain engineering terms: this feature is the safety rail between strict data quality and real-world front-desk chaos. The frontend lets the operator mark a field as missing and inject a configured default. The backend persists an unresolved `LabMissingDetails` row. Later, when a real value lands through patient update, order update, accession, or manual resolve, the system flips that row to resolved and keeps the trail intact. Related Jira Tickets [#related-jira-tickets] | Ticket | Title | Notes | | :--------------------------------------------------------- | :-------------- | :----------------------------------------------- | | [`EN-10733`](https://crelio.atlassian.net/browse/EN-10733) | Missing Details | Primary feature ticket referenced for this guide | Prerequisites [#prerequisites] | Requirement | Why it matters | Where it is enforced | | :--------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Template data must exist in `MissingDetailsTemplates` | The config modal can only enable fields that exist as master templates | [models/missing\_details\_templates.py](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/models/missing_details_templates.py), [fixtures/missing\_details\_templates.json](https://github.com/CrelioHealth/crelio-app/blob/develop/fixtures/missing_details_templates.json) | | Lab-level config must be created in `LabMissingDetailsConfiguration` | Checkboxes only appear for configured fields | [LabMissingDetailsConfigView (API)](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/views/missing_details_view.py) | | User must have `missing_details_access` to see the Missing Data module | The sidebar entry and primary UI are permission-gated | [Registration SideBar](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/Registration/SideBar/index.tsx), [Organisation SideBar](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/OrganisationLogin/SideBar/index.tsx) | | Frontend must preload config into model state | The checkbox component hides itself if config is not loaded or the field is not configured | [`fetchLabMissingDetailsConfiguration`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/utils/helpers.ts), [`MissingFieldCustomCheckBox`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/components/MissingFieldsCheckBox/index.tsx) | | For entity-backed defaults, the relevant create flow must be available | Some configured defaults bootstrap a referral/org/test/ICD/insurance/group entity behind the scenes | [`LabMissingDetailsConfiguration.process_field(...)`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/models/lab_missing_details_configuration.py) | | Session must identify lab and sometimes org scope | Missing detail rows are always lab-scoped and optionally org-scoped | [`LabMissingDetailsView`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/views/lab_missing_details_view.py), PY-2 registration/billing flows | What Is It For [#what-is-it-for] Frontend perspective [#frontend-perspective] * Let staff keep the workflow moving instead of blocking registration or billing because one field is missing. * Show a `Missing` checkbox only for fields the lab has explicitly configured. * Auto-fill a known placeholder/default value when the operator marks a field as missing. * Keep enough generic state to know which config entry was used, what value was injected, and whether the UI should still show the checkbox or switch to a `Missing Data` tag. * Show unresolved missing details on the waiting list, dashboards, accession pages, and the Missing Data tab. * Give staff one-click routes into patient update or order update from the Missing Data tab. Backend perspective [#backend-perspective] * Store lab-level defaults separately from actual missing-detail incidents. * Create unresolved `LabMissingDetails` rows when a patient or order is saved with configured missing fields. * Resolve those rows automatically when a real value later differs from the stored placeholder value. * Resolve those rows manually when the UI explicitly marks them resolved. * Attach `has_missing_fields` flags back onto bill and patient payloads so other screens can show red tags. * Create and log dependent entities for default values that are not plain strings, such as referral, organization, test, ICD, insurance, or group. Types Of Missing Entities [#types-of-missing-entities] | Type | Examples | What the default looks like | Extra entity creation needed | Runtime behavior | | :--------------------------------------------- | :------------------------------------------------------------------------------ | :-------------------------------------------------------------------- | :--------------------------- | :---------------------------------------------------------------------------- | | Static / value-only fields | Address, Email, Age, Patient ID, Contact Number, Order Number, Report Remark | Usually a plain string, number, or date | No | Frontend injects the configured value directly into the form | | Select-backed but still straightforward fields | Nationality, Ethnicity, Race, Patient Type | A label/value that the frontend can construct locally | No backend entity creation | Frontend maps the configured value into the expected select object | | Entity-backed patient fields | Referral, Organization, Allergies, Diseases, Clinical History, Insurance, Group | A name or code that should eventually exist as a proper domain entity | Yes, in many cases | Config save can bootstrap the entity; registration/billing then references it | | Entity-backed order fields | Test, Test ICD code, Bill ICD codes, Consulting doctor | A catalog entity or coded object | Yes | Config save can create the entity, then billing/order update can use it | Structure Of Missing Details [#structure-of-missing-details] | Layer | What it stores | Table / state | Why it exists | | :------------------------------------------ | :-------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------- | | Master template layer | Which fields are even eligible for Missing Details | [`MissingDetailsTemplates`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/models/missing_details_templates.py) | Global catalog | | Lab configuration layer | Which templates this lab has enabled and what default value to inject | [`LabMissingDetailsConfiguration`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/models/lab_missing_details_configuration.py) | Per-lab behavior switchboard | | Runtime incident layer (Transactional Data) | Every actual patient/order missing-field event | [`LabMissingDetails`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/models/lab_missing_details.py) | Audit + resolution lifecycle | | Entity creation trace layer | Which config caused a domain entity to be created | [`MissingDetailsEntityCreation`](https://github.com/CrelioHealth/livehealthapp/blob/develop/labs/models.py) | Observability + traceability | There is also a fifth unofficial layer in practice: activity logs. They are not the source of truth for missing details, but they are the breadcrumb trail for entity creation activity, queueing, success, and failure. Key Features [#key-features] * Lab-specific configuration instead of a hardcoded global rule set. * Soft-disable semantics for config rows, so historical references are not blown away. * Runtime tracking at both patient level and order level. * Automatic resolution when new real data differs from the stored placeholder. * Manual resolution through Missing Data UI and related update flows. * `has_missing_fields` flag propagation into report/waiting/search style screens. * Organization scoping support through `org_id` on runtime rows. * Collection-centre-specific UI gating so the feature does not leak into unsupported contexts. * Legacy PY-2 compatibility for entity creation that still depends on old controllers. * Local PY-3 entity creation for some insurance/group flows where migration work has already happened. # Workflow Guide Workflow Guide [#workflow-guide] This section provides a practical walkthrough that gives context to understand the feature's behavior without digging through implementation details. import Image from 'next/image' import missingTag from '@/images/missing-tag.png' Configuring (Enabling/Disabling) The Missing Details Default Values [#configuring-enablingdisabling-the-missing-details-default-values] The configuration UI lives under advanced registration settings and is the source of truth for which fields can be marked missing. Where the user goes [#where-the-user-goes] 1. Open the registration settings / advanced settings area. 2. Look for the section labeled `Registration/Order Creation With Missing Details`. 3. If nothing is configured yet, the button shown is `Enable`. 4. If configs already exist, the buttons shown are `Edit` and `Disable`. Source references: * [`AdvanceSettings.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/components/AdvanceSettings.tsx) * [`MissingDetailsModal.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/components/MissingDetailsModal.tsx) What the buttons do [#what-the-buttons-do] | Button | What it does | Frontend path | Backend path | | :-------- | :-------------------------------------------------------------- | :------------------------------------ | :-------------------------------------------------------------------------------------- | | `Enable` | Opens the Missing Details modal with no existing lab config | `setToggleModal(true)` | No API yet | | `Edit` | Opens the same modal preloaded with current active config | `loadData()` in `MissingDetailsModal` | `GET /api-v3/registration/patients/lab-missing-details-config` | | `Disable` | Opens a confirmation modal and soft-disables all active configs | `handleDisableAll` | `POST /api-v3/registration/patients/lab-missing-details-config` with `disable_all=true` | | `Cancel` | Closes the modal without saving | modal state only | No API | What happens inside the Missing Details modal [#what-happens-inside-the-missing-details-modal] 1. The frontend calls `getTemplates()` to fetch `MissingDetailsTemplates`. 2. In parallel it calls `getLabConfigs()` to fetch the current active lab config. 3. The modal shows: * a category dropdown, * a field selector with `Select All`, * a table of selected fields, * a `Default Value If Missing` column, * action icons for edit/remove. 4. The operator can: * add one field, * add all fields in the filtered category, * edit a default value, * remove a field from active config. Validation checks before save [#validation-checks-before-save] These checks happen in the frontend before `saveLabConfigs(...)` posts the payload. | Check | Rule | Why it exists | Source | | :----------------------- | :------------------------------------------------------ | :------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | MRN validation | `Patient ID (MRN)` only allows `[A-Za-z0-9\\s\\-\\.,]+` | Prevent garbage placeholder values for a high-signal identifier | [`MissingDetailsModal.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/components/MissingDetailsModal.tsx), [`MissingDetailsHelpers.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/utils/missingFields.ts) | | Email validation | `Email` must pass `EMAIL_REGEX` | Avoid storing broken email defaults | [`constants.ts`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/constants.ts) | | Date validation | Date-type defaults must parse as ISO-compatible dates | `Date of Birth` and similar defaults need a valid date string | [`MissingDetailsModal.tsx`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/components/MissingDetailsModal.tsx) | | Non-editable field guard | Certain fields cannot be edited once configured | These fields are intended to map to catalog/entity semantics rather than arbitrary free text | `NON_EDITABLE_FIELD_NAMES` | Fields that are effectively non-editable after config exists [#fields-that-are-effectively-non-editable-after-config-exists] The UI disables default-value editing for fields listed in `NON_EDITABLE_FIELD_NAMES`. These include items like `Clinical History`, `Allergies`, `Referral`, `Organization`, `Test`, `Bill ICD codes`, `Consulting doctor`, and a few more. Save payload structure [#save-payload-structure] The modal computes three buckets before save: | Bucket | Meaning | | :------------- | :----------------------------------------------- | | `create` | New lab configs not currently active | | `update` | Existing configs whose value or metadata changed | | `disabled_ids` | Existing active configs removed by the user | Posted to: * `POST /api-v3/registration/patients/lab-missing-details-config` Backend behavior on config save [#backend-behavior-on-config-save] The PY-3 config view performs these steps: 1. Resolve `lab_id` from session. 2. Validate payload with `LabMissingDetailsBulkUpdateSerializer`. 3. Create new `LabMissingDetailsConfiguration` rows for genuinely new templates. 4. Soft-disable rows listed in `disabled_ids`. 5. Bulk-update changed rows. 6. For each created config, call [`LabMissingDetailsConfiguration.process_field(...)`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/models/lab_missing_details_configuration.py). `process_field(...)` decides whether the configured default is plain data or a real entity that the system should create somewhere else. For example, referrals, organizations, tests, and some insurance/group entities may be created or queued for creation. Enabling vs disabling in real DB terms [#enabling-vs-disabling-in-real-db-terms] This is not hard delete. | Action | DB behavior | | ----------------- | ------------------------------------------------------------ | | Enable/add field | Create a `LabMissingDetailsConfiguration` row | | Edit field | Update the same active row | | Disable one field | Set `is_disabled = 1` on that row | | Disable all | Bulk update all active rows for the lab to `is_disabled = 1` | Registering A Patient With Missing Details [#registering-a-patient-with-missing-details] The operator flow in registration [#the-operator-flow-in-registration] 1. Open the patient registration form. 2. The frontend loads `labMissingDetailsConfiguration` into model state through [`fetchLabMissingDetailsConfiguration(...)`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/utils/helpers.ts). 3. For configured fields, the UI can show a `Missing` checkbox near the field. 4. The operator clicks the `Missing` checkbox. 5. [`MissingFieldCustomCheckBox`](https://github.com/CrelioHealth/livehealth-frontend/blob/develop/src/components/reusable/Registration/components/MissingFieldsCheckBox/index.tsx) immediately updates: * `missingFieldState[field] = true` * `missingFieldUIVisible[field] = false` 6. The checkbox's `onToggle` handler calls `handleMissingToggle(...)`. 7. `handleMissingToggle(...)` looks up the configured default value and config id. 8. The correct form field is populated. 9. The helper persists: * `missingFieldSavedValue[field] = injected value` * `missingFieldConfigUsed[field] = config id` 10. The visible UI switches from the checkbox to a `Missing Data` label in many places. What happens if the user later types a real value [#what-happens-if-the-user-later-types-a-real-value] If the field was marked missing and the current form value no longer matches the injected saved value, the frontend clears the missing state via `checkAndResetMissingField(...)`. Registration submit payload [#registration-submit-payload] When patient registration is submitted, the payload carries the selected missing config ids (e.g., `labMissingDetailsIds`, `labMissingFieldIds`). Backend registration path [#backend-registration-path] In PY-3: 1. Patient save lands in `UserDetails.after_save(...)`. 2. [`build_value_mapper(self, insurance_details)`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/utils.py#L1) converts the saved object into a flat string map of fields. 3. [`create_or_update_missing_details_entries(...)`](https://github.com/CrelioHealth/crelio-app/blob/develop/admin/account/utils.py) is called with `lab_missing_field_ids`, `lab_id`, `user_details_id`, and context. 4. The function decides whether to create new unresolved rows or resolve existing ones. Registering / Updating An Order With Missing Details [#registering--updating-an-order-with-missing-details] Order-level missing fields follow the same idea but run through billing/update surfaces instead of pure patient registration. Typical order-level examples include `Test`, `Order Number`, `Report Remark`, `Bill ICD codes`, `Test ICD codes`, `Consulting doctor`. Billing flows populate `missingFieldConfigUsed` and send `labMissingFieldIds` on create/update so backend paths can resolve or create runtime rows. Resolving Missing Data [#resolving-missing-data] Resolution modes: * Automatic resolution: user edits patient/order data and saves a different real value. * Manual bulk/manual resolution: user clicks resolve in Missing Data UI. * Clinical-info resolution: updates from accession clinical info. * Waiting-list resolve modal: review and mark resolved. Patch API: `PATCH /api-v3/account/missing-details/lab-fields` expects: ```json { "ids": [101, 102], "is_resolved": true } ``` Behavior: * `resolved_at` is set when `is_resolved=true`. * `resolved_at` is cleared when `is_resolved=false`. Watch the short walkthrough showing how to resolve missing rows from the Missing Data UI: