Running an HL7 FHIR server in production requires operational discipline. Five runbooks cover the common scenarios teams face.
Runbook 1: Terminology update procedure.
1. SNOMED CT (twice yearly), LOINC (twice yearly), RxNorm (weekly) releases. 2. Test loading in staging. 3. Regenerate cached ValueSet expansions. 4. Verify sample validations pass. 5. Deploy to production during maintenance window.
Runbook 2: Postgres index tuning.
1. Monitor query performance for slow queries. 2. HAPI JPA index recommendations. 3. Create indexes during low-traffic window. 4. Verify query plan improvement.
Runbook 3: Bulk export failure recovery.
1. Detect: manifest error entries, timeouts. 2. Diagnose: check server load, storage availability, terminology server responsiveness. 3. Recover: retry with smaller time window (_since), verify. 4. Escalate: if repeated failures, root-cause analysis.
Runbook 4: Subscription delivery investigation.
1. Monitor delivery success rate metric. 2. Failed deliveries: check subscriber endpoint reachability. 3. Persistent failures: pause subscription, notify subscriber. 4. Recovery: fix subscriber, resume with backfill.
Runbook 5: Backup and restore.
1. Nightly pg_dump to versioned backup storage. 2. Weekly full snapshot to secondary region. 3. Quarterly restore drill on staging. 4. Documented RTO/RPO.
Alerting thresholds
| Metric | Alert |
|---|---|
$validate pass rate <95% |
Investigate upstream |
| Postgres pool >85% utilization | Scale or tune |
| Bulk export queue >20 | Investigate load |
| Terminology latency >300ms | Terminology server issue |
| Subscription delivery <95% | Subscriber outreach |
On-call preparation
On-call engineer needs: FHIR server ops docs, runbook access, alerting dashboards, escalation contacts, psql access, container exec access.
HL7 FHIR server operations are well-understood in 2026. Sites that write the runbooks and drill them ship reliably; sites without them scramble during incidents.
