Operations

Phoenix Search Operations

This page collects the day-to-day commands and on-call checks for the Phoenix Search API and its CDC subsystem.

For the full data migration sequence, cdc-ctl, and the Go backfill tool, see CDC Tools and Backfill. For request-to-trace debugging, see API Debugging.

Run Locally

API Only

make install
cp .env.example .env
cp docker/.env.example docker/.env
make up-dev
make seed
make run

Full Local Stack

make up-all
make register-pipeline
make run-cdc
make register-source-connector

Test Infra

make test
make test-docker
make test-docker-up
make test-docker-down

API Health Checks

/health intentionally returns HTTP 200 even when degraded. Alert on the body fields, especially dependency statuses and cdc.status.

Example:

curl -s http://localhost:8000/health
curl -s http://localhost:8000/health/ready
curl -s http://localhost:8000/metrics

CDC Health Checks

The CDC consumer exposes its own health server, default port 8080.

Example:

curl http://<TASK_IP>:8080/health
curl http://<TASK_IP>:8080/ready
curl http://<TASK_IP>:8080/metrics

CDC Runbook Order

When Phoenix Search data looks stale, check the planes in this order. It prevents chasing Elasticsearch symptoms when the problem is actually a connector, Redpanda, or consumer-group issue.

The operator CLI writes artifacts for every run under tools/cdc-ctl/runs/<timestamp>_<subcommand>/. Use those artifacts when handing an incident to another engineer.

Redpanda Runbook

Access Pattern

In production, run rpk from a Redpanda broker or another host inside the VPC. Laptop access to the internal SASL listener is not expected.

ssh -i ~/Documents/pem-files/redpanda.pem ubuntu@<REDPANDA_PUBLIC_IP>

export REDPANDA_BROKERS=<BROKER_1_PRIVATE_IP>:9093,<BROKER_2_PRIVATE_IP>:9093,<BROKER_3_PRIVATE_IP>:9093

rpk topic list \
  --user admin \
  --password '<ADMIN_PASSWORD>' \
  --sasl-mechanism SCRAM-SHA-256

Cluster and Topic Checks

# Broker and partition health.
rpk cluster health \
  --user admin \
  --password '<ADMIN_PASSWORD>' \
  --sasl-mechanism SCRAM-SHA-256

# Expected Phoenix topics.
rpk topic list \
  --user admin \
  --password '<ADMIN_PASSWORD>' \
  --sasl-mechanism SCRAM-SHA-256

# Partition, leader, replica, and high-watermark details.
rpk topic describe phoenix.livehealthapp.userDetails -p \
  --user admin \
  --password '<ADMIN_PASSWORD>' \
  --sasl-mechanism SCRAM-SHA-256

Expected Phoenix data topics:

Production Replication Guardrails

On the 3-node Redpanda cluster, Phoenix CDC topics should use replication factor 3. Critical topics should also use min.insync.replicas=2, so writes fail fast if too many brokers are unavailable.

Check the expected topic policy with the operator CLI first:

CDC_CTL_ENV=production ./cdc-ctl topics list
CDC_CTL_ENV=production ./cdc-ctl topics verify
CDC_CTL_ENV=production ./cdc-ctl topics describe phoenix.livehealthapp.userDetails

If a cluster was expanded from RF=1 to RF=3, existing topics do not automatically become RF=3. The recovery notes in cdc/CDC_RECOVERY_LOG.md document this as an explicit migration step.

Consumer Group and Rebalance Checks

rpk group describe phoenix-cdc-unified \
  --user admin \
  --password '<ADMIN_PASSWORD>' \
  --sasl-mechanism SCRAM-SHA-256

Consumer group rebalance happens when the CDC service scales, a task restarts, or a member stops polling. During rebalance, Redpanda revokes and reassigns partitions, and the Phoenix consumer pauses processing briefly to preserve in-partition ordering.

The safe scaling ceiling is the topic partition count: 6 consumer tasks. More than 6 tasks are usually idle because Phoenix topics are created with 6 partitions.

aws ecs update-service \
  --cluster phoenix \
  --service cdc-consumer \
  --desired-count 6

Lag Interpretation

Sample messages when you need to confirm whether a lag spike is live traffic or a snapshot:

rpk topic consume phoenix.livehealthapp.userDetails \
  --offset end \
  -n 1 \
  --user admin \
  --password '<ADMIN_PASSWORD>' \
  --sasl-mechanism SCRAM-SHA-256 \
  -f '%T p%p:o%o %v\n'

In Debezium envelopes, op: "r" means snapshot/read, op: "c" means insert, op: "u" means update, and op: "d" means delete.

Debezium Runbook

Connector Status

Phoenix Search has two Debezium MySQL connectors.

CDC_CTL_ENV=production ./cdc-ctl status
CDC_CTL_ENV=production ./cdc-ctl offsets

curl -s http://<CONNECT_HOST>:8083/connectors/phoenix-source-existing/status | jq .
curl -s http://<CONNECT_HOST>:8083/connectors/phoenix-source-projection/status | jq .

Check these fields in the status response:

Restart failed tasks first:

CDC_CTL_ENV=production ./cdc-ctl restart --only-failed

Use connector recreation only when status or config is corrupted. Connector deletion preserves offsets in connect-offsets; it does not clear them.

CDC_CTL_ENV=production ./cdc-ctl recreate --dry-run
CDC_CTL_ENV=production ./cdc-ctl recreate --yes --settle-seconds 30

Debezium Signals

Debezium signals are for asking a running connector to re-read rows into Kafka without resetting connector offsets. Phoenix enables source signals through livehealthapp.debezium_signal on both connectors.

SELECT *
FROM livehealthapp.debezium_signal
ORDER BY id DESC
LIMIT 5;

Use an incremental snapshot signal when the connector needs to re-capture rows:

INSERT INTO livehealthapp.debezium_signal (id, type, data)
VALUES (
  'resync-user-details-20260520',
  'execute-snapshot',
  '{"data-collections": ["livehealthapp.userDetails"], "type": "incremental"}'
);

Use a projection-table snapshot when user_meta was repaired and Elasticsearch needs to receive projection events:

INSERT INTO livehealthapp.debezium_signal (id, type, data)
VALUES (
  'resync-user-meta-20260520',
  'execute-snapshot',
  '{"data-collections": ["livehealthapp.user_meta"], "type": "incremental"}'
);

Signals produce snapshot read events into the same Redpanda topics as live binlog events, so lag can spike while the snapshot runs. For normal projection recomputation, prefer the backfill repair modes; signals re-emit rows, they do not recompute billing/sample aggregates by themselves.

For the detailed signal flow and external Debezium references, see CDC.

Connector Recovery Guardrails

Before touching offsets, capture connector status, offsets, topic state, MySQL binlogs, and a debug bundle:

CDC_CTL_ENV=production ./cdc-ctl status
CDC_CTL_ENV=production ./cdc-ctl offsets
CDC_CTL_ENV=production ./cdc-ctl topics verify
CDC_CTL_ENV=production ./cdc-ctl mysql check
CDC_CTL_ENV=production ./cdc-ctl debug --tarball

Deployment References

API Runtime

The API starts from the Phoenix Search package entrypoint and reads configuration from SEARCH_* environment variables. In production, the ALB can forward traffic with /phoenix-search/*; the service strips that prefix internally.

Core runtime dependencies:

API Configuration Variables

Defaults are defined in search/settings.py. The settings prefix is SEARCH_, so a field named mysql_host is configured as SEARCH_MYSQL_HOST.

Elasticsearch:

Search behavior and freshness:

Redis, auth, and rate limiting:

MySQL and telemetry:

CDC Consumer Runtime

The CDC consumer should be deployed as an ECS task with:

Operational Make Targets

Metrics and Alerts

API Metrics

CDC freshness is controlled by:

CDC Consumer Metrics

Alert Conditions

Production Metrics Snapshot

These values are from the Phoenix Search production IN dashboard screenshots. Treat this section as observed Phase 1 production evidence, not a permanent SLO target.

For the full migration outcome, old-vs-new index comparison, request-count reduction, analyzer mapping notes, and OpenTelemetry advantages, see Post-Migration Results.

Success Signals

API Traffic and Latency

POST /api/v1/users/search is the only endpoint that materially drives API cost in the observed window. Detail lookup traffic is much smaller and faster.

Search, MySQL, and ES Query Performance

The API p95 is higher than raw ES/MySQL query latency because it includes request handling, auth/session work, filter resolution, query building, response shaping, and network/runtime overhead.

Elasticsearch Cluster and Index Capacity

The index is large enough that routing matters operationally. For point checks, always use the lab_id route:

curl -s -u elastic:<PASSWORD> \
  "https://<ES_HOST>:9200/user_details/_doc/<USER_DETAILS_ID>?routing=<LAB_ID>"

Dashboard Evidence

Dashboards

Dashboard definitions are maintained as code in the Phoenix Search repo.

Common commands:

phoenix-search dashboards push --api-key YOUR_KEY --env production
phoenix-search dashboards alerts-push \
  --api-key YOUR_KEY \
  --env production \
  --alert-channel slack_webhook \
  --alert-webhook-id YOUR_WEBHOOK_ID
phoenix-search dashboards diff --api-key YOUR_KEY

Makefile shortcut:

make dashboards-push API_KEY=your_key ENV=production

Common Debugging

API returns no search results

Check:

1. Session resolves to the expected lab, branch, organization, and collection-center scope.
2. The selected search_key includes the field being searched.
3. search_fields does not filter out every valid field for the selected mapping.
4. Elasticsearch routing matches the expected lab ID.
5. The ES circuit breaker is not open.

Useful commands:

curl -s http://localhost:8000/health
curl -s http://localhost:8000/metrics | grep search
curl -s -u elastic:<PASSWORD> "http://localhost:9210/user_details/_count"
curl -s -u elastic:<PASSWORD> \
  "http://localhost:9210/user_details/_doc/<USER_DETAILS_ID>?routing=<LAB_ID>"

Always include routing=<LAB_ID> when checking a specific ES document. The user_details index requires routing, and Phoenix Search indexes each document under its lab_id.

/health/ready returns 503

Read the dependency section in the response body. The readiness endpoint checks Elasticsearch, Redis, and MySQL.

curl -s http://localhost:8000/health/ready

Typical fixes:

CDC consumer not processing messages

Symptoms: consumer /health is 200, /ready is 503, or processed-message counters stop increasing.

curl -s http://<CONNECT_HOST>:8083/connectors/phoenix-source-existing/status
rpk group describe phoenix-cdc-unified
docker logs <container_id> --tail 100

Common causes:

• Debezium connector is down.
• Consumer group rebalance is stuck.
• All messages are failing and retries are exhausting.

Debezium connector failed

curl -s http://<CONNECT_HOST>:8083/connectors/phoenix-source-existing/status
curl -X POST http://<CONNECT_HOST>:8083/connectors/phoenix-source-existing/tasks/0/restart

Common causes:

• MySQL binlog expired and the connector lost its position.
• MySQL network or credential failure.
• Schema change on a captured table.

Messages are going to DLQ

rpk topic consume phoenix.cdc.dead-letter-queue --num 5

Check the message headers for the original topic, partition, and offset. Common causes are malformed Debezium events, MySQL pool exhaustion, and Elasticsearch mapping conflicts.

Consumer lag is increasing

rpk group describe phoenix-cdc-unified
curl -s http://<TASK_IP>:8080/metrics | grep cdc_messages_processed

Possible fixes:

• Scale the consumer up to the partition count.
• Increase CDC_MYSQL_POOL_SIZE if MySQL writes are bottlenecked.
• Check MySQL with SHOW PROCESSLIST.
• Check Elasticsearch write thread pool and mapping failures.

CDC phase detection is wrong

mysql -e "SHOW COLUMNS FROM user_meta" | grep full_name

Force the phase when needed:

CDC_PHASE_OVERRIDE=running
CDC_PHASE_OVERRIDE=migration

DDL is blocked on captured tables

Debezium can hold metadata locks on captured tables. Stop both connectors before DDL, then resume them after the migration.

curl -X PUT http://<CONNECT_HOST>:8083/connectors/phoenix-source-projection/stop
curl -X PUT http://<CONNECT_HOST>:8083/connectors/phoenix-source-existing/stop

# Run DDL here.

curl -X PUT http://<CONNECT_HOST>:8083/connectors/phoenix-source-projection/resume
curl -X PUT http://<CONNECT_HOST>:8083/connectors/phoenix-source-existing/resume

If the Kafka Connect version does not support stop, delete and recreate the connectors with the same names. Offsets are preserved in the Connect offsets topic as long as they are not purged.

Backfill and Recovery

Use the Go backfill tool for initial migration, full reindexing, or recovery when CDC is offline.

make backfill-build
make backfill-migrate
make backfill-run BACKFILL_ENV=dev
make backfill-run BACKFILL_ARGS="--verify --verify-sample 50"

For a fresh migration:

make backfill-migrate-reset
make backfill-run-tuned BACKFILL_ENV=production

The migration first populates user_meta, then the ES backfill pushes projected rows to Elasticsearch with preflight checks. CDC should be coordinated around backfill so live changes do not fight the rebuild.

Clean Restart of CDC

Use only when CDC state is corrupted and normal connector or consumer restarts do not recover processing.

aws ecs update-service \
  --cluster phoenix \
  --service cdc-consumer \
  --desired-count 0

curl -X DELETE http://<CONNECT_HOST>:8083/connectors/phoenix-source-existing
curl -X DELETE http://<CONNECT_HOST>:8083/connectors/phoenix-source-projection

rpk group delete phoenix-cdc-unified

curl -X POST http://<CONNECT_HOST>:8083/connectors \
  -H "Content-Type: application/json" \
  -d @tools/cdc-ctl/connectors/source-connector-existing.production.json

curl -X POST http://<CONNECT_HOST>:8083/connectors \
  -H "Content-Type: application/json" \
  -d @tools/cdc-ctl/connectors/source-connector-projection.production.json

aws ecs update-service \
  --cluster phoenix \
  --service cdc-consumer \
  --desired-count 1

Source References