Service Integration

Every service should integrate with ClickStack HyperDX in the same basic shape: configure OpenTelemetry, enable automatic instrumentation, add domain attributes, and verify the result in HyperDX with a real request.

Required Configuration

Setting	Purpose
`OTEL_SERVICE_NAME`	Stable service name shown in HyperDX
`OTEL_EXPORTER_OTLP_ENDPOINT`	OpenTelemetry collector endpoint
`OTEL_RESOURCE_ATTRIBUTES`	Environment, deployment, region, and ownership metadata
Service log format	Must include trace and span IDs when a span is active
Trace sampling policy	Must be explicit per environment

Collector endpoints are provided by the ClickStack HyperDX infra:

Protocol	Internal Endpoint	Port
OTLP gRPC	`otel-collector.hdx-infra.local:4317`	`4317`
OTLP HTTP	`otel-collector.hdx-infra.local:4318`	`4318`

Use the internal endpoint for services running inside the VPC. If an external ingest domain is configured, use the DNS name managed by the ClickStack infra instead of the Cloud Map name.

Instrumentation Requirements

Area	Requirement
Entrypoints	Instrument HTTP routes, worker jobs, consumers, scheduled jobs, or CLI tasks
Dependencies	Instrument database, cache, search, queue, and outbound HTTP clients
Errors	Record exceptions on the active span before returning or retrying
Logs	Inject `trace_id` and `span_id` into structured logs
Metrics	Emit service and domain metrics using stable names
Trace propagation	Forward incoming trace context and return a trace ID for support/debugging

Domain Context Requirements

Automatic spans are not enough. Each service must add the fields that explain why a request behaved the way it did.

Examples of useful domain attributes:

Service Type	Useful Attributes
API service	route, authenticated account, organization, lab, feature flag, request mode
Search service	search key, query shape, routing, target index, hit count, zero-result flag
Worker service	queue, job type, retry count, tenant, source event ID
Sync service	source table, destination table, lag, batch size, offset, DLQ status
Integration service	vendor, message type, external request ID, response code, retry state

Do not add secrets, tokens, PHI, raw patient payloads, or unrestricted request bodies as span attributes. Query capture must be explicitly controlled by service configuration.

Implementation Checklist

Step	Done When
Configure exporter	Service sends OTLP to the ClickStack HyperDX collector
Set service identity	HyperDX shows the expected `OTEL_SERVICE_NAME`
Enable framework instrumentation	Requests or jobs create root spans
Enable dependency instrumentation	Downstream calls create child spans
Add domain attributes	A trace explains tenant/scope/query/job context
Correlate logs	Logs from the request appear with trace and span IDs
Return trace ID	HTTP APIs expose `X-Trace-Id` or an equivalent support-safe correlation ID
Validate in HyperDX	A real request shows timeline, spans, logs, metrics, and errors if present

Rollout Validation

Before marking a service as integrated:

Validation	Expected Result
Healthy request	Root span is visible with correct service name and route/job name
Dependency call	Child span appears with duration and status
Error path	Failed request records error status and exception context
Log correlation	Logs are searchable from the trace
Domain context	Required service attributes are present
Support handoff	Trace ID from response or logs opens the same request in HyperDX

Phoenix Search Baseline

Phoenix Search is the reference integration for this setup. It should be used as the baseline for new services because it connects the browser request, X-Trace-Id, API route span, Elasticsearch span, logs, metrics, and search-specific attributes in one debugging flow.

For the current Phoenix Search flow, see Phoenix Search API Debugging.

Service Integration