ClickStack HyperDX Infra
Self-hosted HyperDX and ClickStack observability service
ClickStack HyperDX Infra
ClickStack HyperDX Infra is CrelioHealth's self-hosted observability service for OpenTelemetry data. Application services send traces, metrics, and logs to the OpenTelemetry collector, HyperDX provides the debugging UI, and ClickHouse stores the telemetry data.
This service exists so engineers can debug production behavior from one request timeline instead of manually matching logs, timestamps, dashboards, and dependency calls.
Repository Information
| Property | Value |
|---|---|
| Repository | clickstack-infra |
| Local Path | /Users/saitharun/Documents/hyperdx |
| Runtime | AWS ECS Fargate + EC2 ClickHouse |
| Region | ap-south-1 |
| Primary UI | HyperDX |
| Ingestion | OpenTelemetry collector over OTLP gRPC and OTLP HTTP |
| Storage | ClickHouse |
| Metadata Store | MongoDB |
What This Service Provides
| Capability | Why It Matters |
|---|---|
| Trace timelines | Shows the full request path with parent and child spans |
| Span duration breakdown | Separates application time from dependency time |
| Log correlation | Connects logs to trace_id and span_id |
| Metrics | Tracks service, dependency, and domain health over time |
| Error investigation | Starts from a failed request and follows the trace to the failing span |
| Service context | Adds domain fields such as lab, route, query shape, routing, and hit count |
Why We Added OpenTelemetry
Before this setup, debugging depended mostly on logs, dashboards, and manual correlation. That made incidents slow to reason about because the team could not reliably see the exact request timeline, which dependency took time, which span failed, or which domain context was attached to the failing request.
OpenTelemetry fixes that by sending structured traces, metrics, and logs from each service into HyperDX. A single trace can show the route span, auth/session work, Redis calls, MySQL calls, Elasticsearch calls, errors, timings, and service-specific attributes for the same user request.
Current Example Integration
Phoenix Search is the first documented integration. It emits request traces, dependency spans, search attributes, Elasticsearch query context, CDC freshness metrics, and trace-aware logs.
Use these pages for implementation and debugging details:
Debugging Principle
For every production issue, start with the most specific request identifier available:
| Starting Point | Next Step |
|---|---|
Browser response has X-Trace-Id | Search that trace ID in HyperDX |
Backend log has trace_id | Open the matching trace and inspect spans |
| Dashboard shows latency or errors | Filter by service, route, and time window, then inspect a representative trace |
| User reports empty or wrong results | Inspect domain attributes and dependency spans for that request |
The expected outcome is a clear answer to where the time, error, or wrong behavior came from.