Observability Setup

---
name: observability-setup
description: |
  Stand up end-to-end observability: traces, metrics, and logs via
  OpenTelemetry plus one vendor backend. Trigger on "observability",
  "distributed tracing", "OpenTelemetry", "OTel", "traces and metrics".
allowed-tools:
  - Bash(npm install *)
  - Bash(pnpm add *)
  - Read
  - Write
  - Edit
---

# Observability Setup

Wire the OpenTelemetry SDK into a service so traces, metrics, and logs
flow to a collector and on to a backend (Honeycomb, Grafana Cloud,
Datadog, or a self-hosted OTel Collector plus Tempo / Loki / Prometheus).

## Prerequisites

- Service language known (Node, Python, Go)
- Backend chosen with endpoint + API key
- Service has a clear entrypoint you can patch before other imports

## Steps

1. **Install the SDK and auto-instrumentations.** For Node:
   ```bash
   pnpm add @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-http
   ```

2. **Bootstrap OTel before app code runs.** Create `tracing.ts` and
   require it first (`node --require ./tracing.ts` or top of entrypoint):
   ```ts
   import { NodeSDK } from "@opentelemetry/sdk-node";
   import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
   import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

   new NodeSDK({
     traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }),
     instrumentations: [getNodeAutoInstrumentations()],
   }).start();
   ```

3. **Set resource attributes.** Service name, version, deploy env. Without
   these, your backend cannot group traces.
   ```
   OTEL_SERVICE_NAME=api
   OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=${GIT_SHA}
   ```

4. **Add one manual span in a hot path** to prove the SDK is live. Wrap
   the slowest DB call or external request.

5. **Correlate logs with traces.** Emit `trace_id` and `span_id` on
   structured log lines so backend click-through works.

6. **Pick a sampler.** Default is 100 percent. At real volume, use a
   parent-based ratio sampler (1 to 10 percent) with always-on for errors.

7. **Dashboards and SLOs.** Build: request rate, error rate, p95 latency,
   saturation. Set one SLO per critical endpoint. Alert on burn rate,
   not on raw thresholds.

## Anti-patterns

- Exporting directly to the vendor from every service; run a collector
- Leaving auto-instrumentation on for every package (noisy, expensive)
- Sampling at 100 percent in production

## Output

- Tracing bootstrap file loaded before app code
- OTel env vars documented in README
- One manual span, correlated logs, one live dashboard, one SLO