Webhook Monitor

---
name: webhook-monitor
description: |
  Monitor webhook reliability end-to-end: delivery status, retry rates,
  payload validation, and latency tracking. Trigger on "webhook monitor",
  "webhook reliability", "track webhook delivery", "webhook retries",
  "webhook latency", "failed webhooks".
allowed-tools:
  - Bash(curl *)
  - Bash(psql *)
  - Read
  - Write
  - Edit
---

# Webhook Monitor

Instrument webhook producers and consumers so you can answer three
questions at any moment: What is my delivery success rate? Where are
payloads getting dropped? How long does the round-trip take?

## Prerequisites

- Webhook producer source access (to emit events)
- Consumer endpoint URL and shared secret
- A datastore for delivery records (Postgres, Redis, or ClickHouse)

## Steps

1. **Define the delivery record schema.** Each attempt writes one row.
   ```sql
   CREATE TABLE webhook_deliveries (
     id UUID PRIMARY KEY,
     event_type TEXT NOT NULL,
     endpoint_url TEXT NOT NULL,
     attempt INT NOT NULL,
     status_code INT,
     latency_ms INT,
     payload_hash TEXT,
     error TEXT,
     created_at TIMESTAMPTZ DEFAULT NOW()
   );
   ```

2. **Sign outgoing payloads** with HMAC-SHA256 using a shared secret so
   the consumer can verify authenticity. Include a timestamp header to
   reject replays older than five minutes.

3. **Wrap the send call** to capture latency and status. Record every
   attempt, not just the final outcome, so retry patterns stay visible.

4. **Validate payloads against a schema** (Zod, Pydantic, or JSON
   Schema) before sending. Log schema violations as a distinct error
   class separate from network failures.

5. **Implement exponential backoff** for retries: 1s, 5s, 30s, 2m, 10m.
   Cap at five attempts. After exhaustion, move the event to a dead
   letter queue for manual replay.

6. **Expose a dashboard** with: success rate per endpoint (last 24h),
   p95 latency, retry rate, and top failing event types. Alert when
   success rate drops below 99 percent for 10 minutes.

7. **Provide a replay endpoint** that can re-send any event from the
   dead letter queue with a single curl call.

## Anti-patterns

- Logging only failures (you lose the success baseline)
- Fixed-interval retries that hammer a recovering consumer
- Storing full payloads forever (hash them, expire raw bodies in 7 days)

## Output

- Delivery records table + indexes on (endpoint_url, created_at)
- Signed payloads with HMAC + timestamp headers
- Dashboard with success rate, latency, retry rate
- Dead letter queue with replay endpoint