ZetrixwebThe metrics that actually matter
Notification systems generate a lot of possible metrics. Most of them don't predict incidents. The handful that do are the ones worth building dashboards and alerts around first.
Queue depth
A rising, unconsumed backlog on any channel's queue means workers can't keep up - either add capacity or something downstream is stuck.
Delivery success/failure rate, per channel
Tracked separately for SMS, email, and push - a drop in one channel's success rate almost always means that specific provider is degraded, not your whole system.
Latency per provider
p50/p95/p99 response time per provider call - a creeping p99 is often the earliest warning sign of an outage, well before the failure rate itself moves.
Structured logging: what to always include
Free-text logs ("Failed to send SMS") are close to useless for debugging a specific customer's complaint at scale. Every log line tied to a notification should carry the same structured fields, so you can filter and correlate across the whole pipeline instantly.
logger.info('notification_attempt', {
notification_id: notif.id,
idempotency_key: notif.idempotencyKey,
channel: notif.channel,
provider: 'twilio',
attempt_count: notif.attemptCount,
status: 'failed',
failure_reason: err.code, // e.g. 'rate_limited', 'invalid_number'
provider_response_time_ms: elapsed,
correlation_id: request.correlationId, // ties back to the originating business event
});The field people forget: correlation_id. Without it, tracing "why didn't this specific payment confirmation arrive" back through the outbox row, the queue message, the retry attempts, and the final provider call requires manually cross-referencing timestamps - with it, it's one query.
Alerting thresholds that don't cry wolf
| Signal | Alert condition | What it usually means |
|---|---|---|
| Queue depth | Sustained growth over 10+ min, not a single spike | Worker capacity too low, or a downstream dependency is stuck |
| Per-channel failure rate | >5% over a 5-minute rolling window | Provider degradation on that specific channel |
| DLQ depth | Any sustained growth (not zero-tolerance, but non-flat) | An integration is failing every retry attempt - needs investigation, not just replay |
| Circuit breaker state change | Any transition to "open" | A provider is degraded enough that the breaker tripped - immediate, not delayed, alert |
| p99 provider latency | >2x the 7-day baseline | Early warning of an outage, often before failure rate moves at all |
Dashboards worth building first
Per-channel health board - queue depth, success rate, and p95 latency for SMS, email, and push, side by side.
DLQ inspector - current depth, growth rate, and a breakdown of failure reasons for whatever's sitting in it right now.
Circuit breaker status board - current state (closed/open/half-open) per provider, with a timeline of recent transitions.
Key takeaways
- Queue depth, per-channel success/failure rate, and provider latency are the three metrics worth alerting on first - most others are noise until those three are solid.
- Structured logs need a shared schema across every service in the pipeline - especially a
correlation_idthat survives from the originating event through to the provider call. - Alert on sustained trends, not single spikes - a 30-second blip in queue depth is normal; 10 minutes of continuous growth is not.
- A circuit breaker opening should be an immediate alert, not a metric you check later - it means the system already decided a provider is unhealthy.
- Build the per-channel health dashboard first - it's the one that turns "customers are complaining" into "we already saw this an hour ago."