Monitoring
How to Stop Alert Fatigue
Alert fatigue is when so many alerts fire that you start ignoring them — and miss the real one. The fix is rarely 'add more alerts' or 'tune thresholds'. It's structural.
1. Alert on symptoms, not causes
Alert when users are affected (latency p95 > 2s error rate > 1%) not when CPU is 80% or disk is 70%. Causes have noise; symptoms have signal.
2. One alert per failure mode
If a database outage triggers 14 alerts (DB app errors queue backlog latency etc.) you've created noise. Use 'inhibit' rules — when DB-down fires suppress the downstream alerts.
3. Pages versus tickets
Page (Telegram/Slack/SMS) only when human action is required NOW. Everything else is a ticket — review it during business hours. Most systems get this wrong by paging on every yellow flag.
4. Track and prune
Weekly review: which alerts fired? Which were actionable? Delete the rest. A monitoring system you trust is one that sends ~1 alert/week and every one of them matters.
Want to verify your setup?
Run the check now