Four alerts that wake someone up, and nothing else

The claim

If more than four conditions in your system are permitted to page a human at three in the morning, none of them will be taken seriously by March. Alert fatigue is not a discipline problem to be solved with better training. It is an arithmetic problem: a person who is woken six times a month for things that turn out to be nothing will, correctly, start assuming the seventh is nothing too.

Alert on symptoms, not causes

The instinct is to alert on everything measurable: CPU above 80%, memory above 90%, disk queue depth, replication lag, container restarts. All of those are causes. None of them is necessarily a problem — a batch job pinning the CPU for twenty minutes is the system doing its job.

Alert instead on the four symptoms a customer could actually experience:

1. The site is not serving requests. An external check from two regions fails three consecutive times. Not one — a single failure is usually the checker's network.
2. Requests are failing. The 5xx rate exceeds 2% of requests over a five-minute window, with a floor of at least 20 requests so that two errors on a quiet Sunday do not page anyone.
3. Requests are unusably slow. The 95th percentile latency exceeds three times its normal value for ten minutes. Use a percentile, never a mean: an average hides the tail, and the tail is where the customers are leaving.
4. Work is not getting done. Your background job queue depth is growing monotonically for fifteen minutes, or the oldest unprocessed job is older than ten minutes. This is the one most teams omit, and it is how invoices silently stop going out.

Everything else goes to a dashboard or a ticket, not a phone.

Give every alert a budget, not a threshold

Thresholds invite argument. Error budgets end it. Pick an availability target and do the division:

99.9%  = 43.8 minutes of downtime per 30 days
99.95% = 21.9 minutes
99.99% =  4.4 minutes

For most small and mid-sized businesses, 99.9% is the honest target: it permits a forty-minute incident once a month without breaching anything, which matches what a two-person team can realistically respond to. Do not claim 99.99% unless someone is on a rotation and paid for it — a single unplanned reboot spends the entire monthly budget.

Once you have a budget, the alert threshold follows from the burn rate rather than from taste. A rule that consumes 5% of the monthly budget in an hour is worth waking someone for; one that would take three weeks to exhaust the budget is a ticket for Monday.

Warning-level alerts that are actually predictions

Some conditions deserve attention without urgency, and the useful form is a forecast rather than a level:

# Disk will be full within four days at the current rate
predict_linear(node_filesystem_avail_bytes[6h], 4*24*3600) < 0

Same for TLS certificate expiry at fourteen days and for a queue of failed background jobs that has not been drained. These go to a work-hours channel. They are the two categories — disk and certificates — that account for a large share of avoidable emergency pages, and both announce themselves days in advance if anyone is looking.

Every page needs a runbook link

An alert that says HighErrorRate firing on api-prod has told the recipient nothing they could not see. The notification should carry the current value, the threshold, the last deploy time, and a link to three specific commands to run first. Writing that runbook takes twenty minutes per alert. With only four alerts, that is eighty minutes of work, which is the strongest practical argument for keeping the list short.

Who receives them

Four alerts still need a named recipient. For a team without a formal rotation, the workable arrangement is a single primary each week, chosen in advance, with a second person listed as escalation after fifteen minutes of no acknowledgement. Publish the schedule where the whole company can see it, so that nobody has to guess who to call, and so the person on duty can decline a Thursday evening commitment without negotiating.

Review the list quarterly

Pull every page from the last ninety days and sort them into three piles: required immediate action, could have waited until morning, or was nothing. If the third pile is more than a fifth of the total, the alert that generated most of it gets tuned or deleted before you add anything new. An alert that has never once caught a real problem is not insurance. It is a tax on your team's sleep, and it makes the real page less likely to be answered.