Your health check should touch the database

The claim

An endpoint that returns 200 unconditionally is worse than having no health check at all. It converts a loud failure into a silent one: your load balancer keeps routing traffic to a process that cannot serve a single request, your uptime monitor stays green, and the first person to notice is a customer. A health check that does not exercise the thing most likely to break is decorative.

Two checks, not one

The common mistake is a single endpoint doing two incompatible jobs. Separate them:

• Liveness answers "is this process wedged?" It must not touch the database. If it does, a brief database blip causes the orchestrator to kill every application container at once, turning a five-second degradation into a full restart storm.
• Readiness answers "should this instance receive traffic right now?" It must touch the database, the cache, and anything else without which requests will fail.

Liveness returns 200 if the event loop is running. Readiness returns 503 when a dependency is down, so the load balancer removes the instance from rotation and puts it back automatically when the dependency recovers.

What a readiness check looks like

func readiness(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
    defer cancel()

    if err := db.PingContext(ctx); err != nil {
        http.Error(w, "db: "+err.Error(), http.StatusServiceUnavailable)
        return
    }
    if err := cache.Ping(ctx).Err(); err != nil {
        http.Error(w, "cache: "+err.Error(), http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

Three details make the difference between a useful check and an outage amplifier.

The timeout is explicit and short. Two seconds. Without it, a hung database connection means the health check itself hangs, the load balancer times out, and you cannot distinguish "slow" from "dead".

The response body names the failing dependency. During an incident, the difference between a blank 503 and db: connection refused is roughly fifteen minutes of your evening.

The query is cheap. SELECT 1 or a driver ping. If your load balancer checks every five seconds across eight instances, that is 96 queries per minute. A check that runs a real business query will show up in your slow query log and, on a bad day, will be the thing that finishes off an already-struggling database.

The dependency you should not check

Readiness should cover dependencies you cannot serve without, and nothing else. A payment gateway is the classic mistake. If Stripe is having a bad afternoon and your readiness check pings it, every instance in your fleet reports 503 and your entire site goes dark — including the pages that have nothing to do with payments. Degrade that dependency inside the request handler instead: show the catalogue, disable the checkout button, and let the rest of the site keep earning.

Cache the result if the fleet is large

Past a dozen instances, put a short in-process cache on the dependency check — five seconds is plenty:

if time.Since(lastCheck) < 5*time.Second {
    return cachedResult
}

This bounds the load your health checks place on a dependency that is already unhappy, which is exactly when you least want to add 96 connections per minute to the pile.

Configure the load balancer to match

The endpoint is half the work. In Nginx:

upstream app {
    server 10.0.1.10:8080 max_fails=3 fail_timeout=15s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=15s;
}

Three failures before removal, fifteen seconds before retry. One failure is too twitchy — a single dropped packet should not eject a healthy instance. Thirty seconds of tolerance is too slow — you are serving errors the whole time. Three checks at five-second intervals means a genuinely dead instance leaves rotation in about fifteen seconds, which is a reasonable place to land.

What to check externally

Your external monitor should not hit the readiness endpoint. It should hit a real page and assert on content:

curl -fsS https://example.ca/products | grep -q 'Add to cart' || exit 1

This catches the class of failure that every internal check misses: the application is up, the database is up, and a bad deploy has replaced the catalogue with an empty state. We have seen a site sit in that condition for two days with 100% uptime reported, because every probe in the stack was asking whether the server responded rather than whether it responded correctly.

A short checklist

1. Liveness checks nothing external. Readiness checks every hard dependency.
2. Every dependency check has an explicit timeout under three seconds.
3. Failures return 503 with the dependency name in the body.
4. Readiness results are cached for a few seconds on large fleets.
5. At least one external check asserts on page content, not status code.

None of this takes more than an afternoon. The payoff is that your monitoring stops agreeing with you when you are wrong.