Skip to content
Infrastructure · 5 min read

Your database restore is untested until you have timed it

The claim Knowing that your database backups exist and even that they are valid is not the same as knowing you can recover from a disaster, because the number that determines wheth...

A Written by Administrator
Your database restore is untested until you have timed it

The claim

Knowing that your database backups exist and even that they are valid is not the same as knowing you can recover from a disaster, because the number that determines whether a recovery is survivable is one most teams have never measured: how long the restore actually takes. A backup that restores in twenty minutes and a backup that restores in nine hours are completely different levels of protection, and if you have never run the restore end to end with a clock on it, you do not know which one you have — and you will find out during the outage, which is the worst time to learn that the answer is nine hours.

Why the backup existing is not the answer

Teams reasonably feel protected once backups are running and verified. But a backup answers only the question "can the data be recovered", and the question that actually governs your business during a disaster is "how long will we be down". Those are different, and the gap between them is the restore time. A perfectly valid backup that takes nine hours to restore means a nine-hour outage, and if your business cannot survive being down for nine hours, then your valid, verified, faithfully-running backup does not actually meet your recovery requirement — you just have not discovered that yet because you have never timed it.

The two numbers a recovery plan needs

Disaster recovery has two governing figures, and both must be decided as business requirements before they can be tested against reality:

  • Recovery time objective is how long you can be down. If the answer is "no more than an hour", then a restore that takes four hours fails the objective no matter how good the backup is.
  • Recovery point objective is how much data you can afford to lose — the gap between the last backup and the moment of failure. If you back up nightly, your recovery point is up to 24 hours, meaning a failure just before the nightly backup could lose a full day of data.

These are decisions, not facts about your current setup: the business decides what it needs, and then you measure whether the current backup strategy meets it. Most teams have never stated either number, which means they are running a backup strategy without knowing what it is supposed to achieve.

Run the restore and put a clock on it

The test that turns a hope into a known quantity is a full restore, from your real backup, into a clean environment, timed from start to serving queries:

Restore drill, timed end to end:
1. provision a fresh database host             [t0]
2. fetch the latest backup from where it lives
3. restore it completely
4. run integrity checks and a sample of real queries
5. record the total elapsed time               [t_end]
Restore time = t_end - t0

The elapsed time is the number you did not have. Fetching the backup over the network is often the surprise — a large backup in remote object storage can take hours to download before the restore even begins, and that download time is part of your real recovery time even though no restore documentation mentions it. Timing the whole thing, including the fetch, is what produces an honest figure.

What to do when the number is too big

If the measured restore time exceeds your objective, you have several levers, and knowing the number tells you which to pull:

Keep a warm copy. A replica that is continuously kept up to date can be promoted in minutes rather than restored in hours, turning a long restore into a short failover. This is the standard answer when the recovery-time objective is tight.

Restore more frequently in miniature. Smaller, more frequent incremental restores, or a standby that is regularly refreshed, keep the gap small and the promotion fast.

Move the backup closer. If the download dominates the restore time, keeping a recent backup on faster or nearer storage cuts the largest term.

Each of these costs money, and the measured restore time is what tells you whether the cost is justified — you are buying down a specific number of hours of downtime, and you cannot make that trade sensibly until you know how many hours are on the table.

Test the restore on a schedule, not once

A restore time measured once is a restore time that was true once. Databases grow, and a restore that took twenty minutes when the database was 5 GB takes far longer at 200 GB — the recovery time degrades silently as the business succeeds and the data accumulates. Schedule the restore drill, quarterly at least, so that the number stays current and you notice when growth has pushed it past your objective. The drill also catches the other failure: a backup that has quietly stopped being restorable because a format changed, a credential expired, or a step in the process broke — discovered on the schedule, on a calm day, instead of during the disaster it was meant to protect against.

The standard to hold

You do not have a tested backup until you have a measured restore time, a stated objective for how long you can be down, and confirmation that the first is smaller than the second. Anything less is a backup you hope will save you, running against a requirement you have never written down, with a recovery time you have never measured. Run the restore, start the clock, and turn all three unknowns into numbers — because the only backup that protects you is one you have proven you can recover from, in a time you can actually afford, before you are forced to do it for real.

#backups #disaster recovery #databases #operations

Keep reading