Disaster Recovery Testing: Essential Cloud Risk Checklist

Disaster recovery testing is the only evidence that a recovery plan works. Everything else is a document. The runbook in the shared drive, the replication dashboard showing green, the vendor’s four-nines commitment and the line in the board pack saying “DR in place” are all statements of intent, and none of them has ever restored a database. The first time an organisation discovers whether it can recover should not be the day it has to.

That gap is uncomfortably common. Plans are written when a system is commissioned, then quietly diverge from reality as the estate changes around them, and the divergence is invisible until somebody tries a real restore. A cloud estate makes this worse rather than better, because the illusion of resilience is stronger: the infrastructure is somebody else’s, the replication is automatic, and the console is full of reassuring green ticks that describe the platform’s health rather than yours.

A disaster recovery test is the cheapest way to close that gap, and it is the one exercise that produces a number rather than an opinion. If you are still working out which of backup, failover and redundancy you actually have, our guide to backup versus disaster recovery versus high availability is the right starting point, and our incident response practice covers what happens in the hour before recovery even begins.

This is a practical checklist for a disaster recovery test in a cloud or hybrid environment: what to test, in what order, how often, who needs to be in the room, what to measure while the clock is running, and what evidence to keep when it is over. It covers the five levels of test from desk walkthrough to full regional failover, the cloud-specific failure points that only surface under a real restore, a twelve-month calendar you can lift straight into a plan, and the failures that turn up again and again in first tests across cloud infrastructure of every size.

Table of contents

Why the disaster recovery test is the control everyone skips
What a disaster recovery test actually proves
The five levels of disaster recovery test, from walkthrough to full failover
Scoping: what to test first when you cannot test everything
RTO and RPO: the numbers a disaster recovery test either confirms or destroys
The disaster recovery test preparation checklist
Cloud-specific failure points a disaster recovery test exposes
Running the test: an hour-by-hour disaster recovery test runbook
What to measure during a disaster recovery test
How often to run a disaster recovery test
The failures a disaster recovery test finds most often
Evidence, audit and regulatory expectations
What a disaster recovery test costs, and what it saves
A twelve-month disaster recovery test calendar
Frequently asked questions about disaster recovery testing
References

Why the disaster recovery test is the control everyone skips

Nobody argues against testing. It simply never reaches the top of a list that also contains a migration, an audit and a release, and it carries a risk profile that makes deferral feel responsible.

Testing looks like a risk, not a control

A disaster recovery test deliberately introduces the failure everybody spends the year avoiding. The instinct to postpone it until “a quieter month” is understandable and entirely self-defeating, because the quiet month never arrives and the untested plan keeps ageing. A controlled test in a maintenance window is the cheapest failure you will ever buy.

The plan ages faster than anyone expects

Estates change weekly. A new managed database, a third-party API added to a checkout path, a certificate moved to a different key vault, a subnet resized. Each change is small; none triggers a plan review. Twelve months of small changes is usually enough to make a recovery runbook wrong in at least three places, and only a disaster recovery test finds which three.

Green dashboards measure the wrong thing

Replication health tells you bytes are moving, which is why no dashboard can substitute for a disaster recovery test. It says nothing about whether the target can be started, whether the application can authenticate against it, whether DNS will point anywhere useful, or whether the person with the permission to trigger failover is contactable at two in the morning. Recovery is an end-to-end property and dashboards measure components.

Assumed knowledge disappears

Recovery plans are written by people who know the system. They quietly assume the reader shares that knowledge, so steps compress into phrases like “restore the database and reconnect the app”. When the author has left, that sentence can cost four hours. Running a disaster recovery test with someone who did not write the plan is how the assumption surfaces, and it is why the operator on the day should be an ordinary member of the rota.

What a disaster recovery test actually proves

Being precise about the claim you are testing keeps the exercise honest and stops it collapsing into a demonstration that everything is fine.

It proves recovery time, not recovery intent

A disaster recovery test replaces a target with a measurement. The output is not “we recovered” but “we recovered the order platform in three hours and fifty minutes with two manual interventions and one undocumented step”. That sentence is worth more than any plan document, because it can be compared, trended and improved against your agreed RTO and RPO targets.

It proves the data is usable, not merely present

Backups that complete are not backups that restore, and only a disaster recovery test tells the two apart. Corruption, partial application consistency, missing transaction logs, encrypted-by-ransomware snapshots and an archive tier that takes twelve hours to rehydrate all present as healthy backup jobs. A restore is the only test of usability, which is why immutability matters and why our guide to immutable backup and the 3-2-1-1-0 rule sits underneath this one.

It proves people can execute under pressure

Half of every recovery is human: who declares the disaster, who authorises the failover, who tells customers, who decides that the partial restore is good enough to open the doors. A disaster recovery test with a perfect runbook and no decision-maker present has proved only half the system works.

It proves dependencies you forgot you had

Identity providers, DNS, certificate authorities, licence servers, payment gateways, SFTP partners and the monitoring platform itself are all dependencies of recovery. Most first disaster recovery tests fail on one of them, and none of them appear on the architecture diagram that was used to write the plan.

Claim	Evidence that supports it	Evidence that does not
We can recover in four hours	A timed restore with a stopwatch and a witness	An RTO written in a contract
Our backups are good	A restored database passing an integrity check	A green backup job history
We can fail over to region B	Traffic served from region B for one hour	Replication lag under one minute
The team knows what to do	A test run by the on-call rota, not the architect	A signed-off runbook
We lose at most 15 minutes of data	A measured recovery point on restored records	A snapshot schedule set to 15 minutes

The five levels of disaster recovery test, from walkthrough to full failover

Testing is not binary. Levels exist so you can build confidence incrementally, and so a small team can prove something useful this quarter without a full regional failover.

Level 1: plan walkthrough

The cheapest form of disaster recovery test is a structured read of the runbook with the people who would run it. Cheap, fast, and surprisingly productive on a first pass because it exposes missing contact details, dead links and steps that only make sense to the author. Budget two hours and expect a dozen corrections.

Level 2: tabletop exercise

A facilitated scenario where the team talks through decisions and timings without touching infrastructure. This is where declaration criteria, communications and decision authority get tested, and it is the cheapest way to rehearse the human half. Our walkthrough of how to run a cyber tabletop exercise covers the format in detail.

Level 3: component restore

An actual restore of one system into an isolated environment: a database, a file share, a virtual machine, a container image and its configuration. This is the first level of disaster recovery test that produces hard numbers, and it is the level most organisations should be running monthly.

Level 4: full application failover

The whole service stood up in the recovery target and proven to work end to end, including identity, DNS, integrations and payment paths. Traffic may be synthetic or a small slice of real users. This is the level that earns a defensible recovery time claim.

Level 5: unannounced or region-wide failover

A live failover of production, ideally with limited warning, occasionally with an injected fault. This is where mature teams end up, borrowing techniques from chaos engineering, and it is not where anybody should start.

Level	What it touches	Typical duration	Production risk	Suggested frequency
1. Walkthrough	Documents only	2 hours	None	Quarterly
2. Tabletop	People and decisions	Half a day	None	Twice a year
3. Component restore	One system, isolated	2–6 hours	Very low	Monthly
4. Application failover	Full stack in recovery target	1–2 days	Low, in a window	Annually per tier-1 service
5. Live or unannounced failover	Production traffic	1–2 days plus failback	Real, managed	Annually, when mature

Scoping: what to test first when you cannot test everything

Nobody has the budget to fully exercise every system every year. Scope is therefore the first design decision, and it should be driven by business impact rather than by which system is easiest to restore.

Rank services by outage cost, not by technical interest

Let the cost of an outage decide which disaster recovery test happens first. Work out what an hour of downtime costs each service in lost revenue, idle staff, contractual penalty and regulatory exposure. The ranking usually surprises people: an unglamorous file share or an identity service often outranks the flagship application, because everything else depends on it.

Test the dependency, not just the crown jewels

If the order platform depends on a directory, a message broker and a payments partner, testing the order platform in isolation proves very little. Draw the dependency chain first and make sure the shared services underneath appear in scope at least once a year.

Pick a scenario, not a system

“Restore the CRM” is a task. “The primary region is unavailable for six hours” is a scenario, and a scenario-shaped disaster recovery test forces decisions about sequencing, capacity and communication that a single-system restore never surfaces. Scenario-driven scope is what makes a test resemble a real event.

Write the scope down before you start

A one-page scope naming the systems, the scenario, the time box, the success criteria and the explicit out-of-scope items prevents the most common failure mode of all, which is a disaster recovery test that quietly expands until it is abandoned at midnight with nothing proved.

RTO and RPO: the numbers a disaster recovery test either confirms or destroys

Recovery time objective and recovery point objective are the two numbers everything else hangs from, and they are usually inherited rather than derived.

Where the targets come from

RTO is how long the business can be down; RPO is how much data it can afford to lose. Both should come from the business, be signed by someone accountable, and be set per service rather than per organisation. A single company-wide four-hour RTO is a sign that nobody has done the analysis.

The difference between the objective and the actual

The measured values have their own names, and using them changes the conversation. Recovery time actual and recovery point actual are what the test produced. A disaster recovery test that reports RTA against RTO turns an argument about capability into a gap you can close with a plan and a budget.

Why a first disaster recovery test misses by so much

The first measured recovery is routinely three to five times the target. The overrun is rarely the restore itself; it is the decision delay, the credential hunt, the DNS propagation and the validation nobody had scripted. Those are all fixable, and they are only visible with a stopwatch.

Measured recovery time against a 4-hour target, first test versus third test

Target RTO 4 hours

First test, unrehearsed 19 hours

Second test, gaps closed 9 hours

Third test, scripted and rehearsed 4.5 hours

The disaster recovery test preparation checklist

Preparation decides whether the day produces evidence or a mess. Work through this list before the test window opens.

Confirm the scope, scenario and time box in writing

Name the systems, the failure being simulated, the start and stop times, and the point at which you will abandon and roll back. An abandoned disaster recovery test is a valid outcome; an open-ended one is not.

Nominate roles, including the ones nobody remembers

Test lead, technical operators, an observer with a stopwatch and a notebook, a business validator who can confirm the recovered service is genuinely usable, and a decision-maker with authority to declare and to stop. The observer is the role most often skipped and the one that produces the evidence.

Verify access before the day, not during it

Break-glass accounts, multi-factor devices, privileged roles in the recovery subscription, console access for the person who will actually click, and a way in if the identity provider is part of the outage. Credential failures cause more overrun in a disaster recovery test than any technical fault.

Freeze changes and warn the right people

No deployments during the window. Tell the service desk, the affected business owners and any third party whose monitoring will light up. An unannounced test is a valid level 5 exercise, but the people who need to know it is a test must still know.

Prepare the rollback before you break anything

Know exactly how you return to normal, how long failback takes, and what happens to data written during the disaster recovery test. Failback is routinely harder than failover and is the step most likely to cause genuine damage.

Set up measurement in advance

Agree what you are timing and who writes it down. A shared timeline document with timestamps for each milestone is enough, and it beats reconstructing the day from memory a week later. Keep your monitoring in view throughout so you can see the recovered service from the outside as a user would.

Cloud-specific failure points a disaster recovery test exposes

Cloud recovery fails in its own characteristic ways, and almost none of them appear in a plan written from an on-premises template.

Capacity is not guaranteed in the recovery region

A pilot-light design assumes the instance types you need will be available when you scale up. During a genuine regional event, everybody else is scaling up in the same neighbouring region at the same time. Reserved capacity or a warm standby is the only real answer, and running the disaster recovery test during a busy period is how you find out whether you have one.

Service quotas follow the account, not the plan

Default limits on cores, addresses, load balancers, database instances and API calls apply per region and are frequently untouched in the recovery region because nothing has ever run there. A quota request can take hours, which is the entire RTO gone.

Identity, keys and secrets do not always replicate

Encrypted volumes and databases are useless without the key, and keys are often regional. The same applies to secret stores, certificate private keys and service principals scoped to a single region or subscription. Restoring data you cannot decrypt is a common and entirely avoidable first-test failure.

DNS, TTL and the propagation you did not plan for

Failover is a DNS change for most estates, and a record with a 24-hour time to live will keep sending users to a dead endpoint long after the recovery is complete. Lower TTLs before the disaster recovery test and confirm the change actually propagates for a real client, not just for your own resolver.

Networking overlaps and hard-coded addresses

Address ranges that overlap between primary and recovery, hard-coded IP addresses in application configuration, firewall rules referencing the old subnet and partner allowlists naming a specific egress address all break silently. Partner allowlists in particular can take days to change.

The control plane may be part of the outage

If your automation, container registry, pipeline or secrets manager lives in the region that has failed, the tooling you planned to recover with is unavailable. This is the dependency loop that turns a four-hour recovery into a two-day one.

Cloud failure point	How it presents	Time it typically costs	Pre-test fix
Instance capacity unavailable	Scale-up fails in recovery region	1–6 hours or unrecoverable	Reserve capacity or run warm standby
Service quota limits	Deployment stops part-built	2–24 hours	Raise quotas in the target ahead of time
Regional encryption keys	Restored data will not mount or decrypt	Hours, sometimes total loss	Replicate keys and test a decrypt
DNS time to live	Users still hitting the dead site	1–24 hours	Lower TTL to 60–300 seconds
Partner IP allowlists	Integrations fail after recovery	1–5 days	Pre-register recovery egress addresses
Archive tier rehydration	Backup exists but is not readable yet	3–12 hours	Keep recent restore points on hot storage
Tooling in the failed region	No pipeline, registry or secrets access	Doubles the recovery	Host recovery tooling independently

Running the test: an hour-by-hour disaster recovery test runbook

A test day runs better with a shape. This one assumes a level 4 application failover in a maintenance window, and compresses well for smaller scopes.

T minus 7 days: readiness review

Confirm the scope, roles and access for the disaster recovery test. Lower DNS TTLs. Raise quotas. Take a fresh backup and note its identifier. Circulate the timeline so nobody discovers their role on the morning.

T minus 1 hour: baseline and freeze

Record the current state: replication lag, last successful backup, open incidents, deployment freeze in effect. Start the timeline document. Confirm the rollback owner is present and awake.

T zero: declare and start the clock

The decision-maker declares the simulated disaster using the same criteria and the same channel they would use for a real one. Start timing here, not at the first technical action, because declaration delay is part of your recovery time and is usually the largest single component.

T plus 0 to 4 hours: recover in priority order

Work the runbook in dependency order: identity, network, data, application, integrations. The observer records every step, every deviation and every moment somebody has to ask a question the runbook should have answered.

Validation: the step teams cut when they run late

A disaster recovery test is not finished when a service responds; it is finished when a business validator confirms real transactions work end to end. Log in, place an order, run a report, send a message to a partner. Unvalidated recovery is the most expensive kind of false confidence.

Failback and stand-down

Return to the primary in a controlled way, reconcile any data written during the test, confirm replication has resumed in the correct direction, and hold a short debrief while the detail is fresh. Then write the findings down within forty-eight hours, before memory tidies them up.

What to measure during a disaster recovery test

Measurement is what separates a test from a demonstration. Six numbers cover most of what matters.

Recovery time actual, split by phase

Total elapsed time is the headline number from any disaster recovery test, but the split is where the improvement lives. Break it into declaration, provisioning, data restore, application configuration, integration and validation. One phase almost always dominates, and that is your next quarter of work.

Recovery point actual

Inspect the recovered data and establish the timestamp of the last committed transaction. Compare it with the moment of simulated failure. This is the only honest measure of data loss, and it frequently differs from the replication interval.

Manual intervention count

Count every step of the disaster recovery test that required a human decision or a hand-typed command. Manual steps are where errors and delays concentrate, and reducing the count is usually the cheapest way to reduce recovery time.

Undocumented step count

Every action taken that was not in the runbook is a defect in the plan. Recording these is the single most valuable output of a first disaster recovery test, and the count should fall towards zero over successive rounds.

Data integrity result

A pass or fail from a defined check: row counts, checksums, a reconciliation report, a sample of records verified by the business. Without this, “the database came up” is all you can claim.

Communication timeliness

Time from declaration to first internal notification, and to first customer-facing statement. Regulators, insurers and customers all care about this, and it is easy to measure and easy to improve.

Where the elapsed time goes in a typical first full failover

Declaration and decision delay 21%

Data restore and rehydration 29%

Application and integration config 24%

Access and credential problems 14%

Validation and sign-off 12%

How often to run a disaster recovery test

Frequency should follow criticality and rate of change, not the calendar year. A system that changes weekly needs testing more often than one that has not been touched since commissioning.

Tie the interval to the tier

Set the disaster recovery test interval per tier and publish it. Tier-1 services earn an annual full failover plus quarterly component restores. Tier-2 services can manage an annual component restore and a tabletop. Tier-3 can be covered by a walkthrough and a restore sample. Write the tiers down and hold to them.

Test after every material change

A migration, a major version upgrade, a new integration or a change of provider invalidates part of the plan. A short targeted disaster recovery test after the change is far cheaper than discovering the gap during an incident, and it fits naturally alongside a cloud migration cutover and rollback plan.

Automate the small tests so the big ones stay affordable

Automated restore verification, scripted integrity checks and scheduled snapshot mounts turn the monthly disaster recovery test into a job rather than an event. That frees the budget and appetite for the annual full exercise that actually proves the claim.

Service tier	Example	Target RTO	Full failover	Restore verification
Tier 1, revenue critical	Order platform, payments	1–4 hours	Annually	Monthly, automated
Tier 1, shared dependency	Identity, DNS, message broker	1–4 hours	Annually	Monthly, automated
Tier 2, operationally important	CRM, ticketing, file services	8–24 hours	Every 2 years	Quarterly
Tier 3, deferrable	Reporting, archives, intranet	2–5 days	Not required	Twice a year, sampled

The failures a disaster recovery test finds most often

The same handful of problems appear across estates of every size and in every sector. Knowing them in advance turns a first test from a humiliation into a checklist.

Nobody was authorised to declare

The technical team waited for a decision that had no named owner, which is the finding a first disaster recovery test produces more often than any technical fault. Fixing it costs one paragraph in a policy and saves hours in every future event.

The runbook referenced things that no longer exist

Dead links, a decommissioned jump host, a monitoring tool replaced last year, a contact who left. Version the runbook, review it quarterly, and make the reviewer someone who did not write it.

The restore worked and the application still did not

Connection strings, feature flags, message queue endpoints, scheduled jobs and cached configuration all point at the primary. Application configuration is where most level 3 successes become level 4 failures.

Backups were replicating the corruption

Ransomware and logical corruption propagate to replicas faithfully and quickly. This is the case for immutable, offline or air-gapped copies with a retention window longer than your detection time, and it is a core part of any serious cybersecurity posture rather than a backup detail.

The disaster recovery test was scoped so narrowly it proved nothing

Restoring one virtual machine into an empty subnet proves the snapshot exists. It does not prove the service recovers. Scope creep is a risk, but scope collapse is far more common.

Failure found	Root cause	Fix	Effort to fix
No declaration authority	Policy gap	Name a role and a deputy, publish criteria	Hours
Credentials unavailable	Break-glass never tested	Quarterly access rehearsal, offline copies	Days
Config pointing at primary	Environment values hard-coded	Externalise config, parameterise endpoints	Weeks
Encryption keys missing	Regional key scope	Replicate keys, test decrypt in target	Days
Integrations rejected traffic	Partner allowlists	Pre-register recovery addresses	Weeks, external
Restore slower than RTO	Archive tier and bandwidth	Hot restore points, parallel restore	Days to weeks

Evidence, audit and regulatory expectations

A test that leaves no record is worth very little to an auditor, an insurer or a customer running due diligence on you.

Keep an evidence pack, not a memory

Close every disaster recovery test with a pack: scope document, timeline with timestamps, screenshots or console output at key milestones, the integrity check result, the list of defects raised and their owners, and a signed summary. That pack answers most audit questions without a second conversation.

Regulated sectors already require this

Financial services firms in the UK operate under specific expectations for outsourcing and operational resilience, and firms in scope of the EU’s digital operational resilience regime face explicit testing obligations with defined frequencies. If you are in scope, the disaster recovery test schedule is not a matter of internal preference.

Insurers and customers ask for it

Cyber insurance questionnaires and enterprise supplier reviews increasingly ask when the last disaster recovery test ran and what the measured recovery time was. A dated evidence pack is a commercial asset, and it pairs naturally with the assurance work in a cloud security posture assessment.

Feed the findings into risk, not just into a ticket queue

Defects with no owner and no date are decoration. Route them into the risk register with an accountable owner and a review date, and report the trend to the board with the recovery time trend beside it. That is what IT governance looks like in practice.

What a disaster recovery test costs, and what it saves

Cost is the usual objection, and it is almost always overstated because the comparison is made against zero rather than against the outage.

The realistic cost of a first full test

For a mid-sized estate, budget three to five engineer-days of preparation, one to two days of execution, a day of write-up, plus any temporary infrastructure in the recovery target. Most organisations land between four and twelve thousand pounds for a level 4 exercise, less if it runs inside an existing managed IT services arrangement.

The cost of the alternative

Compare that with a single day of unplanned downtime. For a business turning over ten million pounds a year with a heavily digital operation, an eight-hour outage runs to tens of thousands in lost trade before considering recovery labour, penalty clauses, regulatory reporting and the customers who do not come back.

The savings are mostly avoided overrun

The value of a disaster recovery test is not that it prevents disasters. It is that it converts a twenty-hour uncontrolled recovery into a four-hour rehearsed one. That difference is the whole return, and it compounds with every test because the fixes are permanent.

Indicative cost of testing against the cost of one untested outage, mid-sized estate

Annual walkthroughs and tabletops £3k

Monthly automated restore checks £6k

Annual full failover exercise £10k

One 8-hour unplanned outage £40k+

Multi-day recovery, untested plan £75k+

A twelve-month disaster recovery test calendar

A calendar removes the annual argument about whether there is time. Publish it, put the dates in diaries, and treat them like an audit date rather than an aspiration.

Build the year around one anchor exercise

Fix the annual full failover first, ideally in a genuinely quiet trading period, and build the smaller disaster recovery tests around it so each one retires a risk that would otherwise surface on the big day.

Alternate scenarios so you do not rehearse one story

Rotate through regional outage, ransomware with corrupted replicas, accidental deletion by an administrator, and third-party provider failure. Each exposes a different weakness, and the ransomware scenario is the one most likely to test whether your backups are genuinely immutable.

Keep a standing agenda item for the findings

Fifteen minutes in the monthly operations meeting to review open defects from the last test keeps the improvement loop alive between exercises. Without it, the report is filed and the same findings reappear next year.

Month	Activity	Level	Owner	Output
January	Runbook review and tier confirmation	1	Service owner	Updated runbook
February	Automated restore verification, tier 1	3	Platform team	Integrity report
March	Tabletop: ransomware with corrupted replicas	2	Security lead	Decision log
April	Component restore: identity and directory	3	Infrastructure	Timed result
June	Full application failover, tier 1	4	Test lead	Evidence pack
August	Access and break-glass rehearsal	3	Security lead	Access report
September	Tabletop: third-party provider failure	2	Service owner	Decision log
November	Component restore: tier 2 systems	3	Platform team	Timed result
December	Annual report to board and risk register update	–	IT leadership	Board paper

Frequently asked questions about disaster recovery testing

How long does a first disaster recovery test take?

Plan for a full day of execution for a single tier-1 service, plus roughly a week of preparation spread across a few people. First tests overrun, which is why the time box and the abandon criteria matter more than the ambition.

Can we test without touching production?

Yes, and most disaster recovery testing should be. Component restores into an isolated environment, replica databases mounted from snapshots and failover tests using synthetic traffic all produce real numbers without customer risk. Only level 5 requires production exposure.

Who should run the test?

The people who would run a real recovery, which usually means the on-call rota rather than the architect who designed the system. If the test only works when one specific person is present, you have found a finding.

What if the test fails badly?

That is a successful disaster recovery test. A failure discovered in a maintenance window with a rollback ready costs a day; the same failure discovered during an outage costs the business. Record it, fix it, retest the specific gap within a quarter.

How does this relate to business continuity planning?

Disaster recovery covers the technology; business continuity covers how the organisation keeps trading, including premises, people and manual workarounds. The disaster recovery test should feed the wider continuity plan, and the two are best exercised together at least once a year.

Do managed service providers test on our behalf?

Some do, many do not, and almost none do it end to end without being asked. Check whether your contract includes a scheduled disaster recovery test, who attends, what evidence you receive, and whether the scope covers your applications or only their platform. If it is silent, it is not happening.

What is the single highest-value first step?

Restore one important database into an isolated environment and time it honestly. It takes an afternoon, needs no policy change, and the result will tell you more about your true position than any plan review.