Disaster recovery for cloud workloads: an RTO and RPO primer

The question behind most disaster recovery conversations is: how much protection do we actually need, and how do we tell whether what we have is enough? You cannot answer either until two numbers exist for each system, and most organisations have neither written down.

Recovery time objective is how long the business can tolerate the system being unavailable before the consequences become unacceptable. It is a measure of downtime.

Recovery point objective is how much recent work the business can tolerate losing. It is a measure of data. If you back up nightly at two in the morning and the system fails at six in the evening, you have lost sixteen hours of work — so your real recovery point objective is sixteen hours, whatever the policy document claims.

The distinction matters because the two are solved by different things and cost different amounts. Downtime is reduced by standby infrastructure and rehearsed failover. Data loss is reduced by replication frequency. A business can quite reasonably decide it can wait four hours to be back up but cannot lose more than five minutes of transactions — and that combination has a specific and quite different architecture from the reverse.

Two further terms are worth having: the actual recovery time and recovery point, measured in a real test. The gap between the objectives on paper and the measured reality is the entire subject of this article.

The most common failure is a single organisation-wide statement: “our recovery time objective is four hours.” It is wrong in both directions simultaneously. It overprotects the internal wiki, which nobody would notice being down for two days, and underprotects the payment path, where four hours may be commercially unsurvivable.

The result is a plan that costs more than it should and still does not cover the thing that matters. Tier the workloads instead. Four tiers is enough for most mid-size organisations; the exact boundaries matter less than the act of forcing every system into one.

• Tier 1 — Critical. Revenue stops or a regulatory obligation is breached when this is down. Minutes to a couple of hours of downtime; minutes of data loss at most.
• Tier 2 — Important. Operations degrade significantly, staff cannot work normally, customers notice. Hours of downtime; up to an hour of data loss.
• Tier 3 — Standard. Inconvenient and workaroundable for a day. A working day of downtime; a day of data loss.
• Tier 4 — Deferrable. Can wait until the important systems are back. Several days of downtime; a day or more of data loss.

Two rules that stop the tiering from collapsing

First, the tier is decided by the business owner and the cost is presented alongside. Everyone declares everything Tier 1 until the price of Tier 1 is on the same page as the request.

Second, a system inherits the tier of anything it depends on. Your Tier 1 order system that authenticates against a Tier 3 directory service is a Tier 3 system wearing a Tier 1 label. Mapping dependencies is where most tiering exercises discover their real problem — the same discovery work described in our cloud migration readiness guide pays for itself twice here.

Backups underpin every tier, including the ones with sophisticated failover, because replication faithfully copies corruption and deletion to the standby site within seconds. Replication protects against infrastructure failure. Backups protect against mistakes and malice. You need both, and they are not substitutes.

What a defensible backup position looks like

• Multiple copies, more than one location. At least one copy separated from the production environment — different region, or different provider entirely.
• At least one copy that production cannot delete. Immutable or write-once storage, with retention that the production credentials cannot shorten. This is the single most valuable control against ransomware, because the attack that matters targets the backups first.
• Separate credentials. If the account that runs the application can also erase the backups, you have one failure domain, not two.
• Retention matched to detection time. Silent data corruption may be noticed weeks later. Seven days of backups is no protection against a problem introduced a month ago.
• Everything needed to rebuild, not just the database. Configuration, infrastructure definitions, secrets, certificates and DNS records. Teams routinely recover the data and then discover they cannot reconstruct the environment to put it in.
• Restores tested on a schedule. An untested backup is a hypothesis. Restore something real, quarterly, and record how long it took — that measured number is your actual recovery time, not the one in the policy.

Four patterns cover almost everything. The right choice per workload falls out of the tier you assigned, and the cost roughly tracks the position in this list.

Backup and restore

Data is backed up off-site; infrastructure is rebuilt when needed. Cheapest by a wide margin. Recovery time is hours to days depending on how automated the rebuild is — infrastructure-as-code is the difference between a day and an hour here. Suitable for Tier 3 and 4.

Pilot light

The core data layer is replicated and running at the recovery site; application servers exist as definitions and start on demand. Moderate standing cost. Recovery in tens of minutes to a couple of hours. A good fit for Tier 2, and the pattern that offers the best value for most mid-size businesses.

Warm standby

A scaled-down but running copy of the full environment, continuously replicated, scaled up on failover. Higher standing cost, recovery in minutes. Appropriate for Tier 1 where the business can tolerate a short interruption.

Active-active

Traffic served from two or more locations at once. Effectively no failover event, and the highest cost and complexity — data consistency across sites is a genuine engineering problem, not a configuration setting, and cross-region transfer is a continuous expense. Justified for the small number of systems where minutes of downtime carry serious cost.

One caution that applies to all four: the failover mechanism itself must be tested, because it is a system too. Health checks that never fire, DNS records with long time-to-live values, and standby databases that quietly stopped replicating three weeks ago are all ordinary findings in a first test. Automating and rehearsing this is continuous engineering work — the same discipline described in our DevOps practice, applied to recovery.

One row per system, filled in with the business owner rather than by IT alone. It fits on a single spreadsheet and takes an afternoon for a mid-size estate.

1. System name and business owner. A named person, not a department.
2. What stops if this is down? One sentence in business language.
3. Cost of an hour of downtime. Lost revenue, idle staff, penalties, remediation. An honest order of magnitude is enough; precision is not the point.
4. Cost of losing one hour of recent data. Often quite different from the downtime cost, and frequently higher for transactional systems.
5. Regulatory or contractual floor. Any commitment that sets a minimum regardless of the economics.
6. Upstream dependencies. What must be running for this to work — authentication, network, integrations, third-party services.
7. Target tier, derived from the four rows above.
8. Current measured recovery time and recovery point. From a real test. “Unknown” is a valid and very informative entry.
9. The gap between target and measured, and the specific change that would close it.
10. Annual cost of closing it, so the owner can decide against the downtime cost in row three.

An illustrative row

Hypothetically, and purely as an illustration rather than a real client: a distributor lists its order entry system with a four-hour downtime tolerance and a fifteen-minute data loss tolerance. Testing shows the actual position is a nightly backup and a manual rebuild — roughly a day of downtime and up to twenty-four hours of lost orders. The gap is stark, the fix is a pilot-light pattern with continuous database replication, and the annual cost of that fix is a number the finance director can weigh against a day of lost orders in peak season. The worksheet has done its job: it has turned a vague anxiety into a priced decision.

Expect the worksheet to reveal that two or three systems are seriously underprotected and that several others are costing money for protection nobody needs. Rebalancing usually funds a good part of the fix.

Business continuity is broader than infrastructure failover, and the non-technical gaps are the ones that turn a two-hour incident into a two-day one. A recovery plan should answer these before it is considered complete.

• Who declares a disaster? A named role with a deputy and the authority to trigger failover without a committee. Hours are routinely lost to nobody being willing to make the call.
• How do people communicate if the usual systems are down? An out-of-band channel and a contact list held somewhere that does not depend on the failed environment.
• What do staff do manually in the meantime? For manufacturing and retail especially, a workable paper fallback for a few hours is often cheaper than the infrastructure that would avoid it.
• Who tells customers and regulators, in what timeframe, and who approves the wording?
• How do you fail back once the primary is healthy, and how is the data written during the incident reconciled? Fail-back is the step most plans omit and the one that generates the worst data problems.
• Where is the plan stored? If the only copy is in the system that is down, there is no plan.

Everything above is documentation until it is exercised. A sensible annual rhythm, scaled to what your organisation can absorb:

1. Quarterly restore test. Restore one real system from backup into an isolated environment and verify the data is usable. Record the elapsed time.
2. Twice-yearly tabletop. Walk a scenario through with the people who would be involved. Cheap, disruptive to nothing, and reliably finds the missing decision authority and the out-of-date contact list.
3. Annual live failover for Tier 1 systems, during a planned window. This is the uncomfortable one, and the only test that produces a number you can defend.
4. Post-test correction. Every test produces findings. Assign them owners and dates, and re-test the fixes. A test whose findings are never closed is theatre.

Update the tiers whenever the business changes — a new product line, a new regulatory obligation or a large customer contract can move a system up a tier overnight, and nothing will tell you except a scheduled review. Where an internal team cannot sustain that cadence, it is exactly the kind of work a managed service is built to carry.