Every data center claims redundancy. Most have never tested it under conditions resembling a real failure.
The gap matters because redundancy is a behavioural property, not a component count. Two power feeds only work as redundancy if the failover completes inside the tolerance of the equipment above them. Two fabrics only work if reconvergence happens before your application gives up. The design document cannot tell you this. Removing the thing can.
Test by removal, not by inspection
The core discipline is simple: for every redundancy claim, remove one side and measure.
- Power: remove feed A on a representative sample of devices, measure whether the device survives and for how long. Then do feed B — because A/B assignment errors are common and testing only A hides them.
- Network: take down a single link, then a full uplink bundle, then a device. Measure reconvergence each time and compare it to your contractual target.
- Cooling: simulate the loss of a unit and observe inlet temperatures against ASHRAE guidance, with logging rather than a handheld reading.
- Storage: pull a path, then a controller, and measure the application-level impact rather than the array's opinion of itself.
Make the failure realistic
Two habits separate useful failover testing from theatre.
Fail it the way it would actually fail. A clean shutdown of a device is not the same as a power loss. Planned removals often hide the exact behaviour you care about, which is what happens during an unplanned one.
Test with load on. A fabric that reconverges in 200ms idle may take three seconds under production traffic. If you cannot test at full production load, test at the highest load you can justify and say so in the report.
When a test fails — the useful part
Failures are the reason you tested. What matters is what happens next.
Record it with severity, owner and a re-test date. Do not mark it passed because go-live is close. If it cannot be fixed before handover, state it in writing as residual risk with a mitigation and a date. A client who knows about a 900ms reconvergence on one path can plan around it; a client who discovers it during an incident cannot.
The best outcome of a failover test is a short list of things that did not work, found before go-live. A commissioning report with zero findings is either a very good design or an untested one.
What to keep
For each scenario: what was removed, the measured recovery time, the target, the observed behaviour, and evidence. Over time that pack becomes something more valuable than a commissioning document — it becomes the answer to a customer question you would otherwise have to answer with adjectives.
Need this applied to your site?
Our engineers run these checks as standard on every delivery. Send us the scope and we will tell you what we would look at first.