Often the testing mindset is mistaken for a disruptive and hacking mindset, while it can be beneficial to explore a system critically the general system owner would prefer avoiding service disruptions. In large enterprise and public services systems the system owners often prefer and sponsor that it’s tested that the solution can indeed withstand and recover from disruptions.
11 software testing mindsets
https://www.ministryoftesting.com/insights/i-think-therefore-i-test-the-importance-of-thinking-for-testers
Disruptions it the mind of a system owner (your sponsor for the testing activities) could be in the form of hacking attempts, ransomware, system availability and other outside forces denying access to the system or primary components of the system. The examples of costly large scale disruptions are well-known and often it’s not a matter about if disruptions will happen – but when. Another concern for your system owner could be regulations around cyber resilience that put heavy fines on those companies who’s digital products have disruptions. A lst risk to bring up could be geopolitical blockades or local regulatory constraints on specific system components. On the later could be a ban to use a specific hardware vendor.
Testing activities for recovery, business continuity, backup and restore might not be classical “testing fields” – but given the above it’s something that especially staff-level testing people should have on their list. The actual testing activities are probably performed in collaboration with the security and operations team, and the DevOps team if you have it.
The key business requirement
There are usually two key requirements in testing for disruption and the terms are a little tricky, while they sound the same (and are often switched) they have each their key message:
- RTO (Recovery Time Objective) is about Time. In a gaming analogy there is a huge difference between respawning instantly to a nearby spot/time on one hand and on the hand other having to exit the game, find save file, reload the game and proceed to the place of disruption. Time to load the latest save-file.
- RPO (Recovery Point Objective) is about data. In a gaming analogy, what do you carry at the time. Your inventory, configuration, units, points etc. What is included in the latest save-file.
Both RTO and RPO are critical for the business as it’s equally (?) costly to have both data loss (business transactions) and time loss (business unavailability). RTO & RPO is also discussed here: https://www.o2sn.dk/2024/06/17/aligning-with-management/
image source: https://www.techtarget.com/it-infrastructure/feature/RPO-vs-RTO-Key-differences-explained-with-examples
Levels of disruption
| Level | Example |
| Business Continuity | have a simulated session with management on how to communicate and collaborate |
| Fail over | shut down your main system, see that the secondary data center takes over (if you have one…) |
| Application recovery | delete the application from the servers, rebuild from build scripts etc. |
| Infrastructure recovery | delete the VM’s, kubernettes and cloud structures, rebuild from pipelines and provisioning scripts |
| Database recovery | take a backup of your database, delete the running database, load the backup |
And there are probably more, and in the TechTarget link above a more formal definition and framing. Special mention to Netflix chaos monkey suite who real-time shuts down services in production. This is the ultimate level of shift-right, which addresses modern observability engineering. But that’s a post on it’s own.
I you look at this and think, “oh that would be costly here” or “that would take like forever“- that is exactly your cue to get the dialog going with your sponsor! Bring it up and plan a disaster party.
One response to “Testing for Disruptionš”
mentioned in the ministry of testing newsletter:
https://www.ministryoftesting.com/newsletter/observatory-weekly-quality-management-systems-testing-for-disruption-and-job-hunting-with-a-bot