Old Test Types Need Not Apply

[ originally at jlottosen.wordpress.com, Dec 2022 ]

I primarily work in situations that are less about application delivery and more about moving the whole system stacks, implementing a standard system, or similarly changing the organizational IT landscape. Some would list these projects as “staff projects” if that helps you. I often find the terms Regression test, SIT, and UAT to be misleading and not helpful to what kind of test is needed for my examples.

Example 1: We are rebuilding all the environments in the delivery stack. All the Dev-, Test-, Integration-, Preprod-, and Prod- environments, and their underlying databases, brokers, and websites. Every time we are constructing, an environment we will be testing the setup from a baseline version of the existing running systems. A baseline that we know is functional already. The classic software development testing types don’t really help us in this situation, as neither regression test, UAT, or SIT conveys the things we want to confirm and the learnings we want to explore.

SIT and UAT has become generic term that it has lost the strength to convey the needed quality narrative. If you do CI/CD (which you should) for your application development that such be sufficient. If you figure out, you need to do a “connection alive” test for your third-party integrations that you move from one environment stack to another that should be accepted with the acknowledgment that you actually considered the challenges ahead.

It’s all about the risks and the mitigations – less about testing everything to the dot. One tool to read the landscape is to communicate curiously about what the stakeholders value more – and value less. And on the other hand, consider the nature of the solution being proposed.

Example 3: Setting up a cloud-based Azure Active Directory – the solution comes with a given security level out of the box (OOTB). As with other OOTB and Software-as-a-service solutions you have little impact on the security features of the solution, besides some simple configurations. While you might think that all security requirements would require 100% acceptance testing coverage, what you want to accept is that they might be provided “by design” – or by a solution decision made long ago.

I would prefer that we can call things what they are and not blindly apply old testing types