Network and DNS. Zero. The biggest issues in my recent projects are integration partnerships, network settings and authentication.
I’m currently doing a major transition project, where we are moving a large public finance system from one data center to another data center. For reasons, mostly to do operational costs. But it could equally be about quality of service, national resilience or a similar high level aspect. The contract is yet another two-digit million dollar deal.
Hence the key business challenge here is not if the solution works. Because it does – both with regards to business features, the software in it self and security levels. That is – it works as well as it did before. It’s same-same, just different. Which trapped even me for a while. For what ever reason I was hooked to introduce better test automation pipelines for the API testing. While interesting in the long run – the current conundrum was about the actual changes of this enterprise lift & shift program.
During the project we have had issues like these as our primary blockers – setting us back weeks:
- Difference in terminating SSL-certificates
- Differences in Internet breakout points
- Difference in partner systems being available
Dealing with partner system
This public finance system has 30 or so integrations to various public services, to various public data brokers and data warehouses. They are all housed at different vendors each of them having their own stack and list of environments. Stubs would be nice, but cannot be taken as a given. There are a few systems that only have a Production System to verify against. And unfortunately only a few, where there is a full “test environment as a service” aka test stubs as a strategic choice.
The later is a dream to work with. We can connect to a range of endpoints besides the actual “business feature”. We can setup data and tear the whole thing down over and over again without interacting with any people on the partner side or actual existing operations . Where there is only a Prod system, it’s more of a nuisance, that require careful planning and can best be performed when switching to the new platform. And require a lot of interactions with the people of the partner systems: when can we meet, what data can we use, etc.
Connection testing: This is about integration with other companies’ online services. Some call it system-to-system integration testing, which is about system-to-system communication with no end-user. It’s about taking input, processing it, and sending it along. That is the easy part. The tricky part is coordinating with the other systems concerning integration points, consistency of data, and synchronizing updates to all the involved systems.
Leading Testing Activities
Dealing with technical issues
At the same time as we are building a new system site for our customer – actual operations are ongoing on the old side. So when we ask the people maintaining the current integrations, our work is it’s not a benefit to them. It’s actually often annoying that we come and request test systems and test data to interact with. And even so for network certifications and updates to various allow lists and certificates. We have an ongoing issue with one integration that only works on the previous system authentication, but when we update it with ours it fails. Yet the same system authentication works fine elsewhere in the environment. It’s probably DNS …
As mentioned it’s a new environment, and going forward the business system will represent itself from another IP address from the new data center (aka internet breakout). Many of the partner systems have an allow list, where the new data center IP needs to be added. If multiple instances of a system is even possible. Sometimes it’s not. Often in advance of the actual final cut over, we need to ask for a service window at the partner. Change the IP, make a small connectivity test and change it back. Just to gain confidence that nothing else is wrong. But often there is.. a network setting or routing somewhere.
Once upon a time, we where half way on moving things from one data center to another when we discovered heavy performance issues. One server was in the UK data center and another in a DK data center. The traffic between the servers was so frequent than even a 20 ms latency slowed the whole thing down to an unusable state. We had to shift the DK server back to the UK site and move them together.
The lesson for you if you do a similar migration or transition test – is to focus on network and integrations. But the key lesson is to primarily look where you make the actual changes that matter. While we have changed the infrastructure, it doesn’t matter (in this story) – but integrations matter in a highly integrated system environment.