Ever had sleepless nights over Tomcat?
Sometimes things should work, though they just do not. And the reason, even for highly knowledgeable and experienced professionals, can remain uncovered for a day or two… It happened to be Tomcat that caused the issue.
In an ongoing project, when deploying the newest version of the solution at test environment all worked flawlessly. The client’s test and production environments were identical as configuration, so the same results were expected when deploying the latest version at production. After moving it though, everything stopped working.
First, we suspected that after all there is some configuration issue, as the code deployed was identical for both environments. Then, we started debugging [the solution is divided into a portal and web services façade – the portal calls the façade (located at a different server) and the portal accesses the WSDL]. The first check was successful – the machine where the portal is deployed duly accesses the other machine. The second check – yes, from the portal server we can access the façade at the port where the application server (Tomcat) is. We felt there was an important piece missing and we couldn’t quite put our finger on it… Then, we just decided that the best place to start when you’re stumped is to go back to the basics.
We finally discovered that Tomcat itself, from the application cannot access the façade. A breakthrough! So we had the following situation – from operating system point of view you can access both the machine and the port where the application is deployed (the issue is not in the firewall, in the network connection, etc.). Thus, the challenge is hidden in the application server.
The setback was in the Java Option at AS level – the portal had a proxy that said: “If somebody tries to access IP 10.0.0**, go through the Internet”. While deploying the application, we followed the instructions in the configuration files where was written that we should access the façade at IP 10.0.0**. Thus, instead of going to the façade, the request was going to the Internet and was failing.
This was a very subtle issue that can always occur when you are dealing with an unknown environment that you should support – always check for hidden configurations. Each application server could have a proxy and in case of stumbles, it is a smart decision to check on that part either through debugging or through sniffing the network traffic.
ScaleFocus IT Infrastructure Team