Issue: Application not able to connect to the oracle database. Connections are not going through and even when you try to telnet to the IP address, you get error message

telnet 172.32.362.171 1521

Connecting To 172.32.362.171…“Could not open connection to the host, on port 1521: Connect failed”

Troubleshooting steps

  • Checked for Firewall setting and checked fire wall logs to confirm that access is being allowed through the firewall. Firewall does show connection attempts going through it.
  • Checked for database listener and listener logs for any errors or connection attempts refused, however there were no errors or information on connection attempts.
  • Checked for listener and tnsnames entries for any incorrect information.
  • Done network packet trace at the database host to confirm
    • Whether or not the database server is seeing the traffic from the application server
    • Whether database server is sending back the response to the application server.
    • After doing network trace it was confirmed that “traffic was only going in towards database, however database host server was not responding or acknowledging any packet requests”
    • On database host, done netstat –a, and from the results we could clearly see that packets are reaching from 172.32.334.100, however their status is SYN_RCVD, which means a remote host has asked for starting a connection however confirmation message is not being send to 172.32.334.100. This may be because of firewall from 172.32.362.171 to 172.32.334.100 or may be because of difference in routing. For successful connection netstat status must be “ESTABLISHED” instead of “SYN_RCVD”.

test1clust.1521      172.32.334.100.3934      0      0 49640      0 SYN_RCVD

test1clust.1521      172.32.334.100.3938      0      0 49640      0 SYN_RCVD

  • Above information proved that there are issues around database host server itself. This would be more likely the case if any of the following are true
    • Firewall issues (which we have already checked and confirmed that there are no issues around this)
    • Check if host-based firewall is not allowing these connections to go up to DB listener (hence the DB listener is not seeing any connection attempts and hence not responding back).
    • Database host doesn’t have static route to correctly route back response traffic back to the required subnet.
    • Upon checking the route, it was found that there is no route for required subnet
    • The responses from database host server should leave via the same network interface (on the host) through which application server access-attempts are hitting database host. Therefore the gateway configured for the destination subnet (172.32.334.100) should be the same as the gateway configured for this network interface.
    • This was then determined to be an in-correct route on the DB host which is causing the problem. Added route on the DB server and that resolved the issue.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s