Recently, at work, faced a strange issue. Thought of noting it down here.. Knowledge Sharing as well as Knowledge Banking 😀
We want to establish a jdbc connection to the MSSQL DB server. That was failing. On looking into the driver’s source, we found that the jdbc driver was sending an UDP packet ‘002’ to the MSSQL DB Server (Port 1433) . (When we send a UDP packet with this data to the MSSQL Server, it will respond with some connection information.) Then we did some wireshark analysis and found that , the udp packet is not received at the server side. (Even though firewall is off everywhere…)
As the same was working from other machines, we decided that, it is not an issue with MSSQL. So, we thought of trying to write a custom UDP server in java and tried running that from different machines and we found that, from some machines we were able to connect and from some we could not. The weird thing is, there was no common denominator across the machines except they are linux machines. (As far as we have observed)
Sooo strange.. So started a mailing thread within office and here and again started comparing the wireshark traces between the machines that work and that doesn’t.
One fine day, (Yes!!! I worked on this issue for days together.. Not fully though!!) I started comparing the wireshark traces. (With a fresh mind) And found that there were some packets reported as checksum failed in wireshark. So, started googling and found this. In short, NIC cards does some checksum verification and if that verification fails, it will not allow the packet. So, wireshark will not even see it. ie. If checksum offloading is enabled, the NICs will do the checksumming. Details on checksum offloading is here.
Now that we found the issue, we need to find a solution. Again, googled and found this.
ethtool –offload eth0 rx off tx off (This will last till the next reboot of the machine. Looks like we need to update the network’s startup script ETHTOOL_OPTIONS for the network interfaces with this info to make it permanent).
With that, we can turn off the checksumming. When we do this, the NICs will allow the packet to be transmitted without checksumming.
Even the strangest issues in computers, will have a reason and a solution.