This weekend was one of those weekends, spent more on Google, looking for potential fixes, than with our families. One problem was a firewall was put in place between two sites and they were not correctly configured to allow extended DNS to pass. The firewalls used DNS guard and a limit of 512 bytes on UDP traffic. DNS resolution traffic uses UDP and would normally fit under the 512 limit. Windows 2003 DNS leveraged extended DNS. The Pix firewall would stop the UDP traffic when it exceeded 512 Bytes and DNS Guard would terminate the session. We were able to isolate the issue through nslookup and reviewing the firewall logs. In nslookup, when the server was changed to one of those behind the firewall, we would get request timed out. When we started looking at the firewall logs, we saw a second DNS request packet coming from the server, using the same session number. After some Google research, we were able to conclude that our issue had to do with the UDP packet size and DNS guard. By increasing the size allowed for UDP to 4096, we eliminated the errors seen in both the firewall logs and nslookup.
Well after we put that problem to bed, we ran into other issues that one by one, we were either able to find an exact fix for or information that point us in the right direction, on Google. Today, a coworker and I were laughing about it, when we started throwing out phrases. Google troubleshooting, trouble Googling; but the one we like the best was troubleshoogling. It is amazing the amount of data related to troubleshooting that can be found by having the right search criteria in Google, which is the hardest part of troubleshoogling.