My most useful debugging tools

Debugging Network Problems

Recently, I have been doing a lot more work on systems integration on quite small systems. You know, 2 database servers, a couple of hypervisors but all connected together by some kind of vlans and operating as a single system with a few public IP addresses to access it from the outside.

In most cases now, we use OPNSense on a VM as a gateway for all virtual machines and we run HA across two hypervisors so we can reboot one and still have functionality.

I already knew a fair bit about networking, or so I thought, but I have also had my fair share of challenges when things are not connecting or otherwise not working properly. Some connections appear completely dead, others are sporadic and the LLMs are only partially useful since they don’t seem to have the same quality of training as programmers have and their suggestions are sometimes useful but other times out of left-field and could easily leave you making very specific low-level changes to your systems. I have very much felt the dangers of trusting LLMs without any knowledge, although fortunately I have mostly avoided this!

Understanding how the network works - IP

Most of us hopefully understand how IP works. I want to connect to a different machine so I might look up its IP address using DNS or I might already have it. I then send a packet addressed to the other end from my IP address and I trust the infrastructure to transport it to the other end and bring any replies back. IP is layer 3 in the 7 layers of the OSI Model and the OSI model is something you should be aware of even if you can’t recall everything in each layer.

TCP adds connection control to IP, so inside an IP packet, we add some TCP headers, which identify mainly the message and which packet this is, as well as the source and destination ports (which allow us to multiplex multiple connections to the same IP address). This allows the other end to know if any packets are missing (so they can be sent again) but also allows the destination to put the packets in the correct order since data can take multiple paths and arrive out of order. For this blog post, TCP is not really relevant since if you have layer 3 working, the rest is usually OK. TCP doesn’t fit neatly into the OSI model (because it postdates the model) but would probably be assigned layers 4. Historically it had been placed between layers and 4 and 5 since it contains both the transport mechanism but also “session” i.e. long-lived message ids so both ends can track progress of the data although this is not really the same “session” that layer 5 is concerned with.

When we are testing networks, most of us are familiar with the ping tool (Or Echo or ICMP), which describe slightly different parts of the operation but we can use the terms interchangeably for now. Most of us at some point have typed something like ping 192.168.1.1 and although this is somewhat useful when it succeeds (to know the destination is online and routeable), it doesn’t tell us much more than that and if it doesn’t work, it doesn’t say why not. The system might be fine but a firewall is blocking the ping request. There might be some routing problems.

Another very useful tool is netcat which is run as nc on the command line and can check for a connection to any port. Its basic usage is just nc -vz <hostname> <portnum>. This will print “Succeeded!” if it works. If not, it will often time out. Most of the other options are a bit more specialist. Imagine you cannot connect to e.g. a SQL Server database from a Linux host, you run nc -vz dbserver 1433 and it shows “Succeeded!”, that saves you a lot of time worrying about networks and firewalls. You can add a -u if you want it to use UDP instead of TCP but since UDP is connectionless, you will only get a useful response if the target actively refuses the packet, otherwise you have to assume it succeeded because it didn’t error.

Going down to layer 2 - ARP and Mac Addresses

This is not something that a lot of Developers will ever worry about because it just works, except of course it doesn’t always just work, somebody has had to make it work. The built-in mechanisms of operating systems are supposed to control all of this but there are places that can go wrong, which is why we sometimes need to go down a layer.

The way that switches work on layer 2 is they learn about MAC addresses. A MAC address is traditionally a unique hardware addresses of network cards but of course we now have virtual network cards too so these have a “virtual hardware address”. A layer 2 switch, therefore, expects layer 2 packets that identify the destination MAC adddress of the packet and route the packet accordingly without any concept of IP addresses or whatever other data might be inside it. They are often called frames rather than packets. The switch learns which of its ports has which MAC addresses and therefore the switching process can be extremely fast. If a switch does not have a mapping for a particular MAC address, it will send out the frame on all ports and wait for a packet sourced from that same MAC to then establish which is the correct port. Since a lot of traffic will travel via the default gateway and the rest to local machines, a switch can usually learn all ports relatively quickly.

So how do our computers know which MAC address to send data to? They use something called ARP (Address Resolution Protocol). This protocol requires the sender, usually on sending the first packet to a particular IP address, to ask e.g. “Who has IP 123.123.123.123”. If the destination is on the local network, that machine, if it is powered on, will reply e.g. “Mac address BC:24:11:8B:69:EE has address 123.123.123.123”. Now this might sound inefficient except these mappings are cached locally for between 10s of seconds up to several minutes (and which will often have the cache extended for every reuse), in the “arp table”, which can be seen with the very useful tool arp by using arp -a which will list all currently cached mappings of IP addresses to MAC addresses. You might be thinking this table would get extremely large but actually it rarely does for the simply reason that if a destination is the other side of your gateway/router, then the IP address you will lookup will be the “next hop” address, which will be the router, not the final destination IP address. The router will worry about that hop when it receives your packet.

Another great tool if you are struggling to get your traffic to the other end is arping which is like ping except it is making arp requests instead of echo requests. If you run this against an IP address that you are targetting, it will tell you which, if any, MAC address/network card is claiming ownership of that IP address. If you see nothing, it is likely a generally routing or firewall problem. But sometimes you might see more than one device responding. If the responses are an HA pair and one of them always responds first, then that is usually expected behaviour. If you are only expecting one result and get two, you have likely assigned the same IP address to more than one machine and that causes of kinds of weird behaviour.

Dumping traffic for better diagnostics

One of the most powerful tools for network debugging is tcpdump this can log to file or, more usually, you can just log to stdout sometimes on both ends of a troublesome link and get information that is really useful to an LLM helping you to track down the source of a problem. For example, I had a range of VMs on about 5 different vlans and when I tried to ssh into them, they mostly all worked fine but the VMs on one particular vlan were really inconsistent. It felt like they were trying to connect and very occasionally they did but most of the time, they timed out and it was really confusing. What causes something to “nearly” work or to work once every 50 attempts? To make it more confusing, if I ssh’d from a VM on the same vlan, it worked every time but from any other vlan it did not.

tcpdump to the rescue. I run tcpdump -ni ens18 on each vm. You can really filter this down as well, otherwise, as you might imagine the network is incredibly noisy. So I used tcpdump -ni ens18 port 22 on the target and tcpdump -ni ens18 host <targetip> on the caller and tried to run ssh. I saw a lot of messages going outbound, as expected, and I saw some messages arrive at the other end. Depending on your expertise you could probably have seen the problem more quickly than me but the most obvious explanation was that the target was replying and the reply was not reaching the other end. I tried again from a VM on the same vlan and saw a much larger set of messages being transferred so I knew how much traffic I should have seen.

Asymmetric connections are a common source of confusing errors but they are a well defined problem and LLMs can now be very helpful with the answer. With our knowledge of the route that the traffic should take back to the origin, I worked out that when the gateway was involved (between different vlans) there was an issue but when the gateway was not involved (a VM on the same vlan) it was fine. In other words, the gateway is causing a problem and when the LLM prompted me to look at the floating virtual IP that is swapped between the gateways when a failover takes place, I realised that a simple typo made both gateways think they were the owner of the IP so both were announcing ownership on the LAN and when the target was trying to route its replies back, it would often hit the wrong gateway, the state of the connection would not be present in the database and it would just get dropped.

Our own specific tools

In my case because I am using OPNSense, there are a number of useful tools there also like the live firewall log. This can show useful things like you might be blocking NTP by default. Not something that surfaces as an error immediately but something that might explain a problem you are seeing elsewhere. It also includes VPN logs to help debug settings problems.

My favourite tools

So in summary, I now use the following a LOT when setting up networks:

• netcat
• arp
• arping
• tcpdump