Network Troubleshooting Methodology

Troubleshooting your network is probably one of the most difficult tasks a network engineer has to perform. Hopefully you are not doing this too much on your network but if you do, having a good methodology will help so you are not going insane. If you are performing several tests in a random order it is easy to make false assumptions and go down a troubleshooting rabbit hole. I use the following methodology to gather all the information I need before making assumptions and trying to put in a fix.
Symptoms:
Connectivity  – probably VLAN/Layer 3 or spanning tree
Latency – could be spanning tree loop or physical connectivity
The following is a list of commands that I use to collect the information necessary to narrow down the culprit.
General Troubleshooting
  1. show logging
  2. show ip interface brief
  3. show cdp neighbor
  4. show ip protocol [vrf <vrf name>]
  5. show ip route [vrf <vrf name>]
  6. ping [vrf <vrf name>]
  7. traceroute
  8. show ip arp
  9. show mac address-table
  10. show run vrf <vrf name>
  11. show ip vrf interface
Services
  1. show ntp status
  2. show netflow
  3. show snmp
  4. show logging
EIGRP
  1. show ip eigrp [vrf <vrf name>] neighbor
  2. show ip eigrp [vrf <vrf name>] interface
  3. show ip eigrp [vrf <vrf name>] topology pending – check for pending routes
OSPF
  1. show ip ospf [vrf <vrf name>] interface brief
  2. show ip ospf [vrf <vrf name>] neighbor
  3. show ip ospf <process #> database
Spanning-tree (perform on both connected switches):
  1. show cdp neighbor – look at the ports connected between switches
  2. show interface status – look at port channel descriptions to match to switches
  3. show etherchannel <port-channel> summary – compare ports listed to cdp neighbor output
VLAN/Layer 3 (perform on both connected switches):
  1. show interface status – look at port channel descriptions to match to switches
  2. show interface port-channel <port-channel> trunk – lists the VLANs that are attached to the trunk
  3. show vlan – make sure the VLANs listed on the trunk are in the VLAN database and active (layer 2)
  4. show ip interface brief – make sure the VLANs listed on the trunk are up/up and have the right IP address and mask (layer3)
  5. ping [vrf <vrf name>] <self IP>
  6. ping [vrf <vrf name>] <neighbor IP>
  7. show ip route [vrf <vrf name>]
  8. show ip eigrp [vrf <vrf name>] neighbor
  9. show ip eigrp [vrf <vrf name>] topology pending – check for pending routes
  10. ping [vrf <vrf name>] 4.2.2.2
Physical Connectivity (perform on both connected switches):
  1. Check to make sure same SFPs are used on both sides
  2. Use flashlight to check fiber paths
  3. show interface status – check SFP type, speed and duplex
  4. clear counter – start at 0 interface statistics before troubleshooting
  5. show interface | include errors – check if errors are incrementing

Leave a Reply