Troubleshooting Static Routes
Followers of the assorted
American political scandals of the past 30 or so years will have heard a congressional investigator ask the question, "What did he know and when did he know it?" The same question serves an internetworking investigator well. When troubleshooting routing problems, the first step should almost always be to examine the route table. What does the router know? Does the router know how to reach the destination in question? Is the information in the route table accurate? Knowing how to trace a route is essential to successfully troubleshooting an internetwork.
Case Study: Tracing a Failed Route
Figure 3.13 shows a previously configured internetwork, with each router's associated static routes. A problem has been discovered. Devices on subnet 192.168.1.0/27, connected to Pooh's Ethernet interface, can communicate with devices on subnet 10.1.0.0/16 just fine. However, when a ping is sent from Pooh itself to subnet 10.1.0.0/16, the ping fails (Figure 3.14). This seems strange. If packets being routed by Pooh successfully reach their destinations, why do packets originated by the same router fail?
Addressing this problem requires tracing the route of the ping. First, Pooh's route table is examined (Figure 3.15). The destination address of 10.1.5.1 matches the route entry for 10.0.0.0/8, which (according to the table) is reached via the next-hop address 192.168.1.34—one of
Eeyore's interfaces.
Next the route table for
Eeyore must be examined (Figure 3.16). The destination address 10.1.5.1 matches the entry 10.1.0.0/16, with a next-hop address of 10.4.6.1. This address is one of Tigger's interfaces.
Figure 3.17 shows Tigger's route table. The destination address matches the entry for 10.1.0.0/16 and will be forwarded to 192.168.1.194, which is at Piglet.
Piglet's route table (Figure 3.18) reveals that the target network, 10.1.0.0, is directly connected. In other words, the packet has arrived. The target address 10.1.5.1 is Piglet's own interface to that network. Because the
path to the address has just been verified as good, we can assume that the ICMP echo packets from Pooh are reaching the destination.
The next step is to trace the path of the responding ICMP echo reply packets. To trace this path, you need to know the source address of the echo packet—that address will be the destination address of the echo reply packet. The source address of a packet that is originated from a router is the address of the interface from which the packet was transmitted.
In this example, Pooh originally forwarded the echo packet to 192.168.1.34. Figure 3.13 shows that the source address of that packet is 192.168.1.33. So this address is the destination address to
which Piglet will send the echo reply.
Referring again to Piglet's route table in Figure 3.18, 192.168.1.33 will match the entry for 192.168.1.0/24 and be forwarded to 192.168.1.193, which is another of
Tigger's interfaces. A reexamination of Tigger's route table in Figure 3.17 at first suggests that there is an entry for 192.168.1.0. However, care must be taken to correctly interpret the information that is actually there.
Compare the entries in Tigger's route table for the 10.0.0.0 subnets to those of the 192.168.1.0 subnets. The heading for the former says that 10.0.0.0 is variably subnetted; in other words, Tigger's static route to subnet 10.4.7.0 uses a 24-bit mask, and the static route to subnet 10.1.0.0 uses a 16-bit mask. The table records the correct mask at each subnet.
The heading for 192.168.1.0 is different. This heading states that Tigger knows of three subnets of 192.168.1.0 and that they all have a mask of 255.255.255.224. This mask will be used on the destination address of 192.168.1.33 to derive a destination network of 192.168.1.32/27. The route table has entries for 192.168.1.64/27, 192.168.1.0/27, and 192.168.1.192/27. There is no entry for 192.168.1.32/27, so the router does not know how to reach this subnet.
The problem, then, is that the ICMP echo reply packet is being dropped at Tigger. One solution is to create another static route entry for network 192.168.1.32, with a mask of 255.255.255.224 and pointing to a next hop of either 192.168.1.65 or 10.4.6.2. Another solution would be to change the mask in the existing static route entry for 192.168.1.0 from 255.255.255.224 to 255.255.255.0.
The moral of this story is that
when you are tracing a route, you must consider the complete communication process. Verify not only that the path to a destination is good but also that the return path is good.
Note
Check both the destination and return path when a route fails.
Case Study: A Protocol Conflict
Figure 3.19 shows two routers
connected by two Ethernet networks, one of which includes a bridge. This bridge handles traffic for several other links not shown and occasionally becomes congested. The host Milne is a mission-critical server; the network administrator is worried about traffic to Milne being delayed by the bridge, so a static host route has been added in Roo that will direct packets destined for Milne across the top Ethernet, avoiding the bridge.
This solution seems to be logical, but it isn't working. After the static route was added, packets routed through Roo no longer reach the server. Not only that, but packets routed through Kanga no longer reach the server, although no changes were made to that router.
The first step, as always, is to check
the route table. Roo's route table (Figure 3.20) indicates that packets with a destination address of 172.16.20.75 will in fact be forwarded to Kanga's E1 interface, as desired. Kanga is directly connected to the destination network, so no more routing should be occurring; a quick check verifies that the Ethernet interfaces
are functioning at both Kanga and at Milne.
In Figure 3.21, a trace is performed from Roo to Milne, and a symptom is found. Instead of delivering the packets to Milne, Kanga is forwarding them to Roo's E0 interface. Roo forwards the packets to Kanga's E1 interface, and Kanga sends the packets right back to Roo. A routing loop seems to have occurred, but why?
The suspicious aspect of all of this is that Kanga should not be routing the packet, which appears to be the case. Kanga should recognize that the destination address of the packet is for its directly connected network 172.16.20.0 and should be using the data link to deliver the packet to the host. Therefore, suspicion should fall on the data link. Just as the route table should be examined to see whether the router has the correct information for reaching a network across a logical path, the
ARP cache should be examined to see whether the router has the correct information for reaching a host across a
physical path.
Note
Check a router's ARP cached when packets do not go to the correct destination on a data link.
Figure 3.22 shows the
ARP cache for
Kanga. The IP address for Milne is in Kanga's cache as expected, with a MAC identifier of 00e0.1e58.dc39. When Milne's interface is checked, though, it shows that it has a MAC identifier of 0002.6779.0f4c; Kanga has somehow acquired incorrect information.
Another look at Kanga's ARP table reveals that the MAC identifier associated with Milne is suspiciously similar to the MAC identifier of Kanga's own Cisco interfaces (the MAC addresses with no ages associated with them are for the router's interfaces). Because Milne is not a Cisco product, the first three octets of its MAC identifier should be different from the first three octets of Kanga's MAC identifier. The only other Cisco product in the internetwork is Roo, so its
ARP cache is examined (Figure 3.23); 00e0.1e58.dc39 is the MAC identifier of
Roo's E0 interface.
So Kanga mistakenly
believes
that the E0 interface of Roo is Milne. It frames packets destined for Milne with a
destination identifier of 00e0.1e58.dc39; Roo accepts the frame, reads the destination address of the enclosed packet, and routes the packet back to Kanga.
But how did Kanga get the wrong information? The answer is proxy ARP. When Kanga first receives a packet for Milne, it will ARP for that device's data link identifier. Milne responds, but Roo also hears the ARP Request on its E0 interface. Because Roo has a route to Milne on a different network from the one on which it received the ARP Request, it
issues a proxy ARP in reply. Kanga receives Milne's ARP reply and enters it into the ARP cache. The proxy ARP reply from Roo arrives later because of the delay of the bridge. The original ARP cache entry is overwritten by what Kanga thinks is new information.
There are two solutions to the problem. The first is to turn off proxy ARP at Roo's E0 with the command:
Roo(config-if)#interface e0
Roo(config)#no ip proxy-arp
The second solution is to configure a static ARP entry for Milne at Kanga with the command:
Kanga(config)#arp 172.16.20.75 0002.6779.0f4c arpa
This entry will not be overwritten
by any ARP reply. Figure 3.24 shows the static ARP entry being entered and the resulting ARP cache at Kanga. Note that because
the entry is static, no age is associated with it.
The circumstances of the network should help determine which of the two solutions is best. A static ARP entry means that if the network interface at Milne is ever replaced, someone must remember to change the ARP entry to reflect the new MAC identifier. On the other hand, turning off proxy ARP is a good
solution only if no hosts make use of it.
| 
