Troubleshooting IP Multicast Routing
Your primary weapon when attacking problems in IP multicast internetworks is a solid understanding of the IP multicast protocols. Without that, no number of troubleshooting tools will help you wend your way through the often confusing, sometimes complex behaviors of IP multicast to the root of a particular problem. And understanding a single protocol is not enough. You also must understand how PIM, IGMP, and unicast routing all interact.
If you have been closely following the troubleshooting sections of each chapter of both Volume I and this volume, you should by now have a well-developed grasp of the approaches and techniques necessary for resolving problems in routed internetworks. So rather than present further case studies illustrating troubleshooting techniques, this section demonstrates the use of the several specialized tools provided for analyzing multicast internets.
Throughout this chapter, you have seen various show and debug commands that are useful for observing the behavior of IP multicast routing on Cisco routers. Table 6-2 lists the show commands available to you, and Table 6-3 lists the important multicast debug commands. Just as show ip route is the primary source of information when troubleshooting IP unicast routing, show ip mroute is the primary source of information when troubleshooting IP multicast routing.
Table 6-2. Important show Commands for Troubleshooting IP Multicast|
| show ip igmp groups [group-name | group-address | type number]
| Displays the addresses of groups that have members on the router's interfaces.
| | show ip igmp interface [type number]
| Displays relevant details of the IGMP-enabled interface.
| | show ip mcache [group [source]]
| Displays the multicast contents of the fast-switching cache.
| | show ip mroute [group-name | group-address] [source] [summary] [count] [active kbps]
| Displays the contents of the multicast routing table.
| | show ip pim bsr
| Displays information about PIM bootstrap routers.
| | show ip pim interface [type number] [count]
| Displays relevant details of PIM-enabled interfaces.
| | show ip pim neighbor [type number]
| Displays PIM neighbors.
| | show ip pim rp [group-name | group-address | mapping]
| Displays the known RPs, and the groups mapped to the RPs.
| | show ip pim rp-hash group
| Displays the RP for the group specified.
| | show ip rpf {source-address | name}
| Displays details of how the router is determining RPF information.
|
Table 6-3. Important debug Commands for Troubleshooting IP Multicast|
| debug ip icmp [hostname | group_address]
| Displays IGMP protocol activity.
| | debug ip mcache [hostname | group_address]
| Displays multicast caching operations.
| | debug ip mpacket [standard_access_list | extended_access_list] [hostname | group_address][detail]
| Displays multicast packets transiting the router.
| | debug ip mrouting [hostname | group_address]
| Displays multicast routing table activity.
| | debug ip pim [hostname | group_address][auto-rp][bsr]
| Displays PIM activity and events.
|
Using mrinfo
The mrinfo command enables you to observe a router's multicast connections and the details of those connections. The command is a part of the tools originally made available as part of mrouted for testing routers in the MBone. Therefore, the command is very useful in multivendor domains. Take, for example, the topology in Figure 6-10.
In Example 6-55, mrinfo is used at router Sombrero. The first line of the output shows the address used as the source of the query, the Cisco IOS Software version running on the router, and a number of flags. Table 6-4 lists the possible flags and their meanings. The next two lines of output show multicast interfaces on the router and any peers that the router may have. On the second line, Sombrero's interface 192.168.10.1 has no peers, indicated by the 0.0.0.0. The 1/0 indicates that the interface has a metric of 1 and that there is no TTL threshold set. PIM is running on the interface, the router is an IGMP querier for the attached subnet, and the subnet is a leaf network (that is, no multicast traffic will transit the network to another multicast router). The third line shows that Sombrero's interface 192.168.200.1 has a peer at address 192.168.200.2 (router Beret), the metric of the interface is 1, there is no TTL threshold, and PIM is running.
Example 6-55 The IP Multicast Connection Information for Sombrero in Figure 6-10
Sombrero#mrinfo
192.168.10.1 [version 12.1] [flags: PMA]:
192.168.10.1 -> 0.0.0.0 [1/0/pim/querier/leaf]
192.168.200.1 -> 192.168.200.2 [1/0/pim]
Sombrero#
Table 6-4. Flags Associated with the mrinfo Command|
| P
| Prune-capable
| | M
| mtrace-capable
| | S
| SNMP-capable
| | A
| Auto-RP-capable
|
The true usefulness of mrinfo, however, is that you can use the command to query other routers in the domain. In Example 6-56, the command is used at Sombrero to query Boater, by specifying one of Boater's IP addresses (in this case, its loopback address). Note that the flags indicate that SNMP is enabled on this router, whereas it is not on Sombrero. The router has five multicast-enabled interfaces, two of which are on leaf networks and three of which have PIM peers. A check of Figure 6-10 shows that this information is accurate.
Example 6-56 mrinfo Is Used at Sombrero to Query Boater About Its Multicast Peers
Sombrero#mrinfo 192.168.224.7
192.168.224.7 [version 12.1] [flags: PMSA]:
192.168.14.1 -> 0.0.0.0 [1/0/pim/querier/leaf]
192.168.15.1 -> 0.0.0.0 [1/0/pim/querier/leaf]
192.168.203.1 -> 192.168.203.2 [1/0/pim]
192.168.206.1 -> 192.168.206.2 [1/0/pim]
192.168.204.1 -> 192.168.204.2 [1/0/pim]
Sombrero#
In Example 6-57, routers Derby and Fez are queried. These two routers share an Ethernet connection, and comparing the results of the queries shows that Derby (192.168.224.4) is the IGMP querier on that subnet.
Example 6-57 Derby (192.168.224.4) and Fez (192.168.224.5) Are Queried from Sombrero
Sombrero#mrinfo 192.168.224.4
192.168.224.4 [version 12.1] [flags: PMA]:
192.168.12.2 -> 192.168.12.1 [1/0/pim/querier]
192.168.205.1 -> 192.168.205.2 [1/0/pim]
192.168.206.2 -> 192.168.206.1 [1/0/pim]
Sombrero#mrinfo 192.168.224.5
192.168.224.5 [version 12.1] [flags: PMA]:
192.168.12.1 -> 192.168.12.2 [1/0/pim]
192.168.205.2 -> 192.168.205.1 [1/0/pim]
192.168.204.2 -> 192.168.204.1 [1/0/pim]
192.168.201.2 -> 192.168.201.1 [1/0/pim]
Sombrero#
Using mtrace and mstat
Another useful tool is the mtrace command, which enables you to trace the RPF path from a specified destination to a specified source. Like mrinfo, mtrace is a UNIX-based MBone tool and can be used in multivendor domains. And also like mrinfo, you can issue the command from any router in the domain—you do not have to be on any router along the RPF path.
When the command is issued, you specify a source address and a destination address. A trace request is sent to the destination, which then uses a unicast trace to the source. The first-hop router on the path toward the source unicasts the results of the trace to the querying router.
Example 6-58 shows an example where a request is issued at Sombrero to trace the RPF path from Derby's 192.168.12.2 interface to Turban's 192.168.11.1 interface. Remember, because this is a reverse-path trace, Turban's interface is the source and Derby's interface is the destination. The output begins at the destination address and displays each intermediate router until the source is reached. The number of hops from the source is indicated, as is the multicast protocol used on that hop.
Example 6-58 mtrace Is Used to Examine the RPF Path from Destination 192.168.12.2 to Source 192.168.11.1
Sombrero#mtrace 192.168.11.1 192.168.12.2
Type escape sequence to abort.
Mtrace from 192.168.11.1 to 192.168.12.2 via RPF
From source (?) to destination (?)
Querying full reverse path...
0 192.168.12.2
-1 192.168.12.2 PIM [192.168.11.0/24]
-2 192.168.206.1 PIM [192.168.11.0/24]
-3 192.168.203.2 PIM [192.168.11.0/24]
-4 192.168.11.1
Sombrero#
Aside from the obvious use of isolating multicast routing failures, mtrace has an additional use of enabling you to examine multicast behavior before you turn up live multicast traffic on your internetwork. Notice in Figure 6-10 that no multicast sources or group members are indicated. Suppose you are going to turn up a multicast source attached to Boater, with an address of 192.168.14.35. This source will originate multicast traffic for group 235.100.20.18, and there will be group members at addresses 192.168.12.15, 192.168.10.8, and 192.168.11.102. Example 6-59 shows the results.
Example 6-59 mtrace Can Be Used to Test the RPF for Source, Destination, and Group Addresses That Do Not Yet Exist in the Multicast Domain
Sombrero#mtrace 192.168.14.35 192.168.12.15 235.100.20.18
Type escape sequence to abort.
Mtrace from 192.168.14.35 to 192.168.12.15 via group 235.100.20.18
From source (?) to destination (?)
Querying full reverse path...
0 192.168.12.15
-1 192.168.201.2 PIM [192.168.14.0/24]
-2 192.168.204.1 PIM [192.168.14.0/24]
-3 192.168.14.35
Sombrero#mtrace 192.168.14.35 192.168.10.8 235.100.20.18
Type escape sequence to abort.
Mtrace from 192.168.14.35 to 192.168.10.8 via group 235.100.20.18
From source (?) to destination (?)
Querying full reverse path...
0 192.168.10.8
-1 192.168.10.1 PIM [192.168.14.0/24]
-2 192.168.200.2 PIM [192.168.14.0/24]
-3 192.168.202.2 PIM [192.168.14.0/24]
-4 192.168.203.1 PIM [192.168.14.0/24]
-5 192.168.14.35
Sombrero#mtrace 192.168.14.35 192.168.11.102 235.100.20.18
Type escape sequence to abort.
Mtrace from 192.168.14.35 to 192.168.11.102 via group 235.100.20.18
From source (?) to destination (?)
Querying full reverse path...
0 192.168.11.102
-1 192.168.202.2 PIM [192.168.14.0/24]
-2 192.168.203.1 PIM [192.168.14.0/24]
-3 192.168.14.35
Sombrero#
The traces in Example 6-59 specify the multicast group along with the source and destination addresses. Although the RPF path would normally be the same for all groups, specifying the group can prove useful in situations where scoping or RP filtering affects the path taken. When no group is specified, as in Example 6-58, the group address 224.2.0.1 (the MBone audio group address) is used by default.
mstat is an adaptation of mtrace and provides not only a trace of the path from a source to a group destination, but also provides statistics about the path. Example 6-60 shows an example where a trace is again requested from source 192.168.14.35 to destination 192.168.10.8 for group 235.100.20.18. Comparing the output in Example 6-60 to the output for the same trace in Example 6-59, you can see that mstat provides not only packet statistics but also a more detailed view of the entire path.
Example 6-60 mstat Provides a More-Detailed Trace of Group Traffic from a Source to a Destination
Sombrero#mstat 192.168.14.35 192.168.10.8 235.100.20.18
Type escape sequence to abort.
Mtrace from 192.168.14.35 to 192.168.10.8 via group 235.100.20.18
From source (?) to destination (?)
Waiting to accumulate statistics......
Results after 10 seconds:
Source Response Dest Packet Statistics For Only For Traffic
192.168.14.35 192.168.200.1 All Multicast Traffic From 192.168.14.35
| __/ rtt 47 ms Lost/Sent = Pct Rate To 235.100.20.18
v / hop 27 ms --------------------- --------------------
192.168.14.1
192.168.203.1 ?
| ^ ttl 0
v | hop 5 ms 0/0 = --% 0 pps 0/0 = --% 0 pps
192.168.203.2
192.168.202.2 ?
| ^ ttl 1
v | hop 7 ms 0/0 = --% 0 pps 0/0 = --% 0 pps
192.168.202.1
192.168.200.2 ?
| ^ ttl 2
v | hop 4 ms 0/0 = --% 0 pps 0/0 = --% 0 pps
192.168.200.1
192.168.10.1 ?
| \__ ttl 3
v \ hop 0 ms 0 0 pps 0 0 pps
192.168.10.8 192.168.200.1
Receiver Query Source
Reading from bottom to top, the display in Example 6-60 shows the query source and response destination, which in this example are both 192.168.200.1 (Sombrero). Notice that there is an ASCII representation of arrows, showing that Sombrero has sent the query to 192.168.10.1 (in this case, its own interface). The reverse path is then traced to the interface on Boater to which the source would be attached, and the response to the query is then sent to Sombrero. At the far left of the display, ASCII arrows also indicate the path multicast traffic will take from the source to the destination. At each hop, the ttl and hop statistics can be a little misleading. ttl actually shows the number of hops from that point to the source, whereas hops shows the delay (in milliseconds) between hops. Notice that the round-trip time (rtt) is indicated below the response destination. Statistics are then shown for all multicast traffic and for the (S, G) pair specified in the command. The first statistic compares the number of packets dropped to the number of packets sent. The second statistic shows the total traffic rate in packets per second. In Example 6-59, all these statistics are zero, of course, because no traffic has passed from the source to the destination. In fact, the source and destination do not even exist yet.
In Example 6-61, the proposed hosts have been installed, the source is generating traffic, and the group member has joined. You can now observe the packet-per-second rates and the drop statistics. An important point to keep in mind when using mstat is that the delay times between routers are valid only if the routers' clocks are synchronized.
Example 6-61 The Same mstat Command Is Used After Multicast Traffic Has Begun Between the Source and Destination
Sombrero#mstat 192.168.14.35 192.168.10.8 235.100.20.18
Type escape sequence to abort.
Mtrace from 192.168.14.35 to 192.168.10.8 via group 235.100.20.18
From source (?) to destination (?)
Waiting to accumulate statistics......
Results after 10 seconds:
Source Response Dest Packet Statistics For Only For Traffic
192.168.14.35 192.168.200.1 All Multicast Traffic From 192.168.14.35
| __/ rtt 48 ms Lost/Sent = Pct Rate To 235.100.20.18
v / hop 48 ms --------------------- --------------------
192.168.14.1
192.168.203.1 ?
| ^ ttl 0
v | hop 10 ms 0/82 = 0% 8 pps 0/81 = --% 8 pps
192.168.203.2
192.168.202.2 ?
| ^ ttl 1
v | hop 6 ms 0/82 = 0% 8 pps 0/81 = 0% 8 pps
192.168.202.1
192.168.200.2 ?
| ^ ttl 2
v | hop 4 ms 0/82 = 0% 8 pps 0/81 = 0% 8 pps
192.168.200.1
192.168.10.1 ?
| \__ ttl 3
v \ hop 0 ms 82 8 pps 81 8 pps
192.168.10.8 192.168.200.1
Receiver Query Source
Sombrero#
Example 6-62 shows what the display might look like if the clocks are not in sync. The trace information and packet rates are still valid, but the delay times for the individual hops are obviously nonsensical. Also in Example 6-62, you can see that one packet has been lost on the hop between Turban and Beret. This may or may not represent a problem; the only way to know is to run several iterations of mstat and observe whether the packet loss is consistent. If so, further investigation using debugging may be required.
Example 6-62 If the Routers' Clocks Are Not Synchronized, the Delays Shown for the Router Hops Are Meaningless
Sombrero#mstat 192.168.14.35 192.168.10.8 228.13.20.216
Type escape sequence to abort.
Mtrace from 192.168.14.35 to 192.168.10.8 via group 228.13.20.216
From source (?) to destination (?)
Waiting to accumulate statistics......
Results after 10 seconds:
Source Response Dest Packet Statistics For Only For Traffic
192.168.14.35 192.168.200.1 All Multicast Traffic From 192.168.14.35
| __/ rtt 44 ms Lost/Sent = Pct Rate To 228.13.20.216
v / hop 44 ms --------------------- --------------------
192.168.14.1
192.168.203.1 ?
| ^ ttl 0
v | hop -222 s 0/82 = 0% 8 pps 0/81 = 0% 8 pps
192.168.203.2
192.168.202.2 ?
| ^ ttl 1
v | hop 113 s 1/82 = 1% 8 pps 1/81 = 1% 8 pps
192.168.202.1
192.168.200.2 ?
| ^ ttl 2
v | hop 108 s 0/80 = 0% 8 pps 0/80 = 0% 8 pps
192.168.200.1
192.168.10.1 ?
| \__ ttl 3
v \ hop 0 ms 80 8 pps 80 8 pps
192.168.10.8 192.168.200.1
Receiver Query Source
Lastly, you may encounter a situation in which mstat shows a negative number of lost packets such as, -3/85. The "negative packet loss" in fact represents a packet gain. In other words, extra packets have been received. This may signify a loop and warrants further investigation.
| 
