Many of the world’s largest service providers, network operators and enterprises use the Packet Design Explorer Suite, which gives us the opportunity to see how they operate. We get to witness the sort of issues they deal with during their everyday operations and how they resolve many of them. We decided to share some of these stories periodically through the Life in the Control Plane blog series. After all, life in the control plane is never dull!
We recently came across the this case in a public email discussion. Though the network in question did not use the Packet Design Explorer Suite, we thought we should share how the Explorer Suite can help solve such issues quickly.
A network engineer managing an enterprise network had reachability issues to prefixes in one of his company’s data centers. The organization relies on two service providers for connectivity – a global service provider and a large regional service provider.
During a routine maintenance schedule, the link connecting the router to the global service provider was taken offline. When this was done, they received a number of reachability alerts for the prefixes in the datacenter. Using BGPMon reporting, the enterprise found that close to eighty Autonomous Systems (AS’s) had noted their routes as withdrawn, making the prefixes unreachable from many major AS’s. But this was unexpected behaviour, because the prefixes should still have been reachable through the large regional service provider.
The network engineer suspected that the regional service provider was not adequately redistributing their prefixes, which made them unreachable from many AS’s. While they conducted the root-cause analysis, the engineer wanted to find out how he could monitor the global reachability of the prefixes via any service provider. This way, he could see the AS paths, and ensure that the paths to the prefixes include the AS’s of all the service providers they use.
One of the options the network engineer had was to run a script that would log in to a router and examine the AS path to their prefixes by querying routeviews. But this requires running the script as a cron job, which is not preferable on a busy router. Another drawback with this solution is that routeviews can be slow to respond to requests, due to the sheer size of the global BGP routing table and multitude of sources.
Another method suggested was to make use of Qrator Radar APIs to do an AS path lookup for a given prefix and see all the paths retrieved from the AS_PATH attribute. While this solution can provide the information required, it can fail if any of the AS’s involved do not export the route information.
The Packet Design Explorer Suite overcomes the limitations of the previous two tools with its peering analysis and BGP path visualization features, making validating BGP route reachability quick and easy.
Using the Explorer Suite, network engineers can see the links and routers in their network connecting the prefix, and next hop information along with the AS path in a map view. Explorer Suite also supports prefix reachability reports that shows a prefix, its destination AS, next hop and the AS path. In this specific case, Explorer Suite could have helped the network engineer quickly find out the prefixes being advertised and their AS paths, helping verify if the regional provider was indeed advertising the prefixes as expected.
Later, the network engineer discovered that the issue was caused by the regional service provider failing to advertise the prefixes due to an error in their export policy – something that could have been verified in minutes using the Explorer Suite.
Want to learn more about the Packet Design Explorer Suite’s peering analysis and other BGP monitoring capabilities?