Packet Design’s use by many of the world’s largest service providers and network operators gives us the opportunity to see how they operate. We get to witness the sort of issues they deal with during their everyday operations and how many of them are resolved. We thought we would share some of these stories periodically through Life in the Control Plane blog series. After all, life in the control plane is never dull!
There are numerous articles and topics that talk about network and configuration changes being the number one reason for network downtime. In fact, we at Packet Design have helped several organizations resolve issues that surfaced from network changes. We have written in this blog before about the service provider whose 10G line card that carried their entire backbone traffic disappeared after an IOS upgrade and about the bank where a network configuration change led to a route leak resulting in network downtime.
While many of the network issues that arise from network changes are quickly resolved by network engineers, reactive troubleshooting is not always the best solution to keeping users happy. When network admins react to network issues, in many cases it means that it is the customer who discovered the issue and raised a ticket complaining about the service or the lack of it.
What organizations, especially service providers, really need is the ability to plan and analyze the impact of a network change before the change is implemented. But this is easier said than done. If the network change only requires making a configuration change or switching the hardware in use, possible issues can be mitigated by analyzing the current and new configuration and ensuring that both match. But when the change requires adding or removing a peer, prefix, route or router, or analyzing how the addition of a new VPN customer and the traffic load they bring will impact existing VPN routes, telecom providers are usually at a dead end. Though it is possible to predict the additional load that a new customer may bring, there are some areas where planning and impact analysis are difficult: How will the additional traffic load impact the traffic path from source to destination, and which links will see higher utilization or result in traffic bottlenecks? Or, when a new router or prefix is added, what will be its impact on the network? Analyzing the impact of such changes can be time consuming and the only change window that the provider may have is the two hours after midnight on a Sunday.
Network providers and operators can now use analytics software to predetermine the impacts of changes like these. From flow data, traffic matrices can be calculated for various times of day, day of week, etc. When combined with IP/MPLS routing analytics for the circuits involved in a particular service, such as a new VPN, engineers can model the impact on link utilization and determine whether congestion will result in path changes and possible service degradation.. These same network analytics tools can be used after the change is made to quickly validate and report on its impact. This kind of efficient network planning removes the unpredictability that often arises from network maintenance. Network operators can optimize their network infrastructure, quickly provision new services to customers, reduce customer complaints while improving the overall customer experience, and avoid the time and cost of troubleshooting issues arising from an incorrect network change.
Here is a look at how a network operator providing Layer 2 and Layer 3 VPN services made use of the Packet Design Explorer Suite to plan the acquisition of a new L3 VPN customer by analyzing the possible impact of the new VPN on their existing network before implementing the changes: