3 Recommendations for Managing Cisco SD-WAN Upgrades in Response to May 2021 Cisco Security Advisories Notifications
On May 5th, Cisco released one critical and three high severity security advisories affecting its Cisco SD-WAN products. Two of these advisories are of particular concern as they affect the Cisco vManage controller, allowing for the exposure of sensitive information and modification of the configuration. Due to the placement of Cisco vManage controllers within network environments – often having ports exposed to the Internet for control connections and administrative ports (web UI) exposed internally within a network – it is critical to update affected systems as quickly as possible.
For full details on each Cisco advisory, see the following publications:
- Cisco SD-WAN vManage Software Vulnerabilities (Critical severity)
- Cisco SD-WAN vManage Software Authentication Bypass Vulnerability (High severity)
- Cisco SD-WAN Software vDaemon Denial of Service Vulnerability (High severity)
- Cisco SD-WAN vEdge Software Buffer Overflow Vulnerabilities (High severity)
Potential Adverse Impacts
A compromised Cisco vManage controller could allow malicious changes to be made to the SD-WAN overlay, resulting in disruption of traffic forwarding or exposure of business data. The vDaemon vulnerability can be caused by an unauthenticated attacker behind a vEdge device capable of sending crafted packets to an affected device (which includes vManage, vBond, vSmart and vEdge devices) resulting in the affected system reloading. This can cause a denial of service (DoS) condition if a vEdge were to reload, or prevent vEdges from joining the overlay if all vBond and/or vSmart appliances were to reload at the same time.
Specific Recommendations for Addressing May 2021 Cisco Security Advisories
Optanix recommends mitigating the exposure of these vulnerabilities on affected devices as quickly as possible. There are no workarounds provided by Cisco. As a result, a software upgrade is the only method available to reduce exposure. This requires upgrading to a minimum SD-WAN software version of 20.4.1 across the controllers and edge devices in the overlay.
Several of these vulnerabilities can have their exposure reduced (but not completely eliminated) by ensuring only the necessary ports and protocols for each controller are exposed. This can be accomplished by placing them behind firewalls which only allow specific ports and protocols to the system. The underlying SD-WAN control ports do not easily allow restricting access based on source IP address ranges. However, limiting access to administrative protocols for the controllers from specific management subnets can help in reducing exposure when vulnerabilities affect these features.
3 Recommendations for Administering Cisco SD-WAN Software Upgrades
Optanix has several recommendations that can help when planning and implementing software upgrades across your Cisco SD-WAN environment. Efficiencies can be achieved by following a regular schedule of software upgrades across your SD-WAN. These efficiencies stem from the experience your IT teams gain by performing these upgrades. They also stem from your IT teams gaining a better understanding of how the business and applications that rely on the SD-WAN react to the upgrades and changing state of the overlay.
Recommendation #1: Review Release Notes
Each major release of Cisco SD-WAN software comes with a release notes document outlining new features, features no longer available, resolved and open bugs, upgrade paths, and a compatibility matrix. By thoroughly reviewing the release notes you can ensure the success of the upgrade by avoiding any complications. For those looking to upgrade to 20.4.1 to remediate the vulnerabilities documented in this post, the SD-WAN 20.4.x Release Notes indicate a direct upgrade can be performed for Cisco vManage from 20.1.x. If you are on an older Cisco vManage version, you must perform a step-upgrade to 20.3. The 20.4 compatibility and server recommendations remain the same if you are upgrading from 20.3.
Recommendation #2: Create a Thorough Verification Plan
Cisco SD-WAN controllers are deployed in a fashion that allows for a high level of redundancy. However, use caution when upgrading the controllers to ensure any issues introduced when upgrading are identified and are not carried forward to the redundant controllers. By creating a thorough verification plan after each controller is upgraded, you can ensure it is capable of handling the control connections while its redundant controllers are upgraded in sequence.
Before any upgrades are performed, be sure to collect general system health (CPU, memory, crashlogs, syslogs, status) as well as overlay health (control connections, OMP peers and routes depending on the controller type) from the controllers and a select list of vEdge devices in the environment. After each controller is upgraded, compare the collected pre-upgrade health data to post-upgrade health data to ensure stability.
The above verification steps can be performed by remote administrators, but it is also important to add additional verification and user testing based on your deployment and the critical applications that use the SD-WAN.
Recommendation #3: Implementation Order of Operations
To ensure there are no unnecessary disruptions to the SD-WAN overlay, it is critical to follow a strict order of operations when upgrading the SD-WAN environment. The components of the SD-WAN should be upgraded in the following order:
- vManage controllers
- vBond controllers
- vSmart controllers
- WAN edge routers
Before performing an upgrade, take a database backup on the vManage controller. Depending on the hypervisor the controllers are running on, snapshots can also be taken for quicker restoration of service.
After Cisco vManage is upgraded, the vBond controllers should be upgraded. This can be done by upgrading half of the vBond controllers at once then, after verifying their state by following recommendation #2, upgrading the remainder of vBond controllers. Once all vBond controllers are upgraded, vSmart controllers can be upgraded in the same fashion.
Finally, once all controllers are upgraded to the target version of software, upgrades can be performed on the WAN edge routers. Each customer environment is unique, but as a general rule, it is best to upgrade test/lab equipment first. Low-risk sites with redundant routers should be upgraded next, followed by medium-risk sites and then high-risk sites. After each group of upgrades, validate the health of the edge routes before proceeding on to the next group.
Security Assistance
Optanix maintains a security-first culture which we extend to all of our customer engagements. If you would like assistance with reviewing security advisories or implementing best practices, our team of enterprise networking professionals can help you. Optanix customers can get in touch by contacting your customer success manager (CSM) or by opening a service request (ESR) in the Optanix Platform. For prospective clients, please click here to engage with Optanix.
This blog post was authored by Brian Yaklin, a senior member of Optanix’s route/switch engineering team. It is part of an ongoing series of posts by Optanix engineers focusing on the importance of security in the IT management space.