Upgrading Catalyst 9500 Switches

Introduction

This document describes the methods for upgrading Catalyst 9500 switches.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on C9500.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

This document covers new and old upgrade procedures for Catalyst 9500 switches that use either BUNDLE or INSTALL modes. ISSU upgrade method is supported on Catalyst 9500 switches.

Recommended Releases

For the recommended software versions based on the downloads page, please consult the following link:

Recommended Releases for Catalyst 9000 Switches

Software Download

To download the software, please visithttps://software.cisco.com/download/homeand select your product.

Essential Criteria for Upgrade

• A maintenance window of 2-3 hours should be sufficient for upgrading to the target version or rolling back to the previous version if any issues arise.

• Ensure you have a 4GB or 8GB USB drive with the .bin files of both the current and target IOS versions. The USB drive should be formatted in FAT32 to copy the IOS image.

• Verify that TFTP is set up with both the current and target IOS versions and is reachable to download these versions to the switch if needed.

• Confirm that console access to the device is available in case any issues occur.

• Ensure there is at least 1GB to 1.5GB of available space in the flash memory for the expansion of the new image. If there is insufficient space, remove old installation files.

Rommon Upgrade and/or Bootloader Upgrade

For Fuji 16.9.x, When you upgrade from the existing release on your switch to a later or newer release for the first time, the boot loader may be automatically upgraded, based on the hardware version of the switch. If the bootloader is automatically upgraded, it will take effect on the next reload. If you go back to the older release after this, the boot loader is not downgraded. The updated boot loader supports all previous releases. For subsequent Cisco IOS XE Everest 16. x.x, or Cisco IOS XE Fuji 16. x.x releases, if there is a new bootloader in that release, it may be automatically upgraded based on the hardware version of the switch when you boot up your switch with the new image for the first time.

For 16.12.x, the ROM monitor (ROMMON), also known as the boot loader is firmware that runs when the device is powered up or reset. It initializes the processor hardware and boots the operating system software (Cisco IOS XE software image). The ROMMON is stored on the following Serial Peripheral Interface (SPI) flash devices in your switch:

Primary: The ROMMON stored here is the one the system boots every time the device is powered on or reset.

Golden: The ROMMON stored here is a backup copy. If the one in the primary is corrupted, the system automatically boots the ROMMON in the golden SPI flash device.

ROMMON upgrades may be required to resolve firmware defects or to support new features, but there may not be new versions with every release. To know the ROMMON or bootloader version that applies to every major and maintenance release, refer to the corresponding subsections and tables below.

• ROMMON Upgrades for C9500-12Q, C9500-16X, C9500-24Q, C9500-40X

• ROMMON Upgrades for C9500-24Y4C, C9500-32C, C9500-32QC, and C9500-48Y4C

For 17.x.x, to know the ROMMON or bootloader version that applies to every major and maintenance release, see ROMMON Versions.

You can upgrade the ROMMON before, or, after upgrading the software version. If a new ROMMON version is available for the software version you are upgrading to, proceed as follows:

• Upgrading the ROMMON in the primary SPI flash device

  This ROMMON is upgraded automatically. When you upgrade from an existing release on your switch to a later or newer release for the first time, and there is a new ROMMON version in the new release, the system automatically upgrades the ROMMON in the primary SPI flash device, based on the hardware version of the switch.

• Upgrading the ROMMON in the golden SPI flash device

  You must manually upgrade this ROMMON.The manual upgrade applies to all models in the series.Enter theupgrade rom-monitor capsule golden switchcommand in privileged EXEC mode.

After the ROMMON is upgraded, it will take effect on the next reload. If you go back to an older release after this, the ROMMON is not downgraded. The updated ROMMON supports all previous releases.

Note: In case of a Cisco StackWise Virtual setup, upgrade the active and standby switch.

Upgrade Methods 

This document covers new and old upgrade procedures for the Catalyst 9500 switch that uses either BUNDLE or INSTALL modes and ISSU.

Install Mode

An install mode upgrade on a Cisco Catalyst 9500 switch is a method of upgrading the switch's software that involves using individual software packages rather than a single monolithic image file.

On Cisco Catalyst 9500 Series Switches, When upgrading from Cisco IOS XE Everest 16.5.1a or Cisco IOS XE Everest 16.6.1 to any newer version in INSTALL mode, the "request platform software" commands are utilized.  

Please follow the outlined steps for an upgrade in Install mode.

1. Cleanup

Remove any inactive installations with the command: 

Switch#request platform software package clean switch all

2. Copying the New Image

Transfer the new .bin image file to the active switch's flash storage using one of the following methods:

Via TFTP:

Switch#copy tftp://Location/directory/<file_name>.bin flash:

Via USB:

Switch#copy usbflash0:<file_name>.bin flash:

Confirm the available file systems with Switch#show file systems

3. Verification

After transferring the IOS to the active switch, check if the image is correctly copied with: 

Switch#dir flash:

(Optional) To verify the MD5 checksum, use the command: 

Switch#verify /md5 flash:<file_name>.bin

Ensure this checksum matches the one provided on the Software Download page.

4. Setting the Boot Variable

Set the boot variable to point to the packages.conf file with the following commands: 

Switch#configure t

Switch(config)#no boot system

Switch(config)#boot system flash:packages.conf

Switch(config)#end

5. Autoboot Configuration

Configure the switch to autoboot by executing:

Switch#configure t

Switch(config)#no boot manual

Switch(config)#end

6. Saving Configuration

Save your current configuration with: 

Switch#write memory

Confirm the boot settings with the command: 

Switch#show boot system

7. Installation of the Image

To install the image, use the command:

Switch#request platform software package install switch all file flash:<file_name>.bin auto-copy

The system will auto-reload. 

8. Verification of the successful upgrade

Switch#show version

Switch#show redundancy

Note: Replace with the actual name of your IOS image file throughout the steps.

On Cisco Catalyst 9500 Series Switches and Catalyst 9500 Series Switches - High-Performance series switches, When upgrading from Cisco IOS XE Everest 16.6.2 and all later releases to any newer version in INSTALL mode, the "install" commands are utilized.

Please follow the outlined steps for an upgrade in Install mode.

1. Cleanup
• Remove any inactive installations with the command:

Switch#install remove inactive

2. Copying the New Image

• Transfer the new .bin image file to the active switch's flash storage using one of the following methods:

• Via TFTP:

Switch#copy tftp://Location/directory/<file_name>.bin flash:

• Via USB:

Switch#copy usbflash0:<file_name>.bin flash:

• Confirm the available file systems with:Switch#show file systems

3. Verification

After transferring the IOS to the active switch, check if the image is correctly copied with:

Switch#dir flash:

(Optional) To verify the MD5 checksum, use the command:

Switch#verify /md5 flash:<file_name>.bin

Ensure this checksum matches the one provided on the Software Download page.

4. Setting the Boot Variable

Set the boot variable to point to the packages.conf file with the following commands:

Switch#configure t

Switch(config)#no boot system

Switch(config)#boot system flash:packages.conf

Switch(config)#end

5. Autoboot Configuration

Configure the switch to autoboot by executing:

Switch#configure t

Switch(config)#no boot manual

Switch(config)#end

6. Saving Configuration:

Save your current configuration with:

Switch#write memory

Confirm the boot settings with the command:

Switch#show boot system

7. Installation of the Image:

To install the image, use the command:

Switch#install add file flash:<file_name>.bin activate commit

When prompted with "This operation requires a reload of the system. Do you want to proceed? [y/n]," respond with "y" to proceed.

8. Verification of the successful upgrade

Switch#show version

Switch#show redundancy

Note: Replace with the actual name of your IOS image file throughout the steps.

Bundle Mode 

A bundle mode upgrade on a Cisco Catalyst 9500 switch refers to a method of upgrading the switch's software where the entire software image is bundled into a single file. This file includes all the necessary components such as the operating system, device drivers, and other essential software required for the switch to operate. The upgrade involves a single software image file, typically with a .bin extension. This contrasts with other methods, such as install mode, which may involve multiple files and packages.

For C9500 we can upgrade directly from 16.x.x to 17.x.x train or within 17.x.x train in INSTALL mode. Please refer to release note of target IOS which is found externally for more understanding.

While upgrading in BUNDLE mode from 16.x.x to 17.x.x it is recommended to go for an intermediate IOS version because of bug CSCwh54386 : Bug Search Tool (cisco.com)

For example, 16.8.x (older) -> 17.3.x  (middle one) -> 17.9.X (newer)

Please follow the outlined steps for an upgrade in Bundle mode:

1. Transfer the new image (.bin file) to the flash memory of each stack member in the standalone switch or stack using one of these methods

Via TFTP:

Switch#copy tftp://location/directory/<file_name> flash: 

Via USB:

Switch#copy usbflash0:<file_name>.bin flash: 

2. Confirm the available file systems by using the command

Switch#show file systems

3. After copying the IOS to all member switches, verify that the image has been correctly copied with

Switch#dir flash: 

4. (Optional) Verify the MD5 checksum with the command

Switch#verify /md5 flash:<file_name>.bin

Ensure that the output matches the MD5 checksum value provided on the Software Download page.

5. Configure the boot variable to point to the new image file with these commands

Switch#configure t

Switch(config)#no boot system

Switch(config)#boot system flash:<file_name>.bin

Switch(config)#end

6. Save the configuration

Switch#write memory

7. Verify the boot settings using

Switch#show boot system

8. Reload the switch to apply the new IOS

Switch#reload

9. Verification of the successful upgrade

Switch#show version

Switch#show redundancy

Note: Replace with the actual name of your IOS image file throughout the steps.

In Service Software Upgrade (ISSU)

In-Service Software Upgrade is a process that upgrades an image to another image on a device while the network continues to forward packets. ISSU helps network administrators avoid a network outage when they perform a software upgrade. The images are upgraded in install mode, wherein, each package is upgraded individually.

For Catalyst 9500 with StackWise Virtual, ISSU support starts from Cisco IOS XE Fuji 16.9.2.

For Catalyst 9500 High Performance, ISSU support starts from Cisco IOS XE Gibraltar 16.12.1c.

For Catalyst 9500X with StackWise Virtual, ISSU support starts from Cisco IOS XE Cupertino 17.12.1.

Please ensure the current SW version and Target SW version is suitable for ISSU upgrade using the link below:

Compatibility Matrix

To verify if the switch is a C9500 series switch or a C9500 high performance series switch, please look into Table 30 of below document:

Cisco Catalyst 9500 Series Switches Datasheet

Note: For ISSU upgrade from 17.3.1, 17.3.2, 17.3.3, or 17.3.4 to 17.6.x in standalone chassis with quad supervisor or high availability setup, you must perform an ISSU upgrade to 17.3.5 and then perform ISSU upgrade to the final target release version. ISSU upgrade to 17.9.1 might fail. See CSCwc54402 for more details. ISSU upgrade from 17.6.4 to 17.9.3 might fail. See CSCwc54402 for more details.

Prerequisites for ISSU

1. Check Current Code Version 

C9500#show version | include IOS XE

2. Check the Boot Mode 

ISSU is supported only if both the switches in Stackwise Virtual are booted in Install mode.

C9500#show version | include INSTALL

3. Check if There is Sufficient Available Memory on Flash 

C9500#dir flash: | include free
11353194496 bytes total (8565174272 bytes free)

C9500#dir stby-flash: | include free
11353980928 bytes total (8566865920 bytes free)

4. Check if Switches are in SSO Mode 

C9500#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 4 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none

Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up

Current Processor Information :
-------------------------------
Active Location = slot 1
Current Software state = ACTIVE <-------
Uptime in current state = 30 minutes
Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Mon 05-Nov-18 19:32 by mcpre
BOOT = flash:packages.conf;
CONFIG_FILE =
Configuration register = 0x102

Peer Processor Information :
----------------------------
Standby Location = slot 2
Current Software state = STANDBY HOT <-------
Uptime in current state = 26 minutes
Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Mon 05-Nov-18 19:32 by mcpre
BOOT = flash:packages.conf;
CONFIG_FILE =
Configuration register = 0x102

5. Check if Auto-Boot is Enabled

C9500#show boot system
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no <------- Manual Boot should be set to "no"
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
---------------------------
Switch 2
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0

If Auto-Boot is not enabled, this can be changed as shown:

C9500(config)#no boot manual

6. Check the Current ISSU and Install States 

C9500#show issu state detail
--- Starting local lock acquisition on switch 1 ---
Finished local lock acquisition on switch 1

No ISSU operation is in progress <------- If see anything else, abort ISSU before proceeding.
Check on how to manually abort ISSU.
C9500#show install summary
[ Switch 1 2 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version
--------------------------------------------------------------------------------
IMG C 16.9.2.0.2433 <------- State should be Activated & Committed for current version alone.
If not clear install state before proceeding. Check on how to clear install state.

--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

Steps To Upgrade 

Please follow the outlined steps to perform an In-Service Software Upgrade (ISSU) upgrade. 

1. Cleanup

Remove any inactive installations with the command: 

Switch#install remove inactive

2. Copying the New Image

Transfer the new .bin image file to the active supervisor's flash storage using one of the following methods:

Via TFTP:

Switch#copy tftp://Location/directory/<file_name>.bin flash:

Via USB:

Switch#copy usbflash0:<file_name>.bin flash:

Confirm the available file systems with: Switch#show file systems

3. Verification

After transferring the IOS to the active supervisor's flash, check if the image is correctly copied with: 

Switch#dir flash:

(Optional) To verify the MD5 checksum, use the command: 

Switch#verify /md5 flash:<file_name>.bin

Ensure this checksum matches the one provided on the Software Download page.

4. Setting the Boot Variable

Set the boot variable to point to the packages.conf file with the following commands: 

Switch#configure t

Switch(config)#no boot system

Switch(config)#boot system flash:packages.conf

Switch(config)#end

5. Autoboot Configuration

Configure the switch to autoboot by executing:

Switch#configure t

Switch(config)#no boot manual

Switch(config)#end

6. Saving Configuration

Save your current configuration with: 

Switch#write memory

Confirm the boot settings with the command: 

Switch#show boot system

7. Installation of the Image

To install the image, use the command:

Switch#install add file flash:<file_name>.bin activate issu commit

8. Verification of the successful upgrade

Switch#show version

Switch#show redundancy

Once you run the command noted here, the process starts and reloads sup automatically. Do not run the command until you are ready for sups to start to reboot. Unlike the normal upgrade process, it does not ask for confirmation from you before the reload happens.

Once you run this command, the ISSU process extracts the files, reloads the standby sup, waits for it to get back to SSO then failover reloads the active. 

Note: Replace with the actual name of your IOS image file throughout the steps.

ISSU Validation Steps

Once ISSU is successfully completed,

• Check if both switches run on the new software using Switch#show version.
• Check show issu state detail output to be clean and not showing any ISSU in progress.
• Check show install issu history output to ensure successful ISSU operation (Command available only with 16.10.1 release and later).

Steps to Recover from ISSU Failure

• If ISSU fails, it is expected that auto-abort can recover the system back to initial state (older image). However, if this fails as well, manual recovery of the chassis is expected.
• During manual recovery, check if both active and standby run the older image (if not, recover the individual chassis).
• After you ensure both chassis run the old image, run install remove inactive to remove any unused image packages.
• Once both chassis run the old software, manually clean all the internal states of ISSU operation. (Refer here on how to clean the internal ISSU states).

Abort ISSU

In the 3-Step Work Flow, during the activate ISSU process, the system can auto-abort to older image if the abort-timer expires. Manual abort is required if the standby does not reach SSO during abort. Also, if for any reason you wish to abort the ISSU in between, manual abort is required.

C9500#install abort issu

Clean ISSU State 

If ISSU upgrade/downgrade/abort/auto-abort is not successful, manual clean up of ISSU internal states is required.

Enable service internal before running the following command:

C9500#configure t
C9500(config)#service internal
C9500(config)#end

C9500#clear install state
clear_install_state: START Tue Nov 13 17:05:47 UTC 2018
--- Starting clear_install_state ---
Performing clear_install_state on all members
[1] clear_install_state package(s) on chassis 1
[1] Finished clear_install_state on chassis 1
Checking status of clear_install_state on [1]
clear_install_state: Passed on [1]
Finished clear_install_state

C9500#sh issu state detail
--- Starting local lock acquisition on chassis 1 ---
Finished local lock acquisition on chassis 1

No ISSU operation is in progress