Field Notice: FN72464 - Cisco Nexus 9300 Switches and APIC Servers Can Experience Memory Failures - Hardware Upgrade Available

A limited number of dual in-line memory modules (DIMMs) shipped from Cisco are impacted by a known deviation in the memory supplier's manufacturing process. This deviation can result in a higher-than-expected rate of failure.

DIMM manufacturers compose their DIMMs of multiple memory modules to reach the desired capacity. In this case, a manufacturing deviation in specific modules impacts 8GB DIMMs and 16GB DIMMs. This deviation was contained to a specific date range, and the DIMMs that were manufactured during the middle to end of 2020.

Since the discovery of this deviation, additional limits have been imposed on the manufacturing process to help prevent future DIMMs from experiencing early failure due to this process variation.

Most DIMMs with this manufacturing deviation will exhibit persistent correctable memory errors. If left untreated, the DIMMs can eventually encounter an uncorrectable memory event. If encountered during runtime, uncorrectable errors will cause an unexpected switch reset.

Various DIMM Reliability, Availability, and Serviceability (RAS) features or even operating system features can mask the extent of these correctable errors. It is recommended to check your DIMMs for exposure using the Serial Number Validation Tool described in the Serial Number Validation section of this field notice. Only specific DIMMs are impacted by this issue.

Solution

Customers should replace the hardware DIMM to avoid the potential for unexpected switch/server failure. For information about requesting a replacement, see the Upgrade Program Information section of this field notice after validating the Serial Number(s) as described in the How To Identify Affected Products section.

Important: For an updated list of impacted products, re-check serial numbers for all products that are listed in this field notice. All 8 GB and 16 GB are included in the Serial Number Validation tool.

All Serial Numbers are the Switch or Server Serial Number, not the DIMM Serial Number.

DIMM Replacement

Cisco is offering field services free of charge for DIMM replacement through a qualified Cisco third-party Field Engineer.

After the replacement DIMMs have arrived onsite and you are ready to schedule the replacement(s), send an email to [email protected] to engage the field services team.

Impacted DIMMs can be identified based on the Product ID (PID) and their serial number. To determine if a serial number is affected, see the Serial Number Validation section of this field notice. 

Note: Cisco recommends notifying our engineers for onsite service to schedule repairs or replace the device. See the Additional Information section.

Switch Logs That Contain Memory Errors

When running NX-OS Standalone, a unit that experiences this issue can show these messages in the syslogs on the device:

• %DAEMON-3-SYSTEM_MSG: Location: SOCKET:0 CHANNEL:? DIMM:? [] - mcelog
• %DEVICE_TEST-3-MCE_24HR_FAIL: Module 1 has exceeded MCE 24 hour correctable threshold of 100 with #### correctable errors within 24 hours.
• %DAEMON-3-SYSTEM_MSG: corrected Socket memory error count exceeded threshold: #### in 24h - mcelog

or

• %DAEMON-3-SYSTEM_MSG: MESSAGE : corrected DIMM memory error count exceeded threshold: #### in 24h - mcelog
• %DAEMON-3-SYSTEM_MSG: MESSAGE_Location: /var/log/mcelog - mcelog

Note: Syslogs that reference the count of errors will be generated whenever 100 errors are corrected and the cumulative count of errors will be printed.

These errors indicate that correctable errors are being generated, which should not impact switch performance. If errors continue, an uncorrectable error can be experienced, and the device will undergo a kernel panic.

When running ACI software, a unit that experiences this issue can show the errors in /mnt/pss/bootlogs/current/dmesg. For example, the following logs confirm that DIMMs are bad and in which DIMM-0 is bad:

• [  167.751610] sbridge: HANDLING MCE MEMORY ERROR
• [  167.751614] CPU 0: Machine Check Exception: 0 Bank 7: 8c00004000010091
• [  168.415928] EDAC MC0: 1 CE memory read error on CPU_SrcID#0_Channel#1_DIMM#0 (channel:1 slot:0 page:0x53232 offset:0xfc0 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0091 socket:0 channel_mask:2 rank:0)

Switch Special Notes

This field notice is to replace memory on site, and it is important to note that if your impacted switch/server has failed or has memory errors and is degraded, use the standard RMA replacement.

Onsite replacements are usually done during maintenance windows and typically scheduled.

Some switches are originally assembled with 24GB of memory, this is one 16GB and one 8GB DIMM. Because of this, we send both DIMMs on those devices. When the switch cover is removed, both DIMMs are replaced even though the 8GB is known to be extremely reliable. This is a proactive action decision made by the Cisco team.

“Cisco highly recommends that you take advantage of the field service offering of a field engineer. Please note that if you decided to change the DIMMs on your own there are some switch models that have a more extensive process to gain access to the DIMM location."

It is highly recommended for a Cisco Field Engineer to complete the replacement for switches with high screw counts.

You can view a list of the switch module and the number of screws involved in this table.

APIC Server Special Notes

• The APIC server serial numbers were added to Serial Number Validation as of Revision 1.2 of this field notice.
• APIC-SERVER-M3 and spare has 6 DIMMs, ask for all 6 when you fill out your form for each server. You will replace ALL the memory.
• APIC-SERVER-L3 and spare has 12 DIMMs, ask for all 12 when you fill out your form for each server. You will replace ALL the memory.
• An onsite Field Engineer is available for free if needed. Customers can choose to do this themselves.
• There is a very important Field Notice CIMC BIOS. See the customer documentation for your current release on how to upgrade Application Centric Infrastructure (ACI) software or CIMC BIOS. The document is titled "Cisco APIC Installation and ACI Upgrade and Downgrade Guide".

  The CIMC BIOS issue is noted in UCS field notice FN72272. This BIOS issue will show higher EC errors counts that are potentially higher than the actual EC error count. You can see Uncorrectable errors because of older BIOS.

Additional Details About Revisions

Revision 3.0

In revision 3.0 Cisco Product number N9K-C93216TC-FX2 was added to the field notice.

Revision 2.0

DIMMS with 8 GB density that were Replace on Failure were moved to Proactive replacements. This is due to a higher fail rate than expected in the field.

Note: Any switch that was repaired previously has had all of its memory replaced. There is no need for the memory to be replaced again.

For Cisco N9K-C93360YC-FX2 or N9K-C93216TC-FX2, the DIMMs are not field replaceable. Cisco will replace the switch, not DIMMs, using the normal RMA process. 

Revision 1.2

Cisco Application Policy Infrastructure Controller (APIC) products were moved from Replace on Failure to Proactive DIMM replacement. 

It is required to replace the DIMMS and update from the older Cisco Integrated Management Controller (CIMC) BIOS (Version 4.1(3c) or earlier) in the same maintenance window.