This document describes the working of Network Time Protocol (NTP), Precision Time Protocol (PTP), and Synchronous Ethernet (SyncE) with concepts, sample configurations, and troubleshooting scenarios for Cisco Nexus 9000 switches.

Background Information

A clock for us is a wall clock or a wristwatch; but for networking devices, it is a periodic signal of alternate 0’s and 1’s which is used to sample the data bits. Just like the seconds hand in the clock has an angular movement to represent a second, a pair of 0 and 1 represents T (time period [T=1/frequency]). To generate this clock, network devices use a crystal oscillator which has a ±100 ppm (parts per million) error. For example, a clock with the frequency of 250 MHz and 100 ppm will have a frequency range of 249.975 MHz to 250.025 MHz in generating the clock signal. So, ideally, the clock is not completely periodic but is sufficient for the requirement of sampling the data signals out of the interfaces. All networks benefit from time synchronization. For more information, see Computer Network Time Synchronization.

Even if there is no strict time synchronization requirement for production applications running on a network, event correction and root cause analysis is greatly improved by precise synchronization of network devices.

Types of Network Clock Synchronization

Frequency Synchronization

Frequency synchronization refers to the coordination of clock rates (the speed at which clocks tick) across different systems/components to ensure they operate at the same frequency. This concept is vital in various fields, including telecommunications, computer networks, and digital audio & video broadcasting, where different devices must work together seamlessly.

Imagine two clocks ticking at different rates. Over time, these discrepancies can lead to significant drifts in the time reported by the two clocks. In systems that rely on precise timing, such as data transfer or signal processing, this can cause serious problems. Frequency synchronization ensures that all clocks in a system are ticking at the same rate, preventing such issues.

Phase Synchronization

Phase synchronization is a fundamental concept in the field of physics and signal processing. It refers to the adjustment of the phase of an oscillating signal, such as a wave or a cycle, so that they align or synchronize with each other.

In a system of two or more interacting oscillators, phase synchronization occurs when the difference in their phases remains constant over time, even though their frequencies may not necessarily be the same. This means that the oscillators are effectively "dancing" in step with each other.

Time Synchronization

Time synchronization is the practice of coordinating clocks in a computer network at the same time. It's crucial for many aspects of system administration, data integrity, security, and digital forensics. Here are some of the principles of time synchronization:

● Coordinated Universal Time (UTC): This is the standard time reference used in time synchronization. It's critical that all devices in a network sync to UTC to ensure consistency across the system.

● Stratum Levels: Stratum levels indicate the distance from the reference clock. A stratum 0 device is a highly accurate timekeeping device such as an atomic clock. A stratum 1 device is a computer that receives time signals directly from a stratum 0 device, and so on. The stratum level impacts the accuracy of the time synchronization. Conversely, a stratum 16 clock is not synchronized and free run without an accurate time source.

● Clock Quality: This value is used to select the best clock source between two clocks and switches dynamically to the clock that has greater accuracy.

● Precision: This refers to the accuracy of the time synchronization. More precise time synchronization requires more resources but can be necessary for certain applications.

● Synchronization Frequency: This refers to how often devices in the network update their time. The frequency can depend on the network conditions, the precision required, and the capabilities of the devices.

● Local Clocks: Every device in a network has a local clock that needs to be synchronized. This local clock can drift due to various factors, such as temperature changes or hardware limitations, so regular synchronization is necessary.

● Clocks Locked: A state where the system's clock is synchronized with an external reference clock, ensuring coordinated data processing and transfer. Typically, this achieved using Phase-Locked Loop (PLL) or Delay Locked Loop (DLL) circuitry.

● Clock Recovery: The process of extracting timing information from a data stream, allowing the timing of the data in the stream to be accurately determined without separate clock information.

● Residence Time: End to end time to forward a synchronization frame through a device.

● Round Trip Time (RTT): This value is divided in half to assume latency in each direction.

Principles of Network Time Protocol

The Network Time Protocol (NTP) is a protocol used to synchronize the clocks of computers over an IP network. NTP was developed to solve the problem of multiple computers working together and needing to have synchronized time.

Principles of NTP

● Stratum Levels: NTP operates in a hierarchical manner, with multiple layers or "strata" of servers. At the top of the hierarchy (Stratum 0) are reference clocks, such as atomic clocks or GPS clocks, which provide the most accurate time. Stratum 1 servers are directly connected to Stratum 0 devices and are the next level of the hierarchy. They provide time to lower-stratum servers (Stratum 2, Stratum 3, etc.) and clients. The hierarchy allows the network to handle large numbers of clients while reducing the load on the Stratum 1 servers.

● Time Synchronization: NTP synchronizes time by exchanging time-stamped messages between the client and server. The client sends a message to the server, the server replies with a time-stamped message, and the client adjusts its clock based on the timestamps received.

● Clock Selection Algorithm: NTP uses a complex set of algorithms to determine the best and most reliable time source to use. It considers factors like network latency, jitter, and the stratum level of the servers.

● Clock Discipline Algorithm: This algorithm slowly adjusts the client's clock to avoid abrupt changes in time. It can adjust both the frequency and the offset (the difference between the local time and the server's time) of the clock.

● Fault Tolerance and Redundancy: NTP clients can be configured to synchronize with multiple servers. If one server becomes unreachable or is found to be unreliable, the client can switch to another server. This provides robustness and reliability in the time synchronization.

● Security: NTP includes some security features, like authentication, to prevent malicious attacks that could alter the time synchronization.

● Accuracy: NTP can achieve very high accuracy, often within a few milliseconds on the public Internet and even better in local area networks. NTP accuracy can be up to 50 milliseconds when residence time is high, which is not acceptable for high performance applications such as video distribution or financial applications.

● UTC and Leap Seconds: NTP uses Coordinated Universal Time (UTC) as its time reference and has mechanisms to handle leap seconds (extra seconds added to UTC to keep it in sync with solar time).

Note: It is important to remember, the NTP protocol uses UDP/IP transport but does not support hardware assistance (hardware clock) to offset end-device delays including residence time. As a result, NTP is only accurate to a few milliseconds time synchronization (best case) and incapable of frequency or phase synchronization. Excessive variation in the one-way delay time negatively impacts synchronization accuracy.

NTP associations

● Peer Association: A device can either synchronize to another device or allow another device to synchronize to it.

● Server Association: A device synchronizes to a server. A server association presumes the association is to a lower stratum (more precise) clock.

NTP Clock Synchronization Algorithm

A typical NTP client regularly polls one or more NTP servers. The client must compute its time offset and round-trip delay. The time offset is passed through filters and analysis ("mitigation"). Outliers are discarded and an estimate of time offset is derived from the best three remaining candidates. The clock frequency is then adjusted to reduce the offset over time using so called "discipline" to prevent abrupt changes.

Accurate synchronization is achieved when both the incoming and outgoing routes between the client and the server have symmetrical delay. If the routes do not have a common delay, a systematic bias exists of half the difference between the forward and backward travel times.

NTP Access Restriction and Security

There have been several CVEs announced pertaining to NTP historically. NTP best practice is to ensure NX-OS is up to date with the latest security fixes, deploy NTP access-groups and/or NTP Authentication to protect NTP from malicious actors.

Please note that these are only a few of the recent CVEs and the list is not exhaustive. For a more exhaustive list based on your platform and software version, see the Cisco Security Adviser.

● CVE-2020-13817: In ntpd in NTP before 4.2.8p15 and 4.3.x before 4.3.101, an authenticated attacker can cause a NULL pointer dereference in ntpd, resulting in Denial of Service.

● CVE-2020-11868: In ntpd in NTP before 4.2.8p15 and 4.3.x before 4.3.101, a missing return statement in the receive() function can cause an authenticated peer to receive a response that was intended for another peer.

● CVE-2020-13857: In ntpd in NTP before 4.2.8p15 and 4.3.x before 4.3.101, a packet with a zero-origin timestamp and nonzero xmt (transmit) timestamp will bypass a test for a "valid" timestamp pair.

● CVE-2020-13858: In ntpd in NTP before 4.2.8p15 and 4.3.x before 4.3.101, a packet with a zero-origin timestamp and zero xmt (transmit) timestamp can cause a NULL pointer dereference, leading to Denial of Service.

NTP Configuration Example

Please reference the NTP configuration guide for more details.

NTP Client Configuration

NTP Server Configuration

NTP Access Groups

Access groups will prevent the control-plane from communicating with unapproved NTP peer devices.

NTP Authentication

Authentication ensures approved peers are not spoofed.

Principles of Precision Time Protocol

Precision Time Protocol (PTP) is an industry-standard protocol used to synchronize clocks in a computer network to a high degree of accuracy and precision. It is defined by the IEEE 1588-2008 standard.

The basic principle of PTP is to have one or more master clocks, which are the candidate reference time sources, and one or more slave clocks, which synchronize their time to the master. The selected master sends synchronization messages containing its current time to the slaves, and the slaves adjust their clocks based on the synchronization information from the elected master.

However, simply sending the master's time to the slaves doesn't guarantee precise synchronization due to inherent delay in network transmission of sync messages. This delay needs to be accounted for to achieve precise synchronization.

PTP uses a delay request-response mechanism where the slave sends a delay request to the master, the master responds with a delay response, and the slave measures the time it took for the round trip. This round-trip delay is then used to correct the time received in the sync messages from the master.

In this way, PTP can synchronize clocks across a network with a precision of nanoseconds which is critical in many industries such as telecommunications, broadcasting, utilities and more. The exact precision and performance can vary based on the specific PTP profile used as well as the quality of the network and the clocks themselves. This mechanism along with the underlying hardware support allows PTP to hold much more precise time synchronization than NTP. With dedicated hardware support, PTP enabled can accurately measure the resistance time between a PTP frame leaving the CPU and leaving the outgoing network interface.

The primary goal of PTP is to isolate the link delay and residence time for each hop in the network to understand a precise round-trip time and thus correction factor to synchronize a local clock to the elected grandmaster.

Main Principles of the PTP Protocol

● Precision: PTP is designed to provide very high precision clock synchronization. It can achieve sub-microsecond accuracy in local area networks under ideal conditions, which is more precise than NTP.

● PTP Domain: Defines a set of PTP enabled devices that will synchronize clocks from the elected grandmaster clock.

● Master-Slave Architecture: PTP operates in a master-slave architecture. The master clock sends synchronization messages to the slave clocks, which adjust their time based on these messages.

● Sync Message: Used by master to start a clock synchronization session with each slave.

● Delay Request-Delay Response Mechanism: To calculate the network delay, the slave clock sends a Delay_Request message to the master. The master responds with a Delay_Response message that contains the receipt time of the Delay_Request. This allows the slave to calculate the round-trip delay and adjust its clock accordingly.

● Best Master Clock Algorithm (BMCA): PTP uses the Best Master Clock Algorithm to automatically select the best master clock from a group of clocks. The BMCA considers factors such as clock accuracy and network path delay when selecting the best master clock.

● Timestamping: PTP uses hardware timestamping (when supported by the network hardware) to improve the precision of the time synchronization. The timestamp is applied when the packet is sent or received at the network interface, which reduces the impact of software and system delays.

● Profiles: PTP includes support for different "profiles" that adapt the protocol for different use cases. For example, the default profile is suitable for general-purpose networks, while other profiles are designed for specific industries or applications.

● Support for Synchronization of Time and Frequency: Unlike NTP, which only synchronizes time, PTP can also synchronize frequency, which is important in some applications.

Note: Nexus 9000 switches generally support PTP transport over UDP/IP while PTP over Ethernet is supported only on Cisco Nexus N9K-C9332D-H2R, N9K-C93400LD-H1, N9K-C9408, 93180YC-FX3 and N9K-C93180YC-FX3S platforms. PTP supports multicast as well as unicast communication with the configuration of unicast mode being optional. See the PTP Transport Options section below for more details.

PTP Transport Options

Multicast Transport Mode

PTP uses multicast destination IP address 224.0.1.129 as per IEEE 1588 standards for communication between devices. For the source IP address, it uses the user-configurable global IP address under the PTP domain. PTP multicast mode is only supported on physical interface for L2 or L3 interfaces. PTP multicast uses the Internetwork Control Block range.

For multicast PTP transport, use the PTP source address configuration rather than the local interface IP.

Specifies the VLAN tag for the interface where PTP is being enabled. You can only enable PTP on one VLAN on an interface. In this example, VLAN 100 is selected.

Unicast Transport Mode

In unicast transport mode, PTP uses configurable unicast source and destination IP addresses that can be configured under an interface. Unicast mode is only supported on physical L3 interfaces. Although only physical interfaces are supported the source IP address can be derived from a logical interface assigned IP address (SVI or Loopback).

In both, the unicast and the multicast modes, PTP uses UDP ports, 319 for event messages and 320 for general messages communication between devices. PTP unicast mode is only supported on physical L3 interfaces.

Ethernet Transport Mode

Ethernet/L2 encapsulation mode is only supported using ITU PTP Profile. L2 transport mode (PTP over Ethernet) is not compatible with PTP over IP peers without translation. Only Nexus 9000 platforms which support ITU profile are capable of translation between PTP over Ethernet and traditional PTP over IP.

Note: Cisco Nexus N9K-C9332D-H2R, N9K-C93400LD-H1, N9K-C9408, 93180YC-FX3 and N9K-C93180YC-FX3S platforms support ITU profile SyncE/PTP.

PTP Port States

A PTP enabled interface will transition between the following states according to the PTP interface state machine:

● Initialization: A PTP port is initializing when it first comes up and is in transient state

● Faulty: Indicates a protocol fault condition. A port in this state will not send PTP messages other than Management messages.

● Disabled: PTP disabled on the port. A port in this state will not send PTP messages other than Management messages.

● Listening: The PTP protocol is waiting for Announce Receipt Timeout to expire.

● Pre-Master: This is the same as Master state except that it shall not transmit any PTP related messages except for Pdelay_Req, Pdelay_Resp, Pdelay_Resp_Follow_Up, Signaling, or management messages that are a required response to a message received from the applicable management mechanism.

● Master: Port is providing time synchronization for downstream Slave. All PTP ports on the elected grandmaster switch that are up in steady state, are in Master state.

● Passive: In this state port will not transmit messages itself and will not respond to messages other than the PTP management messages. However, it will process all incoming Announce messages according to the rules of the BMCA. Redundant paths toward the grandmaster clock will be in a passive state.

● Uncalibrated: One or more PTP Ports in the Master state have been detected in the domain. The appropriate PTP Port in the Master state has been selected, and the local PTP Port is preparing to perform clock adjustments with respect to the selected PTP Port in the Master state.

● Slave: Port is receiving time synchronization for upstream Master. The PTP port is performing clock adjustments with respect to the selected PTP port in the Master state. It shall transmit any PTP message required by the protocol for the Slave state or any required response to a message received from the applicable management mechanism.

PTP Message Types

PTP messages are categorized in two separate categories, Event and General messages. Not all message types are required for PTP operation.

Event Messages

● Sync: Sent by the Master to distribute the time of day and start the synchronization session.

● Delay_Req: Sent by the Slave to the Master for end-to-end delay measurement (request-response delay mechanism).

● Pdelay_Req: Sent by link node A for peer-to-peer delay measurement (Not supported in NX-OS).

● Pdelay_Resp: Sent by link node B for peer-to-peer delay measurement (Not supported in NX-OS).

General Messages.

● Follow_Up: Sent by two-step clock Master following the sync message.

● Delay_Resp: Sent by primary for end-to-end delay measurement.

● Pdelay_Resp_FollowUp: Sent by link node B for peer-to-peer delay measurement (Not supported in NX-OS).

● Announce: Sent by the primary to establish a synchronization hierarchy. The Best Primary Clock Algorithm decides the best primary clock based on the clock properties in the announce message.

● Management: Sent by the management node to the clock for updating and querying the PTP datasets maintained by the clock. Management messages are optional depending on which PTP endpoint features are used.

● Signaling: Sent between peers for indication such as the unicast negotiation.

PTP Grandmaster Clock Election

A PTP Domain defines the scope of synchronization for the elected grandmaster clock (GM). The BMCA (Best Master Clock Algorithm) defines how a grandmaster clock is elected within the PTP domain. Below are the default parameters for BMCA as defined in IEEE 1588. Modification of these parameters may exist while running specific non-default PTP profiles.

● Default domain ID is 0

● All PTP nodes are synchronized to the elected grandmaster clock via a tree hierarchy

● Grandmaster clock identifier is derived from the device MAC Address

● After 3x Announce timeout will cause BMCA to re-run

● Announce Message caries BMCA parameters:

◦ Priority 1 (0-255) lower is better.

◦ Clock Accuracy (Set by Device) – Example: Locked on GPS, Class 6.

◦ Priority 2 (0-255) lower is better.

◦ MAC address of device (Tie Breaker).

PTP Clock Roles

In a Precision Time Protocol (PTP) system, nodes may have different roles to ensure accurate and synchronized time distribution across the network, these are the PTP roles:

● Grandmaster Clock (GMC): This is the primary reference clock in a PTP network. It generates the precise time that all other clocks in the system sync to. If a network has multiple grandmaster clocks, the one with the highest quality (based on factors like stability, accuracy, and priority settings) becomes the active grandmaster.

● Boundary Clock (BC): A boundary clock has both master and slave functionalities. It acts as a slave clock to its upstream master clock and as a master to its downstream clocks. This helps in reducing the number of sessions the grandmaster must maintain and can improve timing accuracy in larger networks. Boundary clocks can reduce the Delay_Request handling load on upstream master clocks, including grandmaster clock.

● Transparent Clock (TC): This type of clock measures the time a PTP event message takes to pass through the device and corrects the timestamp in the message to improve timing accuracy. Nexus 9000 switches do not support the transparent clock role.

● Ordinary Clock (OC): An ordinary clock has only a single PTP port. In a PTP system, an ordinary clock could be either a master or a slave device.

PTP clock roles are central in distributing precise time across the network. Nexus 9000 series switches support both PTP master and slave port roles. The PTP topology is computed using the elected grandmaster clock at the root of the PTP domain. Each subsequent PTP peer is represented by a Master/Slave relationship depending on its locality from the currently elected grandmaster. Slave ports recover a clock signal from master ports closer to the grandmaster clock.

Transparent clocks update their internal clocks based on PTP sync messages passing through the device without peering into the PTP control plane. Transparent clocks can also update the Correction Factor (CF) field as Sync messages traverse the transparent switch.

The goal of transparent clock synchronization is to update residence time along the PTP session path without participating in the PTP peer control-plane explicitly.

Nexus 9000 Series switches run exclusively as boundary clocks. Switches, or other network devices, in the PTP path that do not participate in PTP will provide unaccounted sources of delay and jitter reducing the accuracy of the downstream clocks. Additionally, a larger number of boundary clocks in series, like any device, will lead to a tolerance “stack-up” effect where the small precision loss of each clock becomes additive across the PTP domain.

In the examples below we can see both normal and boundary-elected grandmaster clocks distributing PTP messaging across the domain. Additionally, below are two separate PTP domains. In domain 0 the grandmaster clock is a normal clock (edge) and a switch in transparent operation is carrying the PTP sessions between the grandmaster and downstream boundary clocks.

Boundary Clock

Transparent Clock

Typical PTP Deployment - Comparison Boundary vs Transparent Clocks

PTP One-Step vs Two-Step PTP on Cisco Nexus Switches

Precision Time Protocol (PTP), defined in IEEE 1588, operates in the following two modes:

● One-step mode: In this mode, the master clock sends a single message (Sync) to the slave clock, which includes the precise time of transmission stamped by the hardware. The slave clock then uses this timestamp to adjust its own clock. This mode is more simple and faster because it only requires one message to be sent. However, it needs the master clock to be capable of generating the precise timestamp at the time of transmission.

● Two-step mode: In this mode, the master clock sends two separate messages. First, it sends a Sync message without a timestamp. Then, it follows up with a Follow_Up message that contains the precise timestamp. The master clock hardware can observe the sync message resistance time and update precise timing using the Follow_Up. The two-step mode can increase the accuracy of time synchronization because the timestamp is generated after the Sync message is sent.

Note: The decision between one-step and two-step mode depends on the capability of the master clock, the participating slave clocks.

In two-step mode, the process is divided into the following two parts:

● Sync Message: The master clock sends a Sync message to the slave clocks. This message contains a sequence ID but does not contain the precise timestamp when the message was transmitted. Instead, it's used to indicate the beginning of a time interval.

● Follow_Up Message: After the Sync message, the master clock sends a Follow_Up message. This message contains the precise timestamp of when the Sync message was transmitted.

The advantage of this two-step mode is to enhance the precision of the timestamp exchange while simplifying the implementation. By splitting the process into two messages, PTP allows the master clock to take its time to precisely determine the timestamp for the Sync message. This is particularly useful in systems where generating the precise timestamp may take longer than the time it takes to send the message. The downside to the two-step mode is it reduces the effective PTP port scale due to increased messaging overhead.

As a result, the advantage of one-step operation is that it requires less messaging and thus allows for higher client scale.

Nexus -R and -RX platforms support one-step mode when offloaded to the line card context. When a one-step operation is configured, offload is automatically enabled. Also note, that Ethanalyzer visibility into PTP messages is lost when offloaded to the line card CPU context. By default, a two-step operation is enabled on Master and Slave ports.

Nexus CloudScale based switches support two-step operation by default when negotiated on Slave ports facing upstream two-step advertised clocks (Typically Grandmasters). CloudScale supports one-step operation only on master ports and thus master/slave peering between Nexus CloudScale devices.

In summary, there is a performance trade on message rate and one-step vs two-step operations and effective PTP port scale. One-step operation along with PTP timer values are two major factors impacting PTP scale and overall performance.

PTP Profiles

PTP profiles define a specific configuration of the PTP protocol that are designed for specific applications. They define a set of features and options to meet the precise timing requirements of a specific use case.

The standard PTP profile, also known as IEEE 1588, allows for many options with respect to messaging, transport protocol, network topology, etc. PTP profiles narrow down these options, providing a more specified and restricted version of the standard that is optimized for a certain type of application.

For example, the default PTP profile is intended for general purpose applications, while telecom profiles (ITU-T G.8275.1/2) are for telecommunication networks.

Each PTP profile includes specifications for parameters like clock accuracy, sync frequency, delay request-response mechanism, time source, and network protocols to be used. This allows for the precise synchronization of clocks in a network of devices, which is crucial in many industries such as telecommunications, power distribution, broadcasting, and more.

NX-OS supports default 1588, AES67, SMPTE 2059-2, and ITU profiles. They all have different ranges of sync and delay request intervals. For information on the default profile, refer to IEEE 1588. For more information on AES67 and SMPTE 2059-2, refer to the respective specifications.

Deploying an ITU profile comes with some specific caveats.

● The allowed domain number range for G.8275.1 profile is between 24 and 43. The default is 24. The allowed domain number range for the G.8275.2 profile is between 44 and 63. Default is 44

● ITU profiles implement an Alternate BMCA (Best Master Clock Algorithm) for grandmaster clock selection.

Log Mean Interval

PTP timers are described in terms of Log Mean Interval or the time in seconds until the next update interval. This approach allows users to interact with PTP enabled devices in terms of integer values which equate to fractions of a second in operation of the PTP protocol stack. By reducing the LMI updates are sent more frequently thus reducing the local clock drifts from the grandmaster.

● = 0.0625s or 16 packets per second

● = 0.125s or 8 packets per second

● = .25s or 4 packets per second

● = .5s or 2 packets per second

● = 32s or 1 packet in 32 seconds

● = 64s or 1 packet in 64 seconds

Sync Interval Timer

Option	Range	Default
aes67-2015	–4 to 5 log seconds	-3
smpte-2059-2	–4 to 5 log seconds	-3
Without the aes67-2015 or smpte-2059-2 option	–1 to 6 log seconds (where –1 = 2 frames every second)	· -2 log seconds · -3 log seconds for Cisco Nexus 3232C, 3264Q, and 9500 platform switches

AES67 and SMTP 2059 default sync interval is -3, while the range is what you have suggested. We also suggest setting the sync timer to -3 in the media PTP deployments:

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/guide-c07-742142.html

PTP and Port-Channels on Nexus Switches

Cisco Nexus switches do not support PTP configuration on port-channel logical interfaces however PTP can be configured on one or more member interfaces for redundancy. Whether configured for L2 or L3, port-channel members participating in PTP will use the configured PTP source, not the L3 port-channel IP address or SVI IP address carried by an L2 port-channel. On the slave side of the PTP peering a single link in the bundle will be in Slave state while the rest will remain Passive.

N9K-3

N9K-4

PTP Boundary Clock & Generalized PTP Configuration

By default, Cisco Nexus switches operate in boundary-clock mode.

Generalized precision time protocol (PTP) is an IEEE 802.1AS standard that provides a mechanism to synchronize the clocks of the bridges and end-point devices in a network. Generalized PTP defines the mechanism to elect the grandmaster clock (using the Best Master Clock Algorithm [BMCA]) among the time-aware bridges and the talker and listener.

PTP and GPS/GNSS on Nexus 9000

Beginning with Cisco NX-OS Release 10.3(2)F, the GPS input is supported only on the Cisco Nexus 93180YC-FX3 switch with the following limitations.

The GNSS receiver is designed to operate on the GPS, Galileo, GLONASS, BeiDou and QZSS L1 frequencies 1551MHz to 1614MHz, standard position service, and Coarse Acquisition code. When connected to an external GNSS antenna, the receiver contains all the circuitry necessary to automatically acquire GNSS satellite signals, track up to 32 GNSS satellites, and compute location, speed, heading, and time. It provides an accurate one pulse-per-second (PPS) and stable 10-MHz frequency output for internal system use.

Cisco Nexus 9000 Switches as a PTP Grandmaster Clock

Nexus 9000 series switches can simultaneously operate as a Boundary Clock and a Grandmaster clock. A high-precision oscillator is required to operate as a stable grandmaster clock. Nexus 9000 93180YC-FX3 or N9K-C93180YC-FX3S hardware is capable of both the boundary clock function and high precision oscillator for stable grandmaster clock operation.

Starting in 10.3(2)F, the ordinary clock grandmaster is available only for Cisco Nexus N9K-C93180YC-FX3 or N9K-C93180YC-FX3S platform.

https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/103x/configuration/system-management/cisco-nexus-9000-series-nx-os-system-management-configuration-guide-103x/m-configuring-ptp-10x.html#task_zrr_fhy_4sb

Operating a Nexus 9000 switch as an ordinary grandmaster clock will ensure PTP ports are in the master role, or else a role mismatch is detected.

PTP in Unicast Mode

This feature allows BMCA to select roles dynamically as follows, rather than assigning static roles:

· Use the ptp peer<ipv4>/<ipv6> command to configure Peer IPs.

· Port remains in a Listening state until the Peer IP is reachable when it transitions to the Master state.

· Announce packets are sent as soon as the peer is reachable.

· Based on Announce packet (using BMCA), the Role will be decided. Port state transitions accordingly.

Isolating a PTP Nexus 9000 Boundary Clock

The Cisco NX-OS PTP boundary clock can be configured to isolate downstream clocks.

PTP Hardening

PTP Role Enforcement in Multicast Transport Mode

NX-OS allows users to force PTP port role master PTP interface basis for multicast transport mode. This can be helpful in preventing a PTP port from transitioning into an undesired state and thus deriving synchronization from an inferior source. Use role enforcement when connecting Nexus switches to untrusted PTP edge devices or mitigating inadvertent role transition corner cases.

Disabling PTP Management Message Propagation

Management messages can be disabled using the following command. By enabling the PTP feature and disabling management messages, they will be consumed by s/w and not forwarded to downstream PTP slave devices.

As Management Messages are optional, given deployment specifics, there may be reasons to prevent erroneous or unwanted managed queries within the PTP domain.

Some customers may want to block PTP management messages using hardware ACLs. This will additionally block PTP management messages in hardware on the first ingress PTP node where ACL is applied. PTP management ACLs require user defined ACL qualifier carved in hardware forwarding.

Multicast Environment Stability

When PTP is deployed using multicast transport the stability of the PTP domain is dependent on the underlying multicast routing for reachability. Ensure all switches within the PTP domain have the PTP feature enabled for proper handling of Internetwork Control Block addresses assigned to PTP. If PTP is disabled on a transit Nexus device in the PTP domain, point-to-point PTP messages will be flooded to indirectly connected PTP devices disrupting the PTP domain.

Introduction to IP Multicast

Tuning Control-Plane Policing for PTP Scale

Control-Plane Policing (CoPP) is a feature on Cisco Nexus switches that allows users to limit and control the traffic rate of certain protocols processing on the CPU, including PTP.

The default PTP policer (280 kbps) is not sufficient for large scale PTP domain scale. Depending on the number of PTP speakers, one-step vs two-step operation, and other factors such as PTP profile, CoPP will require tuning for PTP to operate nominally.

Consult the PTP scale supported for your platform and NX-OS version to understand the maximum verified scale. In a typical 64 ports PTP switch configuration, with a ‘-3’ update interval, a two-step operation (8 x 4, including 8 management) will average 2304 packets per second. At a typical 80 bytes per frame with 20% overhead, ~1769 kbps.

Monitoring the CoPP System class and verifying drops are not occurring is one of the first steps in validating PTP domain stability.

Default Strict CoPP Policy for PTP contained class:

The NX-OS guide to tune CoPP on Nexus 9000

Note: Remember, CoPP should be configured according to your specific network requirements and the above is just a general guide. If you're not sure about the correct settings, you should contact support for assistance.

Starting in NX-OS 10.5(1) the default CoPP policy has been increased to accommodate larger PTP scale. This does not eliminate the need for CoPP tuning under all circumstances but reduces the need in many typical PTP deployments.

Enhanced Multicast Scale

When operating in Nexus switches in high PTP client scale mode CoPP should be tuned accordingly as well as disabling PTP debugs. The following command allows 64 peers per supported ToR and 64 peers per line card on the supported EoR chassis. Consult the PTP scale supported for your platform and NX-OS version to understand the maximum verified scale. This feature is to be used only in specific deployment scenarios where higher scaling of PTP multicast secondary devices is required even though the ability to debug is very limited.

It is also recommended, at an enhanced multicast scale, to ensure the slave port facing grandmaster source does not share MAC with downstream device peering. Additionally, not more than two master ports should be enabled per hardware MAC. The hardware MAC for ports on any given switch can be checked using the following command:

In this example on C93360YC-FX2, the following highlighted ports share MAC instance 24:

PTP Grandmaster Switchover Events

Multiple candidate grandmaster clocks can exist in each PTP domain but only one is selected as the elected grandmaster clock. During a grandmaster election, either during initial setup or grandmaster failover events from link failures or device failures, there is a period of time where individual device clocks must “free-run” and can drift. This can cause a transient loss in clock precision throughout the PTP domain.

An optimization can be designed to allow fallback to high precision “feeder” switches during the period during grandmaster elections. These feeder switches, such as Nexus 93180YC-FX3, contain high-precision oscillators, just like elected grandmaster clocks. This will reduce transient jitter during re-election events.

Only Cisco Nexus N9K-C9332D-H2R, N9K-C93400LD-H1, N9K-C9408, 93180YC-FX3, N9K-C93180YC-FX3S platform switches using ITU profile can take advantage of high precision DPLL oscillators. These grandmaster clock quality oscillators can maintain high stability while free running during grandmaster convergence events. Additionally, frequency synchronization capable switches can translate between SyncE and PTP over IP. This requires PTP to operate using the ITU profile along with the associated requirements, such as ITU profile valid domain IDs.

Principles of SyncE

Synchronous Ethernet (SyncE) is a method for distributing a clock signal through an Ethernet network. The goal of SyncE is to enable Ethernet networks to carry synchronized clock signals which are necessary for time-sensitive applications like voice and video services.

The principle of SyncE is based on the traditional Ethernet physical layer but it adds a clock signal into the Ethernet scheme. In other words, SyncE extends the functionality of Ethernet by adding the ability to deliver a clock signal from one end of the network to the other thus emulating clocking distribution in a legacy TDM network.

The clock source sends its timing information to the Ethernet equipment, which then embeds the clock signal into the Ethernet data stream. This clock signal is then propagated throughout the network, allowing all the devices on the network to synchronize their clocks to the same reference clock.

This synchronization is crucial for many applications, particularly in telecommunications and networking, where precise timing is required across multiple devices. It ensures that data packets are transmitted and received at the correct times, reducing jitter, and improving the quality of service. It's important to note that SyncE only provides frequency synchronization, not time-of-day synchronization. For full time and phase synchronization, SyncE is often used in combination with PTP.

Starting in NX-OS 9.3(5) on Nexus 93180YC-FX3 or N9K-C93180YC-FX3S a PTP/SyncE hybrid solution is available to provide an end-to-end synchronization solution.

Hybrid PTP Design

Profile Default: mode { hybrid | non-hybrid | none }

Example:

Configures the PTP operational mode for the switch:

· hybrid: The SyncE source acts as the PTP source.

· default: The local/1588 clock acts as the PTP source.

o This command is automatically configured when the ptp profile command is set. The configuration value cannot be changed.

o The only mode that the 8275-2 profile supports is non-hybrid.

o The profile default mode is hybrid.

Relationship between PTP and SyncE

PTP (IEEE 1588) is a protocol used to synchronize clocks throughout a computer network to a high degree of accuracy and precision. PTP can provide sub-microsecond synchronization, making it suitable for applications that require very precise timing. However, PTP distributes time-of-day information, but not a continuous synchronization signal.

SyncE, on the other hand, is an Ethernet-based synchronization standard. It provides a method for transmitting a continuous synchronization signal over the physical layer of the Ethernet network. SyncE can provide frequency synchronization, but not time-of-day or phase synchronization.

PTP and SyncE is that they can work together to provide comprehensive time and frequency synchronization in a network.

While PTP is used for phase and time-of-day synchronization. The combination of SyncE and PTP can provide the accurate and reliable synchronization needed for many time-sensitive applications.

In this combined model, SyncE helps to stabilize the frequency of the local oscillator of the network devices, reducing the amount of phase variation that PTP must compensate for, resulting in better overall synchronization performance.

SyncE operation of PTP over Ethernet cannot interop with PTP over IP, unicast, or multicast mode, without conversion. A Nexus 93180YC-FX3 or N9K-C93180YC-FX3S can operate as both PTP over Ethernet and PTP over IP modes and translate between PTP transport methods.

NX-OS Clock Manager

Multiple clock synchronization protocols along with multiple hardware clocks can exist on a single Nexus switch. The Clock Manager software component is designed to abstract hardware clocks from individual synchronization protocols to allow multiple protocols to coexist at once. Clocks are resources that need to be shared across different processes and across different VDCs. Thus, we require an arbitrator that chooses just one-time synchronization protocol instance at a given time whose inputs should be used to control the various clocks in the system. In the case where no time synchronization protocol instance is running, we still perform intra-chassis synchronization and syntonization so that latency measurement protocols can get accurate results.

Users configure the “clock controller” responsible for synchronizing clocks across the system. For instance, you may want PTP to be the clock controller while running NTP to distribute clocking across your network environment.

Specify the desired clock protocol to be primary on Nexus 9000

Note: Generally, running NTP and PTP simultaneously is not supported on Nexus 9000 switches. If a user wants to run PTP, configure the clock protocol ptp. If a user runs NTP, configure the clock protocol ntp.

When PTP clock source (i.e Master) connectivity gets disconnected, it's hard for slaves (acting as Master for other slaves in the network) to synchronize time to its downstream slaves. To overcome this problem, system clock time can be fed to the PTP slave for it to propagate/sync to its slaves.

When a GNSS source is configured for PTP grandmaster configuration the clock protocol should be set to GNSS:

Nexus 9000 Switches as PTP Grandmaster with Cisco NDFC

The PTP disciplined hardware clock is either disciplined by GM or “freerun” off the internal oscillator. Nexus switches are not meant to be grandmasters besides C93180YC-FX3 or N9K-C93180YC-FX3S platforms operating in ITU profile thus enabling their DPPL high precision oscillators.

However, for use with DCNM/ND Flow Telemetry Analytics, Nexus 9000 switches can be operated as grandmaster and configured to synchronize their hardware clock via the NTP disciplined system clock. In such deployments, Cisco Nexus 9000 hardware clocks, which would typically be synchronized by an external PTP grandmaster, can now be synchronized by the internal system clock periodically. This allows PTP deployment in NDFC requirements to not require an external grandmaster clock to support the Flow Telemetry feature. Please also refer to the NX-OS Clock Manager selection below for more details on NTP/PTP interoperation.

Starting in NX-OS 9.3(10) and 10.2(3) NTP synchronization and PTP coexistence are supported using a new command to periodically update internal clocks. This command allows the NTP synchronized system clock RTC (Realtime Clock) to update the PTP synchronized hardware clock.

The default system time synchronization is every 60 seconds, the Interval is configurable in seconds format, 0 seconds to 3600 seconds. This command is ignored when any of the switch ports is configured as PTP a slave.

PTP Troubleshooting

Data Collection

Whenever troubleshooting a technical issue, the first order of business is data collection as soon as possible after the incident occurs. Additionally, note the timestamp of the event using the show clock. If the issue is cyclic or reoccurring annotate a few example timestamps for later analysis.

Increasing visibility into the PTP domain includes state transitions as well as high correction values and other PTP events can help troubleshoot issues. PTP protocol stability is very important to clock stability, as a result, notification of PTP state churn can be an important clue in tracking down PTP performance issues.

Troubleshooting High PTP Correction Values

Persistent high correction values indicate instability in PTP clock source or the switch’s ability to synchronize (lock) to elected source. Note that Ethernet is not a lossless transport medium, and therefore infrequent random variances in precision are expected.

The best practice is to define and configure threshold and notification of high correction events. The threshold is driven by the network operating requirements but typical is defined around 450ns of drift, with a goal to not exceed 500ns over 7 hops. This will yield a less than <1ms +/- variance in end-to-end drift.

cIf high correction values are observed, the following steps can be taken to troubleshoot the issue further:

· Collect data as shown above.

· Validate PTP port stability working back toward grandmaster.

· Identify any state changes in the PTP domain, grandmaster elections, PTP parent state changes etc.

· Ensure PTP messages aren’t dropped or delayed between PTP peers.

o PTP message rates are traded between message frequency (higher precision) and processing delay/impact at scale.

o One-step operation reduces message rates to help accommodate the PTP scale.

o Increasing the sync timer reduces messages to help accommodate the PTP scale.

Note: If PTP message drops or jitter observed considering increasing the PTP sync timer interval, especially at large PTP scale.

Nominal corrections should be +/- small values in 10s of nanoseconds indicating a stable PTP environment.

Persistent bad corrections indicate a large amount of clock drift between the selected PTP master and the downstream slave. This usually means instability in the PTP domain clock roles or transport/processing issues with PTP updates.

Validate Upstream

How to Identify Grandmaster Clock and Verify GM Path Stability

For a given boundary clock, all upstream PTP clocks in the path of the grandmaster must be stable. Working backward from the switch(es) experiencing high corrections will allow us to isolate the source of instability. Verifying PTP port state stability, counters, and potentially captures to look for missing or late messages will finally isolate the root cause.

Leveraging PTP Message Counters

PTP counters are a powerful tool to identify and isolate problems within the PTP domain. In a two-step operation, sync and follow-up messages should be increasing at the same rate for a given PTP peer.

In the following example, a pair of N9K-X9636C-RX modules are connected back-to-back in a two-step operation. Here a user can confirm the slave port is processing an equivalent number of Sync and Follow-up messages. This analysis only applies to two-step enabled ports.

A difference in the Sync and Follow-up Rx counters indicates a processing issue that can lead to a loss of PTP precision and high correction values.

Nexus -R platforms in offload operation require the use of offload-counters command:

In this example, offload counters can be used to see what messages are sent and received by the line card offload engine.

Debugging PTP Messages on a Specific Interface

It may be necessary to analyze a specific PTP peering relationship. Using the port identity field, a user can filter messages for a specific PTP peering. In this case the user is investigating the peering on port Ethernet 1/1.

Also note, this approach will not work if the PTP function is offloaded to the line card. In such cases PTP offload counters can be used to analyze PTP operation.

Specialized PTP Data Collection and Troubleshooting

While working with Cisco Services it may be necessary to collect specific PTP logs to assist in troubleshooting PTP or SyncE issues.

Below is a summary of these collection techniques:

Show PTP Troubleshooting

A troubleshooting command was developed to help identify and resolve basic configuration and performance issues with the PTP protocol. This tool can be especially useful in isolating configuration mismatches when deploying PTP in static unicast mode.

In the example below, two Nexus 9000 switches are connected via Eth1/1 respectively. The user observes both peers are in the Master state, which is not correct. This troubleshooting command can be run to isolate a configuration problem. In this case, the troubleshooting tool isolated a PTP domain mismatch as the root cause.

PTP Auto-Collect Logging

PTP auto-collect logs are enabled by default and are written to boot flash. The auto-collect mechanism will trigger when high correction values are observed based on the high-correction threshold command described below. Auto-collection defaults to 200 ns correction limit and will ignore the first 20 bad correction values on PTP start.

The goal is to preserve log data for a specific high-correction event in steady-state operation. These values can be adjusted using the following test commands and are independent of the high-correction syslog alert function described in the previous section.

PTP Servo Log

As discussed earlier some Nexus 9000 platforms implement a high-precision time algorithm to synchronize Time/Frequency/Phase of the internal clock. Such platforms can write a separate servo log to boot flash containing servo algorithm event records.

Verify a particular Nexus 9000 switches contains Servo:

Using EEM to Collect Event Logs

Specific event logs, such as Best Master Clock (BMC), can also roll over rapidly. As a result, it may be necessary to trigger log collection using EEM (Embedded Event Manager) automatically. In the below example, EEM is used to match PTP GM clock change log messages, collect specific event messages and save to boot flash.