ICS Essentials: IEEE-1588 Precise Time Protocol (PTP)

IEC 61850-compliant substations require ~1 microsecond timing accuracy. The goal of Precise Time Protocol (PTP) is to accommodate this demand; however, PTP accuracy is dependent upon hardware, such as the processor type - physical or virtual CPU. Some vendors, most notably Belden (i.e., Hirschmann), has had success with integrating PTP on Linux, VxWorks and Windows platforms. For the hobbyist, you can simply tweak the ACPI Linux kernel's boot parameters to reduce the amount of drift between the internal clock and the PTPd daemon:

• Edit the Linux Grub configuration file: $ sudo vi /etc/default/grub
• Add the following ACPI Linux kernel parameter to the file: nohz=off
• Update Grub: $ sudo update-grub
• I recommend a system restart.

Not all devices support PTP due to their chosen hardware CPU implementation. Additionally, not all switch expansion modules support PTP, e.g. Cisco IE 3000 switch's expansion modules. This highlights the need for a proper understanding of the device's supported capabilities as the default PTP configuration may need adjustment. For example, the IE 3000 has a default delay request interval of 32 seconds and the default sync interval is 1 second. Obviously, this could be a deal breaker should your field devices have more granular time requirements. Note: Cisco's IE 3000's PTP clock properties can be enumerated via the IOS "show ptp clock" command.

Currently, there are two standards of PTP available: IEEE1588 (PTPv1 circa 2002) and IEEE1588-2008 (PTPv2). If version specificity was not enough, there are different profiles - Telecom, Power, and others:

• G.8265.1 (Telecom & Frequency Profile)
• G.8275.1 (Time & Phase Profile)
• G.8275.2 (Time and Phase Profile with partial support from the network)
• Furthermore, several recommendations exist for wireless applications in the form of G.8265.x

PTP can be encapsulated within IPv4, IPv6, Ethernet, and even IPSEC. There is a cost to the latter that we will discuss in a subsequent post. PTP is configurable in either unicast (frequency profile only) or multicast (over UDP). Multicast addresses are specific to the PTP version. For example, multicast PTP uses the following IPv4 address schemes:

• 224.0.0.107/32 Mcast_PTP_v2
• 224.0.1.129/32 Mcast_PTP_v2_messages. This multicast IPv4 address has an IPv6 companion -  FF0x:0:0:0:0:0:0:181 for Ipv6, where 'x' can be a value between 0x0 and 0xF. Reference IEEE1588, Annex E, Section E.3.
• 224.0.1.130/32 Mcast_PTP_v1_messages
• 224.0.1.131/32 Mcast_PTP_v1_messages
• 224.0.1.132/32 Mcast_PTP_v1_messages

The above multicast addresses should not be confused with mDNS (224.0.0.251/32) or multicast NTP (224.0.1.1/32). PTPv2 is not backwards compatible with version one.

To minimize inaccuracies, PTP's algorithm supports "best master clock" options and calculates the slave's offset and delay correct time via the below sequence:

1. The master clock sends a SYNC packet containing the grandmasters annotated real-time.
2. A second packet can be sent by the master (subject to its hardware capabilities) to facilitate the slave's time synchronization to the master; thus, it can determine the delay of transmission from the master. This step is referred to as a "two step process" and is optional per implementation. Unlike NTP, PTP sends sync and follow-up messages at a minimum of 1 packet/16 seconds to a maximum of 128 packets per second (pps).
3. The slave transmits a delay request to the master in order to define the latency for its return path.
4. In return, the master sends a delay response to the slave. Now, both systems know the round-trip time (RTT) propagation delay. The slave will divide the total (RTT) by two and adds the result to its current time. Delay requests and responses are also transmitted between 1 packet/16 seconds to 128 pps.
5. Both of the clocks are in sync and the master will instruct the slave to increment or decrement time to ensure accuracy.

This negotiation can be viewed in Wireshark - https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=ptpv2.pcap

The IEEE-1588 Grand Master serves as the primary reference time clock (PRTC) and it often receives its timing via GPS. It is uncommon to find Grand Masters within a substation as its typically, an edge device. Other architectures may require a Boundary Clock (BC) or multiple BCs  to forward PTP timing information to field devices or relay across the topology without delaying time as packets may be queued. These services can be found within some vendors' networking devices. Lastly, we have the clients that are referred to as slaves or "Other Clocks (OC)".

It is trivial to setup PTP within your lab; although, accuracy will drift in this virtual setup.

• Install the Linux PTP daemon on at least two systems (grand master and slave) via APT: $ sudo apt-get -y install ptpd
• Start the PTP daemon on the grand master VM: $ sudo ptpd -CPWjb eth0
• Start the PTP daemon on the slave VM: $ sudo ptpd -CPjb eth0
• In this example, the PTP slave will be listening on UDP ports 319 and 320.

In a subsequent post, we will discuss the attack surface of PTP. Stay tuned!

References:
https://www.youtube.com/watch?v=yw-gd01aOYg
http://tf.nist.gov/seminars/WSTS/PDFs/3-4-IDT_Rodrigues-IEEE%201588-profiles%20at%20ITU-T%20.pdf
http://www.cisco.com/c/en/us/td/docs/switches/lan/cisco_ie3000/software/release/12-2_46_se1/configuration/guide/scg1/swptp.html
https://splash.riverbed.com/thread/8141
http://ubuntuforums.org/showthread.php?t=1366354
https://wiki.wireshark.org/Protocols/ptp
https://www.belden.com/docs/upload/Precision_Clock_Synchronization_WP.pdf