PTP hardware timestamping appears broken on Pi 5 about linux

[pelwell]

The Pi 5 PHY does not support hardware timestamping.

[davidv1992]

I have played around a bit with the hardware timestamping available on the PI 5 and there are definitely a few strange things going on which suggest to me there are some driver and/or kernel bugs involved.

When looking at IPv4 ptp packets, when configuring the driver to timestamp only layer 4 v2 ptp packets (passing HWTSTAMP_FILTER_PTP_V2_L4_EVENT to the SIOCSHWTSTAMP ioctl), some received packets are not properly timestamped. However, when enabling the HWTSTAMP_FILTER_ALL filter (which is according to ethool -T the only filter that is supported, and hence what HWTSTAMP_FILTER_PTP_V2_L4_EVENT should be upgraded to) send timestamps for delay requests are suddenly not properly timestamped.

However, looking at ethernet packets, due to a bug I was configuring this also with HWTSTAMP_FILTER_PTP_V2_L4_EVENT, and there all ptp packets seem to get timestamped. So the upgrade process at least partially works. Not sure what is going on in the driver exactly but definitely some hardware timestamping bugs.

All this testing was done on the 6.1.0-rpi8-rpi-2712 kernel

[dsseng]

What? I've seen (perhaps 3rd party) reviews about it supporting PTP hardware timestamping. Cadence GEM MACs support PTP (unsure about the PHY support). Perhaps this should be reopened to properly detect hardware timestamping support in the GEM/macb driver. However PHC itself should be kept as we tested it to be functional and properly adjustable even with userspace timestamping.

[P33M]

Works on my machine(s)? CM4+CM4IO plugged into a dumb switch with a Pi 5. CM4 assumes master role, Pi 5 syncs to master.

pi@raspberrypi:~$ sudo ptp4l -i eth0 -l 7 -q -m -q -s
ptp4l[13637.903]: config item (null).assume_two_step is 0
ptp4l[13637.903]: config item (null).check_fup_sync is 0
ptp4l[13637.903]: config item (null).tx_timestamp_timeout is 1
ptp4l[13637.903]: config item (null).hwts_filter is 0
ptp4l[13637.903]: config item (null).clock_servo is 0
ptp4l[13637.903]: config item (null).clock_type is 32768
ptp4l[13637.903]: config item (null).clock_servo is 0
ptp4l[13637.903]: config item (null).clockClass is 248
ptp4l[13637.903]: config item (null).clockAccuracy is 254
ptp4l[13637.903]: config item (null).offsetScaledLogVariance is 65535
ptp4l[13637.903]: config item (null).productDescription is ';;'
ptp4l[13637.903]: config item (null).revisionData is ';;'
ptp4l[13637.903]: config item (null).userDescription is ''
ptp4l[13637.903]: config item (null).manufacturerIdentity is '00:00:00'
ptp4l[13637.903]: config item (null).domainNumber is 0
ptp4l[13637.903]: config item (null).slaveOnly is 1
ptp4l[13637.903]: config item (null).gmCapable is 1
ptp4l[13637.903]: config item (null).gmCapable is 1
ptp4l[13637.903]: config item (null).G.8275.defaultDS.localPriority is 128
ptp4l[13637.903]: config item (null).maxStepsRemoved is 255
ptp4l[13637.903]: config item (null).time_stamping is 1
ptp4l[13637.903]: config item (null).twoStepFlag is 1
ptp4l[13637.903]: config item (null).twoStepFlag is 1
ptp4l[13637.903]: config item (null).time_stamping is 1
ptp4l[13637.903]: config item (null).priority1 is 128
ptp4l[13637.903]: config item (null).priority2 is 128
ptp4l[13637.903]: interface index 2 is up
ptp4l[13637.903]: config item (null).free_running is 0
ptp4l[13637.903]: selected /dev/ptp0 as PTP clock
ptp4l[13637.903]: config item (null).clockIdentity is '000000.0000.000000'
ptp4l[13637.903]: config item (null).uds_address is '/var/run/ptp4l'
ptp4l[13637.903]: section item /var/run/ptp4l.announceReceiptTimeout now 0
ptp4l[13637.903]: section item /var/run/ptp4l.delay_mechanism now 0
ptp4l[13637.903]: section item /var/run/ptp4l.network_transport now 0
ptp4l[13637.903]: section item /var/run/ptp4l.delay_filter_length now 1
ptp4l[13637.903]: config item (null).free_running is 0
ptp4l[13637.903]: config item (null).freq_est_interval is 1
ptp4l[13637.903]: config item (null).write_phase_mode is 0
ptp4l[13637.903]: config item (null).gmCapable is 1
ptp4l[13637.903]: config item (null).kernel_leap is 1
ptp4l[13637.903]: config item (null).utc_offset is 37
ptp4l[13637.903]: config item (null).timeSource is 160
ptp4l[13637.903]: config item (null).pi_proportional_const is 0.000000
ptp4l[13637.903]: config item (null).pi_integral_const is 0.000000
ptp4l[13637.903]: config item (null).pi_proportional_scale is 0.000000
ptp4l[13637.903]: config item (null).pi_proportional_exponent is -0.300000
ptp4l[13637.903]: config item (null).pi_proportional_norm_max is 0.700000
ptp4l[13637.903]: config item (null).pi_integral_scale is 0.000000
ptp4l[13637.903]: config item (null).pi_integral_exponent is 0.400000
ptp4l[13637.903]: config item (null).pi_integral_norm_max is 0.300000
ptp4l[13637.903]: config item (null).step_threshold is 0.000000
ptp4l[13637.903]: config item (null).first_step_threshold is 0.000020
ptp4l[13637.903]: config item (null).max_frequency is 900000000
ptp4l[13637.903]: config item (null).servo_offset_threshold is 0
ptp4l[13637.903]: config item (null).servo_num_offset_values is 10
ptp4l[13637.903]: config item (null).dataset_comparison is 0
ptp4l[13637.903]: config item (null).tsproc_mode is 0
ptp4l[13637.903]: config item (null).delay_filter is 1
ptp4l[13637.903]: config item (null).delay_filter_length is 10
ptp4l[13637.903]: config item (null).initial_delay is 0
ptp4l[13637.903]: config item (null).summary_interval is 0
ptp4l[13637.903]: config item (null).sanity_freq_limit is 200000000
ptp4l[13637.903]: PI servo: sync interval 1.000 kp 0.700 ki 0.300000
ptp4l[13637.903]: config item /var/run/ptp4l.boundary_clock_jbod is 0
ptp4l[13637.903]: config item /var/run/ptp4l.network_transport is 0
ptp4l[13637.903]: config item /var/run/ptp4l.masterOnly is 0
ptp4l[13637.903]: config item /var/run/ptp4l.BMCA is 0
ptp4l[13637.903]: config item /var/run/ptp4l.delayAsymmetry is 0
ptp4l[13637.903]: config item /var/run/ptp4l.follow_up_info is 0
ptp4l[13637.903]: config item /var/run/ptp4l.freq_est_interval is 1
ptp4l[13637.903]: config item /var/run/ptp4l.msg_interval_request is 0
ptp4l[13637.903]: config item /var/run/ptp4l.net_sync_monitor is 0
ptp4l[13637.903]: config item /var/run/ptp4l.path_trace_enabled is 0
ptp4l[13637.903]: config item /var/run/ptp4l.tc_spanning_tree is 0
ptp4l[13637.903]: config item /var/run/ptp4l.ingressLatency is 0
ptp4l[13637.903]: config item /var/run/ptp4l.egressLatency is 0
ptp4l[13637.903]: config item /var/run/ptp4l.delay_mechanism is 0
ptp4l[13637.903]: config item /var/run/ptp4l.hybrid_e2e is 0
ptp4l[13637.903]: config item /var/run/ptp4l.fault_badpeernet_interval is 16
ptp4l[13637.903]: config item /var/run/ptp4l.fault_reset_interval is 4
ptp4l[13637.903]: config item /var/run/ptp4l.tsproc_mode is 0
ptp4l[13637.903]: config item /var/run/ptp4l.delay_filter is 1
ptp4l[13637.903]: config item /var/run/ptp4l.delay_filter_length is 1
ptp4l[13637.903]: config item (null).slave_event_monitor is ''
ptp4l[13637.903]: config item eth0.boundary_clock_jbod is 0
ptp4l[13637.903]: config item eth0.network_transport is 1
ptp4l[13637.903]: config item eth0.masterOnly is 0
ptp4l[13637.903]: config item eth0.BMCA is 0
ptp4l[13637.903]: config item eth0.delayAsymmetry is 0
ptp4l[13637.903]: config item eth0.follow_up_info is 0
ptp4l[13637.903]: config item eth0.freq_est_interval is 1
ptp4l[13637.903]: config item eth0.msg_interval_request is 0
ptp4l[13637.903]: config item eth0.net_sync_monitor is 0
ptp4l[13637.903]: config item eth0.path_trace_enabled is 0
ptp4l[13637.903]: config item eth0.tc_spanning_tree is 0
ptp4l[13637.903]: config item eth0.ingressLatency is 0
ptp4l[13637.903]: config item eth0.egressLatency is 0
ptp4l[13637.903]: config item eth0.delay_mechanism is 1
ptp4l[13637.903]: config item eth0.unicast_master_table is 0
ptp4l[13637.903]: config item eth0.unicast_listen is 0
ptp4l[13637.903]: config item eth0.hybrid_e2e is 0
ptp4l[13637.903]: config item eth0.fault_badpeernet_interval is 16
ptp4l[13637.903]: config item eth0.fault_reset_interval is 4
ptp4l[13637.903]: config item eth0.tsproc_mode is 0
ptp4l[13637.903]: config item eth0.delay_filter is 1
ptp4l[13637.903]: config item eth0.delay_filter_length is 10
ptp4l[13637.903]: config item eth0.logMinDelayReqInterval is 0
ptp4l[13637.903]: config item eth0.logAnnounceInterval is 1
ptp4l[13637.903]: config item eth0.inhibit_announce is 0
ptp4l[13637.903]: config item eth0.ignore_source_id is 0
ptp4l[13637.903]: config item eth0.announceReceiptTimeout is 3
ptp4l[13637.903]: config item eth0.syncReceiptTimeout is 0
ptp4l[13637.903]: config item eth0.transportSpecific is 0
ptp4l[13637.903]: config item eth0.ignore_transport_specific is 0
ptp4l[13637.903]: config item eth0.G.8275.portDS.localPriority is 128
ptp4l[13637.903]: config item eth0.logSyncInterval is 0
ptp4l[13637.903]: config item eth0.operLogSyncInterval is 0
ptp4l[13637.903]: config item eth0.logMinPdelayReqInterval is 0
ptp4l[13637.903]: config item eth0.operLogPdelayReqInterval is 0
ptp4l[13637.903]: config item eth0.neighborPropDelayThresh is 20000000
ptp4l[13637.903]: config item eth0.min_neighbor_prop_delay is -20000000
ptp4l[13637.903]: config item eth0.asCapable is 1
ptp4l[13637.903]: config item eth0.inhibit_delay_req is 0
ptp4l[13637.903]: config item eth0.udp_ttl is 1
ptp4l[13637.904]: config item (null).dscp_event is 0
ptp4l[13637.904]: config item (null).dscp_general is 0
ptp4l[13637.904]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[13637.904]: config item /var/run/ptp4l.logMinDelayReqInterval is 0
ptp4l[13637.904]: config item /var/run/ptp4l.logAnnounceInterval is 1
ptp4l[13637.904]: config item /var/run/ptp4l.inhibit_announce is 0
ptp4l[13637.904]: config item /var/run/ptp4l.ignore_source_id is 0
ptp4l[13637.904]: config item /var/run/ptp4l.announceReceiptTimeout is 0
ptp4l[13637.904]: config item /var/run/ptp4l.syncReceiptTimeout is 0
ptp4l[13637.904]: config item /var/run/ptp4l.transportSpecific is 0
ptp4l[13637.904]: config item /var/run/ptp4l.ignore_transport_specific is 0
ptp4l[13637.904]: config item /var/run/ptp4l.G.8275.portDS.localPriority is 128
ptp4l[13637.904]: config item /var/run/ptp4l.logSyncInterval is 0
ptp4l[13637.904]: config item /var/run/ptp4l.operLogSyncInterval is 0
ptp4l[13637.904]: config item /var/run/ptp4l.logMinPdelayReqInterval is 0
ptp4l[13637.904]: config item /var/run/ptp4l.operLogPdelayReqInterval is 0
ptp4l[13637.904]: config item /var/run/ptp4l.neighborPropDelayThresh is 20000000
ptp4l[13637.904]: config item /var/run/ptp4l.min_neighbor_prop_delay is -20000000
ptp4l[13637.904]: config item /var/run/ptp4l.asCapable is 1
ptp4l[13637.904]: config item /var/run/ptp4l.inhibit_delay_req is 0
ptp4l[13637.904]: config item (null).uds_address is '/var/run/ptp4l'
ptp4l[13637.904]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[13637.904]: port 1: received link status notification
ptp4l[13637.904]: interface index 2 is up
ptp4l[13645.611]: port 1: announce timeout
ptp4l[13645.611]: config item eth0.udp_ttl is 1
ptp4l[13645.612]: config item (null).dscp_event is 0
ptp4l[13645.612]: config item (null).dscp_general is 0
ptp4l[13645.612]: selected local clock d83add.fffe.7bc6ba as best master
ptp4l[13647.571]: port 1: received link status notification
ptp4l[13649.708]: port 1: setting asCapable
ptp4l[13649.708]: port 1: new foreign master b827eb.fffe.379f04-1
ptp4l[13653.115]: port 1: announce timeout
ptp4l[13653.115]: config item eth0.udp_ttl is 1
ptp4l[13653.115]: config item (null).dscp_event is 0
ptp4l[13653.115]: config item (null).dscp_general is 0
ptp4l[13653.115]: selected local clock d83add.fffe.7bc6ba as best master
ptp4l[13653.709]: selected best master clock b827eb.fffe.379f04
ptp4l[13653.709]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[13654.664]: port 1: delay timeout
ptp4l[13655.232]: port 1: delay timeout
ptp4l[13655.233]: delay   filtered      32766   raw      32766
ptp4l[13655.709]: master offset    -207368 s0 freq  -55511 path delay     32766
ptp4l[13656.709]: master offset    -212656 s1 freq  -60798 path delay     32766
ptp4l[13656.864]: port 1: delay timeout
ptp4l[13657.710]: master offset       7859 s2 freq  -52939 path delay     32766
ptp4l[13657.710]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[13657.907]: port 1: delay timeout
ptp4l[13657.907]: delay   filtered      34117   raw      35468
ptp4l[13657.993]: port 1: delay timeout
ptp4l[13657.993]: delay   filtered      35468   raw      36740
ptp4l[13658.245]: port 1: delay timeout
ptp4l[13658.246]: delay   filtered      36104   raw      36922
ptp4l[13658.710]: master offset       1022 s2 freq  -57419 path delay     36104
ptp4l[13659.710]: master offset      -2447 s2 freq  -60581 path delay     36104
ptp4l[13659.941]: port 1: delay timeout
ptp4l[13659.942]: delay   filtered      35468   raw      33249
ptp4l[13660.235]: port 1: delay timeout
ptp4l[13660.235]: delay   filtered      34664   raw      33860
ptp4l[13660.710]: master offset       4692 s2 freq  -54176 path delay     34664
ptp4l[13660.767]: port 1: delay timeout
ptp4l[13660.767]: delay   filtered      34677   raw      34677
ptp4l[13661.592]: port 1: delay timeout
ptp4l[13661.592]: delay   filtered      35072   raw      35589
ptp4l[13661.710]: master offset      -1452 s2 freq  -58913 path delay     35072
ptp4l[13662.711]: master offset       5393 s2 freq  -52503 path delay     35072
ptp4l[13663.477]: port 1: delay timeout
ptp4l[13663.478]: delay   filtered      35468   raw      38094
ptp4l[13663.712]: master offset       -866 s2 freq  -57144 path delay     35468
ptp4l[13664.314]: port 1: delay timeout
ptp4l[13664.314]: delay   filtered      35416   raw      35365
ptp4l[13664.401]: port 1: delay timeout
ptp4l[13664.401]: delay   filtered      35416   raw      33737
ptp4l[13664.711]: master offset       -139 s2 freq  -56677 path delay     35416
ptp4l[13665.711]: master offset      -2685 s2 freq  -59265 path delay     35416
ptp4l[13665.938]: port 1: delay timeout
ptp4l[13665.938]: delay   filtered      35058   raw      34752
ptp4l[13666.014]: port 1: delay timeout
ptp4l[13666.014]: delay   filtered      34714   raw      34021
ptp4l[13666.711]: master offset       2808 s2 freq  -54577 path delay     34714
ptp4l[13667.559]: port 1: delay timeout
ptp4l[13667.560]: delay   filtered      34714   raw      36832
ptp4l[13667.711]: master offset      -2181 s2 freq  -58724 path delay     34714
ptp4l[13668.712]: master offset      -1636 s2 freq  -58833 path delay     34714
ptp4l[13668.786]: port 1: delay timeout
ptp4l[13668.786]: delay   filtered      34714   raw      34554
ptp4l[13669.712]: master offset       1542 s2 freq  -56146 path delay     34714
ptp4l[13670.649]: port 1: delay timeout
ptp4l[13670.649]: delay   filtered      34944   raw      35136
ptp4l[13670.712]: master offset       1377 s2 freq  -55848 path delay     34944
ptp4l[13671.712]: master offset      -1329 s2 freq  -58141 path delay     34944
ptp4l[13671.936]: port 1: delay timeout
ptp4l[13671.936]: delay   filtered      34944   raw      34705

[P33M]

The MAC driver registers the ptp device in this case, not the PHY driver. What is your server configuration? I see you have

ptp4l[1220.606]: selected best master clock f8e43b.fffe.a6665f 
ptp4l[1220.606]: foreign master not using PTP timescale 

in the log.

[dsseng]

Both ptp4l and statime (with all 4 combinations of these two on the either side) produce the same: missing RX timestamps. The server is a PC with USB NIC using software (kernel) timestamping.

@P33M what are your kernel and ptp4l versions?

[P33M]

Pi 5:

pi@raspberrypi:~$ uname -a
Linux raspberrypi 6.6.13-v8-16k+ #14 SMP PREEMPT Wed Jan 24 16:31:17 GMT 2024 aarch64 GNU/Linux
pi@raspberrypi:~$ ptp4l -v
3.1.1

Local kernel build, but reflects latest rpi-update.

CM4:

pi@raspberrypi:~$ uname -a
Linux raspberrypi 6.1.63-v8+ #1700 SMP PREEMPT Thu Nov 23 14:10:01 GMT 2023 aarch64 GNU/Linux
pi@raspberrypi:~$ ptp4l -v
3.1

Interestingly, if I swap CM4 to slave mode and the Pi 5 assumes master role then the CM4 reports sync errors similar to yours.

ptp4l[4256.733]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[4256.733]: port 1: received link status notification
ptp4l[4256.733]: interface index 2 is up
ptp4l[4262.613]: port 1: announce timeout
ptp4l[4262.613]: config item eth0.udp_ttl is 1
ptp4l[4262.614]: config item (null).dscp_event is 0
ptp4l[4262.614]: config item (null).dscp_general is 0
ptp4l[4262.614]: selected local clock b827eb.fffe.379f04 as best master
ptp4l[4270.365]: port 1: announce timeout
ptp4l[4270.365]: config item eth0.udp_ttl is 1
ptp4l[4270.366]: config item (null).dscp_event is 0
ptp4l[4270.366]: config item (null).dscp_general is 0
ptp4l[4270.366]: selected local clock b827eb.fffe.379f04 as best master
ptp4l[4270.570]: port 1: setting asCapable
ptp4l[4270.570]: port 1: new foreign master d83add.fffe.7bc6ba-1
ptp4l[4274.570]: selected best master clock d83add.fffe.7bc6ba
ptp4l[4274.570]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[4275.564]: port 1: delay timeout
ptp4l[4275.724]: port 1: delay timeout
ptp4l[4275.985]: port 1: delay timeout
ptp4l[4276.569]: port 1: have SYNC 4, expecting FOLLOW_UP but got SYNC 5, dropping
ptp4l[4277.125]: port 1: delay timeout
ptp4l[4277.569]: port 1: have SYNC 5, expecting FOLLOW_UP but got SYNC 6, dropping
ptp4l[4277.797]: port 1: delay timeout
ptp4l[4277.801]: port 1: delay timeout
ptp4l[4278.569]: port 1: have SYNC 6, expecting FOLLOW_UP but got SYNC 7, dropping
ptp4l[4279.325]: port 1: delay timeout
ptp4l[4279.569]: port 1: have SYNC 7, expecting FOLLOW_UP but got SYNC 8, dropping
ptp4l[4280.569]: port 1: have SYNC 8, expecting FOLLOW_UP but got SYNC 9, dropping
ptp4l[4280.687]: port 1: delay timeout
ptp4l[4281.569]: port 1: have SYNC 9, expecting FOLLOW_UP but got SYNC 10, dropping
ptp4l[4282.429]: port 1: delay timeout
ptp4l[4282.430]: delay   filtered      34295   raw      34295
ptp4l[4283.569]: port 1: have SYNC 11, expecting FOLLOW_UP but got SYNC 12, dropping
ptp4l[4283.857]: port 1: delay timeout
ptp4l[4283.858]: delay   filtered      35490   raw      36686

The MAC supports both one-step and two-step timestamping. It appears that the Pi 4 is expecting two-step but the Pi 5 is transmitting one-step frames.

[dsseng]

I'll try a bit more and could maybe also do a 3.1.1 linuxptp build (it's missing some of the features I use)

[dsseng]

ptp4l from Git v3.1.1 exhibits the same behavior. Current kernel: 6.1.0-rpi8-rpi-2712

If PC with software timestamps is client and Pi is server it shows received DELAY_REQ without timestamp unless I disable HW timestamping on the Pi 5.

[davidv1992]

Further update: for ethernet packets, the send timestamping not working when using HWTSTAMP_FILTER_ALL also occurs for ptp over raw ethernet.

[dsseng]

Thanks for looking into this!