camilleg / clockkit

Sub-msec accurate distributed timestamps.

Clockkit provides timestamps to distributed networked PCs with guaranteed bounds on latency and jitter, typically under 10 microseconds, as described in the conference paper Synchronous data collection from diverse hardware.

It runs on Linux, Windows, and Raspi, and needs neither extra hardware nor elevated privileges.

It includes bindings for Python, Ruby, and TCL. It also has a Rust API.

It can measure a system's realtime behavior, by providing a common time reference for events recorded by different sensors (audio, video, gamepad, GPS, SMS, MIDI, biometrics), and for triggering outputs (audio, video, LEDs, servos, motion bases).
It did this originally for a full-motion driving simulator with eye tracking and a quickly churning set of other sensors and outputs, for over a decade.

Clockkit was published in 2004 on http://zx81.isl.uiuc.edu/camilleg/clockkit (defunct).
It was revised and moved to GitHub in 2020.

The source code is licensed under the MIT License.

Installing

Ubuntu 22

sudo apt install g++ libpython3.10-dev make pkgconf ruby ruby3.0-dev swig tcl8.6-dev
cd ClockKit && make

Ubuntu 20

sudo apt install g++ libpython3.8-dev make pkg-config psmisc ruby ruby2.7-dev swig tcl8.6-dev
cd ClockKit && make

Ubuntu 18 and older

Install a g++ that supports C++20: sudo apt install g++-8.
Change the Makefile's -std=c++20 to -std=c++2a.
Proceed as with Raspberry Pi.

Raspberry Pi 3 and 4 (Debian/Raspbian)

sudo apt install g++ libpython3.8-dev make pkg-config psmisc ruby ruby-dev swig tcl tcl8.6-dev
cd ClockKit && make
(The package ruby-2.5dev vanished after Debian 10 "Buster" and Ubuntu 18.)

Windows 10

Install Windows Subsystem for Linux, using the Ubuntu 20 distro.
sudo apt install tcl
Proceed as with Ubuntu 20.
(We may restore native builds for older versions of Windows, but no older than Windows XP.)

Using Python, Ruby, or TCL

make bindings builds the modules used by python/ckphaselock.py, ruby/ckphaselock.rb, and tcl/ckphaselock.tcl.
To build for only one language, e.g. Python, make python/_clockkit.so. For details, in the Makefile look for SWIGEXES.

Running

To test on localhost:

cd ClockKit && make test

To sync host B to host A:

On host A, ckserver <IP address to bind to> <port>
On host B:

• cp clockkit.conf my-clockkit.conf
• Edit my-clockkit.conf. Set the server to host A, e.g., 192.168.1.1 or myhost.example.com. Set the port to 4567, or whatever port you told ckserver to use.
• ./ckphaselock my-clockkit.conf
  (make test-remote automates this, using an ssh key.)

Of course, these steps for host B can be repeated on other hosts C, D, E,... to sync them all. Here, "syncing" means providing synchronized timestamps, not adjusting the hosts' own clocks.

To plot performance:

sudo apt install gnuplot
cd simulation && make

Citing

Camille Goudeseune and Braden Kowitz. 2004. "Synchronous data collection from diverse hardware."
Driving Simulation Conference - Europe (Conférence Simulation de Conduite), pp. 245-252.

Contributing

• To maintain the formatting style, sudo apt install clang-format, and use clang-format through the provided git hook:
  git config core.hooksPath .git_managed_hooks
• New code should follow the C++ Core Guidelines.
• For profiling and code coverage, sudo apt install gcovr. See also man gcovr.
  To collect and print statistics, make clean && make profile, run some tests (but not test-bindings), gcovr.
  To reset statistics before another profile, make purge.
  To cease profiling, make purge && make.

Roadmap

When this software launched in 2004, lab software was pretty much restricted to desktop OSes. But by now, labs and makerspaces use many more software development environments, especially for hardware I/O: Arduino, musl, Raspi, and smartphones to name a few. The choice of mature scripting languages has grown similarly.

Also, private wired 10baseT subnets have been pretty much replaced by WLAN, with much more bandwidth but less predictable performance.

Finally, since 2004, C++ standards have improved and software engineering in general has matured.

Therefore, these steps are proposed.

• Keep modernizing the code.
• Clean up the interface to other languages.
• Implement integration testing.
• Make reproducible performance tests for some use cases.
• Extend multiplatform support beyond POSIX, for other microarchitectures.
• For some use cases, reduce energy consumption, file size, bandwidth.
• To better exploit the strengths and manage the weaknesses of WLAN, replace the generic network stack's lower OSI layers with specific ones for Wi-Fi, Bluetooth LE, Zigbee, 6LoWPAN, etc.
• Throughout all these, insert optimization passes.
• Explore more distant use cases that need clock sync, such as high performance computing, logfile evaluation, and security breach detection.

clockkit's Issues

Fix broken Python (example)

Currently the example in ckphaselock.py is not working with >=Python-3.

Make command on Linux(Redhat) yieldspthreads_undefined error. Also gcc compatiblity note

Hi.
This issue is half FYI and half confirmation of the solution.
Just wanted to point out that I when I ran the mkae file in my rhel 8 enterprise linux with gcc 12, the following error came up :
undefined reference to pthread_create
I solved it by adjusting Makefile line to
CFLAGS := -O3 -fPIC -Wall -Werror -Weffc++ -std=c++20 -pedantic -pthread ( after seeing that these flags were used for .c to .o compilation only)
The other fix would be add #include <pthreads.h> to corresonding c files
Please do let me know if this is indeed the right thing to do and also confused as to why this issue wasnt raised before coz I am very sure it would be a common error unless ofcourse I am doing something wrong
Note: the current c++2a standard (which provides definitions for erase_if among others) is only supported by gcc 9 + versions)

testing two raspberry pi

Hi everyone,
I'm was looking for a library to sync two raspberry pi in a local network and I have found this one which looks very promising.
I tried to check the delay between two computers using this custom script for the client:

// Only for testing.
// A self-contained program with threads for a server and some clients.
// An example that does not read any config file.

#include <thread>
#include <vector>

#include "ClockClient.h"
#include "HighResolutionClock.h"
#include "PhaseLockedClock.h"

using namespace dex;
using namespace std::chrono_literals;

int main(int argc, char* argv[])
{
    if (argc != 3) {
        std::cerr << "usage: " << argv[0] << " port duration\n";
        return 1;
    }
    const auto port = atoi(argv[1]);
    std::chrono::microseconds runtime(int64_t(1000000 * atof(argv[2])));

    auto& clockHiRes = HighResolutionClock::instance();

    ClockClient client(ost::InetHostAddress("192.168.1.1"), port);
    client.setTimeout(1000);
    client.setAcknowledge(true);
    PhaseLockedClock plc(clockHiRes, client);
    plc.setPhasePanic(5000);
    plc.setUpdatePanic(5000000);

    std::atomic_bool end_clocks(false);

    std::thread plcThread(&PhaseLockedClock::run, &plc, std::ref(end_clocks));

    while (runtime.count() > 0) {
        std::cout << "offset: " << plc.getOffset() << "\n" << timestampToString(plc.getValue()) << std::endl;
        std::cout << std::endl;
        constexpr auto wait = 600ms;
        std::this_thread::sleep_for(wait);
        runtime -= wait;
    }

    plc.die();
    plcThread.join();
    return 0;
}

And running the ClockServerMain.cpp executable on the other PC.
However, this is the output I get:

<time 1631805921 304074> 192.168.1.2    3       907
<time 1631805921 337063> 192.168.1.2    -9221740230933440132    0
<time 1631805921 340981> 192.168.1.2    -9221740230933436065    0
<time 1631805921 516315> 192.168.1.2    -9221740230933263086    0
<time 1631805921 552744> 192.168.1.2    -9221740230933224251    0
<time 1631805921 560615> 192.168.1.2    -14     897
<time 1631805921 562499> 192.168.1.2    -9221740230933214455    0
<time 1631805921 666530> 192.168.1.2    -9221740230933110745    0
<time 1631805921 693150> 192.168.1.2    -9221740230933083913    0
<time 1631805921 695964> 192.168.1.2    -9221740230933081202    0
<time 1631805921 697906> 192.168.1.2    -9221740230933079108    0
<time 1631805921 749100> 192.168.1.2    -9221740230933028002    0
<time 1631805921 751206> 192.168.1.2    -9221740230933025967    0
<time 1631805921 753392> 192.168.1.2    -73     983

When I run the same script locally, the offsets and RTT times are reasonable:

<time 1631807223 681889> localhost	-36	148
<time 1631807223 681889> offsetMax	110	---
<time 1631807223 879107> localhost	-7	159
<time 1631807224  81756> localhost	-6	157
<time 1631807224 291728> localhost	-5	152
<time 1631807224 485152> localhost	-21	129
<time 1631807224 681626> localhost	-17	167

Could you please help me to find out what am I doing wrong?

Thank you for your time,

Simone

remove exceptions

At low level, they're not rare errors, but expected events: network dropouts, drifting crystals.
At high level, they may not spill into an FFI.

Installation and build issues on latest raspbian bullseye and raspberry 3

Hello tried to install this app and at first when I did try to install prerequisite packages this package no longer exist: ruby2.5-dev so I just used ruby-dev instead.

Then I tried to make in Clockit folder and got this error:

clockkit.cpp:(.text+0x1e8): undefined reference to `pthread_create'
/usr/bin/ld: clockkit.o: in function `ckInitialize(char const*)':
clockkit.cpp:(.text+0x1e8): undefined reference to `pthread_create'
collect2: error: ld returned 1 exit status
make: *** [Makefile:100: ckserver] Error 1
make: *** Waiting for unfinished jobs....
collect2: error: ld returned 1 exit status
make: *** [Makefile:103: ckphaselock] Error 1
collect2: error: ld returned 1 exit status
make: *** [Makefile:97: cktest-unit] Error 1
/usr/bin/ld: /tmp/ccuLpkEv.o: in function `void std::vector<std::thread, std::allocator<std::thread> >::_M_realloc_insert<void (dex::PhaseLockedClock::*)(std::atomic<bool>&), dex::PhaseLockedClock*&, std::reference_wrapper<std::atomic<bool> > >(__gnu_cxx::__normal_iterator<std::thread*, std::vector<std::thread, std::allocator<std::thread> > >, void (dex::PhaseLockedClock::*&&)(std::atomic<bool>&), dex::PhaseLockedClock*&, std::reference_wrapper<std::atomic<bool> >&&)':
test-standalone.cpp:(.text._ZNSt6vectorISt6threadSaIS0_EE17_M_realloc_insertIJMN3dex16PhaseLockedClockEFvRSt6atomicIbEERPS5_St17reference_wrapperIS7_EEEEvN9__gnu_cxx17__normal_iteratorIPS0_S2_EEDpOT_[_ZNSt6vectorISt6threadSaIS0_EE17_M_realloc_insertIJMN3dex16PhaseLockedClockEFvRSt6atomicIbEERPS5_St17reference_wrapperIS7_EEEEvN9__gnu_cxx17__normal_iteratorIPS0_S2_EEDpOT_]+0x1e4): undefined reference to `pthread_create'
/usr/bin/ld: /tmp/ccuLpkEv.o: in function `main':
test-standalone.cpp:(.text.startup+0x298): undefined reference to `pthread_create'
/usr/bin/ld: clockkit.o: in function `ckInitialize(char const*)':
clockkit.cpp:(.text+0x1e8): undefined reference to `pthread_create'
collect2: error: ld returned 1 exit status
make: *** [Makefile:94: cktest-standalone] Error 1

Any help ?

[Feature Suggestion] FFI for Rust

Hello,

currently I'm using clockkit in a project which I wrote in Rust. SWIG is not supporting Rust at the moment. To use it, I made a temporary FFI which works fine as long as there are no exceptions thrown (Rust doesn't have those). For now that is fine in my case. But in order to make it useful for others too, I wouldn'd mind doing things properly. Before investing the time though, I wanted to ask whether you would be willing to include the wrapper file in clockkit (and probably answer some questions so the interface ends up sensible).

My two main concerns here are:

1. My proposed FFI is a bit different from the ones you are currently providing for Ruby and Tcl, as follows:
   1.1 In order to create a PhaseLockedClock with a configuration file for which one does not need root access, I exposed the corresponding method in ConfigReader.
   1.2 To handle the lifetime properly I made a clock an object like it is in C++ too.
   1.3 Methods currently exposed are:
   • is_synchronized()
   • get_offset()
   • get_value() (raw timesstamp as integer)
   • and some methods to give a DateTime value as a timestamp
2. From what I've read, it is good practice to make so called sys-crates buildable without network access. The easiest way to do that would be to simply put the third party code into the crate.
   • Would that be acceptable to you?
   • I would ask you to tag at least one version with a number so it can be properly referenced

To indicate various state changes, the Rust methods would return Results with enum instances corresponding to the exceptions, e.g. clockkit::error::Error::OutOfSync. The candidates I have identified so far are:
- OutOfSync
- Timeout
- PacketError
As this will be a high level interface I am unsure about the value of "lower level" exceptions like the performance counter frequency error or a clock rollover.

In addition to that, I would probably provide a patch to move the printing of the configuration in ConfigReader to a method. That method could be called in clockkit.cpp so the output would not change, yet the exposed FFI would not print anything when creating a clock as part of another project.

I would appreciate to hear your feedback on this; and hope that, with another easy interface to this cool project, more people will be able to make use of it.

Running clockkit on Arduino

I see Arduino on the roadmap. Is it possible to run clockkit on Arduino compatible board with a pretty beefy microcontroller, i.e. Atmel SAMD51P20? My goal is to synchronize clocks (find dt) between a PC and SAMD51P20. I need only about 1ms accuracy.

Segfault in ClockServer on ackData_cleanup

I ran into a SIGSEGV while using the PhaseLockedClock with a remote server.

After some investigation I found this to happen in ClockServer::updateEntry. First the erase on the std::map deletes an entry, after that the loop moves the iterator further and invalidates it by skipping an element.

EDIT:
My setup is Linux, amd64, gcc-9.3 with exactly two clients, where one is running on the same machine as the ckserver.

I'll offer a PR with a simple fix. The PR provided fixes this in my case.

How to interpret output of ckserver?

Hello,
I am pretty new to time synchronization.
I have installed and successfully ran the clockkit sync.

However, I don't understand how to interprete the results.
As I understand, the software should give me two timestamps, one on the host and one on the client that correspond to each other.
How should I then understand the command line ouputs?

I would really appreciate your help!

It gives me the follwing results on the host.
<time 1671706538 646372> offsetMax 958 ---
<time 1671706538 841717> 192.168.1.128 200 992
<time 1671706539 51321> 192.168.1.128 41 734
<time 1671706539 245279> 192.168.1.128 37 834
<time 1671706539 245279> offsetMax 454 ---
<time 1671706539 455172> 192.168.1.128 27 841
<time 1671706539 649876> 192.168.1.128 226 1256
<time 1671706539 855896> 192.168.1.128 68 823
<time 1671706539 855896> offsetMax 479 ---
<time 1671706540 52855> 192.168.1.128 119 1002
<time 1671706540 262814> 192.168.1.128 62 890
<time 1671706540 459118> 192.168.1.128 44 728
<time 1671706540 459118> offsetMax 408 ---
<time 1671706540 654233> 192.168.1.128 2 661
<time 1671706540 861885> 192.168.1.128 194 1115
<time 1671706541 63815> 192.168.1.128 38 911
<time 1671706541 63815> offsetMax 493 ---
<time 1671706541 258405> 192.168.1.128 -53 1039
<time 1671706541 457905> 192.168.1.128 23 916
<time 1671706541 665429> 192.168.1.128 205 1245
<time 1671706541 665429> offsetMax 827 ---
<time 1671706541 868078> 192.168.1.128 61 969
<time 1671706542 67013> 192.168.1.128 11 888
<time 1671706542 271299> 192.168.1.128 11 954

FFIs shouldn't require apt install world-plus-dog

Someone with no need of, say, Tcl and Ruby, shouldn't have to reverse engineer how to not build them and to exclude their dependencies.

replace commoncpp with ucommon

The underlying library Common C++ aka commoncpp was a good choice 17 years ago.
But by now, ucommon is better maintained, more broadly used, and more portable to e.g. MacOS and musl.

ModuleNotFoundError: No module named '_clockkit'

Hi, I am trying to run the ckphaselock.py script and keep getting this error. Can you please tell me where to get the _clockkit module?