go-daq / tdaq

tdaq: a toolkit for building DAQ systems

tdaq

tdaq is a toolkit to create distributed DAQ systems, over TCP/IP.

Installation

$> go get github.com/go-daq/tdaq/...

Example

In a terminal, launch the run-control:

$> tdaq-runctl -web=:8080 -i -lvl dbg
tdaq-runctl          INFO listening on ":44000"...

::::::::::::::::::::::::::
:::  RunControl shell  :::
::::::::::::::::::::::::::

- /config -> configure tdaq processes
- /init   -> initialize tdaq processes
- /run    -> start a new run
- /stop   -> stop current run
- /reset  -> reset tdaq processes
- /status -> display status of all tdaq processes
- /quit   -> terminate tdaq processes (and quit)

tdaq-runctl          INFO waiting for commands...
tdaq-runctl>>
tdaq-runctl          INFO starting web run-ctl server on ":8080"...
tdaq-runctl          INFO received /join from conn 127.0.0.1:44066
tdaq-runctl          INFO   proc: "tdaq-datasrc"
tdaq-runctl          INFO    - outputs:
tdaq-runctl          INFO      - name: "/adc"
tdaq-runctl          INFO        addr: "[::]:34373"
tdaq-runctl          INFO received /join from conn 127.0.0.1:44112
tdaq-runctl          INFO   proc: "tdaq-datasink"
tdaq-runctl          INFO    - inputs:
tdaq-runctl          INFO      - name: "/adc"
tdaq-runctl>> /config
tdaq-runctl          INFO /config processes...
tdaq-runctl          DBG  sending /config to "tdaq-datasrc"...
tdaq-runctl          DBG  sending /config to "tdaq-datasink"...
tdaq-runctl          DBG  sending /config to "tdaq-datasrc"... [ok]
tdaq-runctl          DBG  sending /config to "tdaq-datasink"... [ok]
tdaq-runctl>> /init
tdaq-runctl          INFO /init processes...
tdaq-runctl          DBG  sending cmd CmdInit to "tdaq-datasrc"...
tdaq-runctl          DBG  sending cmd CmdInit... [ok]
tdaq-runctl          DBG  sending cmd CmdInit to "tdaq-datasrc"... [ok]
tdaq-runctl          DBG  sending cmd CmdInit to "tdaq-datasink"...
tdaq-runctl          DBG  sending cmd CmdInit... [ok]
tdaq-runctl          DBG  sending cmd CmdInit to "tdaq-datasink"... [ok]
tdaq-runctl>> /run
tdaq-runctl          INFO /start processes...
tdaq-runctl          DBG  sending cmd CmdStart to "tdaq-datasrc"...
tdaq-runctl          DBG  sending cmd CmdStart... [ok]
tdaq-runctl          DBG  sending cmd CmdStart to "tdaq-datasrc"... [ok]
tdaq-runctl          DBG  sending cmd CmdStart to "tdaq-datasink"...
tdaq-runctl          DBG  sending cmd CmdStart... [ok]
tdaq-runctl          DBG  sending cmd CmdStart to "tdaq-datasink"... [ok]
tdaq-runctl>> /stop
tdaq-runctl          INFO /stop processes...
tdaq-runctl          DBG  sending cmd CmdStop to "tdaq-datasrc"...
tdaq-runctl          DBG  sending cmd CmdStop... [ok]
tdaq-runctl          DBG  sending cmd CmdStop to "tdaq-datasrc"... [ok]
tdaq-runctl          DBG  sending cmd CmdStop to "tdaq-datasink"...
tdaq-runctl          DBG  sending cmd CmdStop... [ok]
tdaq-runctl          DBG  sending cmd CmdStop to "tdaq-datasink"... [ok]
tdaq-runctl>> /run
tdaq-runctl          INFO /start processes...
tdaq-runctl          DBG  sending cmd CmdStart to "tdaq-datasrc"...
tdaq-runctl          DBG  sending cmd CmdStart... [ok]
tdaq-runctl          DBG  sending cmd CmdStart to "tdaq-datasrc"... [ok]
tdaq-runctl          DBG  sending cmd CmdStart to "tdaq-datasink"...
tdaq-runctl          DBG  sending cmd CmdStart... [ok]
tdaq-runctl          DBG  sending cmd CmdStart to "tdaq-datasink"... [ok]
tdaq-runctl>> /stop
tdaq-runctl          INFO /stop processes...
tdaq-runctl          DBG  sending cmd CmdStop to "tdaq-datasink"...
tdaq-runctl          DBG  sending cmd CmdStop... [ok]
tdaq-runctl          DBG  sending cmd CmdStop to "tdaq-datasink"... [ok]
tdaq-runctl          DBG  sending cmd CmdStop to "tdaq-datasrc"...
tdaq-runctl          DBG  sending cmd CmdStop... [ok]
tdaq-runctl          DBG  sending cmd CmdStop to "tdaq-datasrc"... [ok]
tdaq-runctl>> /quit
tdaq-runctl          INFO /quit processes...
tdaq-runctl          DBG  sending cmd CmdQuit to "tdaq-datasrc"...
tdaq-runctl          DBG  sending cmd CmdQuit... [ok]
tdaq-runctl          DBG  sending cmd CmdQuit to "tdaq-datasrc"... [ok]
tdaq-runctl          DBG  sending cmd CmdQuit to "tdaq-datasink"...
tdaq-runctl          DBG  sending cmd CmdQuit... [ok]
tdaq-runctl          DBG  sending cmd CmdQuit to "tdaq-datasink"... [ok]
tdaq-runctl          INFO shutting down...
tdaq-runctl          INFO closing...

In a second terminal, launch the tdaq-datasrc data producer application:

$> tdaq-datasrc -lvl dbg
tdaq-datasrc         DBG  received /config command...
tdaq-datasrc         DBG  received /init command...
tdaq-datasrc         DBG  received /start command...
tdaq-datasrc         DBG  received /stop command... -> n=57
tdaq-datasrc         DBG  received /start command...
tdaq-datasrc         DBG  received /stop command... -> n=457
tdaq-datasrc         DBG  received "/quit" command...

In a third terminal, launch the tdaq-datasink data consumer application:

$> tdaq-datasink -lvl dbg
tdaq-datasink        DBG  received /config command...
tdaq-datasink        DBG  received /init command...
tdaq-datasink        DBG  received /start command...
tdaq-datasink        DBG  received /stop command... -> n=57
tdaq-datasink        DBG  received /start command...
tdaq-datasink        DBG  received /stop command... -> n=457
tdaq-datasink        DBG  received "/quit" command...

One has also access to a web-based control UI for the run-ctl:

tdaq's Issues

tdaq: ensure I/O end-points are type-compatible

we need to make sure that e.g.:

• proc1 with output end-point "/adc"
• proc2 with input end-point "/adc"

have compatible types (e.g.: []int32)

tdaq: add log/monitor servers

• log server
• monitor server

tdaq: add GUI run-ctl

tdaq: add high-level API to create+schedule run-ctl+servers

tdaq: add support for processes w/ multiple inputs

tdaq: add support for net.Listen("unix")

tdaq: add a splitter/fan-out server

tdaq: add benchmarks

add throughput benchmarks to assess validity/feasability of migrating to mangos.

tdaq: add general handling of "events"

• run-nbr
• evt-nbr
• trigger nbr
• timestamps (start/stop)

run-ctl: create DAG of data-dependencies

• create DAG of data-dependencies
• validate DAG (cycles, missing providers)
• broadcast commands according to DAG

tdaq: consider using mangos as underlying protocol

depending on performances, it might be better to use mangos as the underlying transport layer and topologies handling (REQ/REP, PUSH/PULL, ...) instead of re-implementing everything.

bonus: get ipc & inproc for free.

run-ctl: consider broadcasting commands according to data DAG

tdaq: maintain consistency of DAG when a process leaves the pool

tdaq: use a poison-pill based strategy to stop runs

tdaq: add a filter/scaler server