voc / aes67-recorder

A Linux/GStreamer-Based AES67 Multitrack Audio Backup Solution

A Linux/GStreamer-Based AES67 Multitrack Audio Recording Solution

Given you have an AES67 based Audio-over-IP Device or -Network and want to contiously record a given set of tracks uncompressed, then this might be for you.

On 34C3 we used tascam SSD-recorder to record all audio-tracks from all sub-groups to SSDs. this had some major drawbacks;

• the recorder did record all 128 tracks and did not name them; so finding the ones of interest was quite hard
• the recorder did not have NTP and their clocks were not set correctly
• Someone had to unload the SSDs when they were full, carry them to an unloading-station, unload them and carry them back
• The 120GB+ had do be copyied at-once every hours (whenever the SSDs were full) to the storage, spiking the network load
• The backup files were hours long of multi-GB .wav-Files, so seeking in them (via network) was quite a challenge

On 35C3 we plan to fix these issues by capturing the Audio via AES67 and gstreamer to chunked, nicely named .wav-files, constantly syncing them to a storage-server.

Configuration

For the time being, just take a look at config.yaml.

Requirements

# for the main recording application
apt-get install gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-tools libgstreamer1.0-0 gir1.2-gstreamer-1.0 gir1.2-gst-plugins-base-1.0
apt-get install python3 python3-yaml python3-paho-mqtt python3-gi

# for the web-gui application
sudo apt-get install python3-socketio-client python3-flask

Running

The recorder itself is a headless application. It publishes status information via tcp on localhost port 9999.

./main.py -i my-config.yaml

nc 127.0.0.1 9999

A second application connects to this status-stream and serves a Web-UI on port 9998.

./web-ui.py -i my-config.yaml

The reason fo this split is, that a Web-UI with Websockets for live level-graphs does not fit nicely with the main recorder, built around GObject/GStreamer.

Troubleshooting

Error-Message from gst_element_request_pad

gst_element_request_pad: assertion 'templ != NULL' failed
WARNING: erroneous pipeline: could not link audiotestsrc0 to mux_0, mux_0 can't handle caps audio/x-raw, format=(string)S24LE, rate=(int)48000, channels=(int)1

Known Bug in GStreamer, a Problem with the splitmuxsink not able to handle simple audio-only codecy, like wavenc a Patch has been merged to master which fixes this problem. Until it landed in your Distribution, you probably need to build your own Version of GStreamer.

aes67-recorder's Issues

PTP-Sync

During 35C3 I was not able to get the GStreamer PTP-Clock to sync with the Audinate AES67-PTP-Clock.
This should be checked against another PTP-Clock.

Still start when not all Multicast-flows are alive

And show which flows are bad/broken (eg. because of a mistyped IP-Address)

Any experience using this in a Dante environment?

Hey there,

as far as I understand it, with newer Dante versions it should be possible to "downgrade" a Dante environment to AES67. Does this work in conjunction with the aes67-recorder, to record sound from a Dante device (e.g. yamah rio, or kt dn9650)? Did anyone try that before?

Kanalnamen als Tool-Tip-Text auf den Levelmetern anzeigen

Recorded files are not of equal length

when recorded from a audinate/dante card in aes67 mode

Web UI doesn't open

I'm trying to run this on Ubuntu 20.04 but when I try to open the web UI in a browser or curl on port 8080 I get ERR_EMPTY_RESPONSE. I also tried port 9998 but that shows Invalid Connection header: ['keep-alive'].

I installed all the packages in the README with apt, then I ran pip3 install --user -r requirements.txt. This produced some errors so I changed these 2 lines to allow newer versions:

pycairo>=1.18.0
PyGObject>=3.30.4

Running the web server starts like this, no errors showing:

$ python3 webgui.py -i config.yaml
webgui.py:86: DeprecationWarning: Application.make_handler(...) is deprecated, use AppRunner API instead
  webserver = loop.create_server(app.make_handler(), '0.0.0.0', 8080)
INFO:webgui:Starting Webserver on port 8080
INFO:webgui:Starting Websocket-Server on port 9998
INFO:webgui:Starting TCP-Client on port 9999
INFO:tcp-server:connected
INFO:tcp-server:received system_config: {"type": "system_config", "source": {"format": "S24BE", "rate": 48000}, "sources": [{"type": "udp", "channels": 8, "address": "239.69.83.67", "iface": "ens224", "port": 6518}], "clocking": {"source": "system", "jitterbuffer": false}, "channelmap": {"0": "ch1", "1": "ch2", "2": "ch3", "3": "ch4", "4": "ch5", "5": "ch6", "6": "ch7", "7": "ch8"}, "capture": {"folder": "./audio-backup/", "segment-length": 5, "format": "S24LE"}, "status_server": {"port": 9999, "bind": "::", "level_interval_ms": 250, "system_health_report_interval_ms": 1000}, "watchdog": {"enabled": true, "check_interval_ms": 1000, "warn_after_missing_signal_for_ms": 2000, "restart_after_missing_signal_for_ms": 5000, "mqtt": {"enabled": true, "host": "mng.c3voc.de", "port": 1883, "username": "aes67-backup", "password": "***"}}}
INFO:tcp-server:received system_health_report: {"type": "system_health_report", "bytes_total": 33155760128, "bytes_free": 20597014528, "bytes_available": 18889199616, "bytes_available_percent": 0.6212197955493666, "bytes_used": 12558745600, "inodes_total": 2064384, "inodes_free": 1759255, "inodes_available": 1759255, "inodes_available_percent": 0.8521936810205853, "inodes_used": 305129, "interfaces": {"ens224": {"rx": {"bytes": 79359392632, "packets": 66884700, "bytes_per_second": 9658005.0, "packets_per_second": 8026.0}, "tx": {"bytes": 1559648814, "packets": 496715, "bytes_per_second": 2938.0, "packets_per_second": 17.0}}, "ens192": {"rx": {"bytes": 70505272, "packets": 574594, "bytes_per_second": 788.0, "packets_per_second": 7.0}, "tx": {"bytes": 26358139, "packets": 274903, "bytes_per_second": 255.0, "packets_per_second": 3.0}}, "lo": {"rx": {"bytes": 1368431193, "packets": 117534, "bytes_per_second": 3603.0, "packets_per_second": 17.0}, "tx": {"bytes": 1368431193, "packets": 117534, "bytes_per_second": 3603.0, "packets_per_second": 17.0}}}}

Support Recording-Format with Timestamps

WAV-Files do not contain any timestamp-information.
BWF/BWAF-Files do contain a single timestamp-information for their start but drift or loss during the file's duration can not be compensated.

Options would be BWF, AC3(lossy) or DTS or anything in a container (PCM in MKV).

Tool to create a config from Avahi/Bonjour

At lease Dante-Config is announced via Bonjour

avahi-browse-a.txt
avahi-browse-ar.txt

Recorded files are registered as 4h long

Windows hält die Files für 4 Stunden lang und die DAW auf dem Windows folglich auch
Wenn ich die Files in die DAW werfe hab ich 4 Stunden Files wo nur in den ersten 15Min was drin ist. Keine Lust das alles zurecht zu schneiden

The DAW used is calles Fairlight Dream, but Windows also shows the length wrong,

show filenames in webgui after initial start

When the WebGUI connects after the Recorder has startet, it will not show the Filenames until a new Set of Filenames are generated.

Unspecified UI-Crashes

At the end of day 1 the UI crashed several times within seconds without useful trace in the logs.

Decklink-Sync

When recording audio for embedding into Video, it would be good to be able to sync the recording's clock to a decklink reference in (BlackBurst or TriSync)

format numbers & sizes so that they don't jump

from 100G to 100.01G, better alwas show 2 places: 100.00G

show file-sizes

create a folder-hierarchy per date

audio-backup/s1/headsets/1/2018-12-27/01-12-56.wav

GstNetClock-Sync

Syncing against another GStreamer Pipeline like Voctomix would help having Recordings exactly matching the recorded Video.

Voctomix already supplies a GstNetClock network-clock.

show uptime

Rename project?

Isn't this project suitable for all AES67 recording requirements? For more general applications a segment length of 30 or 60 minutes might be more appropriate, but as all this is configurable I don't see any reason this would not work as a general purpose AES67 audio recorder.

With AES67 compatibility enabled via Dante Controller this should also work in all Dante environments as recorder.

Schema der MQTT-Warn-Component an VOC-Konvention anpassen

Konvention ist type/instance, nicht host:service.

08:13 <             viri > WARN encoder.muc.c3voc.de:aes67-recorder: No Data received within 2001 ms
08:13 <             viri > WARN encoder.muc.c3voc.de:aes67-recorder: No Data received within 3001 ms
08:13 <             viri > WARN encoder.muc.c3voc.de:aes67-recorder: No Data received within 4002 ms
08:13 <             viri > WARN encoder.muc.c3voc.de:aes67-recorder: No Data received within 5003 ms
08:13 <             viri > WARN encoder.muc.c3voc.de:aes67-recorder: No Data received within 2001 ms
08:13 <             viri > WARN encoder.muc.c3voc.de:aes67-recorder: No Data received within 3001 ms
08:13 <             viri > WARN encoder.muc.c3voc.de:aes67-recorder: No Data received within 4002 ms
08:13 <             viri > WARN encoder.muc.c3voc.de:aes67-recorder: No Data received within 5002 ms
08:13 <             viri > ERROR encoder.muc.c3voc.de:aes67-recorder: No Data received within 5002 ms, restarting recorder

Check for Output-File collisions

Check if a filename is occupied by more then one stream and fail to start with a useful warning.