torsion-audio / nn-inference-template

Neural network inference template for real-time cricital audio environments - presented at ADC23

A Template Audio Plugin for Real-time Neural Network Inference

This repository was started as a companion repository to the talk Real-time inference of neural networks: a practical approach for DSP engineers at the Audio Developer Conference 2023. The video can be found here.

Since the conference, we have continued to refine and extend the codebase. For more flexible and easier use of the inference architecture, we have consolidated this work into a library called anira, which is now used in this repository. For those interested in the state of the repository as presented at ADC23, it can be found under the tag ADC23-talk.

Authors: Valentin Ackva & Fares Schulz

Overview

This repository provides a comprehensive JUCE plugin template that demonstrates the use of anira to implement neural network inference in real-time audio applications. In this template, we use all three inference engines currently supported by the library:

• TensorFlow Lite
• LibTorch
• ONNXRuntime.

anira, an architecture for neural network inference in real-time audio applications, covers all critical aspects of ensuring real-time safety and seamless signal flow for real-time inference. Detailed information can be found on the library's github repo.

Build instruction

On Apple Silicon Macs you need to install the OpenMP library via Homebrew (brew install libomp)

Build with CMake

# clone the repository
git clone https://github.com/Torsion-Audio/nn-inference-template/
cd nn-inference-template/

# initialize and set up submodules
git submodule update --init --recursive

# use cmake to build debug
cmake . -B cmake-build-debug -DCMAKE_BUILD_TYPE=Debug
cmake --build cmake-build-debug --config Debug

# use cmake to build release
cmake . -B cmake-build-release -DCMAKE_BUILD_TYPE=Release
cmake --build cmake-build-release --config Release

To run the plugin or standalone application with the default model, you need to copy the model files to the correct location. The application will attempt to load the neural models from the user's application data directory at runtime. Therefore, you must copy the folder ./modules/GuitarLSTM/* to the following locations, depending on your operating system:

Linux: ~/.config/nn-inference-template/GuitarLSTM/*

macOS: ~/Library/Application Support/nn-inference-template/GuitarLSTM/*

Windows: %APPDATA%\nn-inference-template\GuitarLSTM\*

Install from release

To install the plugin, please follow the instructions provided for your operating system:

Linux	macOS	Windows

Unit Test / Benchmark

The previous unit test for benchmarking the plugin performance across different audio configurations and inference backends is replaced by the benchmarking options within anira. These new benchmarks have been improved in many ways and provide a range of simple to complex benchmarks that can be used to compare the inference time for different models, inference engines, and audio configurations.

Licenses

The primary license for the code of this project is the MIT license, but be aware of the licenses of the submodules:

• The anira library is licensed under the Apache 2.0
• The GuitarLSTM fork is licensed under the GPLv3
• The JUCE library is licensed under the JUCE License
• The ONNXRuntime library is licensed under the MIT
• The Libtorch library is licensed under the Modified BSD
• The TensorflowLite library is licensed under the Apache 2.0
• All other code within this project is licensed under the MIT License.

nn-inference-template's Issues

What is this project bringing over anira ?

Hi all,

Just watched the ADC talk and came skimming through this repo to understand it a bit better. I see that the guts of the template is the anira project from TU-Studio and I'm not 100% sure if this repo just brings a pretty UI or if there's something more to it ? Just asking because anira seems to already provide a plugin example, and receives update sfaster than this repo so I'm wondering which one I should fork :)

Thanks in advance!

Linker error on windows

Following the instructions, I get a linker error, presumably from libTorch trying to find MKL. I have MKL installed. I tried to fix it but no joy yet

LINK : fatal error LNK1181: cannot open input file 'mkl_intel_ilp64.lib'