mimic-sussex / sema

Sema – A Playground for Live Coding Music and Machine Learning

Sema – A Playground for Live Coding Music and Machine Learning

Sema is a playground where you can rapidly prototype live coding mini-languages for signal synthesis, machine learning and machine listening.

Sema aims to provide an online integrated environment for designing both abstract high-level languages and more powerful low-level languages.

Sema implements a set of core design principles:

• Integrated signal engine – In terms language and signal engine integration, there is no conceptual split. Everything is a signal. However, for the sake of modularity, reusability, and a sound architecture, sema's signal engine is implemented by the sema-engine library.

• Single sample signal processing – Per-sample sound processing for supporting techniques that use feedback loops, such as physical modelling, reverberation and IIR filtering.

• Sample rate transduction – It is simpler to do signal processing with one principal sample rate, the audio rate. Different sample rate requirements of dependent objects can be resolved by upsampling and downsampling, using a transducer. The transducer concept enables us to accommodate a variety of processes with varying sample rates (video, spectral rate, sensors, ML model inference) within a single engine.

• Minimal abstractions – There are no high-level abstractions such as buses, synths, nodes, servers, or any language scaffolding in our signal engine. Such abstractions sit within the end-user language design space.

Dependencies

Sema requires the following dependencies to be installed:

• Chrome browser or any Chromium-based browser (e.g. Brave, Microsoft Edge, Opera)
• Node.js active LTS version (currently v14.4.0). To switch between node versions, you can use nvm.
• NPM cli OR Yarn

In order to run, Sema must connect with a project url and api key to a Supabase backend.

• You can either install supabase locally by following these instructions, or set up a free hosted project.
• Next, follow the steps on how to connect to a supabase backend.

How to build and run the Sema playground on your machine

If you decide to use npm to build sema, you can follow this list of commands:

$ cd sema
$ npm install
$ npm run build
$ npm run dev

If you decide to go with the Yarn package manager instead, you can use the following list of commands:

To use Yarn:

$ cd sema
$ yarn
$ yarn build
$ yarn dev

Once you have sema running as a node application, you can load it on your browser on the following ports

• npm run dev, go to http://localhost:5000 on your browser

Hardware acceleration:

Hardware acceleration will have a drastic effect in Tensorflow.js model training speed.

To enable it in Chrome:

• Navigate to chrome://settings
• Click the Advanced ▼ button at the bottom of the page
• In the System section, ensure the Use hardware acceleration when available checkbox is checked (relaunch Chrome for changes to take effect)

To enable in Firefox:

• Go to about:preferences
• Scroll till you reach the Performance section, or simply search for "performance"
• Enable recommended performance settings, this will enable hardware acceleration if and when it is available

Linux Users

Sema uses Web Audio API Audio Worklets. Their performance seems very sensitive to CPU power scaling. If you are experiencing sound quality issues, try setting the CPU governor to performance mode. e.g on Ubuntu,

$ cpupower frequency-set --governor performance

Documentation

Sema's internal documentation aims at supporting the users learning experience. It is integrated within the application and comprises the following sections:

• Getting started

• Playground

• Live Coding

• Machine Learning

• Language Creation

Sema's Wiki documentation aims at supporting contributions. It focuses on how Sema is designed and built:

• What is the architecture of Sema?

• How does Sema implement and use web services

• How to setup Sema on my own web server?

• How do I add a new ML library to Sema?

• How do I create and add a new widget to Sema?

• How do I add my own documentation to Sema?

• How do the Svelte stores in Sema work?

Contributing

Sema is an open-source project and hopefully the underlying vision, aims and structure will motivate you to contribute to it. Check the following:

• How do I contribute to Sema?

• CONTRIBUTING.md

• Getting set up for development

• Debugging Sema

• Design Guide

Publications

Bernardo, F., Kiefer, C., Magnusson, T. (2021). Assessing the Support for Creativity of a Playground for Live Coding Machine Learning, In: Baalsrud Hauge J., C. S. Cardoso J., Roque L., Gonzalez-Calero P.A. (eds) Entertainment Computing – ICEC 2021. ICEC 2021. Lecture Notes in Computer Science, vol 13056. Springer, Cham. https://doi.org/10.1007/978-3-030-89394-1_38

Bernardo, F., Kiefer, C., Magnusson, T. (2020). A Signal Engine for a Live Coding Language Ecosystem, J. Audio Eng. Soc., vol. 68, no. 10, pp. 756-766. doi: https://doi.org/10.17743/jaes.2020.0016

Bernardo, F., Kiefer, C., Magnusson, T. (2020). Designing for a Pluralist and User-Friendly Live Code Language Ecosystem with Sema. 5th International Conference on Live Coding, University of Limerick, Limerick, Ireland

Bernardo, F., Kiefer, C., Magnusson, T. (2019). An AudioWorklet-based Signal Engine for a Live Coding Language Ecosystem. In Proceedings of Web Audio Conference 2019, Norwegian University of Science and Technology (NTNU), Trondheim, Norway (Best Paper Award at Web Audio Conference 2019)

sema's Issues

Import MMLL library for onset detector

serial port access

ref: https://github.com/garrows/browser-serialport

check for undefineds to stop audio engine blowout

MIDI implementation

Git mirror eppEditor repo to sema repo

Make option to download saved data (now it is only in the browser cache)

grammar editor: decide how to deal with 'require moo' error

this is an item from raw .ne scripts that creates an error in sema. Should we search for this line and automatically ignore it? otherwise might be confusing to users to deal with this error

Add bracket match

file saving/loading from model : allow to query the available files/datasets

Create a pattern oscillator

Integrate NexusUI

Language Interpreter: e++ AST converter to audio engine tasks

automatic syntax highlighting

based on the grammar file

look for generative models (piano genie, performanc-rnn, char-rnn) add to

save data using filesaver.js

https://github.com/eligrey/FileSaver.js/

Undo function

The live coding editor could have an undo function that would revert back to previously running code replacing the signal graph and the code in the code window.

Use cmd+z for Undo and cmd+Shift+z to revert back to what was undone. (Like in a regular text editor)

Change colour of cursor in live code editor (problem in Linux Ubuntu 18.4 Chromium 75.037****)

After executing a long line in livecode window, it doesn't wrap around anymore.

Utilities for visualizing and editing/trimming training data

add noise generators

An audio processing language is not complete without some noise generators.

Add basic functionality for recording and managing training sequences

This is cumbersome to write in JS, would be nice to have these available as functions / classes.

To help the whole workflow for:

• recording a sequence from incoming data (growing buffer for a full sequence; circular buffer to create frames for use in realtime prediction)
• loading / storing these sequences to disk - also not just as single sequences, but as a set.
• formatting these sequences/sets as input for TensorFlow based models.

Implement pulse oscillator with variable pulse width and/or a single pulse generator

Referring to issue #14, to trigger envelopes, we need an audio unit to have an oscillator with variable pulse width.
Then you can have a pattern going on where the duration of the note and the time between notes is different from each other.

Alternatively, the option to have a single pulse with a variable width, that is triggered within a sequence would be very useful.

Audio Engine: Audio Worklet Signal Processing Function injector

add overlay with Start button to start Audio Context

make some higher level synths in maximilian C++ lib

fm operator, fm operator network, subtractive synth variations, additive, atomic synthesis

grammar editor: block commenting in the language

Integrate RapidLib

dissapearing cursor in editor sometimes

grammar does not accept negative numbers

{2.0,-1.0}mul

gives error:

index.js:235 DEBUG:Main:evalLiveEditorExpression: {2.0,-1.0}mul
nearley.worker.js:28 ErrorError: invalid syntax at line 1 col 6:

  {2.0,-
       ^
Unexpected hyphen token: "-"

workaround:

{2.0,{0.0,1.0}sub}mul

grammar editor: code highlighting for .ne format

Email github for repo name transfer

Document methods of including JS files into tfjs worker

Custom post window to post messages to from the live code (other than the browser dev console)

Live Code Editor: Implement commands (e.g. CMD-. CMD-enter)

error when parsing grammar with sum?

executing
{2,3,4}sum
gives an error in nearly.worker.js:28
ErrorTypeError: Cannot read property 'setup' of undefined

Change audio engine init to async/await

write README file

bracket matching while typing

There is a plugin for CodeMirror for this: https://codemirror.net/addon/edit/matchbrackets.js

Add/define 'comment' token to base lexer/tokenizer

highlight a block of code that has just been executed

fade out highlight - like SC editor

Live Code Editor: Responsive layout

Setup grammar playground

Sema-electron

Create an electron container for Sema

Chris to add prototype language to langSketch.js (hardcoded)

all editors: text search function

sample load syntax - \x gives error

{4}\xylophone

results in error:
Error in worklet evaluation: SyntaxError – Invalid hexadecimal escape sequence

Add envelope to the language

add option to execute a block of code in the live code window

method for uploading datasets

Well commented and working examples of machine learning models