Source code repository for the Fragment app. which can be found at : https://www.fsynth.com
- Fragment - The Collaborative Spectral Synthesizer
Fragment is a collaborative audiovisual live coding web. environment with pixels based (image-synth) live additive/spectral/granular synthesis and sequencing, the sound synthesis is powered by pixels data generated from live GLSL code
Many videos of most features are available on YouTube
Fragment has only one fragment shader which has the particularity to be shared between the users of an online session, it update and compile on-the-fly as you or other peoples type, some settings are also synchronized between users such as slices and some global settings.
Fragment has many features making it a bliss to produce any kind of sounds associated (or not) with visuals, it is aimed at artists seeking a creative environment with few limitations to experiment with, a programmable noise-of-all-kinds software.
With WebGL 2 capable web browser, audio can be produced independently from the visuals with the synthOutput vec4 uniform, with WebGL 1 you can use the default pixels output (gl_FragColor or fragColor) to produce sounds and visuals, use the monophonic mode if you want independent audio & visuals with WebGL 1.
For any questions, a message board is available here
You can support this project through Patreon
- Recent web browser such as Chrome, Opera, Safari or Firefox (MIDI is not supported by Firefox at the moment)
- Mid-range GPU, this app. was made and tested with a GeForce GTX 970
- Mid-range multi-core CPU (a dual core should be ok with the FAS, a beefy CPU is needed if you use more than one output channel)
- Not necessary but a MIDI device such as a MIDI keyboard is recommended
Note on performances : Fragment has excellent performances with a modern multi-core system and a browser such as Chrome, if you experience crackles or need advanced audio features, it is recommended that you use the sound server and the independent code editor, the things that may cause poor performances is due to the browser reflow (UI side)
Fragment is also able to do real-time distributed sound synthesis with the sound server, it support any number of machines over the wire and multicore support, this feature also need the fas_relay to work (see below)
- Additive, spectral, granular, subtractive, phase modulation synthesizer powered by WebAudio oscillators (additive only, work best in Chrome), a Wavetable (additive only, slow/worst) OR a C native audio server (fastest/best)
- Live coding/JIT compilation of shader code
- Real-time, collaborative app.
- Distributed sound synthesis, multi-machines/multi-core support (Fragment Audio Server with fas_relay)
- Stereophonic or monaural
- Polyphonic
- Multitimbral
- MIDI inputs support with Web MIDI compatible web browsers, MPE supported
- Adjustable audio output channel per slices
- Real-time frames by frames recording with export as an image or Fragment input (export back into itself, this can be used to build complex brushes for drawing canvas inputs)
- .wav export (without FAS : additive synthesis only)
- WebGL 2.0 and GLSL 3.0 support when compatibility is detected
- RGBA Live visuals with stereophonic sound generation (WebGL 2.0) or monophonic sound generation (WebGL 1)
- Synthesis data processed in 32-bit precision (WebGL 2.0 & EXT_color_buffer_float extension) or 8-bit precision
- Slices can be added/deleted anywhere on the canvas without limits, move left or right automatically and have independent pitch offset
- Feedback via framebuffer (for fx like reverb, delay, spectral distortion etc)
- OSC in/out support (a SuperCollider port of the additive synthesis engine use this)
- Shader inputs:
- Webcam (no audio support)
- Images
- Videos with rate & rewind & loop settings (can import the audio track as an image)
- Audio files (translated to images)
- Drawing canvas with drawing and compositing operations which use images Fragment input as brushes, Fragment is bundled with 20 high-quality brushes, a pack of 969 high-quality brushes is also available as a separate download
- Uniform controllers via OSC Open Stage Control is recommended
- Per-sessions discussion system
- Global and per sessions settings smart save/load; make use of localStorage
- No authentifications needed, anonymous (make use of localStorage and is sessions based)
Fragment capture pixels data (1px wide slices) from a WebGL drawing surface at the browser display refresh rate and translate the RGBA pixels value to notes, the notes are then interpreted and played by one or more synthesis method in real-time.
Common to all synthesis methods, the canvas represent frequencies (exponential mapping) on the vertical axis and time on the horizontal axis.
The Fragment audio server is necessary for very fast sound synthesis and other features such as different output channels per slices.
With the web browser, Fragment is limited to additive synthesis and two output channels.
Slices data can be sent via OSC bundles to extend the possibilities of this synthesizer.
Fragment default sound synthesis is additive, with additive synthesis, Fragment become a fully capable spectral synthesizer able to do re-synthesis based on imported videos or audio files, an image-synth where any number of partials can be played, there is no limits except the canvas height and the computing resources available.
With additive synthesis and stereo mode, the Red channel is the left oscillator amplitude and the Green channel is the right oscillator amplitude.
With FAS (Fragment Audio Server), the Blue channel value is used to add bandwidth noise to the oscillator which may enhance the sound by adding a noise component to the oscillator.
In monophonic mode, the alpha value is used for both L&R oscillator amplitude.
Granular Synthesis (FAS only)
FAS load all samples in a "grains" folder and try to guess their pitch to map it on the canvas so that it match the canvas freq. mapping.
Fragment let you manipulate granular synthesis parameters, there is two type of parameters, channels based and dynamic parameters from the fragment shader output.
Channels based parameters :
- Granular envelope
- sine
- hann
- hamming
- tukey
- gaussian
- confined gaussian
- trapezoidal
- blackman
- blackman harris
- parzen
- nutall
- flattop
- kaiser
- Minimum grain length (ms)
- Maximum grain length (ms)
Dynamics parameters
- Red = Left amplitude
- Green = Right amplitude
- Blue = Sample index bounded to [0, 1] (cyclic) and grains density when > 2
- Alpha = Grains start index bounded [0, 1] (cyclic), grains start index random [0, 1] factor when > 1, play the grain backward when negative
Sampler (FAS only)
Granular synthesis with grain start index of 0 and min/max duration of the sample duration can be used to trigger samples as-is like a regular sampler.
PM (FAS only)
Phase modulation (PM) is a mean to generate sounds by modulating the phase of an oscillator (carrier) from another oscillator (modulator), it is very similar to frequency modulation (FM).
PM synthesis in Fragment work by giving an oscillator index (based on image-height) to the note, this oscillator will be used as a modulator.
No filtering is applied.
Dynamics parameters
- Red = Amplitude value of the LEFT channel
- Green = Amplitude value of the RIGHT channel
- Blue = Fractionnal part : Modulator amplitude
- Alpha = Modulator frequency
Subtractive (FAS only)
Subtractive synthesis start from harmonically rich waveforms which are then filtered.
It is somewhat slow due to additive synthesized waveforms and there is only one high quality low-pass filter (Moog type) implemented.
There is three type of band-limited (no aliasing!) waveforms : sawtooth, square, triangle
This type of synthesis may improve gradually with more waveforms (and a faster way to generate them) and more filters.
Note : The waveforms are constitued of a maximum of 64 partials
Dynamics parameters
- Red = Amplitude value of the LEFT channel
- Green = Amplitude value of the RIGHT channel
- Blue = Moog filter cutoff multiplier; the cutoff is set to the fundamental frequency, 1.0 = cutoff at fundamental frequency
- Alpha = Moog filter resonance [0, 1] & waveform selection on integral part (0.x, 1.x, 2.x etc)
Fragment support MIDI inputs.
- MIDI keyboard support, list of MIDI parameters accessible in the fragment shader :
- note-on/note-off messages
- note frequency
- velocity / release velocity
- aftertouch
- pitch bend
- CC74
- MIDI channel
- elapsed time since the key was pressed
- Polyphony is automatically detected from the GPU capabilities (704 notes with a GeForce GTX 970 GPU, 16 notes is the minimum, maximum notes depend on the GPU capability/shader complexity)
- Multidimensional Polyphonic Expression (MPE) support
- Hot plugging of MIDI devices
If you need to control more parameters, see OSC below.
Fragment support OSC input and output, an OSC relay which translate WebSockets data to UDP packets should be used for this feature to work.
Fragment uniforms can be defined through OSC with two methods :
- Message with an address starting with i such as /iarr
- This will create/update a float array uniform, the message should contain an array with index to update at index 0 and the value at index 1
- If the array does not exist, it will create it and grow the array as needed
- Update whole float array with message starting with a address such as /aarr
- This will create/update a whole **float **array uniform, the message should just contain all the array values
You can send a message to the /clear address to clear all OSC defined uniforms
Open Stage Control can be used to control partials or more parameters through OSC via faders etc.
Many tools are available to enhance Fragment.
-
A graphical launcher for Fragment and the audio server program is available here.
-
Independent GLSL editor which can directly connect to the sharedb server
-
SuperCollider port of the additive synthesis engine (use OSC)
- The main limitation of Fragment is the events granularity caused by the monitor refresh rate (60 or 120 FPS), this can be solved by running the browser without VSYNC, example for Chrome with the command-line parameter --disable-gpu-vsync this may be not ideal for visuals however.
- Discrete mapping of frequencies can be seen as a limitation, especially for additive synthesis, this is mostly solved by increasing the resolution of the canvas.
- Additive synthesis : Phases are randomly assigned
- Many synthesis parameters cannot be mapped well due to RGBA, this is a limitation which apply mainly to PM/FM, this limit the number of algorithm for example.
- WebAudio oscillators and the WebAudio Wavetable mode can only have two output channels (L/R) due to performances issues (this may change in the future!)
- If you enable the monophonic setting, you have the RGB output for live coding visuals which can be fully synchronized with the synthesized sounds, sounds will be synthesized by using the alpha channel
- Pressing F11 in the GLSL code editor make the editor fullscreen (as an overlay)
- You can feed the display content of any apps on your desktop (such as GIMP or Krita) by streaming your desktop as a camera (v4l2loopback and ffmpeg is useful to pull of this on Linux)
- client - main application
- www - landing page
- fss - main server (discuss. system, slices)
- fsdb - sharedb server (collaborative features)
- fsws - web. server (only used for development or local installation)
- osc_relay - an OSC relay which use the osc.js library (must be launched to use OSC features)
- fas_relay - distributed multi-machines/multi-core realtime sound synthesis
- editor - external GLSL code editor
- supercollider - the SuperCollider port of the additive synthesis engine (fed through OSC)
- documentation - MkDocs documentation and previous MAML (Minimalist Anubis Markup Language) documentation with the latest HTML and PDF doc.
- main.js - Electron app. file
- common.js - Server config. file
All servers are clustered for scalability and smooth updates.
Fragment is built with a custom build system scanning for changes in real-time and which include files when it read /*#include file*/, it execute several programs on the output files such as code minifier, the build system was made with the functional Anubis programming language, a programming language based on cartesian closed category theory.
Since the Anubis language is relatively unknown, a simplified (without live check & build) Python port of the build system is available, check out pyNut
If you want to build it by yourself, install pyNut script somewhere in your PATH then call pynutbuild shell script in the client root directory.
_app_fs_ and _app_cm_ are the entry point files used by the build system to produce a single file and a production ready file in the dist directory.
The Anubis build system can be found here and this build system is called by the shell script named nutbuild (root folder)
Fragment make use of NodeJS, NPM, MongoDB and Redis database, once those are installed, it is easy to run it locally:
- clone this repository
- cd fss & npm install & node fss
- cd fsdb & npm install & node fsdb
- cd fsws & npm install & node fsws
- point your browser to http://127.0.0.1:3000
If you just want to try it out without the collaborative feature and GLSL code save, you don't need MongoDB and Redis, you just need "fsws" then point your browser to http://127.0.0.1:3000
If you want to use it with an OSC app like the SuperCollider fs.sc file or Open Stage Control, please look at the osc_relay directory.
To use the OSC relay :
- cd osc_relay & npm install & node osc_relay
To use the FAS relay :
- cd fas_relay & npm install & node fas_relay
- prod_files contain a list of files and directories that will be copied to the production system
- prod is a shell script which produce an archive from prod_files list, perform additional cleanup and unarchive over SSH
- setup is a script which is executed on the server after everything has been uploaded, this configure Fragment for the production system
Maybe a VST/LV2 plugin for accessibility and of course many new features are coming soon. ;)
A native app. will be done soon but with a totally different paradigm, it may be extremely simple technically and very flexible/accessible, it will also fix the main limitation of Fragment by allowing > 60 FPS capture (configurable so not limited to the display refresh rate...) which mean basically unlimited granularity as the hardware get faster.
Client :
- Vanilla JS yup!
- WUI vanilla collection of UI widgets for the web
- CodeMirror for the awesome editor and its addons/modes
- osc.js
- glsl-simulator the GLSL parser is based on glsl-simulator
- ShareDB for the collaborative features
- mpejs
- FileSaver.js
- Recorderjs
- Normalize
- Skeleton for the landing page
- Showdown in-app markdown
- Mikola Lysenko stft (enhanced version)
Papers :
- The Scientist and Engineer's Guide to Digital Signal Processing
- L'audionumérique 3°ed by Curtis Road
- Fabrice Neyret Desmos page
Servers :
Utilities :
- FontAwesome
- pegjs
- fa2png
- Brackets
- Atom
- desmos
- libwebsockets for fas
- portaudio for fas
- libflds for fas
- SimpleScreenRecorder for videos recording
- KDEnlive to edit the videos
- Geogebra for the logo
- Inkscape for the logo and some graphics
- GIMP some graphics
- MkDocs Documentation
- The Anubis programming language
- Minimalist Anubis Markup Language
- Nut
- HotShots for the UI quick reference
Data :
The repository for the early proof of concept can be found here.
Simplified BSD license
The main inspiration for all of this is Alexander Zolotov Virtual ANS software, thank to him.
Heavily inspired by Shadertoy as well.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent