vcync / modv

modular audio visualisation powered by JavaScript

modV

modV is a modular audio visualisation environment built upon web technologies.

Download

Find the latest modV binaries available for download in the releases section.

https://github.com/vcync/modV/releases/latest

Project development setup

yarn

Compiles and hot-reloads for development

yarn electron:serve

Builds for release

yarn electron:build

Caveats

Removing a package

If you want to remove a package with yarn remove, make sure to run yarn afterwards as we are using patch-package without postinstall-postinstall because of #554.

Platform specifics for building and development

Windows

Shell

CMD or PowerShell is required as native binaries need to be compiled or fetched for Windows. WSL (Windows Subsystem for Linux) or WSL2 are not supported as they will try to compile those binaries for Ubuntu.

Dependancies

Visual Studio Tools and Python are required by node-gyp, which build native node modules. Since Node.js 16, the official Windows installer can install them.

If you're using nvm-windows or another headless Node.js install, you can easily install the required packages with Chocolatey:

choco install python visualstudio2022-workload-vctools

'vue-cli-service' is not recognized

Please see issue 122 (vcync/modv-3/issues/122#issuecomment-640100114) on how to resolve this.

Ubuntu/Debian

libndi is required for NDI sources and must be installed for modV to build. You can find that available to download here: Palakis/obs-ndi/releases

Last successful build was with libndi4_4.5.1-1_amd64.deb.

Other Linux flavours

Untested. NDI is provided by grandiose, our fork is here: vcync/grandiose This fork of grandiose has other libndi supported platforms, however even on Ubuntu we needed the above libndi package to be installed.

Let us know how you get on (good or bad) and we'll update the repo and docs accordingly.

Contributing

Contributions, issues and feature requests are welcome! Feel free to check issues page.

Show your support

Give a ⭐️ if this project helped you!

Acknowledgements

Thank you to:

• Tim Pietrusky for his continued support, help, mentorship and kindness
• Live:JS for inspiration, motivation and advice
• Hugh Rawlinson, Nevo Segal and Jakub Fiala for the incredible audio analysis engine, meyda
• Dario Villanueva for his advice and introduction to live visuals which inspired this whole project

License

Copyright © 2022 vcync. This project is MIT licensed.

modv's Issues

Use OffscreenCanvas for main draw loop composition

Render speeds are good in modV at the moment, but could be better still.

Using OffscreenCanvas to handle the draw loop should allow all drawing features to run freely and not block each other (in theory?).

Though OffscreenCanvas isn't shipped in Chrome yet, it is behind the Experimental Web Platform Features flag (chrome://flags/#enable-experimental-web-platform-features) which can be enabled by the user in the browser or added to the NWJS build.

This is related to #35.

Use jsdoc

As suggested by pashutk, modV should use jsdoc for documentation.

This is an excellent idea. This issue will be kept open until the whole codebase has been documented.

Video is not working

It would be awesome if the video-module would work.

ModuleShader should use different shader file input

ModuleShader should use separate .fs and .vs files for Shader code instead of HTML.

Module removal from layer

If Module is picked up from a layer and moved to another layer without dropping, then dropped into the gallery to remove the module it causes a fatal error rendering the gallery useless.

Update to webGL 2

Updating to webGL2 will bring some performance improvements, GLSL 300 for compaibility with ShaderToy and the ability to tile NPOT textures

Username never gets stored

Set username doesn't get stored in Global Preferences

Save MIDI assignments into preset

Issue description

When doing some MIDI assignments and creating a new preset afterwards, the assignments are not saved into the preset.

This makes using MIDI impossible, because I don't want to loose the assignments every time I reload modV.

Steps to reproduce

• Add a module
• Add a MIDI assignment to the module, for example for the "enable" button and a slider inside the module
• Create a new preset
• Reload modV
• Load the preset

Versions

• modV 2.0-grabCanvas

Consider moving to Electron

Electron has updated its APIs since I started building standalone versions of modV and Window looks like it is supported exactly how it is in the browser.

Currently, the only reliant part of modV to NWJS when built standalone is its Context Menu, which is very similar to Electron's.

Electron is now also smaller in size, has huge community backing and its documentation is far better .

Potentially a 1.7 update consideration.

How to run the production build?

I created the build with npm build but I have no idea how to run it based on what is written in the README.md.

Error while building modV on Windows 10

When installing modV by using yarn or npm install on Windows 10 I was receiving the following error because of node-sass:

MSBUILD : error MSB3428: Could not load the Visual C++ component "VCBuild.exe"

This can be solved by installing the windows-build-tools like this in a terminal with Admin-rights:

npm install --global --production windows-build-tools

This should be mentioned somewhere so that people on Windows don't run into the same problem.

Save the settings of a plugin

Issue description

It would be nice so also save the settings for a plugin permanently, for example into preset or localStorage or another settings-file.

Right now the settings are lost when the app is reloaded

Version

modV: 2.0.0

Preset does not save when previous preset load has happened

Title

Dynamic Module and Plugin loading

The Media Manager should watch modV's modules folder and tell modV to register Modules when found.

modV's upcoming Plugin structure should also use this.

Horrendous drawImage performance in Chrome 55

Currently working on a bug report example to Chromium about this.

Mean-time fix: downgrade to Chrome 54 and don't update.

fails at npm start

Hi,

after cloning the repo and running npm install, I Ran the npm start command. it gave me the following error(attached txt file)

my node version is 6.9.0 as suggested.

log.txt

Update to Webpack 2

The title.

MediaManager is not started when running the build version

I was able to build modV on Windows 10 and run it. But the MediaManager is not started, I had to manually start it by executing node .\node_modules\modv-media-manager\run.js.

I started modV like this: modV\nwjs\build\modV\win64

Context menu for some controls

There are some controls that don't have a context menu:

• Layers
  • Clearing
  • Inherit
  • Pipeline
• Module
  • Use detected BPM

It would be awesome if they could get a context menu too, so it is possible to use them for MIDI assignments.

Layer Lock

Add the ability to lock Layers so they cannot be edited or removed when loading a preset (add seperate preset loading option near preset loading controls)

Why is there no MIDIModule?

the title

Unify Gallery rendering

Gallery items do not use the same rendering pipeline as the main loop, however they share the same drawtime as the main loop.

This causes a small but sometimes noticeable performance issue, especially on lower-end machines.

A main cause of this is Canvas2D's drawImage and webGL's bindBuffer and bufferData for setting the draw area (setRectangle).

Canvas2D's drawImage function is just generally slow, so it should be used as sparingly as possible and the webGL environment's setRectangle function should only be called on resolution updates (I think), currently setRectangle is in the main draw loop so the gallery items draw correctly.

There's not much we can do about the drawImage function at the moment, maybe when Offscreen Canvas hits (Chrome 60?) this could leverage some threaded action?

setRectangle could potentially be pushed out to a separate gl instance specifically for the gallery, there's possible memory concerns there, but since we're only pushing relatively small programs to GPU memory it may not be a problem (but it IS doubling up the memory essentially...).

Some expert knowledge would be helpful here.

Missing font

Error in console: missing Arimo-regular font. It's used in nwjs styles.
Needs to be restored or removed.

Meyda version increment

Hola 🎸

Meyda's quite a lot further along now than where it was in the initial integration. We're on npm now, and we've done a lot of thinking about improvements to the API, so it's likely that you'll find them very useful 😄 We haven't added any new audio features, but that's the next step, so it's probably worth pulling in the dependency from npm so you can keep them up to date. I'm pretty sure your implementation doesn't care what audio features meyda provides, so if we add them you can just write modV patches that use them without having to modify modV core at all?

Multiple window output

The ability to use more than one output window would be nice.

Eventually some sort of window configurator for different window positions from a main canvas would be great too!

Performance issue after loading presets with shaders rapidly more than 20-25 times in a row

I've no idea why this occurs. The bug is only noticeable on large display outputs (> 1280x720).
Something to do with the texture upload to WebGL I think.

Add docs serving

In #54 @2xAA added generate-docs task, which (surprisingly) generate docs from source code (by using jsdoc annotations).

It would be nice to serve that docs when you launch modV. For example, on localhost:3131/docs, or on another port.

Palette controls wrongfully get deleted on module removal

The title

Plugin: Shadertoy Search/Import

• Createa plugin to search and import Shadertoy Shaders using their API: https://www.shadertoy.com/api
• Add iMouse default controls to every imported shader
• Save the loaded shader into a preset

Module info SaveData key

Add the ability to define custom data sets to be saved within a Preset in the info Object of a Module's super call.

Initial idea is to use a Mongoose Model-esque map.
Example:

info: {
	name: 'Ball',
	author: '2xAA',
	version: 1.0,
	meyda: ['rms', 'zcr'],
	saveData: {
		balls: {
			type: Array,
			eachKey: { // Map Array's child keys if Array contains Object children (omit if Array contains JSON serialisable content such as plain Objects (no functions), Numbers, Booleans, Strings.
				bounds: {
					type: Object
				},
				position: {
					type: Object,
					keys: [ // Specify which keys within the Object to save
						'x'
					]
				},
				velocity: true, // save the whole variable, detect type (may cause an error?)
				wrap: {
					type: Boolean
				},
				speed: {
					type: Number
				},
				xReverse: {
					type: Boolean
				},
				yReverse: {
					type: Boolean
				}
			}
		}
	}
}```

Use OS specific data stores for options.json and media directory

Let the Media Manager use %APPDATA%, ~/Library/Application Support, etc. for data and media storage.

gulp watch

https://pastebin.com/iMN530kM

GrabCanvas: Resize preview

Right now the preview view will not be resized when the output view is resized.

The output view only gets resized after changing the size of the main window.

Leverage WebWorkers to reduce work in main loop

Palette and LFO cycling could potentially be handled by a secondary (or more) loop(s) to keep the main modV loop for audio processing and drawing.

When Audio Workers become available, Meyda will hopefully be able to use those for its audio processing (I have talked with the collaborators about this in person) which will leave only BeatDetektor in the main loop along with each Module's draw/process loop.

Potentially Modules could have a special script-loop for pushing heavy work out of the main loop (such as the Perl Module)? - something to consider.

However, I think Palettes and LFOs should be shunted out of the main loop by the release of 1.5.

Remove all modV- and modV. prefixing on .js files

It's unnecessary and very difficult to find the files you need.

what was I thinking? 😫

Save module state into preset

• Disable a module
• Change a configuration value of a module
• Save a a preset
• Load the preset

-> The module is enabled again & all the configurations have default values.

The desired behaviour should be, that the module is disabled and the configuration is loaded from the saved preset.

Maybe add a button "default values" into every module, so someone can overwrite the loaded configuration values?

Allow Layers to draw to specific Output Windows

Branch 2.0-outputwindow-targeting has been created to implement this feature.

Allow Layers to Inherit from any Layer

Sort of a Layer to Layer routing type thing.
Select box should go in the Layers UI panel.

Define modV Plugin Structure (+modV's future)

After talking with @TimPietrusky at FrankenJS about modV's future, it became clear a plugin API would be very useful.

Features such as a live code editor, Palette and asset managers etc. would benefit from being separate from modV's core code. As it is, modV has a fairly large codebase already.
Writing these as Plugins will allow future extension of modV and keep extra features written be potential users outside of modV Module types.

To begin with, a full writeup of modV's current API and inner workings is essential.
This (the writeup) will not happen until the end of the 1.5 milestone.

Plugin functionality may come before 2.0 as I'm thinking 2.0 may become the "UI" update and the final move to NWJS as the main target for modV.

It is clear a roadmap is required.

Node not finding local gulp on fresh install

package.json installs gulp locally, but even runninggulp watch through npm start node isn't seeing the local gulp installation. What's up with that?

Rewrite Media Manager from ground up

The code for the Media Manager is a mess.

All the elements are there, it just needs to be a Class and probably have its own sub-build process.

Presets don't load/save correctly

Image and Video Controls lose their source.
Palette Controls lose the loaded colours, BPM and timing settings also.

Creating more than one module in the active list and deleting the 'original' module makes the next module created 'module (1)'

In short: handle the module titling better.

NWJS build cannot access media files

NWJS provides a filesystem module through require to allow filesystem access.
This is great, however I think some work regarding the building of the NWJS app and the Chrome Browser versions needs to be done to prevent any unnecessary code duplication and/or errors between versions.

Loading 3D module from preset does not initialise the 3D module correctly

The title - only affects the 1.5 branch.

MIDI & OSC

Implement MIDI and OSC I/O.

MIDI:

• modV has assignable parameters using webMIDI already
• MIDI Clock & MTC for BPM control (this should work for LFOs also)
• Layers could potentially each use a channel? (might get confusing)
• Media Manager could be renamed to 'modV helper' and initiate its own MIDI I/O ports to make things easier (use: https://github.com/justinlatimer/node-midi )

OSC:

• Support for automatic routing for the OSC I/O (e.g. /moduleName/controlName/param )
• use https://www.npmjs.com/package/osc
• Does media manager/modV helper need to be involved?

PS: never used OSC, the documentation is pretty awful.

Add doubleclick on output window to make fullscreen

The title.

ModuleScript is broken in GrabCanvas

It doesn't work!

I think you should remove it and "GrabCanvas" should be a plugin.

Is that a good idea?

Event Emitters for modV Plugins

I've decided for the 1.6 Plugin system Event Emitters are a fairly good idea.
It will keep modV Plugins abstract which will hopefully allow the creation of very complex or very minimal plugins.

Events are going to be added into 1.5, but the Plugin loader (Issue #38) will not be added until 1.6.

The reasoning behind this is that I can work on getting modV's API tidied up in 1.5 which will introduce a lot of breaking changes in one go (though none of these changes will break users' experience unless they've modified the code somehow).
I can add the Emitters into the rebuilt internal API at the same time and all the code will be updated to ES6 also.

Let the madness ensue!

jsfft breaking Meyda breaking modV

modV is currently broken an all platforms, please bear with this while Meyda is broken.
Currently trying to diagnose the problem myself and create a PR, however the issue seems elusive.

Track this progress on Meyda's issues page: https://github.com/hughrawlinson/meyda/issues
Currently keeping up to date on this in meyda/meyda#138 :)