brianchirls / seriously.js Goto Github PK
View Code? Open in Web Editor NEWA real-time, node-based video effects compositor for the web built with HTML5, Javascript and WebGL
License: MIT License
A real-time, node-based video effects compositor for the web built with HTML5, Javascript and WebGL
License: MIT License
It looks like IE doesn't like the way pre-processor macros are written. Need to fix this and apply fixes to Accumulator as well. (#23)
i'm running nodejs. how can i save the rendered effect to a file?
The camera source (640x480) will render to a target of a different size (1280x720). It works as expected until I connect an effect node and try to render the result on that same target.
This may apply to all video sources, but I haven't checked.
var seriously = new Seriously();
var effectTarget = seriously.target('#effect');
var effect = seriously.effect('edge');
effect.source = seriously.source('camera');
effectTarget.source = effect;
seriously.go();
<!-- size does not match webcam feed. nothing is rendered -->
<canvas id="effect" width="1280" height="720"></canvas>
<!-- size matches webcam feed. all is well -->
<canvas id="effect" width="640" height="480"></canvas>
Any idea what causes this?
Due to this Firefox bug, which is a few months old, without much attention. We need a workaround.
Similar to how you can create plugins for Source types
d83bd7f
WebGL canvas targets will remain built-in. Other types like FBO/Texture and 2D canvas, may be moved out into plugins. The goal is to eventually build target plugins for other frameworks like Three.js, Babylon.js, so output from Seriously.js can be used as textures for 3D scenes. Those will be filed under separate issues.
In SourceNode.prototype.renderVideo method there's try-catch statement. The catch part is incorrect - the name of the Error may probably vary among browsers - for Safari for instance it is "SecurityError" while you're checking against "SECURITY_ERR". Also, there should definitely be some "else" there - otherwise the error can't be spotted in any way.
To sum up, if you try CORS request in Safari you'll fail but also you won't get any feedback whatsoever.
reference:
I have been doing image processing in workers. I'm wondering if there could be a no-webgl alternative for devices without webgl to process still images with a Seriously.js graph.
Would need to send the fragment shader + the image pixel data to the worker, then interpret the shader in plain js and run it on each pixel. Possible?
Fire notifications on significant state changes.
Should this include all values on nodes? Probably not, at least not at first, as it might slow things down to be firing triggers/events on every frame.
But this might be a simple way to let nodes output non-image values.
Similar to the requestAnimationFrame
shim, perhaps you should consider a setImmediate
shim that uses setImmediate
where available (which is actually only in latest IE... 10, maybe 9? I'm not sure TBH ) and polyfills where it's missing (which would just be your current setTimeoutZero
impl, renamed)
Just a thought
The edges I get in Safari in iOS 8 are for some reason much nicer than edges I get on a Mac with Safari or Chrome with the same footage and settings.
Part of this is probably due to the denser retina display, but it seems more pronounced with that. I'm wondering if there's some smoothing/antialiasing setting with different defaults? gl.antialias defaults to true on the laptop and false on the tablet, but I didn't see a difference when I disabled it. I'm not sure where else to start looking.
Thoughts?
Depends on #60
I hear music but don't see anything.
Uncaught Unknown source type seriously.js:2727
Seriously.SourceNode seriously.js:2727
findInputNode seriously.js:1065
Seriously.EffectNode.setInput seriously.js:1949
setInput seriously.js:1512
(anonymous function) seriously.js:1518
setUpVideo script.js:31
Loaderator.Category.checkForAllLoaded loaderator.js:155
Loaderator.resLoadCallback loaderator.js:725
resource.element.onload
My user-agent is: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1530.4 Safari/537.36
(Chrome Canary on a MBP)
Chrome dev tools timeline shows memory ramping up. Most of it gets garbage collected every 10-15 seconds, but possibly not all of it. And, if possible, it would be great to avoid allocating any unnecessary new memory in frame updates, even if it does get GC'ed.
It may be that the bulk of the allocated/collected memory is in the video frames, in which case it's probably not possible to fix.
Simplifcation of the render callback system may mitigate a little bit of this.
Things to test:
.go()
)Hi there !
I'm building a node-webkit application doing chromakeying on a mjpg stream. While I can have my stream displays properly in my canvas with a requestAnimationFrame and a drawImage() method, when I implemented the chromakey effect, target.render()
would not update the source, and only show the first jpg of the stream. Here is a sampe code:
var seriously,
image,
target,
chroma;
seriously = new Seriously();
image = seriously.source('#image');
target = seriously.target('#canvas');
chroma = seriously.effect('chroma');
function play(){
target.width = image.width;
target.height = image.height;
chroma.source = image;
target.source = chroma;
target.render();
requestAnimationFrame(play);
}
requestAnimationFrame(play);
I checked and the play function is looped correctly.
Has anyone an idea on why the source where chroma apply its effect is not updated ?
All inputs for effect nodes are validated by type and converted to a usable value, optionally within a given range. Transform node inputs should do the same.
Could Seriously.js be made to support IEWebGL (http://iewebgl.com/)?
...instead of just objects, so private variables can be defined in closures.
I noticed that the color node wasn't doing anything in a test I wrote, and then I looked at it in the Seriously.js/index.html demo and it's also failing there.
The following code doesn't render anything.
var seriously = new Seriously();
var black = seriously.effect('color');
var target = seriously.target('#seriously-target');
target.source = black;
seriously.go();
Every effect input has a default value, but it would be convenient to be able to override the defaults in case someone wants to make many nodes with the same value.
var seriously = new Seriously({
defaults: {
blur: {
amount: 0.5
},
tvglitch: {
lineSync: 0,
verticalSync: 0
}
});
defaults
method on the Seriously instance: seriously.defaults({
color: {
color: 'darkgrey',
width: 960,
height: 540
},
blend: {
bottom: seriously.effect('color')
}
});
// or...
seriously.defaults('edge', {
mode: 'frei-chan'
});
Reset defaults to default defaults:
seriously.defaults(null); // all effects, all fields
seriously.defaults('edge', null); // just one effect, all fields
It would be nice to have an FFT node. Personally, I just want to play with it... but in general it's incredibly useful for signal processing.
Probably, the best solution would be to find a version of FFTW that runs as shader code. I'm not sure how feasible this is, but it seems like FFTW is the fastest FFT algorithm out there.
Transform nodes need to do their own rendering when there is a target node that is not an inPlace
effect. A FrameBuffer object is only needed in this case. Since inPlace
can be dynamic (e.g. "blend" effect), depending on the input values of the target node, we don't know whether we need the FrameBuffer until render time. This causes visible jank, as in the "Whip Pan" example, because the FrameBuffer object takes some time to create.
We need to be able to determine inPlace
earlier, as soon as the target node is connected and again any time an input on the target node may change the inPlace
value if inPlace
is a function.
While we're at it, it's also worth looking into whether we can release the FrameBuffer when it's not needed anymore, either when the target node is disconnected or when inPlace
changes back to true
.
It'll be cool if API for plugins will be documented.
This may not be necessary if MS fixes this before our next release.
For reference:
https://twitter.com/frankolivier/status/494162110935359488
Looks like the slider polyfill isn't needed any more.
http://caniuse.com/#feat=input-range
Fryn says slider support was added in firefox version 22.
https://github.com/fryn/html5slider
Don't load that whole big mess if you don't need it.
I'm gonna wait until we have a proper build tool for this.
Originally reported here: processing/p5.js-video#40
This is tricky to replicate, as it seems to be a problem only for certain devices. It first manifested as complete lack of any alpha channel on images processed by the Chroma effect. Testing with a split effect showed that the same users were seeing white dots along the edge where the shader had branching. A version of the shader without branching solved the problem.
This ticket is to remove branching from effects wherever possible, starting with Split and Chroma.
Tried in both Firefox & Chrome, neither work. I have AdBlock & FlashBlock plugins, but disabled both for this page and that still didn't seem to change anything.
Uncaught Not a valid HTML element: EMBED (must be img, video or canvas) seriously.js:2634
Stacktrace:
Seriously.SourceNode seriously.js:2634
findInputNode seriously.js:1065
Seriously.EffectNode.setInput seriously.js:1949
setInput seriously.js:1512
(anonymous function) seriously.js:1518
setUpVideo script.js:20
Loaderator.Category.checkForAllLoaded loaderator.js:155
Loaderator.resLoadCallback loaderator.js:725
obj.onload
Requested by @lselden
Similar to the Quartz Composer "Accumulator" effect:
http://quartzcomposer.com/patches/366-accumulator
Refer to discussion in issue #22
Need to account for different dimensions of inputs.
I'm trying to get a "stream" of base64 encoded jpeg images drawn on a canvas with some effects.
I first tried to set the targets source property to a image element:
var seriously = new Seriously();
var target = seriously.target('#image-canvas');
seriously.go();
var image = new Image();
target.source = image;
function onRecieveImage(data) {
image.src = 'data:image/jpeg;base64,' + data;
}
The onRecieveImage is called around 15 times/s.
This only showed the first loaded image. If i added the image to the page i could see the live stream in the img tag.
I'm drawing the image on a second canvas and then setting the imageData from that canvas to my seriously target:
var canvas = document.getElementById('image-canvas');
var helperCanvas = document.createElement('canvas');
var helperContext = helperCanvas.getContext('2d');
helperCanvas.width = canvas.width;
helperCanvas.height = canvas.height;
var seriously = new Seriously();
var target = seriously.target(canvas);
seriously.go();
var image = new Image();
image.onload = function() {
helperContext.drawImage(this, 0, 0);
var imageData = helperContext.getImageData(0, 0, helperCanvas.width, helperCanvas.height);
target.source = imageData;
};
function onRecieveImage(data) {
image.src = 'data:image/jpeg;base64,' + data;
}
This works as intended, however this approach introduces a major memory leak, and Chrome crashes within a few minutes.
When taking heap snapshots in the Chrome dev tools i can see a raise in memory usage by around 0.1mb every 2-4 second until Chrome crashes.
Removing the target.source = imageData;
inside the onload function and showing the helperCanvas on the document will not introduce any memory leaks.
Currently, it is possible to build a single shader shared by all node instances of a given effect, so we don't have to create a new, identical copy for each node. This is great, but some effects have multiple shaders, and we should support that. They should be indexable by name and automatically cleaned up when the last node is deleted.
Feedback effects can be created pretty easily if the "getTexture" method of the Target class is duplicated by the Effect class:
this.getTexture = function () {
if (me && me.frameBuffer) {
return me.frameBuffer.texture;
}
};
Source already allows WebGLTexture as an input type, so the result can be used as any other source node.
The result is pretty useful - I've created a game of life shader that uses this method, and you can also take a look at a video feedback effect here:
http://www.lukeselden.com/vj/
"framing is not allowed" on gh-pages 404
Camera Shake is cool but leaves some space at the edges unless you scale the image up. We should automagically calculate how much you need to scale it and do it for you.
Not always possible, but should at least try. At the very least, issue some kind of notification or warning. Requires rebuilding of all WebGL assets on every node.
Uncaught Could not initialise shader: (56,22): error X5608: Compiled shader code uses too many arithmetic instruction slots (74). Max. allowed by the target (ps_2_0) is 64. Consider increasing optimization level to reduce instruction count.
Warning: D3D shader compilation failed with default flags. Retrying with avoid flow control.
(56,22): error X5608: Compiled shader code uses too many arithmetic instruction slots (74). Max. allowed by the target (ps_2_0) is 64. Consider increasing optimization level to reduce instruction count.
Warning: D3D shader compilation failed with avoid flow control flags. Retrying with prefer flow control.
(56,22): error X5608: Compiled shader code uses too many arithmetic instruction slots (74). Max. allowed by the target (ps_2_0) is 64. Consider increasing optimization level to reduce instruction count.
Warning: D3D shader compilation failed with prefer flow control flags.
Not sure why. They should be the same. Need to check Safari and IE too, even though I don't wanna.
in seriously.js and in plugins
and noConflict
Right now, 'blend' only supports two layers: top and bottom. Should have a plugin that supports multiple layers - at least four, ideally unlimited, with a blend mode and opacity on each layer.
Edit: now working on NoFlo + editor
I finally have native nodes working in Meemoo. The i/o definitions are quite similar to Seriously.js, so I think that I might be able to make some kind of automatic translation.
The bigger issue is how to parse a Meemoo graph:
nodes: [
{id: 1},
{id: 2},
{id: 3}
],
edges: [
{source: 1, target: 2},
{source: 2, target: 3}
]
into Seriously JavaScript code.
I'll probably need to have some kind of hooks that listen for dis/connect events, and then look for Seriously-specific nodes and rebuild the graph every time.
Are there any thoughts or examples out there of translating a visual node system to a code-based node system?
Blend (normal mode) with low opacity renders incorrectly
While we're at it:
The API needs to provide access to the list of inputs accepted by each effect/transform node, with type information for each one. Some effects have fixed inputs, but others are calculated at instantiation time based on configuration options.
Info for each input should include:
Additionally, provide effect/transform title/name, id and possibly creation options for each node.
Hi, I've a problem, there is a button on my app that will create a new instance of Seriously in a canvas, then save some images and destroy the seriously object with seriously.destroy()
I'm calling seriously without new
but it keeps throwing this warning Target already use by another instance
What can I do for ensure that there is only one instance?
Thanks!
Can we allow one instance of Seriously to render to multiple targets? Currently only the first call to seriously.target()
is successful.
var t1 = seriously.target('#t1'); // renders
var t2 = seriously.target('#t2'); // does not render
A workaround is to create another instance of Seriously
and point it to the second target.
var s2 = new Seriously();
var t2 = s2.target('#t2');
If Seriously
instances only accept one render target, should it be an argument required for instantiation?
var s = new Seriously('#render-target')
If we remove the option of creating a target with the .target() method, it would prevent devs from mistakenly trying to create multiple targets.
Another option would be to throw an error when the dev attempts to create a second target.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.