brianchirls / seriously.js Goto Github PK

Discovered by @cbodin and referenced in #62.

Similar to the requestAnimationFrame shim, perhaps you should consider a setImmediate shim that uses setImmediate where available (which is actually only in latest IE... 10, maybe 9? I'm not sure TBH ) and polyfills where it's missing (which would just be your current setTimeoutZero impl, renamed)

Just a thought

"framing is not allowed" on gh-pages 404

http://seriouslyjs.org/

All inputs for effect nodes are validated by type and converted to a usable value, optionally within a given range. Transform node inputs should do the same.

I noticed that the color node wasn't doing anything in a test I wrote, and then I looked at it in the Seriously.js/index.html demo and it's also failing there.

The following code doesn't render anything.

var seriously = new Seriously();
var black = seriously.effect('color');
var target = seriously.target('#seriously-target');
target.source = black;
seriously.go();

Blend (normal mode) with low opacity renders incorrectly

While we're at it:

• Build example for testing all blend modes
• Skip draw call for top layer if opacity is set to 0

Requested by @lselden

Similar to the Quartz Composer "Accumulator" effect:
http://quartzcomposer.com/patches/366-accumulator

• blend mode
• opacity amount.
• boolean options for whether to reset the buffer and to turn accumulation on and off.

Refer to discussion in issue #22

Need to account for different dimensions of inputs.

Not always possible, but should at least try. At the very least, issue some kind of notification or warning. Requires rebuilding of all WebGL assets on every node.

• Gaussian (approximate - real thing might be too slow)
• Motion Blur
• Zoom Blur

Could Seriously.js be made to support IEWebGL (http://iewebgl.com/)?

Fire notifications on significant state changes.

Should this include all values on nodes? Probably not, at least not at first, as it might slow things down to be firing triggers/events on every frame.

But this might be a simple way to let nodes output non-image values.

Not sure why. They should be the same. Need to check Safari and IE too, even though I don't wanna.

Looks like the slider polyfill isn't needed any more.
http://caniuse.com/#feat=input-range

Fryn says slider support was added in firefox version 22.
https://github.com/fryn/html5slider

Originally reported here: processing/p5.js-video#40

This is tricky to replicate, as it seems to be a problem only for certain devices. It first manifested as complete lack of any alpha channel on images processed by the Chroma effect. Testing with a split effect showed that the same users were seeing white dots along the edge where the shader had branching. A version of the shader without branching solved the problem.

This ticket is to remove branching from effects wherever possible, starting with Split and Chroma.

The edges I get in Safari in iOS 8 are for some reason much nicer than edges I get on a Mac with Safari or Chrome with the same footage and settings.

Part of this is probably due to the denser retina display, but it seems more pronounced with that. I'm wondering if there's some smoothing/antialiasing setting with different defaults? gl.antialias defaults to true on the laptop and false on the tablet, but I didn't see a difference when I disabled it. I'm not sure where else to start looking.

Thoughts?

• Safari 8.0 in 10.10 beta
  
• Safari in iOS 8.0 beta
  

It looks like IE doesn't like the way pre-processor macros are written. Need to fix this and apply fixes to Accumulator as well. (#23)

Similar to how you can create plugins for Source types
d83bd7f

WebGL canvas targets will remain built-in. Other types like FBO/Texture and 2D canvas, may be moved out into plugins. The goal is to eventually build target plugins for other frameworks like Three.js, Babylon.js, so output from Seriously.js can be used as textures for 3D scenes. Those will be filed under separate issues.

Due to this Firefox bug, which is a few months old, without much attention. We need a workaround.

https://bugzilla.mozilla.org/show_bug.cgi?id=926753

Camera Shake is cool but leaves some space at the edges unless you scale the image up. We should automagically calculate how much you need to scale it and do it for you.

I'm trying to get a "stream" of base64 encoded jpeg images drawn on a canvas with some effects.

Approach 1

I first tried to set the targets source property to a image element:

var seriously = new Seriously();
var target = seriously.target('#image-canvas');
seriously.go();

var image = new Image();
target.source = image;

function onRecieveImage(data) {
  image.src = 'data:image/jpeg;base64,' + data;
}

The onRecieveImage is called around 15 times/s.

This only showed the first loaded image. If i added the image to the page i could see the live stream in the img tag.

Approach 2

I'm drawing the image on a second canvas and then setting the imageData from that canvas to my seriously target:

var canvas = document.getElementById('image-canvas');

var helperCanvas = document.createElement('canvas');
var helperContext = helperCanvas.getContext('2d');
helperCanvas.width = canvas.width;
helperCanvas.height = canvas.height;

var seriously = new Seriously();
var target = seriously.target(canvas);
seriously.go();

var image = new Image();
image.onload = function() {
  helperContext.drawImage(this, 0, 0);
  var imageData = helperContext.getImageData(0, 0, helperCanvas.width, helperCanvas.height);
  target.source = imageData;
};

function onRecieveImage(data) {
  image.src = 'data:image/jpeg;base64,' + data;
}

This works as intended, however this approach introduces a major memory leak, and Chrome crashes within a few minutes.

When taking heap snapshots in the Chrome dev tools i can see a raise in memory usage by around 0.1mb every 2-4 second until Chrome crashes.

Removing the target.source = imageData; inside the onload function and showing the helperCanvas on the document will not introduce any memory leaks.

• Hue
• Saturation
• Color
• Luminosity
• Divide
• Lighter Color
• Darker Color

reference:

• http://photoblogstop.com/photoshop/photoshop-blend-modes-explained
• http://helpx.adobe.com/en/photoshop/using/blending-modes.html

Edit: now working on NoFlo + editor

I finally have native nodes working in Meemoo. The i/o definitions are quite similar to Seriously.js, so I think that I might be able to make some kind of automatic translation.

The bigger issue is how to parse a Meemoo graph:

nodes: [
  {id: 1},
  {id: 2},
  {id: 3}
],
edges: [
  {source: 1, target: 2},
  {source: 2, target: 3}
]

into Seriously JavaScript code.

I'll probably need to have some kind of hooks that listen for dis/connect events, and then look for Seriously-specific nodes and rebuild the graph every time.

Are there any thoughts or examples out there of translating a visual node system to a code-based node system?

Currently, it is possible to build a single shader shared by all node instances of a given effect, so we don't have to create a new, identical copy for each node. This is great, but some effects have multiple shaders, and we should support that. They should be indexable by name and automatically cleaned up when the last node is deleted.

In SourceNode.prototype.renderVideo method there's try-catch statement. The catch part is incorrect - the name of the Error may probably vary among browsers - for Safari for instance it is "SecurityError" while you're checking against "SECURITY_ERR". Also, there should definitely be some "else" there - otherwise the error can't be spotted in any way.

To sum up, if you try CORS request in Safari you'll fail but also you won't get any feedback whatsoever.

and noConflict

Tried in both Firefox & Chrome, neither work. I have AdBlock & FlashBlock plugins, but disabled both for this page and that still didn't seem to change anything.

Uncaught Not a valid HTML element: EMBED (must be img, video or canvas) seriously.js:2634

Stacktrace:
Seriously.SourceNode seriously.js:2634
findInputNode seriously.js:1065
Seriously.EffectNode.setInput seriously.js:1949
setInput seriously.js:1512
(anonymous function) seriously.js:1518
setUpVideo script.js:20
Loaderator.Category.checkForAllLoaded loaderator.js:155
Loaderator.resLoadCallback loaderator.js:725
obj.onload

Transform nodes need to do their own rendering when there is a target node that is not an inPlace effect. A FrameBuffer object is only needed in this case. Since inPlace can be dynamic (e.g. "blend" effect), depending on the input values of the target node, we don't know whether we need the FrameBuffer until render time. This causes visible jank, as in the "Whip Pan" example, because the FrameBuffer object takes some time to create.

We need to be able to determine inPlace earlier, as soon as the target node is connected and again any time an input on the target node may change the inPlace value if inPlace is a function.

While we're at it, it's also worth looking into whether we can release the FrameBuffer when it's not needed anymore, either when the target node is disconnected or when inPlace changes back to true.

Hi there !

I'm building a node-webkit application doing chromakeying on a mjpg stream. While I can have my stream displays properly in my canvas with a requestAnimationFrame and a drawImage() method, when I implemented the chromakey effect, target.render() would not update the source, and only show the first jpg of the stream. Here is a sampe code:

    var seriously,
        image,
        target,
        chroma;

    seriously = new Seriously();
    image = seriously.source('#image');
    target = seriously.target('#canvas');
    chroma = seriously.effect('chroma');

    function play(){
        target.width = image.width;
        target.height = image.height;

        chroma.source = image;
        target.source = chroma;

        target.render();
        requestAnimationFrame(play);
    }

    requestAnimationFrame(play);

I checked and the play function is looped correctly.

Has anyone an idea on why the source where chroma apply its effect is not updated ?

Don't load that whole big mess if you don't need it.

I'm gonna wait until we have a proper build tool for this.

This may not be necessary if MS fixes this before our next release.

For reference:
https://twitter.com/frankolivier/status/494162110935359488

i'm running nodejs. how can i save the rendered effect to a file?

in seriously.js and in plugins

I hear music but don't see anything.

Uncaught Unknown source type seriously.js:2727
Seriously.SourceNode seriously.js:2727
findInputNode seriously.js:1065
Seriously.EffectNode.setInput seriously.js:1949
setInput seriously.js:1512
(anonymous function) seriously.js:1518
setUpVideo script.js:31
Loaderator.Category.checkForAllLoaded loaderator.js:155
Loaderator.resLoadCallback loaderator.js:725
resource.element.onload

My user-agent is: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1530.4 Safari/537.36

(Chrome Canary on a MBP)

Can we allow one instance of Seriously to render to multiple targets? Currently only the first call to seriously.target() is successful.

var t1 = seriously.target('#t1'); // renders
var t2 = seriously.target('#t2'); // does not render

A workaround is to create another instance of Seriously and point it to the second target.

var s2 = new Seriously();
var t2 = s2.target('#t2');

If Seriously instances only accept one render target, should it be an argument required for instantiation?

var s = new Seriously('#render-target')

If we remove the option of creating a target with the .target() method, it would prevent devs from mistakenly trying to create multiple targets.

Another option would be to throw an error when the dev attempts to create a second target.

Uncaught Could not initialise shader: (56,22): error X5608: Compiled shader code uses too many arithmetic instruction slots (74). Max. allowed by the target (ps_2_0) is 64. Consider increasing optimization level to reduce instruction count.

Warning: D3D shader compilation failed with default flags. Retrying with avoid flow control.
(56,22): error X5608: Compiled shader code uses too many arithmetic instruction slots (74). Max. allowed by the target (ps_2_0) is 64. Consider increasing optimization level to reduce instruction count.

Warning: D3D shader compilation failed with avoid flow control flags. Retrying with prefer flow control.
(56,22): error X5608: Compiled shader code uses too many arithmetic instruction slots (74). Max. allowed by the target (ps_2_0) is 64. Consider increasing optimization level to reduce instruction count.

Warning: D3D shader compilation failed with prefer flow control flags.

...instead of just objects, so private variables can be defined in closures.

Chrome dev tools timeline shows memory ramping up. Most of it gets garbage collected every 10-15 seconds, but possibly not all of it. And, if possible, it would be great to avoid allocating any unnecessary new memory in frame updates, even if it does get GC'ed.

It may be that the bulk of the allocated/collected memory is in the video frames, in which case it's probably not possible to fix.

Simplifcation of the render callback system may mitigate a little bit of this.

Things to test:

• don't use videos (just static images, or color generator node).
• use manual render loop (not .go())

The API needs to provide access to the list of inputs accepted by each effect/transform node, with type information for each one. Some effects have fixed inputs, but others are calculated at instantiation time based on configuration options.

Info for each input should include:

• name
• type
• min
• max
• dimensions
• defaultvalue

Additionally, provide effect/transform title/name, id and possibly creation options for each node.

Hi, I've a problem, there is a button on my app that will create a new instance of Seriously in a canvas, then save some images and destroy the seriously object with seriously.destroy()

I'm calling seriously without new but it keeps throwing this warning Target already use by another instance

What can I do for ensure that there is only one instance?

Thanks!

I have been doing image processing in workers. I'm wondering if there could be a no-webgl alternative for devices without webgl to process still images with a Seriously.js graph.

Would need to send the fragment shader + the image pixel data to the worker, then interpret the shader in plain js and run it on each pixel. Possible?

It'll be cool if API for plugins will be documented.

Feedback effects can be created pretty easily if the "getTexture" method of the Target class is duplicated by the Effect class:

this.getTexture = function () {
  if (me && me.frameBuffer) {
    return me.frameBuffer.texture;
  }
};

Source already allows WebGLTexture as an input type, so the result can be used as any other source node.

The result is pretty useful - I've created a game of life shader that uses this method, and you can also take a look at a video feedback effect here:
http://www.lukeselden.com/vj/

It would be nice to have an FFT node. Personally, I just want to play with it... but in general it's incredibly useful for signal processing.

Probably, the best solution would be to find a version of FFTW that runs as shader code. I'm not sure how feasible this is, but it seems like FFTW is the fastest FFT algorithm out there.

Depends on #60

The camera source (640x480) will render to a target of a different size (1280x720). It works as expected until I connect an effect node and try to render the result on that same target.

This may apply to all video sources, but I haven't checked.

    var seriously = new Seriously();

    var effectTarget = seriously.target('#effect');
    var effect = seriously.effect('edge');

    effect.source = seriously.source('camera');
    effectTarget.source = effect;
    seriously.go();

<!-- size does not match webcam feed. nothing is rendered -->
<canvas id="effect" width="1280" height="720"></canvas>

<!-- size matches webcam feed. all is well -->
<canvas id="effect" width="640" height="480"></canvas>

Any idea what causes this?

• Use grunt.js
• Separate out packaged transforms
• Minify with uglify.js or Google Closure Compiler
• Run unit tests on command line
• Run jshint and unit tests before building
• Build with r.js
• option to build with all/some/none of effect and transform plugins included
• In-browser build tool in addition to command line

Every effect input has a default value, but it would be convenient to be able to override the defaults in case someone wants to make many nodes with the same value.

• Passed as an option to the Seriously constructor:

    var seriously = new Seriously({
        defaults: {
            blur: {
                amount: 0.5
            },
            tvglitch: {
                lineSync: 0,
                verticalSync: 0
            }
    });

• defaults method on the Seriously instance:

    seriously.defaults({
        color: {
            color: 'darkgrey',
            width: 960,
            height: 540
        },
        blend: {
            bottom: seriously.effect('color')
        }
    });

    // or...
    seriously.defaults('edge', {
        mode: 'frei-chan'
    });

Reset defaults to default defaults:

    seriously.defaults(null); // all effects, all fields
    seriously.defaults('edge', null); // just one effect, all fields

Right now, 'blend' only supports two layers: top and bottom. Should have a plugin that supports multiple layers - at least four, ideally unlimited, with a blend mode and opacity on each layer.