dishmint / arcanepropagation

Exploring image convolution and fragment shaders.

Hey! I'm Fez.

Check out what I've been up to!

---

Wolfram Language

Paclets

Paclet	Description
	Extract substrings matching a lexical pattern
	Experimental Wolfram Language functions

Functions

Function	Description
	Map a function over elements at specified positions and another function to the rest
	Map functions to specified parts of an expression

---

Stats

arcanepropagation's Issues

Add mazeImage function

a function that creates a pixel maze

Use HSB instead of RGB

With RGB transmission is based on the value of each channel, when I'm more curious about color being its own entity, with saturation and brightness.

Hue => which 'frequencey'
Saturation => which 'multilple'
Brightness => what 'amplitude'

This makes 'color' a standalone feature, and provides two other interesting features for adjusting behavior that don't affect the base color.

Port Processing 4 code to P5js

Port Processing 4 code to P5js and add controller GUIs for selecting files and modifying parameters.

[Feature] Implement GUI with Tweakpane

None of the GUI libraries were easy to setup, and I like everything about Tweakpane.

Have the sketch launch a browser window with Tweakpane inside. Then have Tweakpane and the sketch talk to eachother via osc or something.

[Feature] Langton's Ant and Game of Life

Implement Langton's ant and migrate gol filter to p4_Arcanepropagator.

Make ximg[loc] an actual matrix instead of a single float

The p5 sketch operates in this manner. x[loc] should be the convolution kernel where each kernel element is the normalized value of the color.

float[][] kernel = {
    {nw,   n, ne},
    {  w,   c,   e}, 
    {sw,   s,  se}
}

Add colorspace option

One of the p5 shader examples uses hsb color space to convert colors to angles. I should try that and see if that changes the appearance of anything.

Implement Image Dispersion

Space out image pixels

One way of doing this is creating a new image double the size,

PImage img = loadImage("path/to/imag.png");
dispersedImage = createImage(img.width * 2, img.height * 2, ARGB);

then when loading the pixels set the pixels I don't care about to zero.

void loadDispersedImage(PImage source, PImage di) {
  source.loadPixels();
  di.loadPixels();
  for (int i = 0; i < di.width; i++){
    for (int j = 0; j < di.height; j++){
    int dindex = (i + j * di.width);
    if(i % 2 == 0 && j % 2 == 0){
      dindex = (i + j * di.width);
      int sindex = ((i * 2) + ((j * 2) *source.width));
      di.pixels[dindex] = source[sindex];
    } else {
      di.pixels[dindex] = color(0);
      }}}
  source.updatePixels();
  di.updatePixels();
}

The shader shouldn't be a problem since the value of the empty pixels is 0, though I'll have to load this image every frame because I'll need the updated source image. Or wait, maybe not, since I'd be applying the kernel to the dispersed image.

Capture image from camera every x seconds

Video doesn't work well because there's no time for the information to propagate. If I sample an image from the camera or video every x seconds, then there's some breathing room.

[Refactor] Improve CA implementation

https://processing.org/examples/wolfram.html

[Feature] Implement Reaction Diffusion

• https://www.karlsims.com/rd.html
• https://www.youtube.com/watch?v=BV9ny785UNc

Markov Filter

Use a transition probability matrix, and the logic of markov chains to create a filter.

Blender implementation

This blender tutorial by khamurai makes me think I could implement AP in Blender pretty easily.

First — I'd need to create a program that applies the AP filter to the image. That's it
Then — Render the nested application of the filter
Then — Run the video into blender and draw the lines with geo-nodes (image below).

Clean up ArcanePropagator

Clean out commented code, and move image generators to a separate file.

Take log of operations to avoid small numbers

Maybe add a param takeLog, which will take the log of kernel results and avoid small numbers. (like vanishing gradients in backprop).

Manage program states with classes

Make a new class to move the kernel functions out of the main file. It will be easier to read/edit/maintain.

class ArcanePropagator {
    float widthfactor;
    float anglefactor;
    
    // Default Constructor
    ArcanePropagator(){
        widthfactor = 1.0;
        anglefactor = 1.0;
    }
    
    ....
}

[Documentation] Add gif to README.md

A gif would be a good opener for the README.md

[Question] What affect does an unconstrained energyAngle have on the sketch

I constrain energyAngle to be between 0 and 360, but maybe it shouldn't be, because values can grow over 255.

Probabilistic Information Spread

A pixel's channel average represents the probability that its information spreads.

if it's greater than 0.5, that pixel will absorb neighboring values. If it's less than 0.5, the pixel will not absorb neighboring values.

For fun, display the probabilities instead as white (>0.5) or black (<0.5)

Implement Perform mode

ArcanePropagation/p4_ArcanePropagatorGUI/p4_ArcanePropagatorGUI.pde

Line 19 in d61b981

/* TODO: Add performance mode so live parameter changes can be recorded and saved */

[Feature] Support Video

Add support for video either from the camera or a file.

Using Movie objects generates error messages

The errors seem benign on the surface. Movie frames are rendered, but I get lots of errors in the console.

Implement preset system

• value of gs
• low, hi in map(gs, _, _, low, hi)
• window size
• image size
• glsl code (?)

Spawning Filter Agents

Spawn filter agents across the canvas that apply the filter.

Rather than applying the filter to the entire image all at once. have the agents run around and do it.

So this is basically async filter application.

Add randomImage function

Add support for a noisy image.

img = randomImage(width, height);

PImage randomImage(int w, int h){
	PImage rimg = createImage(w,h, ARGB);
	rimg.loadPixels();
	for (int i = 0; i < rimg.width; i++){
		for (int j = 0; j < rimg.height; j++){
			color c = color(random(255.));
			int index = (i + j * rimg.width);
			rimg.pixels[index] = c;
		}
	}
	rimg.updatePixels();
	return rimg;
}

Use OpenCV to apply filters

Right now I'm applying filters with for loops.

Presumably OpenCV would have be able to apply filters in an optimized way, across the matrix, with multiple cores etc.

If possible this would really come in handy on the web, because the p5js code is really slow atm.

Shader not centered when using display density of 2

Setting pixel density to 2 in p4_ArcanePropagation_Shader makes the shader uncentered. I know it's because more pixels are being drawn, but changing img.width/height to img.pixelWidth/pixelHeight didn't seem to fix the issue.

Channel Oscillator

Create a ChannelOsc class which creates an oscillator for each color channel, and otherwise behaves like any other Osc class.

Port to p5js

Clean up current codebase, then implement in p5js. I'd like to systematically explore the parameters with tweakpane.

nannou implementation

[Feature] Implement Collatz Filter

Expose parameters to a GUI interface

This example shows adding a swing UI to a Processing sketch.

https://fablab.ruc.dk/integrate-javaswing-gui-in-your-processing-sketch/

For the p5 implementation: tweakpane/plugin-essentials#8 (comment)

GUI needed:

• File Browser to select image
• Field to change PImage and Window dimensions, downsample, modfac
• slider for sf, xsmnfactor, displayscale, frameRate
• checkbox dispersed or not
• Restart Button to reevaluate setup to pick up gui changes
• Save button to save presets

smear shifts the image northwest

the smear function appears to push pixels up and to the left, and I haven't figured out yet how to control the direction.

Multithreaded Image Processing

[Improvement] Filter Optimization

If I can apply the convolution / filter in parallel that would help speed up the processing sketches, but also probably drastically improve the plus sketches.

Clean up PointOrbit p4 sketch

Clean out comments and refactor code to be more efficient.

Add 2D CA support

I could try to access the Wolfram Language via J/Link or I can see if there's a CA library available already.

[Feature] Use a Neural Network to determine optimal parameters

A NN could be used to tune parameters, but I don't know what the goals are yet.

[GLSL] radius and thickness change how many points are displayed

		// Not enough points
		// float radius    = .0000001;
		// float thickness = .0000001;
		
		// Bottom Left Corner is weird
		// float radius    = .00000001;
		// float thickness = .00000001;
		
		// Bottom Left Corner is weird and larger
		// float radius    = .00000002;
		// float thickness = .00000002;
		
		// Bottom Left Corner is weird but smaller
		// float radius    = .000000001;
		// float thickness = .000000001;
		
		// Bottom Left Corner is weird and really small
		// float radius    = .0000000001;
		// float thickness = .0000000001;
		
		// Perfect!
		// float radius    = .00000000001;
		// float thickness = .00000000001;

		// pixel isn't small enough to capture enough details
		// float radius    = pixel.x;
		// float thickness = pixel.x;
		
		// blank screen
		// float radius    = pixel.x/pixel.y;
		// float thickness = pixel.x/pixel.y;

Process Image with Tensorflow

https://www.oreilly.com/library/view/learning-tensorflowjs/9781492090786/ch04.html

• Convert image to tf
• apply p-function to pixels
• Convert tf to image

Try using inverse square law for fall off

In the transmit function I use l to scale the added color.

See what happens when using 1/(l^2).

[Feature] Implement with canvas-sketch instead of p5

Having issues with the p5js implementation. I might just implement this with canvas for now then.

For shaders I can look at this: https://github.com/patriciogonzalezvivo/glslCanvas

Emit sound per frame

Add up all sin(pixel) and emit the sound at the end of the frame.

Gradient Descent Filters

idk, something like this..:

pixel = pixel_rate * pixel_gradient

[Feature] Channel selector

    Could be interesting to have agents for each color channel.  imagine sending out a green agent, where the seed image is preserved, but only the green channel is affected.

Originally posted by @dishmint in #42 (comment)

Choose which channels to apply filtering to. r,g,b or a.

Clean up p4_ArcaneShader

Clean up ArcaneShader

Modulate transmission with audio

Hook up the modular and let the wave form be the transmission.

Implement custom filters via shaders

To implement the custom filtered in shaders I'd need:

• An image for transmission strength (or could also just be dynamically sampled)
• A single shader containing all filter functions
• A PGraphics for storing the filtered image. This PGraphics could then be the input for the blueline shader.

Implement ArcaneViewer

Adding video doesn't make sense for ArcanePropagator, but it does for ArcaneViewer. ArcaneViewer will just render the video according to the scheme, that is, orbit a color-point around a pixels origin.

Bayesian Filters?

Might not be very interesting to look at but, for each pixel take the P(color|cdist) to get the new color.