nasser / zajal

Experimental creative coding framework

Zajal 0.6 - Atlantic

Atlantic is the next major rewrite of the Zajal creative coding framework. It is built on ClojureScript, Pixi.js, ThreeJS and the ongoing progress of the Arcadia project. The goal, as always, is to create a highly expressive, completely live creative coding experience suitable for beginners and advanced programmers alike.

This fourth rewrite, continues the exploration of Clojure and Lisp in the service of creative coding, but builds on a web-based stack of functional reactive tools that have become practical in recent years. This incarnation of Zajal, like the previous one and the Arcadia project, experiments with the effects of functional programming on high performance interactive graphics programming. The hope is to provide a real semantic alternative to the imperative painters-algorithm style tools available today.

Status

Extremely early, nothing beyond the basic concept has been demonstrated to work.

(defn step [x] (inc x))

(defn draw [t]
  (renderer
    {:width 400
     :height 400}
    [(graphics {:shape (circle 20)
                :line-width 4 
                :line-color 0xffffff
                :x 100
                :y 100})
     (graphics {:shape (polygon [0 -15 -10 10 0 5 10 10 0 -15])
                :fill 0
                :line-width 1
                :line-color 0xffffff
                :x 100
                :y 75
                :rotation (* 0.05 t)})
     (text {:text "Zajal!"
            :fill "white"
            :x 50})
     (graphics {:shape (polygon [0 -15 -10 10 0 5 10 10 0 -15])
                :fill 0
                :line-width 1
                :line-color 0xffffff
                :x 100
                :y 125
                :rotation (* 0.05 t)
                })]))

(sketch 0 #'step #'draw)

Plan

The trajectory of the project is towards a functional approach to creative coding that uses a "virtual scene graph" representation akin to React's. The current implementation is faster than any other similar technology for our purposes and can target the HTML DOM, ThreeJS scene graph, Pixi,js scene graph, or any other mutable tree-like data structure. The current prototype a minimal JavaScript Electron application that loads ClojureScript and sets up a socket REPL and filewatcher.

Using

Electron needs to be installed.

git clone https://github.com/nasser/zajal.git
cd zajal
git checkout atlantic
npm install
electron . zajal/draw/pixi.cljs

Legal

Zajal is a labor of love by Ramsey Nasser. Use it for good, not evil.

Provided under the MIT License.

Support

This project has been generously supported by the following institutions. They believed in it, challenged it, and pushed it forward. Zajal would be nowhere without them, and I thank them all deeply.

• Parsons The New School for Design, where Zajal was my thesis project towards an MFA in Design and Technology
• Karaj Beirut, where Zajal was the focus of my residency
• Eyebeam Art and Technology Center, where Zajal was part of my fellowship exploring code as self expression

zajal's Issues

Time methods

Splitting up time into more explicit seconds and milliseconds methods and possibly keeping time which will default to returning milliseconds, but can be overridden with the time_mode setting

time_mode :seconds
text time # displays 15.65

time_mode :milliseconds
text time # displays 15650

Headless mode in terminal frontend

should be able to run the cli app without a window popping up. all other functionality should be intact.

Color Ranges

Not sure about the syntax, but making color ranges work the way number ranges do with respect to interpolation would be awesome.

c = Color.new(:red)..Color.new(:blue) # ew, clunky syntax

color c * 0.1 # set current color to 10% red, 90% blue
color c * 0.5 # set current color to 50/50 red/blue
color c * 0.9 # set current color to 90% red, 10% blue

Event hooks for internal maintenance

Each module needs hooks to be able to run code at specific events in order to take care of things for the user. For example, ofArduino requires that its update function be called once a frame. Zajal can do this for the user by implementing an update hook in the Hardware module and calling #update on each active Arduino.

Clean up bookeeping variables

Zajal was started with OF 6, and a lot of internal variables weren't exposed, leading to code like this:

/*  internal global variables */
/* TODO move these to Internals? they're all normally accessible... */
VALUE _zj_curve_resolution = INT2FIX(20);
VALUE _zj_circle_resolution = INT2FIX(22);
VALUE _zj_smoothing = Qfalse;
VALUE _zj_alpha_blending = Qfalse;
VALUE _zj_arb_textures = Qtrue;
VALUE _zj_line_width = INT2FIX(1);
VALUE _zj_background_auto = Qtrue;
VALUE _zj_fill = Qtrue;

OF 7 exposes a lot of these values through functions, so this old code should be removed.

Documentation generation

Decide on an in-source documentation syntax and write a doc generator

Serial Support

Support arbitrary serial communication in the Hardware module. Wrap ofSerial.

FBO Support

Wrap ofxFBO. This is important for live coding bare sketches and possibly implementing better smoothing.

HSL/HSB/HSV Support

As mentioned on the forums, this is needed for badass coloring. Proposed syntax is

color_mode :hsb
color 23, 20, 80 # interpreted as HSB

Where the color_mode method takes symbols like :hsb and :rgb to change how the color method works.

The main open question is the ranges of the parameters for HS* colors. Is 0-255 ok?

Range interpolation operator

The * should interpolate over a range instead of %

Audio support

Support audio playback. Wrap ofSoundPlayer.

Vector class

Wrap ofVec2d, 3d. Should have a Point alias.

Named Colors

Should use names from an established palette like the X11 colors or maybe just the HTML colors. Syntax could use Symbols and possibly class constants of the Color class.

# shorthand
color :red
color :purple
color :crimson
color :dark_sea_green

# longhand
my_color = Color.new :red
my_color.use

my_color = Color.new :dark_sea_green
my_color.use

# possible use of class constants. they're a bit chunky.
Color::Red
Color::DarkSeaGreen

Remove animation in reduced mode

Totally confusing

Live coding

It is still confusing in parts, where the past seems to leak into the present.

Possible solution: The so-called "settings methods" like rectangle_mode and background that affect the global state of the interpreter could be implemented differently. When called in the setup event, they should only modify the default settings. Then, at the start of each loop, all settings are reverted to their defaults. This should achieve the expected live-coding behavior of changing settings outside of setup.

RubyGems support

Support RubyGems, including native extensions.

They should be loadable via language syntax like require. Also, they should be manageable with an external tool, like zajal-gem on the command like and a panel in the GUI.

Unit testing

Establish a unit testing strategy

Docs formatting

The documentation should be formatted more like Processing, openFrameworks and Arduino.

circle_resolution should convert arguments to integers

eliminates to_i at the application level

Arduino Support

Support Arduino over Firmata in the Hardware module. Wrap ofArduino.

Font#size

The size of the font object. Stored as a protected variable fontSize in ofTrueTypeFont

Error reporting broken in current build

Might be the FBO rendering, but errors aren't being drawn to the window anymore.

Better argument parsing

Argument parsing has gotten complex, with optional arguments, heavy overloading and hash parameters. rb_scan_args is not enough. There should be a unified internal mechanism to deal with this.

Mathematics#noise should take Vectors

like Graphics methods

Webcam support

Support webcams, wrap ofVideoGrabber

CSS style font loading

Should be able to load fonts off the system as well as from files using CSS style syntax ("Georgia, Times, sans-serif"). This could mean bundling standard fonts with Zajal (sans-serif, serif, monospace etc).

A guess at how font loading would work on a Mac, but missing FT_GetFilePath_From_Mac_ATS_Name symbol in libof.a prevents testing it out:

VALUE fontname; // the name of the font to load as a Ruby string
char path[255]; // the path of the font file to pass to OF's font loading functions
long face_index; // not sure
FT_GetFilePath_From_Mac_ATS_Name(StringValuePtr(fontname), (UInt8*)&path, 255, &face_index);

Support HSB Colors

Not sure about the syntax. This OF forum post could serve as a jumping off point.

Code Style and Naming Conventions

Normalize the naming conventions and style of C functions in the Zajal implementation.

Things to look at

• Functions that can be called from Ruby and C without extra meaningless arguments.
• Hash argument extraction
• Overloaded methods
• C++ class names

key_pressed not being called

key_up and key_down seem fine, but key_pressed is not firing.

Hex Color Syntax

color 0xff02b8 # easy to implement
color 0xf6b # needs a bit more work

size method should set size of window before ofSetupOpenGL is called

This is the correct behavior, and would avoid that nasty flicker at the start of every sketch as the window snaps from arbitrary 500x500 to whatever the first size call is.

ZajalInterpreter.cpp:56 is the relevant line.

Font#draw never uses contours

The implementation should check if the font was created with contours and then use drawStringAsShapes instead of drawString. This is complicated by the fact that bMakeContours, the ofTrueTypeFont variable that keeps track of this, is protected and hence not accessible to the Font#draw implementation.

Defaults mechanism cleanup

The implementation of defaults works, but is very messy internally. Clean up.

C/Ruby Refactoring

C code should be much more minimal, and only implement basic functionality with fixed arguments. Variable arguments and more advanced functionality should be implemented in Ruby. The documentation of C and Ruby code needs to be merged as well.

Better examples

Examples that are neither straight ports from Processing or boring numerical demonstrations are needed to show off the language and act as active documentation.

Rewind videos using shorthand syntax

Shorthand video playing syntax eg video "file.mp4" currently always continues the video when resuming after an interruption. as mentioned on the forums, it is desirable to make the video restart on resumption as well. this should be controllable with video_mode syntax.

Console interception cleanup

Data written to stdout is intercepted by the interpreter so it can be passed to the GUI consols in the Cocoa app. The current implementation breaks in the terminal app though.

Relevant files: lib/zajal/console.rb, src/ZajalInterpreter.cc, frontends/cli/main.cc

Range#each is not the same as Array#each

"supercharged" Array#each functionality does not apply to Range#each

Generic blend mode method

blend_mode :none
blend_mode :alpha
blend_mode :add
blend_mode :subtract
blend_mode :multiply
blend_mode :screen

wrap ofEnableBlendMode

Fonts don't draw UTF8 strings

Could be an OF issue?

vertical and horizontal methods

To draw vertical and horizontal lines

Classes should use same cache as shorthand methods

Image.new "somefile.png" should cache the image just like image "somefile.png" does.

Better smoothing with FBOs

Get rid of code like this

  if(_zj_smoothing == Qtrue && _zj_fill == Qtrue) {
    /*  smoothing is on and the rectangle was filled. draw a smooth outline. */
    ofNoFill();
    ofRect(NUM2DBL(x1), NUM2DBL(y1), NUM2DBL(w), NUM2DBL(h));
    ofFill();

  }

and use FBO multisampling to implement smoothing. render everything to an FBO and then draw that to the window. the smoothing method will control the number of samples in the FBO.

Better defaults implementation

When methods like background, color, line_width and so on are run during the setup event, their behavior should be different than in other events. Instead of setting the current color, for example, color should change the default color that is reset at the start of each frame. This results in the most intuitive user experience.

The current implementation of this logic is a mess and should be improved. Two possible approaches are:

• Create an alternate "setup mode" module with default-setting implementations. This module is loaded at startup then swapped out once drawing starts for the normal implementations.
• Leave method implementations as is then, in between setup and the first update call, figure out the values of these special functions and store them as defaults.

The second approach is the cleanest.

Turn on alpha blending by default

A common source of confusion. Not as big a performance killer as smoothing, and those interested in performance tuning can turn it off.

Multiple file support in live coding

The require method should build a dependency graph of imported files that are then watched for updates. It remains to be decided how the sketch would react to required files being changed (e.g. restarting or sculpting).

Triangle overloads

triangle 20, 40, 50 # draws an equilateral triangle centered at 20, 40 with a radius of 50
triangle 20, 40, 50, 30 # draws an isosceles triangle centered at 20, 40 with a radius of 50 and an angle of 30

Proof of concept iOS support

Make initial steps towards support for iOS devices.

color method should return current color

color 128, 64, 32
color # => [128, 64, 32, 255]

color 32
color # => [32, 32, 32, 255]

color :blue
color # => [0, 0, 255, 255]

when color classes are done, this should return a color class instance

Color class

Colors should get a Color class that will support color arithmetic, interpolation, conversion and so on along with a use or apply method to set it as the current color.

# this
color 240, 128, 64

# would be the same as
my_red = Color.new 240, 128, 64
my_red.use

This would allow more advanced uses of colors that the present color method can't support. The color method would then act as the shorthand to the Color class.