jd557 / minart

A minimal Scala creative coding library targeting JVM/Javascript/Native

Minart

Minart is a minimalistic Scala library for creative coding. It provides simple methods to draw on the screen and play sound.

It's mostly useful for small toy projects or prototypes that deal with generative art or software rendering.

To know more about the library and how to get started check the microsite and the examples.

Features

• JVM, JS and Native support
• Small footprint
• Double buffered canvas
• Integer scaling
• Keyboard and pointer input
• Surface blitting (with multiple blending modes)
• Surface views and infinite planes
• Reading and Writing PPM, BMP and QOI images
• Audio playback (mono)
• Procedural audio generation
• Reading and Writing RTTL, WAV, AIFF and QOA sound files

minart's Issues

Add an arrow abstraction to minart-pure

Right now, there's a huge performance difference between having something like val renderImage: CanvasIO[Unit] and def renderImage(x: Int, y: Int): CanvasIO[Unit], due to the amount of allocations required (at least one CanvasIO is allocated for each pixel).

The easiest solution is to define renderImage as a simple wrapper over an impure implementation (e.g. def renderImage(x: Int, y: Int) = CanvasIO.accessCanvas(canvas => unsafeRenderImage(canvas, x, y))), but I think that it might be possible to avoid some of those problems by adding an Arrow abstraction (see https://github.com/hot-crew/zio-arrow/ for an example).

Fix audio sync problems in Firefox

In some situations (with long audio playback on CPU intensive applications) the audio playback in Firefox starts glitching out.

Some issues that might be interesting to take a look at:

• WebAudio/web-audio-api#2410
• WebAudio/web-audio-api#2467 (includes multiple Firefox bug reports)

Increasing the buffer size and adding a grace start period don't seem to solve the problem.
Capping the framerate seems to work for a bit, but as frames are missed the problem starts to compound.

Add unit tests

While most code is not trivial to test, it should be possible to at least test some basic things, such as the RenderLoop

Add an async RenderLoop

Right now, the RenderLoop mixes the state update and render logic, so it's not possible to cleanly write code with a render thread and a logic thread.

I'm not sure if having a loop with multiple threads is supported by all platforms, so this might require splitting the RenderLoop into a SyncRenderLoop and an AsyncRenderLoop.

Remove dependency on scalanative-graphics-bindings

#50 required bundling a custom version of https://github.com/regb/scalanative-graphics-bindings/.

Since Minart doesn't use that much from SDL, it might be a better idea to simply declare the used SDL bindings by on another file and remove the library. I have a feeling that it might be possible to get some performance improvements by exploiting how some bindings are defined.

Add Android Support

Add a JavaFX backend.

According to https://github.com/makingthematrix/scala_android_dummyapp, this might allow minart apps to run on Android.

Add audio support

Audio support is still missing.

One of the main blockers for this is that, to fit into the "procedural art engine" philosophy, so it should be possible to play proceduraly generated audio on each sound channel.

Something like minart-image but for audio would also hepl a lot. Sound formats don't seem that easy to implement, but it might not be too hard to support AIFF, some limited WAV (without any compression) and maybe MOD files.

See:

• https://aneescraftsmanship.com/wav-file-format/
• https://www.eblong.com/zarf/blorb/mod-spec.txt

Add instructions on how to compile native applications on windows

Native applications on windows require some custom flags to correctly link SDL and the SDL.dll files.

Provide cross-platform ExecutionContext

Provide a cross-platform execution context.

JVM: scala.concurrent.ExecutionContext.global
JS: org.scalajs.macrotaskexecutor.MacrotaskExecutor
Native: Custom EC similar to scala.scala-native.ExecutionContext.QueueExecutionContext, that runs one (or more?) tasks on each loop iteration. When the loop exits, it should just push the queued tasks to the global EC.

Fix cross-compilation in examples

Fix the artifact path on the examples to allow compilation with scala-js and scala-native.

Looks like I should have used //> using lib "eu.joaocosta::minart::0.4.0" instead of //> using lib "eu.joaocosta::minart:0.4.0".

Stereo sound output

Right now, it's only possible to play and manipulate Mono sound.

It should be possible to at least use stereo clips.

Maybe more channels would be ideal, although I don't think it's worth going over 6 channels (surround 5.1).
Still, if this for some reason is cumbersome , let's go first with just 2 or 3 channels and call it a day.

Publish scaladoc

It would be nice if the scaladoc showed up in scaladex.

Fix Java Renderling in Linux

There are apparently some non-deterministic rendering bugs in linux.

This is mostly apparent on the examples-colorsquareJVM example, which most times won't render. It also seems that sometimes the rendering is not correctly aligned.

I've had this problem in Pop OS!

I'm not sure if this is related to the tiling manager, but I doubt it, since it's probably the same problem as https://gist.github.com/JD557/6dabe2af946bba8887be60f9b75a3380

Add more keys to the keyboard input

Right now, the keyboard input only supports arrow keys.

While I don't think that this project needs to be as complete as something like SDL, it would be nice to at least support:

• All latin letters
• Common whitespace characters (enter, tab, space, backspace, and delete)
• All numbers
• Common modifiers (shift, option and control)

Convert examples to scala-cli

Convert the examples to use scala-cli an to work a bit more like a tutorial.

It's important that the examples can still be compiled by the CI, to make sure that they are up to date.

Set sbt versionScheme

Just a reminder to configure https://www.scala-sbt.org/1.x/docs/Publishing.html#Version+scheme

I guess either strict or semver-spec will do.

Explore ammonite integration/live-coding possibilities

Minart can be called from ammonite, which can lead to some interesting live-coding usages.

I haven't thought about this in detail, but I'm thinking about adding a new minart-live package with some utilities such a special RenderLoop, like:

class LiveRenderLoop[S](innerRenderLoop: RenderLoop) {
  private[this] var renderFrame: (Canvas, S) => S = _._2

  def changeRenderFrame(newRenderFrame:  (Canvas, S) => S): Unit

  def init(canvasManager: CanvasManager, initialState: S, frameRate: FrameRate) {
    innerRenderLoop.infiniteRenderLoop(canvasManager, initialState, {(c, s) => renderFrame(c, s)}, frameRate)
  }
}

This would allow a render loop to keep the state and run in the background, while the rendering logic was updated in the REPL.

Fix janky rendering on scaled HTML canvas

Calling createImageBitmap can make the rendering of scaled canvas janky, especially on Firefox.

I think that, at least for non-full screen canvas, this can be improved by scaling the canvas via CSS (see https://github.com/JD557/WakuWakuBuffet/blob/master/src/html/index.html#L21)

Fix artifact publishing

Right now, neither minart-pure nor the scala-native release are being published.

Fixing the minart-pure package should be trivial. Fixing the native release is a bit more cumbersome, due to the lack of scala 2.12/2.13 support.

Support resizable windows

Ideally one should be able to define a max/min width/height in the canvas settings, and that would allow it to be resized.

It would be up to the application to check the width and height at the beginning of every frame to render things correctly.

I think forcing the window to keep the aspect ratio can be out of scope for now. It's more important to ensure that operations like redraw() don't break in the middle of a resize.

Also, I'm not sure how this would work on JS. I think it should be possible, but maybe it's simpler to just focus on JVM and Native and then leave JS to a future version.

Add default implementations to each platform

While I think that it's important to manually choose the RenderLoop strategy and Canvas, it's a pain to have to change the code for each backend.

Ideally, each platform should have provide a default implementation.

Add Skia backends

It might be interesting to try a Skia backend both in the JVM (using skija) and Native backends, to see if there are some performance benifits.

Calling this from the JVM seems to require some system dependent libs, and I think that Skia doesn't handle events/window creation (so SDL would still be needed in the native backend). Those might be a dealbreaker.

Full-screen support

It would be nice to have fullscreen support, even if that would have to be defined at canvas creation.

I'm not sure if it's possible to go fullscreen in javascript without user interaction, though.

Add minart-pure interop with cats-effect/zio

It would be nice if minart-pure could be used with cats-effect and zio with minimal boilerplate.
I want to keep that library pretty small, so I'm thinking of adding a minart-pure-cats and a minart-pure-zio.

The CanvasIO part should be pretty easy, but I'm not yet sure how to port the RenderLoop logic.

Mouse input support

Add something like the KeyboardInput interface, but for mouse inputs.

It would be pretty nice if this could also get touch inputs when compiling with scala.js.

Alpha channel support

Right now there's pretty much no support for transparencies.

The Color class only supports 24-bit colors, although it would be trivial to extend it to 32-bit (it kind of already does).

I'm not sure if the blit operation should take the alpha into account, due to performance reasons. Unlike the mask, here we would need to check every pixel to know if we could use an optimized loop or not.

Also, it might be interesting to support multiple blending modes (e.g. "normal" blending, premultiplied alpha, alpha test...).

One initial way to support this could be to just add the alpha channel to Color and not do anything else. An end user could then handle the blending manually (e.g. with SurfaceView#zipWith).
I don't think this is that bad, as long as we had an analogous to blit on SurfaceView (e.g. something like SufaceView#zipWithAt(surface: Surface)(x: Int, y: Int)(f: (Color, Color) => Color)). I need to think how this would look like (maybe there's already a name for an operation like that).

Add GamePad support

Similar to the touch support, some GamePad support could be included in the KeyboardInput

Improve FullScreen ergonomics

Full screen is kind of a pain to use.

It should be easy to go full screen AND fill the screen (without having to manually compute the scale).
Also, leaving the fullscreen with ESC on JavaScript can leave things in a weird state.

Better Scastie support

It would be nice to have a cleaner API for demos like this:

https://scastie.scala-lang.org/mt95s4WbSgqip5YezoLgjQ

Uncouple keyboard clearing logic from canvas clearing logic

Right now, the KeyboardInput is cleared when the canvas is cleared.

This might not always be desired (e.g. for performance reasons, a library might now want to clear the canvas).

Replace scala.math with java.lang.math

According to Daniel Spiewak - Micro-Optimization for Fun and Profit, there should be quite a performance difference between these two.

There are a lot of performance sensitive places using math.min/math.max, so it might be interesting to check if there's indeed a difference.

Surface#fillRegion implementations should check bounds

Right now the Surface#fillRegion implementations are not checking the bounds, so it's very easy to crash an application with them.

Simply adjusting the w and h values before entering the loop should be enough

Add a subproject for each example

The current example structure is not very convenient and it doesn't allow for multiple javascript examples.

Having a subproject for each example should fix this.

Add automatic frameskip support

I think it should be easy to add a frameskip definition to the RenderLoop. If some frame takes too long to render, the RenderLoop just has to switch the canvas to a FrameSkipCanvas that ignores any rendering operations (clear(Backbuffer), putPixel,redraw, ...) and applies all other operations on the real canvas.

This might need some smart logic to avoid constant frameskipping (maybe setting a maxFrameskip instead of frameskip).

Add more keys

The KeyboardInput.Key enum is quite lackluster.

It should be possible to at least write a URL, so I would say the following are a must: (., ,, /, -).
Other characters like ?, &, _ and = are also desirable, but I need to think how to handle the shift

Update scala.js to 1.x

Add bulk operations

There might be some performance benefits of using bulk operations.

At least having a getBackbuffer to avoid multiple calls to getBackbufferPixel would be much faster on some backends.

I would also like to have something like blitBuffer(x: Int, y: Int, buffer: Buffer), but I'm not yet sure what the buffer should look like (should it support transparent pixels? what about alpha blending?)

Update scala-native to 0.4

As stated in #8, there's no easy way to add unit tests and keep scala-native 0.3.x.

Unfortunately, this is blocked by regb/scalanative-graphics-bindings#2