livebook-dev / kino

Client-driven interactive widgets for Livebook

Kino is the library used by Livebook to render rich and interactive output directly from your Elixir code. You can learn more about Kino by installing Livebook and heading into the "Learn" section of your sidebar.

Installation

To bring Kino to Livebook all you need to do is Mix.install/2:

Mix.install([
  {:kino, "~> 0.12.3"}
])

Additionally, there are packages with components designed for a specific use cases. The officially supported ones are:

• kino_bumblebee - for Bumblebee integration
• kino_db - for database integrations
• kino_explorer - for Explorer integration
• kino_maplibre - for map plotting
• kino_slack - for Slack integration
• kino_vega_lite - for data charting

License

Copyright (C) 2021 Dashbit

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

kino's Issues

Integration with Explorer

Add clickable rows to dynamic tables

One idea is to introduce a meta attribute, called :link@kino, that when present makes the row clickable.

Mechanism to pull Kino.Input values into a module

Hello friends 👋🏼

We started using Livebook for technical interviews at Simplebet, and its a really awesome experience! Both candidates so far have praised it for being nicer than other alternatives.

While preparing the interview, I wanted to combine a text input with usage of ExUnit, so that I could provide a large test input with all benefits of ExUnit. Ultimately I realized that Kino.Input does not support the ability to read at the top level and make that value available to a compiled module, which is not really a suprise. However, I thought that this might be a nice feature as I don't imagine the usecase is that unique. See this elixir slack: https://elixir-lang.slack.com/archives/C01UKP7DG7K/p1641260554068900

Here's an example of the problem I'm trying to describe

ExUnit.start()
feed = Kino.Input.textarea("feed")

defmodule Test do
  use ExUnit.Case

  setup do
    # Won't work because of the variable scope
    %{data: Kino.Input.read(feed)}
  end

  test "it works", %{data: data} do
    
  end
end

ExUnit.run()

A solution we found was to put into :persistent_term and pull out in the test

ExUnit.start()
feed = Kino.Input.textarea("feed")
:persistent_term.put(:input, Kino.Input.read(feed))

defmodule Test do
  use ExUnit.Case

  setup do
    %{data: :persistent_term.get(:input)}
  end

  test "it works", %{data: data} do
    
  end
end

ExUnit.run()

However, I'd love to see an out of the box solution, where you can read from named inputs. Something like:

ExUnit.start()
_feed = Kino.Input.textarea("feed")

defmodule Test do
  use ExUnit.Case

  setup do
    %{data: Kino.Input.read("feed")}
  end

  test "it works", %{data: data} do
    
  end
end

ExUnit.run()

Is it possible to support an API like this? Maybe there's a better way and theres just a documentation update? Please advise and I'm happy to make a PR either way 😄

`Kino.Input.read` does not get the current value when used inside `Kino.VegaLite.periodically`

I wanted to use a checkbox to control what is happening inside a Kino.VegaLite.periodically but Kino.Input.read always returns the value that was set when the iteration started.

These are the Livebook code cells:

frame = Kino.Frame.new()

checkbox = Kino.Input.checkbox("My Checkbox")

Kino.VegaLite.periodically(frame, 1000, 1, fn i ->
  IO.inspect Kino.Input.read(checkbox )

  {:cont, i + 1}
end)

If run as is, the initial checkbox value will be false which will be printed to the output. Toggling the checkbox does not reflect in the new output.

Add `Kino.Process.trace/2`

Another idea is Kino.Process.trace/2:

Kino.Process.trace(pid, fn ->

end)

The above will trace all messages received by pid (or list of pids) and build a sequence diagram.

Producing sound

It would be interesting if Kino could support producing sound for notifications or playing midi or sounds loaded from the file system.

Add origin monitoring

Control and input events include client origin, we should be able to monitor it. The same mechanism should cover ctx.origin in Kino.JS.Live.

Add support for rendering LiveViews

This could make it a great playground for teaching LiveView. This will require two features on the Phoenix side:

1. The ability to sanitize HTML from LiveView side
2. The ability in Phoenix to spawn channels in remote nodes

Probably via a socket configuration here: https://github.com/phoenixframework/phoenix/blob/master/lib/phoenix/channel/server.ex#L26

Widgets that export computational representations

There is an excellent paper and talk on the matter. Examples we want to consider in the future are:

1. API widget - a widget to the API requests which then emits a library code that does such request

2. Confusion widget - a widget for emitting confusion matrices to compare ML results

3. Chart widget - a widget that builds charts on data and then emits a vegalite specification (Deepnote has something equivalent)

4. Table widget - a widget where you can change the table and it reflects in the dataframe/query

5. SQL widget - a widget that allows you to write SQL and then populates a variable (Hex does something similar)

The chart widget is likely the most straightforward starting point for us (and also one of the most impressive ones).

Introduce a geo builder smart cell

Add file input

Kino.DataTable does not preserve the order of columns when specified with option `:keys`

The documentation of Kino.DataTable.new/2 mentions that the keys also dictate the order of the rendered columns:

:keys - a list of keys to include in the table for each record. The order is reflected in the rendered table. Optional

However, the order of keys does not seem to be reflected in the table. Using this simple example shows the issue:

data = [%{one: 1, two: 2}, %{one: 3, two: 4}]
keys = [:two, :one]
Kino.DataTable.new(data, keys: keys)

Which renders the table with columns :one and :two (and not in the expected order :two and :one).

Looking at the implementation, the delegation to Table.to_rows_with_info/2 makes no guarantees about the order of columns.

Either the documentation promises too much, or the order should be preserved somehow.

Add a table function that works with any enums

As long as we have an enum of tuples or enum of maps, we can render them with pagination and what not.

For maps, field ordering is going to be sorted. For keywords, we respect the keyword ordering. For tuples, it is the indexes as done in ETS. And we can also accept options such as :fields when calling DataTable/data_table.

Pandas-esque `describe` method

Are there any plans for a pandas-like describe method that would combine some of the aggregate methods (count, mean, standard deviation, peak) into a table to give you a rough understanding of the data, or is that out of scope of this library?

Provide a mechanism to abort execution

It should also be supported within branched sections and only interrupt that branch.

@jonatanklosko mentioned that Process.exit may be enough.

Print supervision trees

If a process represents a supervisor (based on Process.info and Supervisor.which_children), we can use Kino.Markdown with Mermaid to print the supervision tree. For example:

graph TD
    A(#PID<0.23.3>) --> B(Foo.Bar.Sup)
    B ---> C(Foo.Baz.Sup)
    C --->D(One)
    C --->E(#PID<1.2.3>)
    C --->F(Three)
    B -.->F
    style A stroke-width:3px

The root has a different stroke to mark the initial PID. Dotted lines can used for links.

Automatic image detection

We can look at a binary and if it looks like an image, we can automatically show it as said image.

Grid of Frames

It'd be great if there was a way to show outputs side-by-side natively somehow.
The easiest way I can think of right now would be a Grid or Row that exposes several frames that can be rendered into.
The actual display would then show these frames side-by-side.

I really like the Matrix example from "Plotting with VegaLite" and think having a way to do this with arbitrary component would be great
For reference, said example:

(From what I can tell there is no way to implement this externally, but please correct me if there is a way)
(I'd be happy to try and contribute something here if desired)

Support for per-user frames

We can support an additional option in Kino.Frame.render/2 for targeting a specific user. Events generally have an origin field that represents the user, so we would use it as the target. This requires changes to Livebook as well.

In combination with Kino.Control.form/2 this will allow for a per-user request-response flow, while having everyone in the same session. An example would be form with parameters, submitting the form results in a query and the result is rendered to whoever submitted the form.

More tables more strict

Today tables are very flexible in that they can render mixed data types, but that's going to conflict with some of the features we want to add in the future. So I think we should do a more explicit modelling, where we choose the data type based on the first entry of the list, as well as the fields, and assume everything is there.

For ETS, we should have a single column, which is the row, and print them as tuples.

Support structs in data_table

We should not show fields starting with underscore by default. We could add a show_underscored: true if someone wants to disable this behaviour. This will allow us to cleanly handle things like __struct__ and __meta__ fields.

Allow sensitive data to be used but not persisted to the .livemd file

For making smart cells like the GitHub GraphQL Smartcell it would be great if we could declare some attributes as sensitive such that they are passed around internally but not persisted to the .livemd file if the user is saving their live book to disk.

How to style Kino.JS and Kino.JS.Live components?

Hi all, forgive me if this isn't the right place to ask this. Please remove the issue if it's not.

How can I effectively style Kino.JS.Live components?

So far I've tried:

• Inline styling ✔
• appending a child to the head. ✔

      const style = document.createElement('style');
      style.type = 'text/css';
      style.innerHTML = '.cssClass { color: red; }';
      document.getElementsByTagName('head')[0].appendChild(style);

• await ctx.importCSS("https://cdn.tailwindcss.com"); ❌
• reusing existing class names applied in the livebook ❌

If it's helpful, I would be happy to create an example in the documentation of the recommended method!

Introduce Kino.Tab

We should change some components to render multiple tabs by default. Like a PID/atom with supervision tree and so forth.

VegaLite.push should validate the data

VegaLite.push does not validate the data, and if we send invalid data, it crashes the graph rendering.

My suggestion would be to call Map.new(arg) on whatever we pass to push. This will both validate it and allow us to pass keyword lists in.

Thoughts @jonatanklosko?

Add data_chart

The goal of data_chart is to receive some data and we will automatically generate a chart with it with options to control how the chart is rendered. Then we can export the chart to Elixir VegaLite source code.

Introduce Kino.start_child

Today, if we call start_link in a cell, as we reevaluate the cell, we will leak processes. The idea is to expose a Kino.start_child, that accepts a child specification, and starts it associated to the current cell. This way, if the cell needs to be re-evaluated, we terminate the processes before re-evaluating it.

Add export to JSON to the table functionality

This is likely a Livebook feature, but opening here in case we need changes to the protocol. The goal is to allow data to be exported, like we do for VegaLite, but for tables.

We could also export CSVs but this may be trickier, because some tables, like ETS, do not have a fixed number of columns.

Add CSV parser For smart cell

The cell Will be an editor but it will also support drag and dropping a file. We will use NimbleCSV for parsing. Although we will most likely need to add a struct that will be returned by the streaming API.

There is an open question all what to do we file uploads. We will write it to disk first? We will load the contents into the editor? We will persist its contents in the livemd file?

VegaLite axis titles disappear when pushing data

I'm rendering a VegaLite graph with axis titles:

widget =
  Vl.new(width: 600, height: 400)
  |> Vl.mark(:line, point: true)
  |> Vl.encode_field(:x, "iteration", type: :quantitative, title: "Iteration")
  |> Vl.encode_field(:y, "time", type: :quantitative, title: "Time (ms)")
  |> Kino.VegaLite.new()

I'm then pushing data to it, in the same way as the demo LiveBook does:

for i <- 1..iterations do
  {usecs, :ok} = :timer.tc(make_request)
  millisecs = usecs/1000
  Kino.VegaLite.push(widget, %{iteration: i, time: millisecs})
end

This works, but the graph seems to lose the axis titles as soon as data is pushed - the x axis title disappears completely, and the y axis title is split vertically making it illegible.

See short video demonstrating this:

vegalite-kino-missing-titles.mov

I installed using:

Mix.install([
  {:kino, "~> 0.2.0"},
  {:vega_lite, "~> 0.1.0"},
  ...
])

Kino outside of LiveBook?

I am finding myself using LiveBook for data exploration more and more, and at one point a question comes up: how to move my findings & code into a backoffice page.

Typically: I have found something insightful via exploration, and would like to easily convert that to something that is a part of our app.

I wonder if Kino could be used outside of LiveBook easily? Or if there are plans to make this easy?

Thanks!

Kino not rendering graph, shows inspect result instead

On testing this code https://hexdocs.pm/kino/Kino.html#module-kino-datatable

On windows10/localhost shows the result as %Kino.DataTable{pid: #PID<0.1496.0>}

The same code, on cloud-vm/linux renders the vegalite correctly.

Make communication upstream use a pubsub

VegaLite does not need to track IDs, instead it will send a message to the group leader with a pubsub topic that is sent to Livebook.

Add configurable timeout to query smart cell

The default timeout of 5s may be too low for some workflows. We should increase it to 30s and make that configurable as part of the smart cell.

Add Dynamic Image widget

Hi Dashbit friends.

I am experimenting with a dynamic image widget and wondering whether it'd make sense to submit a pull request to have this be part of Kino / Livebook.

Rationale

Initially I needed it show a video feed in one of my nerves notebooks with:

video = fn y ->
  # Picam  captures stills using the camera module on Raspberry Pi
  Kino.Image.new(Picam.next_frame, :jpeg)
  |> Kino.render

  Process.sleep(100)
  y.(y)
end

video.(video)

My intention was to take a photo every 100ms, then update an image output cell with that image, but rendering Kino.Image appends a new image to the notebook instead.

There are though numerous use-cases, especially educational, where a dynamic image can be used with an svg to visualise an algorithm in an animated fashion. I'm am already working on visualising towers of hanoi and game of life.

Demo

marquee.mp4

Implementation

Currently, my implementation consists of a LivebookWeb.Output.ImageLive view in livebook and a Kino.ImageDynamic GenServer and minor changes in a few other files.

Please let me know your thoughts and thank you for Livebook and Kino 🙌

Better image support

There are some ideas flying around:

1. Support Nx.Tensor in Kino.Image by using a custom content type that is loaded into a canvas
2. Have a canvas input
3. Have a camera input
4. Have a specific image tensor upload, where the image is loaded into a canvas to be converted into pixels (and potentially resized)

Make Kino.render/1 return do not display result

This is theoretically a breaking change, so we need to release v0.3.0 and update the existing explore notebooks afterwards.

Provide a default solution for map/geo printing

It may be Vega, deck.gl, maplibre, or something else. If multiple renderings with multiple formats are supported, maybe we could provide something similar to the chart builder. There was a discussion on #44.

Add a form builder smart cell

Drag and drop input components to build a Kino.Control.form(...) with some sort of layout.

Do not show fields starting with underscore by default unless included

We could also have an option called show_underscored which you can set to true.

sorting status in data tables is not passed to assistive technologies

When viewing a Kino data table with a screen reader, it's not possible to see which column the table is sorted by, or whether the sort is increasing or decreasing. This information should probably be included in the column header as a textual label.

The column headers should also be made reachable with the keyboard and turned into links, buttons or some other clickable control.

Handle tables with mixed column number (as found in ETS)

[Question] - Steps to create new component

Hi!
I'm not sure where to direct my question so apologies if this isn't the correct forum or place.

I'm trying to add a dynamic map component to Kino to interact with geographical data. It is loosely based on Folium

I'm trying to understand the flow and extension points in Kino and Livebook.
My understanding so far:

Kino

• add a new type and mapping in Kino.Output

• add a defimpl in Kino.Render (to_livebook)

• implement a server in Kino

Livebook

• introduce a LiveComponent in Livebook for rendering the output

• create a hook for the client side JS part of it

• add a type in Livebook.Notebook.Cell.Elixir

Are there any other points to consider?

Also I would like to thank all of your building such a fantastic tool! Keep up the good work!

kino 0.2.2 and ecto

After updating to kino 0.2.2, I started getting compile errors:

==> kino
Compiling 13 files (.ex)

== Compilation error in file lib/kino/ecto.ex ==
** (CompileError) lib/kino/ecto.ex:28: module Ecto.Query is not loaded and could not be found

The workaround for this isn't that hard, depend on :ecto_sql, but I wasn't sure if you wanted to make :ecto_sql a hard dependency or something else.

This also looks like it breaks the kino examples in Livebook right now as well.

Automatically render benchee results

Benchee.run returns a %Benchee.Suite{} struct with a ton of numbers. We can implement the Kino.Render protocol to show some plots. As for what to show, there's benchee_html to take inspiration from. Maybe alternatively we could embed the result of benchee_html.

Add a Kino.Download widget

The goal is to have a button that, when the user clicks, it downloads a file with dynamically generated content, such as: Kino.Download.start("filename.json", fn -> file_contents end). We can get the content type from the filename.

Add Kino.inspect/2 and return input on Kino.render/1

Kino.inspect/2 should behave just like IO.inspect, but with color.

Reduce the amount of generated modules

We want to keep Kino's footprint small as that is going to be used by almost all notebooks. Today it generates too many modules but all of the Kino internal modules can likely be refactored to use one internal Kino.Output struct.

Make Kino.Input compatible with CLI

As a follow up to #53 we should detect when not in Livebook and read+parse the given input.

Feature Idea: Kino.ignore to avoid an Elixir cell outputting anything

Sometimes an Elixir cell in a LiveBook has a pretty meaningless output and it's just the side effects that you want - eg. setting variables or rendering something else. One option would be to put an :ok or nil at the end so that becomes the output, but that's a bit unclear about intent and would be a bit annoying with the export to script feature that's been talked about.

One example is where you use Kino.render() with a VegaLite graph and then add data to it in the loop - I end up with output of a long list like `[:ok, :ok, ...]

Would it make sense to instead be able to pipe to a Kino function that returns what's piped in but renders as nothing?

Two examples where I'd like to use this:

url = IO.gets("URL: ") |> String.trim()
{iterations, _} = IO.gets("Iterations: ") |> String.trim() |> Integer.parse() |> Kino.ignore

And

widget =
  Vl.new(width: 600, height: 400, title: "Performance for #{url}")
  ...
  |> Kino.VegaLite.new()
  |> Kino.render()

(for i <- 1..iterations do
  point = %{iteration: i, ...}
  Kino.VegaLite.push(widget, point)
end) |> Kino.ignore