Comments (18)
Yes, after a reevaluation, I am thinking that maplibre plus built-in support for the geo
types/encodings may be the way to go. deck.gl looks very nice but my understanding is that it builds on top of of a map library, so we have to start with a map library first.
from kino.
Hey @sweco-semtne, the points you listed are accurate. There are two points sides to components.
The first one is the output specification (currently documented in Kino.Output
) - these are the recognised formats and on Livebook side this translates to a piece of UI for the given type.
The other side is rendering the given Elixir term into that specific output (implementing Kino.Render
), in most cases we have dedicated Kino.*
structs.
The output format may be more generic, for example we have the :table_dynamic
format (rendered as a table in Livebook), but we have several components using this: Kino.ETS
, Kino.DataTable
, Kino.Ecto
. A server process (on Kino side) is only necessary for components that need to dynamically exchange data with Livebook, in other cases we simply convert structs to static output like {:vega_lite, %{plot definition here}}
.
As for the UI, it depends on the case, we currently use JS only for rendering Vega-Lite plots (and that does require a hook).
[1/2]
from kino.
Hi everyone! We have pushed an initial implementation of Kino.JS
, which allows anyone to create a Kino widget! So go ahead with your wildest ideas! :D You can see #64 as an example.
from kino.
[2/2]
We are definitely open to adding new components, but in cases where it requires integrating an external JS package we need to be careful, because every such package may significantly increase the bundle size. What's the exact idea behind the map component? What kind of Elixir data would that render? Also, see a relevant question here livebook-dev/livebook#305 :)
from kino.
Looking at Folium, I can see that they manipulate the map using leaflet.js, so based on the Python code they generate some JavaScript from a template and then let this JavaScript run. So to do something similar we would need either:
- let the output be arbitrary HTML/JS, which we don't want given the security risks
- have a lot of custom JS on Livebook side, or rather create a package that takes a JSON specification and renders the map; however it would need to be something standardized like Vega-Lite
from kino.
@jonatanklosko thank you for a very thorough answer.
I was thinking that the output would basically be wrapping the concept of a data table in the sense that it could take a table as input along with a definition of which columns contains the geographical coordinates.
Or a data frame similar to GeoPandas.
The reason for having a Server is to be able to push live data on to the map canvas for for instance GPS positions.
It could also possible be used to filter tables on geography if a map input component is built.
That would need additional libraries like turf to do the spatial filtering.
I'm not looking to use leaflet but instead the MapLibre library which is a fork of Mapbox 1.x
The library has the concept of data sources that can be updated with a setData function that would map well to hook.
Under the hood the input data could be converted to GeoJSON features.
I agree that some kind of specification on what background maps, styles etc to use should be formalized but I'm not sure there is any standard worth sticking to. There is the old OGC Web Map Context and several other proprietary means of serializing the map context.
I'll try to come up with an example flow to better describe what I'm thinking.
Again, thank you for taking the time to respond and also investigate!
from kino.
After some discussions with @jonatanklosko and @thbar yesterday, we think it is work adding some support for map rendering in Livebook/Kino. We mentioned VegaLite before but it seems Vega does not have support for tiling yet.
My initial suggestion is to support GeoJSON rendering exclusively, so the API would rather evolve around GeoJSON, with some basic configuration for picking the tile provider. Would that be helpful to you @thbar and @sweco-semtne?
Also, does anyone have a rundown on the differences or solid preferences towards MapLibre or Leaflet?
from kino.
Hi @josevalim
That's great news.
GeoJSON as data format is a good middleground.
I've worked with both OpenLayers, Leaflet and MapLibre(MapBox 1.0) and for me personally I like the data source + multiple layer abstraction in MapLibre.
I would say that Leaflet and OL has a richer set of plugins. But for this use case I would go with MapLibre.
Or maybe take a look at vis.gl and the tools around luma?
On a side note, we've been using
Liveview with Broadway to push thousands of vehicle position updates to a Mapbox/MapLibre map without any performance issues. So thank you very much for all of these fantastic libraries/frameworks!
from kino.
The Style specification for MapLibre can be used as a basis for a map "context".
The GeoJSON specifies the data but there still is a need to specify how to visualize it, ie layer order, colors, shapes etc
https://maplibre.org/maplibre-gl-js-docs/style-spec/
from kino.
My initial suggestion is to support GeoJSON rendering exclusively, so the API would rather evolve around GeoJSON, with some basic configuration for picking the tile provider. Would that be helpful to you(...) ?
I believe a GeoJSON layer + tile background rendering would be a natural first step, and would probably take us quite far already. It would definitely be useful to the various data explorations we do at https://transport.data.gouv.fr.
Let me share extra ideas that could be useful for later (these are things we are also looking at) ; these are currently overkill, but keeping them as options in mind could help design something that can go further later:
- Support code generating GeoJSON on the fly, based on the currently viewed area (bounding box), so that we can have more dynamic scenarios in the future (clustering of large data, moving around etc)
- Support for MVT (MapBox Vector Tiles), which are protobuf-compressed dynamically generated data (for the main data, not the background) ; this supports much larger datasets, and I am mentioning this because we see more large-data stuff coming out recently (in the spirit of Explorer, Nx/Axon etc). I have not found native support for this in Elixir yet, but will most likely dive into it (interesting link: https://github.com/urbica/martin)
Also, does anyone have a rundown on the differences or solid preferences towards MapLibre or Leaflet?
I am not familiar at all with MapLibre. At https://transport.data.gouv.fr, we currently use LeafLet and mostly quite happy with it.
I'll add a couple of interesting links to the discussion:
- https://github.com/leaflet-extras/leaflet-providers -> having different tiling servers support will be helpful
- http://leaflet-extras.github.io/leaflet-providers/preview/index.html -> demo of the above
- https://github.com/topojson/topojson -> I have seen this used as layer with VegaLite, a middle-ground of optimization
from kino.
@thbar MapLibre is the community fork of Mapbox pre 2.0. So it supports vector tiles out of the box. Or did you mean the library should support generating it in Elixir-land?
I was also thinking about having a map input. Is that what you mean with the bounding box?
And it would be interesting to wrap a data table with a function to transform it to GeoJSON.
from kino.
MapLibre is the community fork of Mapbox pre 2.0. So it supports vector tiles out of the box.
Neat, thanks for clarifying this!
Or did you mean the library should support generating it in Elixir-land?
This is a great question! Yes, I meant "Support /Elixir/ code generating GeoJSON on the fly", definitely. Having "server-side" (Elixir-land) data generation will allow to process large amounts of data, cluster them if needed, clip as needed, and only send what is relevant to the rendering part.
Is that what you mean with the bounding box?
By bounding box I meant: the end user could in the future be able to control the map with arrows, and the resulting viewport (lat/lon top left and bottom right) gives coordinates, which can be "sent back" to the Elixir part, so that it only generates the right bits of data.
If needed, I'll emphasise that a first version with no clipping of any sort, "just" rendering an Elixir-generated GeoJSON on a map, and letting the user configure a tiling background where the coordinates would automatically use the same referential and be well aligned with the GeoJSON, would already provide a huge value (the bounding box idea is more fuel for thoughts for later in my head!).
from kino.
I'll just add one more link for inspiration: https://geopandas.org/docs/user_guide/interactive_mapping.html
The GeoPandas library just introduced the explore function that creates an interactive leaflet map based on a dataframe.
from kino.
Very interested in this conversation as I work in the web mapping / data science field where I use and develop mapping components regularly. New to Elixir and loving livebook/kino so I would like to add deck.gl to the conversation. deck.gl supports a declarative system for describing layers and their props, and this declarative API can be trivially exposed via the JSON API. Hoping this may make it easier to integrate.
My opinion is a webgl-based mapping library (maplibre, mapbox v1, or deck.gl) would improve livebook more than a leaflet-based solution.
see bindings for python (pydeck), Dash Deck , R (mapdeck), Vega(vega-deck.gl) at https://github.com/visgl/deck.gl
Brent
from kino.
Currently in elixir there are no libraries equivalent to Geopandas, which in its internal development makes use of other mature and stable projects such as shapely to work with geometries, pandas for DataFrame, etc.
The first thing would be to create a similar project in elixir that will enhance the use of the platform in this niche. There are two real options:
1- Being Rust a platform that has projects similar to python in this niche, create elixir NIFs with Rustler that brings functionalities to the elixir ecosystem and use them in a transparent way. Creating similar projects from scratch would be very expensive.
2- Find an efficient way to integrate python into livebook and call these functionalities without much cost to the system
Later it would be used to display Deck.gl or Maplibrejs, due to its webgl rendering capabilities.
from kino.
Being new to Elixir, I have been spending some time getting up-to-speed with the Elixir geospatial libraries. Wow! there are some really good libraries available to build the core of a geospatial data handling/indexing solution.
geo - Encodes and decodes WKB, WKT, and GeoJSON formats. Will also handle standard geometry formats from databases (PostGIS, MySQL)
topo - Geometry library for determining spatial relationships between geometries
envelope - A library for calculating envelopes of geometries and tools to compare them.
spatial_map - Geospatial feature storage for fast intersection checks
Seems like pkinney would be someone that might have some good ideas into this conversation since he developed quite a few of these libraries that cover core geospatial data formats and storage/indexing requirements.
So we already have the libraries above to use within livebook. The missing part is still the map feature and tile rendering - A package that takes a JSON specification and renders the map like @jonatanklosko mentioned above seems like a really good solution when you take into account the existing elixir geo libraries.
Brent
from kino.
Maplibre.js and geo type support would be amazing. This is really great news.
from kino.
Also just came across this component which is the beginnings of a simple SVG map renderer that handles OSM map tiles. https://hex.pm/packages/live_map
from kino.
Related Issues (20)
- Bug happening when running remote execution smart cell
- Add callback to `Kino.interrupt!` HOT 5
- Postgre columns of type array can crash SQL smart cells.
- Support `:list` in Kino Table
- Feature request: Multi-select HOT 2
- Refactoring DataTable assets
- Allow Kino.DataTable to be updated
- Rerender an input will make it losing focus HOT 1
- Expand tuples in Kino.Tree by default HOT 7
- Update to Glide Data Grid v6
- Reevaluating a cell containing Kino.start_child! sometimes errors with "Already started" HOT 5
- Kino.Layout.grid hangs forever when you pass another Kino to it HOT 1
- Kino input field not appearing (bug?) HOT 1
- Add a process tracing JS.Live widget
- Add Kino.LiveFrame HOT 1
- Add clients monitoring
- Custom Message in `Kino.Process.seq_trace` HOT 2
- Sortable columns for Kino.DataTable
- Add Kino.Hub.on_join HOT 8
- Trouble Switching Tabs HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kino.