tsupinie / autumnplot-gl

Hardware-accelerated geospatial data plotting in the browser

autumnplot-gl

Hardware-accelerated geospatial data plotting in the browser

Links

Github | API docs | NPM

What is this?

Lots of meteorological data web sites have a model where the data live on a central server, get plotted on the server, and then the server serves static images to the client. This creates a bottleneck where adding fields and view sectors takes exponentially more processing power for the server. One way around this is to offload the plotting to the client and to have the browser plot the data on a pan-and-zoomable map. Unfortunately, in the past, this has required developing low-level plotting code, and depending on the mapping library, the performance may be poor.

autumnplot-gl provides a solution to this problem by making hardware-accelerated data plotting in the browser easy. This was designed with meteorological data in mind, but anyone wanting to contour geospatial data on a map can use autumnplot-gl.

Usage

autumnplot-gl is designed to be used with either Mapbox GL JS or MapLibre GL JS mapping libraries. If you're using webpack or another node-based build tool, you can install by running

npm i autumnplot-gl

Unfortunately, you may have to modify your build tool configuration to include the WebAssembly binary. For webpack, I've found adding this to webpack.config.js works:

{
    "module": {
        "rules": [
            {
                test: /\.wasm$/,
                type: "asset/resource",
                generator: {
                    filename: "[name].wasm"
                }
            }
        ]
    }
}

In addition to the Typescript library, pre-built autumnplot-gl javascript files area available here. Adding them to your page exposes the API via the apgl global variable (e.g., instead of new PlateCarreeGrid(...) in the examples, you'd call new apgl.PlateCarreeGrid(...)).

A basic contour plot

The first step in plotting data is to create a grid. Currently, the only supported grids are PlateCarreeGrid (a.k.a. Lat/Lon), RotatedPlateCarreeGrid, and LambertGrid (a.k.a. Lambert Conformal Conic).

// Create a grid object that covers the continental United States
const nx = 121, ny = 61;
const grid = new PlateCarreeGrid(nx, ny, -130, 20, -65, 55);

Next, create a RawScalarField with the data. autumnplot-gl doesn't care about how data get to the browser, but it should end up in a Float32Array or Float16Array in row-major order with the first element being at the southwest corner of the grid. If you're using zarr.js, you can use the getRaw() function on a ZarrArray to get data in the correct format. Also, Float16Arrays are not in the Javascript standard library (for now), so for the time being, you'll need to use this library. However, the nice part about using a Float16Array is that your data will be stored as float16s in VRAM, so they'll take up half the space as the same data as float32s. Once you have your data in that format, to create the raw data field:

// Create the raw data field
const height_field = new RawScalarField(grid, height_data);

Next, to contour the field, create a Contour object and pass it some options. See here for a full list of options.

// Contour the data
const height_contour = new Contour(height_field, {color: '#000000', interval: 30});

Next, create the actual layer that gets added to the map. The first argument ('height-contour' here) is an id. It doesn't mean much, but it does need to be unique between the different PlotLayers you add to the map.

// Create the map layer
const height_layer = new PlotLayer('height-contour', height_contour);

Finally, add it to the map. The interface for Mapbox and MapLibre are the same, at least currently, though there's nothing that says they'll stay that way in the future. Assuming you're using MapLibre:

const map = new maplibregl.Map({
    container: 'map',
    style: 'https://api.maptiler.com/maps/basic-v2/style.json?key=' + maptiler_api_key,
    center: [-97.5, 38.5],
    zoom: 4
});

map.on('load', () => {
    map.addLayer(height_layer, 'railway_transit_tunnel');
});

The 'railway_transit_tunnel' argument is a layer in the map style, and this means to add your layer just below that layer on the map. This usually produces better results than just blindly slapping all your layers on top of all the map (though the map style itself may require some tweaking to produce the best results).

Contour Labeling

Typically, when plotting meteorological data, contours are labeled with their values, which you can do with ContourLabels:

const labels = new ContourLabels(height_contour, {text_color: '#ffffff', halo: true});
const label_layer = new PlotLayer('label', labels);

map.on('load', () => {
    map.addLayer(label_layer, 'railway_transit_tunnel');
});

Barbs

Wind barb plotting is similar to the contours, but it requires using a RawVectorField with u and v data.

const vector_field = new RawVectorField(grid, u_data, v_data);
const barbs = new Barbs(vector_field, {color: '#000000', thin_fac: 16});
const barb_layer = new PlotLayer('barbs', barbs);

map.on('load', () => {
    map.addLayer(barb_layer, 'railway_transit_tunnel');
});

The wind barbs are automatically rotated based on the grid projection. Also, the density of the wind barbs is automatically varied based on the map zoom level. The 'thin_fac': 16 option means to plot every 16th wind barb in the i and j directions, and this is defined at zoom level 1. So at zoom level 2, it will plot every 8th wind barb, and at zoom level 3 every 4th wind barb, and so on. Because it divides in 2 for every deeper zoom level, 'thin_fac' should be a power of 2.

Filled contours or raster plots

Plotting filled contours is also similar to plotting regular contours, but there's some additional steps for the color map. A couple color maps are available by default (see here for more details), but if you have the colors you want, creating your own is (relatively) painless (hopefully). First, set up the colormap. Here, we'll just use the bluered colormap included by default.

// colormaps is imported via `import {colormaps} from 'autumnplot-gl'`
const colormap = colormaps.bluered(-10, 10, 20);
const fills = new ContourFilled(height, {cmap: colormap, opacity: 0.6});
const height_fill_layer = new PlotLayer('height-fill', fills);

map.on('load', () => {
    map.addLayer(height_fill_layer, 'railway_transit_tunnel');
});

Making a raster plot is very similar (the two classes support the same options):

const raster = new Raster(height, {cmap: colormap, opacity: 0.6});

Normally, when you have a color fill, you have a color bar on the plot. To create an SVG color bar:

const colorbar_svg = makeColorBar(colormap, {label: "Height Perturbation (m)", 
                                             ticks: [-10, -8, -6, -4, -2, 0, 2, 4, 6, 8, 10],
                                             orientation: 'horizontal', 
                                             tick_direction: 'bottom'});

document.getElementById('colorbar-container').appendChild(colorbar_svg);

Varying the data plots

The previous steps have gone through plotting a static dataset on a map, but in many instances, you want to view a dataset that changes, say over time. In order to switch the data currently plotted, you can call updateField() on the plot component (e.g, ContourFilled, Barbs, etc.) to switch what field is plotted.

// Create the initial field
const height_field_f00 = new RawScalarField(grid, height_data_f00);
const fills = new Contour(height_data_f00, {interval: 30});

// Update the field plotted
const height_field_f01 = new RawScalarField(grid, height_data_f01);
fills.updateField(height_field_f01);

Another way to vary the data is to use MultiPlotLayer. The main difference between this and using updateField() is that MultiPlotLayer will put all data for all times onto VRAM at once. In contrast, with PlotLayer and updateField(), only the data currently plotted are stored on VRAM. For large grids, this may take up a large amount of VRAM, so you may not want to use this, and it may be removed in later versions.

// Contour some data
const height_contour_f00 = new Contour(height_f00);
const height_contour_f01 = new Contour(height_f01);
const height_contour_f02 = new Contour(height_f02);

// Create a varying map layer
const height_layer_time = new MultiPlotLayer('height-contour-time');

// Add the contoured data to it
height_layer_time.addField(height_contour_f00, '20230112_1200');
height_layer_time.addField(height_contour_f01, '20230112_1300');
height_layer_time.addField(height_contour_f02, '20230112_1400');

// Add to the map like normal
map.on('load', () => {
    map.addLayer(height_layer_time, 'railway_transit_tunnel');
});

The second argument to addField() is the key to associate with this field. This example uses the absolute time, but you could just as easily use 'f00', 'f01', ... or anything else that's relevant as long as it's unique. Now to set the active time (i.e., the time that gets plotted):

// Set the active field in the map layer (the map updates automatically)
height_layer.setActiveKey('20230112_1200');

Typescript Considerations

autumnplot-gl is written in Typescript to facilitate type info in large projects. Typescript isn't necessary to use autumnplot-gl, but if you want to use it, there are some considerations.

Many of the plot component classes have generic types. The Typescript compiler can generally figure out the generic type parameters, but if you're declaring a variable to be a plot component, you'll probably need to specify those ahead of time. The first type parameter is the array type (either Float32Array or Float16Array), and the second is the type of the Map you're using.

// Import the map from maplibre-gl, if that's what you're using. Mapbox should be similar.
import { Map } from 'maplibre-gl';

// Declare a contour field which contours an array of float float32s with the MapLibre map.
const cntr: Contour<Float32Array, Map>;

Built-in color maps

autumnplot-gl comes with several built-in color maps, accessible via import {colormaps} from 'autumnplot-gl'. These are basic blue/red and red/blue diverging color maps plus a selection from PivotalWeather. The blue/red and red/blue are functions that take a minimum contour level, a maximum contour level, and a number of colors. For example, this creates a blue/red colormap starting at -10, ending at 10, and with 20 colors:

const colormap = colormaps.bluered(-10, 10, 20);

Here are all the colormaps available:

Map tiles

The above exmple uses map tiles from Maptiler. Map tiles from Maptiler or Mapbox or others are free up to a (reasonably generous) limit, but the pricing can be a tad steep after reaching the limit. The tiles from these services are extremely detailed, and really what you're paying for there is the hardware to store, process, and serve that data. While these tiles are very nice, the detail is way overkill for a lot of uses in meteorology.

So, I've created some less-detailed map tiles that are small enough that they can be hosted without dedicated hardware. However the tradeoff is that they're only useful down to zoom level 8 or 9 on the map, such that the viewport is somewhere between half a US state and a few counties in size. If that's good enough for you, then these tiles could be useful.

Closing thoughts

Even though autumnplot-gl is currently an extremely new package with relatively limited capability, I hope folks see potential and find it useful. Any contributions to fill out some missing features are welcome.

autumnplot-gl's Issues

Map Tiling Issue (or perhaps feature request)

I'm not sure if this is a bug, or just an overlooked possibility, but is it possible to have the map continuously tile data. For example, on the image attached, GFS data is loaded in but the map only displays the "tile" once.

In general usage, I believe everyone would agree that it should continuously tile side to side. Additionally, if you try to zoom in on a specific part within the US, the tile disappears from the screen.

An example of how I am displaying this is here:

const grid_tmp = new apgl.PlateCarreeGrid(1440, 721, 0, 90, 360, -90);

const resp = await fetch('tmp.dat');
const blob = await resp.blob();
let data = new Float32Array(await blob.arrayBuffer());
data = data.map(el => { return (el - 273.15) * (9 / 5) + 32; });
console.log('tmp grid data: ', data);

const tmp_field = new apgl.RawScalarField(grid_tmp, data);

const filled = new apgl.ContourFill(tmp_field, { 'cmap': apgl.colormaps.pw_t2m });
const raster_layer = new apgl.PlotLayer('tmp', filled);

const svg = apgl.makeColorBar(apgl.colormaps.pw_t2m, {
    label: "Temperatures", fontface: 'Trebuchet MS',
    ticks: [-40, -50, -30, -20, -10, 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120],
    orientation: 'horizontal', tick_direction: 'bottom'
})

return { layers: [raster_layer], colorbar: svg };

Is this expected behavior or is this a "bug"? Any possible work arounds?

Additionally, I noted that in this example, the bottom of the GFS output doesn't reach the bottom of the map. I'm not sure if that's a problem on my end or a problem with map tiling.

Overall, this package is amazing! Great work so far. The speed and features of this package are incredible, can't wait to see where it goes.

NBM Data Projection - Compatible?

I was wondering if you have had any experience plotting NBM data using autumnplot-gl. I have some data available here https://wxlogic-models.nyc3.cdn.digitaloceanspaces.com/nbm-test/18z-asnow.dat (compressed gzip).

I have tried adding using the LambertGrid, however no data is being displayed.

const fac = 1e-6;
const ni = 2345;
const nj = 1597;
const dx = 2539703;
const dy = 2539703;
const lon_0 = 265000000 * fac;
const lat_0 = 25000000 * fac;
const lat_std = [25000000 * fac, 25000000 * fac];

const grid1 = new apgl.LambertGrid(ni, nj, lon_0, lat_0, lat_std, -ni * dx / 2, -nj * dy / 2, ni * dx / 2, nj * dy / 2);

I modeled this off from the HRRR demo to no avail.

Any thoughts on this?

Feature Request - Precipitation Type

This package has been amazing to work with! I'm wondering about the possibility of adding precipitation data to model data. Models have reflectivity, and they also have information about precip type. I'm wondering about the possibility of supporting precip type down the road. I know this is a rather complex task, but would we be able to achieve this using a "filter" in WebGL to apply a separate color palette based on the precipitation type? Finally, for the required data, in terms of the current project, this would require three or four items in addition to reflectivity - CSNOW, CICEP, CFRZR, and possibly CRAIN. Perhaps CRAIN may not be needed as if your precip type is not snow, mix, or freezing rain, it's going to be rain?

Thoughts on this? In terms of structure - perhaps this should be an additional layer type such as a "Shaded Layer" and perhaps the inputs could be data, and then the layer types. I think that this should also be applicable to both ContourFilled and Raster.

I'm willing to help out any way possible, but my knowledge of WebGL is lacking.