fsspec / gsoc-kechunk-2022 Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
@peterm790 , Check out this GSoC repo from 2019 as a possible template for tracking activity. I think it worked pretty well!
https://nbviewer.org/gist/136480d389e948b0756ee2f7c5ee9564
This would be another good one for the kerchunk examples.
This GSoC was chosen by the IOOS organization, so I just wanted to report that @peterm790's help with understanding a problem with the fill_value
in led to a Kerchunk PR which was then successfully applied to an IOOS model collection -- the LiveOcean forecast collection from the NaNOOS Regional Association of IOOS.
We took 24 sample hourly LiveOcean NetCDF from ROMS, put them on the Open Storage Network and kerchunked them into a single virtual Zarr dataset. We were able to modify the metadata, adding the standard_name='time'
attribute to the ocean_time
variable.
Here is a notebook demonstrating access and simple visualization:
Snapshot:
I didn't remember this quite right -- we didn't read the original geotiff files here because kerchunk doesn't create xarray spatial
coordinates yet when reading geotiff. So we first convered the geotiff files to netcdf, and it's the NetCDF files we are using in this example:
https://nbviewer.org/gist/cea8e1ee16e318e05c128353482836b9
It would be good to add this example to kerchunk (it also shows accessing an S3 endpoint that is not on AWS!)
The NetCDF Markup Language (NcML) has been widely used for decades to create virtual NetCDF datasets that point to collections of NetCDF files and add/modify global and variable metadata and/or values.
It would be very useful to have a package that parses this info from an NcML XML file and generates the equivalent Kerchunk JSON virtual dataset.
A fairly complicated example of NcML that aggregates a collection of ROMS NetCDF files and makes them CF and SGRID compliant is here:
https://github.com/rsignell-usgs/xml/blob/master/THREDDS/geoport/COAWST_catalog.xml
@peterm790 , I think it would be useful to create pangeo-forge recipes for ERA5 and other datasets.
I didn't really see the point of using pangeo-forge for our workflows up until now, because:
But @sharkinsspatial convinced me at SciPy2022 that even if we didn't need pangeo-forge there was value in capturing the workflow for others using pangeo-forge formalism. And they are apparently working on the updating part.
At the scipy sprint on pangeo-forge, I created my first recipe, following the example here: https://pangeo-forge.readthedocs.io/en/latest/pangeo_forge_recipes/tutorials/hdf_reference/reference_cmip6.html
@martindurant, two questions:
GRIB2ReferenceRecipe
method, or should it be for a more generic ReferenceRecipe
method?submit midterm GSoC evaluation due July 29
@peterm790, I've been immersed in SciPy and ESIP Summer Meeting last week and this one, and thought I'd better catch up on what you've been up to: https://github.com/fsspec/GSoC-kechunk-2022/blob/main/GSOC_monitor.md
Got time for a quick update there?
@peterm790, please take a look at https://discourse.pangeo.io/t/efficient-access-of-ensemble-data-on-aws/2530 and see if you think the current version of kerchunk could handle it!
The ESIP JupyterHub on AWS (https://jupyter.qhub.esipfed.org) is a great place to run workflows that analyze and visualize kerchunked data because:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.