Comments (4)
I think this also relates to #495 because I feel a lot of the information that could be retrieved from an inspect()
call would be a good base filtering mechanism.
from quilt.
We are interested in implementing this feature and would like to gather more examples. The above filters by node_name. More generally, we could support regex matching for node names. What are other examples that you encounter in your workflows? The queries that we need to service will strongly effect the design.
Does it make sense to support the attachment of a metadata field to each node (e.g. a single JSON doc) and then filtering on the same?
from quilt.
I thought about this over the weekend and I think my ideal system would be as follows:
Introduce a new keyword to the build.yml file system. 'metadata'
, works like a 'file'
keyword but only accepts json files (json.load()
must not return a failure). On construction of the package, you use whatever json is given by this field and inject the attributes like node_name
or others found in an inspect()
call on that node. An example is below.
If a person provides a metadata file it would look like this.
contents:
README:
file: this_is_a_readme.md
transform: id
real_file:
file: actual_file.tiff
transform: id
metadata: this_files_meta.json
The above example would append the contents of an inspect()
call to the provided metadata.
If a person doesn't provide a metadata file it would look like this.
contents:
README:
file: this_is_a_readme.md
transform: id
real_file:
file: actual_file.tiff
transform: id
The above example would generate a base metadata file that contains the contents of an inspect()
call.
Why do I like this system:
It means that not only would people be able to filter on basic quilt node attributes but also allow for custom filtering by the package creator using the same system.
Other things I would really like are things like a number of rows option. Maybe I provide a query but I only want the first 10 that match. More complex would be give me depth_rows
so go through each group_node and filter the files present but only return the first 10 for that group_node.
There are some other things but I would like to hear back first on what you think before I go too deep.
from quilt.
Quilt 3 packages now provide predicate filtering over package metadata.
from quilt.
Related Issues (20)
- package ignoring locally installed versions HOT 9
- Catalog failing to render thumbnails on package view but succeeding on file view HOT 2
- Catalog does not render large Vega JSON files (created by altair) HOT 2
- Accessing package causes ascii decoding error HOT 3
- [question] Quilt and min.io? HOT 15
- Add package created and last updated timestamps part of the API response HOT 2
- Can't resolve 'jsoneditor/dist/jsoneditor.css' HOT 4
- Dependency Dashboard HOT 2
- Add a way to pre-fill metadata on push HOT 4
- Cannot retrieve the old version data HOT 1
- quilt3 verify enhancement
- Remove $ref restriction HOT 1
- Remove schema version restriction HOT 3
- add kms docs
- Quilt 2.9.15 within AWS Lambda execution HOT 3
- Support for running Quilt in an EKS pod with OpenID authentication to AWS disappeared/missing HOT 2
- Workflows should enforce push-level tag completeness
- quilt3 push: --dir is a required argument
- Config resolution at import
- Object level metadata from JSON schema
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from quilt.