Comments (2)
I was able to implement the CMR query which returns a list of granule URs present in CMR for a specified time range. I started to implement the Hydrocron query but ran into some confusion around DynamoDB queries.
I would like to query the hydrocron tables in a way that returns all granuleUR
with a range_start_time
between a specified range which I can determine from the CMR query. So something this:
items = hydrocron_table.query(
IndexName="GranuleURIndex",
KeyConditionExpression=(Key("range_start_time").between(start_time, end_time))
)
But when I try to do this the query produces an error and complains that I have not included a "key schema element" in the query:
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the Query operation: Query condition missed key schema element
From my understanding you cannot query DynamoDB on a date range only and instead would need to perform a scan which is costly.
For now I am going to query Hydrocron by granuleUR
which I can get by iterating over the CMR query results to perform the query. This will allow me to automatically see what is in CMR but not in Hydrocron.
I won't be able to easily determine what is in Hydrocron but not in CMR (or overlap between the two). I think we would to set up another global secondary index on the "range_start_time" with a range key of "granuleUR" if we want to facilitate a different query. Although I am not sure that quite fits with the DynamoDB use case.
@torimcd - Does this fit with your understanding of DynamoDB queries?
Any objections to de-prioritizing determining what is in Hydrocron but not CMR since we were only going to log those results for now?
from hydrocron.
We will de-scope determining overlap and what exists in Hydrocron but not CMR for now. Create #206 to track future work.
from hydrocron.
Related Issues (20)
- Add new option for the "feature" request parameter for prior lakes HOT 4
- Cost model impact for new prior lake database table HOT 2
- Setup new table for ingest status tracking HOT 1
- (Optional) add checksum during forward processing ingest HOT 1
- Create table for tracking granule ingest status HOT 1
- Determine overlap of data in cmr and data in hydrocron HOT 1
- Construct CNM to trigger load data
- Cost model impact for new track status database table
- Define a usage plan and API key for the Confluence workflow HOT 6
- Complete track status reconcilliation of Hydrocron with CMR
- Using spaces in requested fields provides default fields instead of requested
- Features with large geometries cannot be loaded in Hydrocron with current methods HOT 3
- Query track ingest table for granules with "to_ingest" status HOT 1
- Update track ingest table with granule status
- Add option to return data for all nodes associated with a given reach id
- Add DOI to documentation HOT 1
- Feature Analysis: Support full geometry returns for SWOT lake data
- Upgrade Terraform AWS provider to 5.0
- Add operations to load granule Lambda to write granule record to track ingest database
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hydrocron.