mdubbert / astro_analysis

Exploring astronomical data with cloud technologies

astro_analysis

Analyze streaming astronomical data

Project Goals

• Explore new and unfamiliar technologies, ranging from libraries to services.
• Explore new and unfamiliar data domains relating to astronomy.
• Experiment with automated build, testing, and deployment.

System Overview

Example Question

What is the combined mass of asteroids, given a streaming set of observations?

Contributing

Pull requests are welcome. Check out the Issues tab to get started.

astro_analysis's Issues

Deploy basic stack with Kinesis data stream & KMS

Configure Actions for deployment (CD)

Automate deployments

1. Trigger upon merge commits to main branch.
2. Deploy CloudFormation stack, producer, and any consumers to single AWS account.
3. Add repo secrets and reference them securely.

Add documentation

1. Populate the README
2. Add diagrams

Develop producer to send records to Kinesis

Define Glue table with Kinesis stream as source

So that streaming computation can be performed, define a table over incoming data, preferably using CloudFormation.

Ignore schema management and evolution for now.

Increase volume and variety of produced data

Currently, the producer job fetches the same set of records and sends them to Kinesis stream.

Adapt the producer to loop through requests, responsibly call the SBDB API using limit-from query parameter.

Create & deploy KDA application for calculating total mass of streaming sets

So that analysis can be performed with less user interaction, convert the Flink SQL consumer from current notebook state to deployable Kinesis Data Analytics application.

1. Define the application in CloudFormation template. Use minimal KPUs for now.
2. Update deployment workflow(s) to include application packaging.

Configure Actions for build & test (CI)

Upon commit to any branch

1. run unit tests
2. validate CloudFormation template(s)
3. lint

Producer: Make function case-insensitive

In src/python/producer/producer.py the to_boolean() function uses a case-sensitive lookup. To make this more robust against potential API changes,

1. Update the lookup to work whether inputs are capitalized or lowercase.
2. Add or update unit test

Producer: Replace hardcoded values for region and stream name

Develop Flink consumer application to analyze stream contents