The goal of the project is two-fold: finding optimal batching and partitioning for a specific dataset and application. Golam Md Muktadir
Type | Description |
---|---|
Dataset | USA Name Data (BigQuery Dataset) |
License | Open for analysis |
Owner | Data.gov |
History | This public dataset was created by the Social Security Administration and contains all names from Social Security card applications for births that occurred in the United States after 1879. |
- What are the most common names?
- What are the most common female names?
- Are there more female or male names?
- Female names by a wide margin?
- Impact of Game of Thrones on US Baby Names! (https://www.kaggle.com/dhanushkishore/impact-of-game-of-thrones-on-us-baby-names)
- Impact of TV series on US Baby Names! (Not yet done) OUTCOME
For each operation an impact of batching and partition on request time and throughput. Targeting to learn spark streaming 2.0.
- One query, and throughput.
- One query, throughput, and request time.
$$ WIP
- Count names per second without stream persistence
- Count names per second with stream persistence (different modes)
- Experiment with different batch intervals