md2korg / cerebralcortex-dataingestion

CerebralCortex Data Ingestion (CC-DI)

Cerebral Cortex is the big data cloud companion of mCerebrum designed to support population-scale data analysis, visualization, model development, and intervention design for mobile sensor data.

CerebralCortex-KafkaStreamPreprocessor (CC-KSP) is a apache-spark based pub/sub system for processing incoming mobile sensor data.

You can find more information about MD2K software on our software website or the MD2K organization on our MD2K website.

How it works?

• CerebralCortex-APIServer uploads data and publish publish file names on Kafka queue.
• CC-DI process Kafka messages published by CerebralCortex-APIServer:
  • Processes uploaded raw files (sensor data), converts CSV/JSON data into CerebralCortex stream format
  • Stores processed raw data and metadata in File-System/HDFS and MySQL using CerebralCortex Data-Manager
  • Stores processed data in InfluxDB for visualization

Install

• CerebralCortex

Contributing

Please read our Contributing Guidelines for details on the process for submitting pull requests to us.

We use the Python PEP 8 Style Guide.

Our Code of Conduct is the Contributor Covenant.

Bug reports can be submitted through JIRA.

Our discussion forum can be found here.

Versioning

We use Semantic Versioning for versioning the software which is based on the following guidelines.

MAJOR.MINOR.PATCH (example: 3.0.12)

1. MAJOR version when incompatible API changes are made,
2. MINOR version when functionality is added in a backwards-compatible manner, and
3. PATCH version when backwards-compatible bug fixes are introduced.

For the versions available, see this repository's tags.

Contributors

Link to the list of contributors who participated in this project.

License

This project is licensed under the BSD 2-Clause - see the license file for details.

Acknowledgments

• National Institutes of Health - Big Data to Knowledge Initiative
  • Grants: R01MD010362, 1UG1DA04030901, 1U54EB020404, 1R01CA190329, 1R01DE02524, R00MD010468, 3UH2DA041713, 10555SC
• National Science Foundation
  • Grants: 1640813, 1722646
• Intelligence Advanced Research Projects Activity
  • Contract: 2017-17042800006