splitsky / resdata2

Full stack implementation of resdata API. Designed to operate and provide a free and open-source data pipeline to use in research for structured and semi-structured data

ResData

THIS IS A PLACEHOLDER. I WILL WRITE IT PROPERLY ONCE SOMETHING WORKS

ResData is a comprehensive full-stack application designed to facilitate efficient data management and analysis for research groups. Built with FastAPI, MongoDB, Svelte, and Kubernetes, ResData aims to provide a robust and scalable solution for handling large datasets generated from lab equipment.

Table of Contents

• Features
• Technology Stack
• Project Structure
• Installation
• Usage
• Deployment
• Contributing
• License
• Contact

Features

• Real-time Data Ingestion: Stream data from lab equipment in real-time.
• Secure Data Storage: Store data efficiently using MongoDB with robust security measures.
• User-friendly Interface: Interact with your data through a modern Svelte-based frontend.
• Scalability: Deploy on-premises or scale effortlessly using Kubernetes on AWS.
• Integration with Jupyter: Seamlessly integrate Jupyter notebooks for advanced data analysis.
• Continuous Development: CI/CD pipelines to ensure continuous integration and deployment.

Technology Stack

• Backend: FastAPI
• Database: MongoDB
• Frontend: Svelte
• Containerization: Docker
• Orchestration: Kubernetes
• Cloud: AWS

Installation

Prerequisites

• Docker
• Kubernetes (kubectl)
• AWS CLI (if deploying to AWS)
• Python 3.9+
• Node.js

Backend

1. Navigate to the backend directory:

   cd resdata/backend

Install the required dependencies:

bash

pip install -r requirements.txt

Run the FastAPI application:

bash

uvicorn app.main:app --reload

Frontend

Navigate to the frontend directory:

bash

cd resdata/frontend

Install the required dependencies:

bash

npm install

Start the Svelte application:

bash

npm run dev

Usage Running the Application Locally

Ensure both backend and frontend services are running.
Access the frontend at http://localhost:5000 (or the port specified in your configuration).

API Endpoints

GET /api/data: Fetch data
POST /api/data: Upload data
WebSocket /ws: Real-time data stream

Deployment On-Premises

Navigate to the scripts directory:

bash

cd resdata/scripts

Run the deployment script:

bash

bash deploy.sh

AWS Deployment with Kubernetes

Ensure you have configured AWS CLI and kubectl.
Update Kubernetes manifests in resdata/k8s.
Apply the Kubernetes configuration:

bash

kubectl apply -f resdata/k8s/

Contributing

Fork the repository.
Create a new branch (git checkout -b feature/your-feature).
Commit your changes (git commit -am 'Add some feature').
Push to the branch (git push origin feature/your-feature).
Create a new Pull Request.

License

This project is licensed under the MIT License. See the LICENSE file for details. Contact

For more information, please contact:

GitHub: SplitSky