vpgits / sdgp-backend

This repository contains the backend component of the SDGP project "quizzifyme"

sdgp-backend

Welcome to the sdgp-backend repository!

This repository contains the source code related to the Software Development Group Project (SDGP) machine learning component. Specifically, it includes a docker compose file and related github actions related to CI-CD pipeline

Requirements

To run the backend successfully, ensure that your machine meets the following requirements:

• A VPS with least 2BG RAM , 1 VCPU and least 5Gb of Storage
• A stable internet connection
• API keys to supabase, fireworks, runpod

Key-Takeaways

• You have to download gte/small[link:https://huggingface.co/thenlper/gte-small] and place it inside celery_workers/src/api
• Manually running without docker will require manually changing the code to configure rabbitmq and redis.

Deployment

• You can try to replicate the given code and make your own version. You are free to explore and use it as you see fit.
• Or if you want to deploy this , locally can be done through docker compose. Make the appropriate changes to make docker compose work.
• If need to be deployed on cloud, setup a reverse proxy and use docker compose for hosting.

sdgp-backend's Issues

Pytest testing

• Perform testing on backend using pytest

Document/Quiz summary

• Create a document/ quiz summary

bare-minimum-api-endpoints

Refactoring

• Refactor the codebase for unit testing

Create supabase client

Create supabase client.
Add CRUD opereations to include document summary to the documents table

Documentation for the backend

• Needs documentation for the python backend
• Have to modify the source code accordingly to use swagger as documentation

upload file conversion pipeline

Using the tesseract ocr pipeline , giving users access to upload a wide variety of files.

serverless lambda function for endpoint generation

The proposed microservice architecture comes with its own issues. Currently the image size for the celery worker is high compared to other images. Invoking the embedding generation might lead to instability. Therefore trying to implement an endpoint to generate embeddings.

migrate from pinecone

• Pinecone is ending their free tier in April
• Migrating back to supabase for vector ops

Create Document Summary