This repository is the official codebase for the arxiv paper search app hosted at: https://docsearch.redisventures.com
Through the RediSearch module, vector data types and search indexes can be added to Redis. This turns Redis into a highly performant, in-memory, vector database, which can be used for many types of applications.
Here we showcase Redis vector similarity search (VSS) applied to a document search/retrieval use case. Read more about AI-powered search in our blog post hosted at Data Science Dojo.
This app was built as a Single Page Application (SPA) with the following components:
- Redis Stack: Vector database + JSON storage
- FastAPI (Python 3.8)
- Pydantic for schema and validation
- React (with Typescript)
- Redis OM for ORM
- Docker Compose for development
- MaterialUI for some UI elements/components
- React-Bootstrap for some UI elements
- Huggingface Tokenizers + Models for vector embedding creation
Some inspiration was taken from this Cookiecutter project and turned into a SPA application instead of a separate front-end server approach.
As a way to expose the different capabilities of embedding providers, this applications supports HuggingFace
, OpenAI
, and Cohere
embeddings out of the box. Interested in a different embedding provider? Feel free to open a PR and make a suggested addition.
Provider | Embedding Model | Required? |
---|---|---|
HuggingFace | sentence-transformers/all-mpnet-base-v2 |
Yes |
OpenAI | text-embedding-ada-002 |
Yes |
Cohere | small |
Yes |
The arXiv dataset was sourced from the the following Kaggle link.
If you wish to modify or work with your own data...download and extract the zip and place the resulting json file (arxiv-metadata-oai-snapshot.json
) in the data/
directory.
Before running the app, install Docker Desktop.
- To get started, make a copy of the
.env.template
file:
$ cp .env.template .env
- Add your
OPENAI_API_KEY
to the.env
file. Need one? Get an API key at https://platform.openai.com. - Add you
COHERE_API_KEY
to the.env
file. Need one? Get an API key at https://cohere.ai.
Both Redis Stack and the application backend run with Docker Compose using pre-built containers. Choose one of the methods below based on your Redis setup.
-
Get a Redis Cloud Database (with the RediSearch module included).
-
Update the
REDIS_HOST
,REDIS_PASSWORD
, andREDIS_PORT
environment variables in the.env
file created above.: -
Run the App:
$ docker compose -f docker-cloud-redis.yml up
Use the provided Dockerfiles and open source containers to run the application locally:
$ docker compose -f docker-local-redis.yml up
You can use the Jupyter Notebooks in the data/
directory to create paper embeddings and metadata. The pickled dataframes will end up stored in the data/
directory and used when creating your own container.
You can the build.sh
script to create your own docker image, and then make sure to update the .yml
file with the right image name if necessary.
If you want to use K8s instead of Docker Compose, we have some resources to help you get started.
It's typically easier to write front end code in an interactive environment, testing changes in realtime.
- Deploy the app using steps above.
- Install packages (you may need to use
npm
to installyarn
)$ cd frontend/ $ yarn install --no-optional
- Use
yarn
to serve the application from your machine$ yarn start
- Navigate to
http://localhost:3000
in a browser.
All changes to your local code will be reflected in your display in semi realtime.
It's typically easier to manipulate front end code in an interactive environment (outside of Docker) where one can test out code changes in real time. In order to use this approach:
- Follow steps from previous section with Docker Compose to deploy the backend API.
cd gui/
directory and useyarn
to install packages:yarn install --no-optional
(you may need to usenpm
to installyarn
).- Use
yarn
to serve the application from your machine:yarn start
. - Navigate to
http://localhost:3000
in a browser. - Make front end changes in realtime.
Sometimes you need to clear out some Docker cached artifacts. Run docker system prune
, restart Docker Desktop, and try again.
Open an issue here on GitHub and we will try to be responsive to these. Additionally, please consider contributing.