vaibhavag18 / bladerunner

Extract Text from PDF Document: Load the PDF document using a library such as PyPDF2. Extract the text from the PDF and concatenate it into a single string. Preprocess Text into Chunks: Split the extracted text into smaller chunks, such as paragraphs or segments, to improve manageability and efficiency during processing. Store these chunks for later use. Generate Embeddings: Use a SentenceTransformer model to generate embeddings for each chunk of text. These embeddings represent the semantic meaning of the text chunks in a high-dimensional space. Connect to Cassandra: Establish a connection to the Cassandra database using the necessary credentials and authentication methods. Store Text Chunks and Embeddings in Cassandra: Store the text chunks and their corresponding embeddings in a Cassandra vector store. This will allow efficient querying of text chunks based on semantic similarity. Query Processing: Take a user query and process it. Encode the user query to obtain its embedding using the SentenceTransformer model. Find Relevant Text Chunks: Query the Cassandra vector store using the query embedding to find the most relevant text chunks. The vector store uses semantic similarity to match the query with the stored text chunks. Generate Answer: Use the LLaMA model to generate an answer based on the relevant text chunk and the user query. This process involves encoding the text chunk and query, passing them through the LLM, and decoding the output to generate the final answer.