mtenenholtz / chat-twitter Goto Github PK

When retrieving additional context for the latest user query, the query string (used for KNN against the pinecone embeddings) includes the system prompt (as well as some of the users previous queries).

Whilst the users previous user queries might be useful (although we already have context for these in theory), the system prompt (AI instructions + initial context) doesn't seem relevant or useful here (querying for additional context) and seems likely to cause the additional context to not be a great match for the latest user query.

Looks like a bug to me, but perhaps I am missing something?

https://github.com/mtenenholtz/chat-twitter/blob/f889b688aa35d38a9823e0dde5c0cff94a6ae1fc/backend/main.py#LL168C17-L168C31

The system prompt, providing the context information from the vector store, is fixed after the first question therefore to answer different question, requiring different context one has to reload the interface (there is only one call to system_message endpoint).

Example conversation looks like this:

Q: What is the ranking algorithm?
A: There are several ranking algorithms mentioned in the provided context, so here is a summary of each:
.... yada yada yada

Q: What is the project license?
A: The provided context does not mention the project license.

Reload the interface:

Q: What is the project license?
A: The project license is the GNU Affero General Public License, version 3 or later.

Q: What is the ranking algorithm?
A: There is no mention of a ranking algorithm in the provided context. The context only provides information about the
 GNU Affero General Public License and its definitions, as well as information about patent licenses.

I experimented with hardcoded splitting - it works but I was thinking of instructing openAI to basically split up the text in manageable chunks.

I would suggest to have some overlap and make it aware of the previous chunk.

This might also be an over-engineered solution proposition for code (I think it works better for website crawled via Puppeteer due to superfluous text of repeating menus, etc.). Keen to hear your thoughts though.

npm run dev spits out:

> [email protected] dev
> next dev

sh: 1: next: not found