Adding metrics to the backend about danswer

[megamorf]

If I were to operate this application in a business context (company with a +20000 user GHES instance) I'd be looking for the following:

• indexing information per connector (last index date, index success/failure, crawling data)
• usage metrics (number of active sessions, tokens generated)

What can I do with those?

The index data and success status tell me if the connectors are working correctly and with the changes in the crawling data I can see if there was suddenly a big change in the ingested content that I otherwise might not be aware of.

With the active sessions I could get a better understanding of how the system is used throughout the day to plan changes or maintenance. And the generated tokens gives me a correlation for usage and cost.

[Weves]

Hey @kavfixnel! Do you have any specific metrics / areas you want observability into?

[lucianosb]

A chat history could be a useful metric specially on the Enterprise scenario with at least the requesting user and the total tokens generated.

[yuhongsun96]

Follow up question, what specific things would be useful to track via prometheus? We're trying to avoid introducing too much extra infra that's not related to the core requirements of the project. Is there a must-have reason for observability via a third party log ingestion tool?

That said we are currently adding better logging on the backend for things such as model response, latency etc.

If this is just better logging, we are already doing it and will have an update soon.

If this is about a third party log ingestion service, please explain the need, thx

[lucianosb]

@yuhongsun96 A better logging is indeed useful since I could parse to ingest in my own analytics system that is already in place in the company. It's a good step.

I wish to use such observability together with DLP solutions to ensure a safe usage of LLM systems across the company.

[kavfixnel]

Although logging could be used to create metrics about usage, typically it makes more sense to add metrics to the a application directly.

Since Danswer seems to specifically focus on containerized Kubernetes applications, I suggested Prometheus since that’s a common way to observe metrics in these types of apps.

For metrics I was looking for:

• How often endpoints were called
• Request duration
• Number of times APIs (OpenAI etc.) were called as well as request duration and number of errors to these APIs
• Scrape durations of data sources

The nice thing about adding metrics is you can balsa add alerting for when things go wrong, which seems like something an Enterprise tool should have