Runtime:loacal is not adding any value in final report.
Instead we can have OS:os_name or OS:os_name:os_version

Description

When execute RAG app, we are getting error "unable to reach pebblo server." but it is generating report as expected on Pebblo Server.

Error message
$ python3 fin_corp_rag_app.py
Loading RAG documents ...
Unable to reach pebblo server.
Loaded 93 documents ...

Hydrating Vector DB ...
Finished hydrating Vector DB ...

Expected behavior
It should call Pebblo APIs and pdf report should get generated without any error.

Additional context
Pebblo server was healthy when this error occured.

System:

• OS: Mac
• GPU/CPU:
• Pebblo version (commit or version number): 0.1.11
• Langchain version: 0.1.9
• DocumentStore:
• Reader:
• Retriever:

Link: #353 (comment)

As of pebblo 0.1.9, we have only pebblo_report.pdf as output of the pebblo package.
It would be useful if we can have simple UI running locally on pebblo server which will show all apps discovered, giving details about each app(equivalent to pebblo_report.pdf).

We would like to have Safe Retriever Details on the local UI. Here two separate tabs would be shown for loader type and retrieval type applications.
This would include below details on Safe Retrieval App listing page:

• Active Users (with mouse over showing actual "Top" users - cut off at N user-ids, N = 3)
• Retrieve Documents (with mouse over showing "Top" documents retrieved - cut off at N user-ids, N = 3)
• Retrievals (i.e. prompt count, cumulative)
• VectorDB (with mouse over showing first N vector db names, N=3)
• Owner(app owner)

Describe the bug
When URL is 0.0.0.0/pebblo while running locally, reports could not be downloaded due to CORS issue.

System:

• OS: Mac
• Pebblo version (commit or version number): 0.0.15

Create a Dockerfile for pebblo to quickly run pebblo in a containerized environment.

Tasks:

• Add Pebblo Dockerfile
• CI: Add docker image push on release/tag

Description

The current filesystem based RAG application state doesn't scale for deeper of SafeLoader and SafeRetriever analytics.

• Introduce sqlite-db based application state
• Deprecate filesystem based application state

As we have ruff linter for python code, we would like to have linter for UI code.

Follow on for #296, which introduced SafeLoader for LlamaIndex.

Add support for SafeRetriever with,

• identity enforcement / filtering on the doc snippets retrieved from vector db
• semantic topic policy enforcement / filtering on the doc snippets retrieved from vector db
• send app-discover and per-prompt stats to pebblo server and pebblo cloud

Description:
We need to streamline our code formatting process by implementing a make command that can format either all files in the repository or only the changed files. This will help maintain consistency in our codebase and make it easier for developers to adhere to our coding standards.

Tasks:

• Configure the formatting commands (format and format-diff) to use the ruff formatting tool
• Document the usage of these commands in the project's README or documentation.

Labels: enhancement, formatting

Feature: Document snippet anonymizer

Anonymize document snippets in Pebblo report. As Pebblo is considered for environments beyond dev, anonymization will help distribute the report to more app stakeholders.

Capture and display RAG document snippet's classifier confidence score in Local UI and PDF report

Tasks

• Capture confidence score in backend schema
• Display confidence score in PDF report
• Display confidence score in Local UI on SafeLoader and SafeRetriever pages

Description: By using context words, it increases result score by matching the text coming from a specific column or a specific user input.
PR: #204

1. Display Pebblo version and Pebblo client versions on JSON Schema and Reports

Description:

• Once Pebblo installation is completed and user tries to run Pebblo
• Below runtime error occurs - OSError: [E050] can't find model en_core_web_lg
• Using Ubuntu VM with Ubuntu 22.04
• Please refer below screenshot for complete error message:

Pebblo (as of 0.1.12 version) supports single data source. Having support multiple data sources within single RAG application would be a good feature.

Description:
When I have multiple data sources to be used in my app, I should be able to see all those data sources and their details in the pebblo report.

As part of this feature, following changes would need to be done in the report:

1. Report Summary: Aggregate details about all data data sources.
2. Top Files With Most Findings: Add new column to show to which data source the file belongs.
3. Data Source: It would show snippets about all data sources.

• Pebblo server is failing on conda environment due to Pango(WeasyPrint) dependency issue
• Once Pebblo installation is completed and user tries to run Pebblo
• Below runtime error occurs -
  WeasyPrint could not import some external libraries. Please carefully follow the installation steps before reporting an issue: https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#installation https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#troubleshooting
• Please refer below screenshot for complete error message:

As of pebblo 0.1.7, we have good information about current state of the app loading. It would be useful to capture history of last 5 loads of the app.
This will give below information about each load:

• Location of the report and report file name
• How many findings were there in that report
• How many files were there with findings
• When the report was generated.

Description

Pebblo currently supports LangChain RAG apps. Extend the client to support another popular RAG development framework: LlamaIndex.

https://www.llamaindex.ai/

Components

• Pebblo SafeLoader for LlamaIndex

1. Separate findings for entities list.
2. Separate findings for topics list.
3. Timezone for reports.

Add delete app support for SafeLoader and SafeRetriever apps

Tasks

• Add delete app API to pebbo server
• Add Delete link in Local UI