[Enhancement] Support for Multiple Data Sources about pebblo

[rahul-trip]

Analysis from the plugin side:

Problem at hand:
What should be the trigger to generate the report?
Earlier (In case of single loader) we used to send a flag, loading_end: bool.
Now with multiple document loaders we have the following scenarios:

1.
loader1 = PebbloSafeLoader(ABCDocumentLoader(abcpath), abcname, abcdescription, abcowner) #discocery
loader2 = PebbloSafeLoader(XYZDocumentLoader(xyzpath), xyzname, xyzdescription, xyzowner)
loader1.load() #classification
loader2.load()
2.
loader1 = PebbloSafeLoader(ABCDocumentLoader(abcpath), abcname, abcdescription, abcowner) #discocery
loader1.load() #classification
loader2 = PebbloSafeLoader(XYZDocumentLoader(xyzpath), xyzname, xyzdescription, xyzowner)
loader2.load()

Note: PebbloSafeLoader instantiation --> app:loader discovery, and it's .load() --> classifies the loaded data.

In 1st case we will get to know the first loader and then the 2nd loader, and then both loaders will be loaded (and their data classified) one by one (generating reports twice, on every loading_end = true from both the loaders).
But in 2nd case when the first loader is discovered and loaded just after its discovery and just before the discovery of 2nd loader, server will get loading_end = True (first loader) and report generation will get triggered, We need to make some adjustments here.

As of today, the pebblo server will generate the report with every new loading_end = true. the 2ne case mentioned above is a kind of race condition where we won't be able to get a trigger point to make sure the loading is actually over in an RAG app. Let's wait for a few seconds before generating the report, a new loader discovery might just kick in. Also, we generate(overwrite) a new report with every new loading_end=True

[shreyas-damle]

In order to support multiple data sources, we need to update out folder structure within scratch space as follows:

.pebblo/
       /<app-name>/
                  /metadata/metadata.json             => app name and run_id to load_id mappings.
                                                         This file will be protected by file lock.
                  /<run-id>/
                           /metadata/metadata.json    => All the details needed to generate report.json.
                                                         This will include details for all successful loads.
                           /report.json               => This is input to reports module.
                                                         This will include details for all successful loads.
                           /pebblo_report.pdf         => Report pdf including details for all successful loads.
                  /<load-id>/
                           /metadata/metadata.json.   => All the details needed to generate report.json.
                                                         This will include details for all successful runs.
                                                         This is only for backward compatibility
                           /report.json               => This is input to reports module.
                                                         This will include details for all successful runs.
                                                         This is only for backward compatibility
                           /pebblo_report.pdf         => Report pdf including details for all successful runs.
                                                         This is only for backward compatibility