edanalytics / lightbeam Goto Github PK

To use lightbeam in an automated data pipeline it would be useful to have it generate structured output noting the number of records sent and skipped by resource and status code, the run timestamp, and possibly other information. Open questions include:

1. what should the flag be? perhaps --results-file output.json
2. what should the output format be? (YAML, JSON, something else?)
3. besides the items noted above, what else should be included in the results? perhaps the Ed-Fi API base_url? anything else?

Tagging @jayckaiser for thoughts.

I am in the process of building out delete functionality for DPS to delete descriptors from their ODS. I want to leverage the lightbeam delete functionality, but it seems that lightbeam is not correctly querying the API.

On the server request logs, we can see a request come through like so:

/data/v3/ed-fi/programTypeDescriptors?codeValue=504&namespace=uri://dpsk12.org/programTypeDescriptor&shortDescription=106

These parameters are not valid to be used against the API as per swagger. This request will return the whole set of PorgramTypeDescriptors that exist in the ODS. so returns a 200 response, but does not subset the descriptors as perhaps intended.

This results in lightbeam not being able to find a 1-1 match for payload and record with this being an output:

lightbeam INFO (reason: [multiple matching payloads found in API]; instances: 2)

There are also no subsequent DELETE requests to the API as per the access logs, so lightbeam is definitely stopping after this point.