microsoft / appcenter-sample-export

This repo contains sample .NET Core library for parsing Export blobs created via App Center Continuous Export.

App Center Export Parser

This repo contains sample .NET Core library for parsing Export blobs created via App Center Continuous Export.

Sample 1 - Console App to forward logs for specific users to their account

This sample is located under sample-console-app. The console app can be used to selectively forward logs from one storage account to another. This can be useful for satisfying one of GDPR export requests.

To build the sample:

dotnet build export-parser.sln

To run the sample:

sample-console-app.exe <installId> <blob storage connection string where export is pointing to> <blob storage connection string where filtered data will go> <output container name>

Note: since the sample downloads each export blob to a local machine, it will be singnificantly faster if the code was executed on Azure VM in the same data center as the storage account.

Sample 2 - Azure Function to forward logs for specific users to their account

This sample is located under sample-azure-function. This Azure Function is similar to the console app sample, however is can be used to selectively forward logs on an ongoing basis.

To build the sample:

dotnet build export-parser.sln

To deploy the function refer to this article.

Sample 3 - Continuously stream export data

This sample shows how to continuously stream data from your export Azure storage container to Event Hubs and to another Azure storage container. This sample is located under export-streaming. This sample is a console application that creates an observable that parses exported data, and multicasts the observable data into both an Event Hub and an Azure storage container.

To build the sample:

dotnet build export-streaming.sln

appcenter-sample-export's Issues

Reading from a blob takes to much time

let say that I want all the logs before 1 year, this sample app will try to access all the possible directory for months [1,12] then it will try to access all the possible directories for days [1,31] then every possible minute [1,59].

In a simple calculation, if accessing a directory will takes 1 sec and reading from a file (inside the minute directory) will takes 4 sec. then to get all the logs before 1 year will takes
(12+ 31)+ (59*4) = 451 seconds = 7.51667 minutes.

let's says I am using this into my API, the user should wait for 7.51667 minutes to get his logs ( without counting the time needed to format the API).

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request