Incomplete data being returned about coronavirus-dashboard-api-v2-server

No, just ends prematurely resulting in malformed json. looks like a size limit in the return. cat of a file downloaded today via the web browser https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=newCasesBySpecimenDateAgeDemographics&format=json it abruptly ends {"age":"40_44","cases":0,"rollingSum":4,"rollingRate":55.5},{"age":"45_49","cases":0,"rollingSum":3,"rollingRate":33.8},{"age":"50_54","cases":0,"rollingSum":6,"rollingRate":62

[xenatisch]

Where does it cut off? Is there a specific date or area?

Will work on getting a list together of any other failing metrics

[colic001]

The case age demographics by LTLA is the only one I have come across with the error, this is what happens to the JSON file in Power Query.

As for the CSV version it misses of random LA's of such as Ashford

[xenatisch]

@melvinio16 It's unlikely to be related to metrics so much as it is related to the volume of the data being returned.

I am disabling the streaming in favour of an alternative approach whereby instead of streaming and caching the data simultaneously, we first cache the in storage, then respond with a 303 redirect code against the cached file. This will reduce the chance of premature terminations.

Additionally, I will be adding 2 new tags to the storage files - in_progress and done. When in_progress=1 and done=0, the request will be held (suspended) with 10-second waiting periods. Once in_progress=0 and done=1, the redirect code will be served. If in_progress=0 and done=0, the cache item will be deleted to break any outstanding wait clause and induce a retry.

The downside here is this approach:

• if the request fails frequently, you'll end up waiting for longer before getting failed request response
• you don't get a response until everything is ready

On the other hand, I expect this to improve consistency and reliability of the API.

Need to make some fine tuning and perform a few tests. Expect to deploy tonight. Will update here.

[xenatisch]

@colic001 As a part of this update, I've implemented a few safeguards to ensure that the beginning, ending, and separating commas of JSON responses are properly positioned.

[xenatisch]

@melvinio16 / @colic001 I have now deployed the new version. Please do give it a go and let me know how if it's made things better or worse.