NWS integration fails to populate 11 sensor entities. about core

[MatthewFlamm]

Is this the endpoint being used?

https://api.weather.gov/stations/Kxxx/observations/latest

It seems to be repeatedly working with a browser (and contains data).

What do you make of this?

Yes. It works in a browser, because requesting via browser gives a fresh user agent for each request. When running through a client based request, we are supposed to use a unique and stable user agent. However NWS has a caching problem in this case from time to time. NWS specifically asks that cache-busting techniques to not be employed.

[Sudo-Rob]

I can't tell if data are old when they're populated. The issue is most of the time the integration is not populating any data. All 11 entities are showing unknown.

[Sudo-Rob]

I've seen good data over the last day. I'm assuming the issue is related to the API.

I recommend closing the issue.

[MatthewFlamm]

Ironically, my local station seems to be more flaky now than it was before. #117109 will help make the integration more robust to momentary problems for observation data. This will be in the next monthly release. Since you are the issue creator, you can close it in the future. But I will do it here.

[MatthewFlamm]

@home-assistant close

[home-assistant]

Hey there @MatthewFlamm, @kamiyo, mind taking a look at this issue as it has been labeled with an integration (nws) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of nws can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign nws Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

nws documentation
nws source
_{^{(message by IssueLinks)}}

[MatthewFlamm]

This seems like NWS server flakiness to me, although the debug log you sent has data in it, at least at the beginning. Did you capture that log during the time period you did not have data?

There are a few relevant "known issues" and "upstream issues" currently in the NWS API service page (under "Updates").

[Sudo-Rob]

You might be correct about the flakiness. I just performed a restart and data are being populated.

I couldn't remember if the file was with or without data because I spent time over the following day trying to get data restored. What I can say with some certainty is that if after a restart or YAML reload data are not populated, it will stay that way until there is a successful restart with data. What's still unknown to me is whether the data are being properly updated--that is, if they're populated. FYI: I have not seen any update of the sensor entities since they were populated about 5 or 6 hours ago after a restart.

As an aside, I think it would be useful to have a datetime entity for the native sensors so we could tell if the data are stale. Also, I'd like to know how often the observation data are being polled, and if there is a way to adjust the polling interval. I realize observation data are typically associated with an observing station, and most update hourly if they are automated.

[MatthewFlamm]

Currently the data is polled every 10 minutes, but if there is a failed update, it will update every minute. There is no way to increase this currently. NWS suggests no more frequent than about 1 minute per request, and we are right there. Currently, if the response is not a 500 error, we treat it as successful, however the NWS servers are lately having a problem with providing a successful response, but with no data. I have started a PR so that the integration will automatically retry fetching data if no data returns, but this actually exacerbates the drop out of data in home assistant, so it is on hold.

The addition of datetime for observation as a sensor makes sense to me as a feature request. PR's welcome :).

[Sudo-Rob]

Is this the endpoint being used?

https://api.weather.gov/stations/Kxxx/observations/latest

It seems to be repeatedly working with a browser (and contains data).

What do you make of this?

[Sudo-Rob]

Is using curl the proper way of generating a client-base request? If so, it seems to be responding with data.

[MatthewFlamm]

Assuming you are using curl -A "my_agent_id" [URL] or something similar that will use a consistent, unique user agent, yes. The issue is that after some amount of time, your user agent will start being served old data in some manner.

[kamiyo]

Are we not returning old cached values if the API request fails?

…

On Fri, May 17, 2024, 14:19 Sudo-Rob ***@***.***> wrote: I can't tell if data are old when they're populated. The issue is most of the time the integration is not populating *any* data. All 11 entities are showing unknown. image.png (view on web) <https://github.com/home-assistant/core/assets/56487711/9e00f344-96d7-495f-83cb-2f16c660801c> — Reply to this email directly, view it on GitHub <#117533 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABWCYT4YQD7SKEFM5XZZZATZCZJ2LAVCNFSM6AAAAABHZPVDFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJYGIZTMMJTGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[MatthewFlamm]

The problem is that the request succeeds, i.e. 200 status code, yet the server returns []. So if you restart and get this the first time, you won't see data until the first successful request. This is probably what you are seeing. If you restart and get data, it will continue showing you that data even if subsequent calls return []. However, this isn't good either as the data is becoming increasingly invalid.

#117109 is intended to mark that data stale after X minutes (and institute more frequent retries in the interim), but it makes the problem more apparent.
I'd argue this reflects reality more, but it is more noticeable to the end user.

[Sudo-Rob]

With curl, I see data approximately 2-hours old. Is that rejected, or should it populate the entity?

[MatthewFlamm]

With curl, I see data approximately 2-hours old. Is that rejected, or should it populate the entity?

This is probably a smoking gun observation here. In HA we also utilize the start parameter, which might cause the API to return nothing instead.

[Sudo-Rob]

After hours of no data, I reloaded the integration and the entities populated. If it behaves like it has previously, I doubt I will see any changes in states over the next several hours.

[Sudo-Rob]

I see no changes in the sensor entity states after getting data following a restart. I've taken a cursory look at NWS API service and see nothing regarding the absence of a 200 status code. Is there any visibility of the issue you're aware of?

I've also looked here..

https://github.com/weather-gov/api/discussions

...and this is the closest post I've seen regarding data expiration.

weather-gov/api#719

The responder doesn't appear to be a civil servant.