owtaylor / flatpak-indexer Goto Github PK

In flatpak_indexer/datasource/fedora/bodhi_change_monitor.py, we want to make sure that the updates thread either keeps running or sets an error so that the main process exits. But in:

        # Then we use a timeout of None (never), until the channel is closed
        for method, properties, body_json in channel.consume(queue_name,
                                                             inactivity_timeout=None):
            self._update_from_message(body_json)

This potentially doesn't happen. "The active generator iterator terminates when the consumer is cancelled by client via BlockingChannel.cancel() or by broker" - we never cancel the consumer, but the broker might - for inactivity, just because, etc.

This seems to possibly have happened for the Fedora indexer:

https://pagure.io/fedora-infrastructure/issue/10415

So we should add logic to reconnect in this case. (If we get a new queue on reconnection, we'll need to requery from scratch - so there will need to be some logic changes in the way that updater.py and bodhi_change_monitor.py interact.)

fedora/release_info.py is a liability, because when things change - e.g., a new Flatpaks release is added in Fedora - the indexer breaks in a hard-to-notice fashion.

We can get the same information directly from Bodhi using the https://bodhi.fedoraproject.org/releases/?exclude_archived=1

Endpoint. Basic idea is:

• Add bodhi_query.py: refresh_updates()
• Have that cache a JSON-object in redis like:

[
{
   "name": "F34F",
   "content_type": "flatpak",
   "branch": "f34"
}
]

• Call refresh_* updates() once per update cycle, then load and use it where we currently use release_info.py. (Probably pass the loaded result as a parameter to bodhi_update.py:_query_updates() to reduce redis round-trips.)

We should clean up deltas that aren't in use any more.

It probably is necessary to have some sort of grace period so that clients using an old downloaded version of the index don't get a 404. But how to determine when a tardiff result or manifest is "recently unused"

A) update the mtime on reuse?
B) keep some sort of "tardiff:expiring" key in redis
C) ?

We actually have the same issue with icons - we clean them up without any sort of grace period.

In a test Fedora instance:

• Delta generation was failing (timing out) for some tardiffs repeatedly, probably because of stuck downloads
• After a day the tardiff:: entries timed out
• The tardiff containers then crashed because of the missing spec entries

There's multiple things to fix here, but in general, I think if tardiffs haven't been processed after a couple of hours, the indexer should consider that they won't be, and mark them temporarily/permanently failed and move on to generating the index.

If resolving rabbitmq.fedoraproject.org fails, then socket.gaierror is thrown. This leaves fedora-monitor in a stuck state because that's not caught by:

and then the exception is rethrown when the main thread tries to retrieve changes from the monitor. Adding socket.gaierror there is likely the right thing to do, though perhaps we also need to think about the handling of exceptions during the update process - perhaps the

            try:
                indexer.index(registry_data)
            except Exception:
                logger.exception("Failed to create index")
                continue

in cli/daemon.py is wrong and we should let the app exit, so that openshift will restart it in a clean state?