[Bug]: Very slow access to apps.nextcloud.com/api/v1/apps.json for checking app updates about server

[bigcat88]

This issue is related more to the appstore and related to infrastructure, imho.

Sometimes getting of json from "api/v1/apps.json" can fail(when appstore is under high load), but in 98-99% cases it should be fine.

[bigcat88]

Here is mine.

I was wrong.
It seems that this is not a problem with the appstore code itself, since during high loads the appstore simply does not serve the file at all, but if it has already started serving, it does it quickly, because the file itself is given by NGINX(reverse proxy).

[joshtrichards]

Any updates guys? :'(

I think the challenge is we're having a hard time reproducing this and there doesn't seem be an explanation for why your environment is experiencing this.

It's tough to fix without a clear (or even suspected possible) cause. :-)

[hammeractual]

This issue is related more to the appstore and related to infrastructure, imho.

Sometimes getting of json from "api/v1/apps.json" can fail(when appstore is under high load), but in 98-99% cases it should be fine.

Could you please tell the infrastructure division/department to check/troubleshoot the issue?

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

[bigcat88]

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

It works for me now and has worked since the very beginning of today.
there was no peak load in the appstore today, there shouldn't have been any slowdowns

[hammeractual]

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

It works for me now and has worked since the very beginning of today. there was no peak load in the appstore today, there shouldn't have been any slowdowns

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

[joshtrichards]

@hammeractual What does curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test from your Nextcloud Server's command line look like? Here's how it looks for me from a test instance:

www-data@ef86c134f021:/tmp$ curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.9M  100 13.9M    0     0  6545k      0  0:00:02  0:00:02 --:--:-- 6548k
www-data@ef86c134f021:/tmp$ 

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 903254 out of 6408674 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

#40082 has a manual approach for doing so. Also it's tracking the feature request.

[hammeractual]

@hammeractual What does curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test from your Nextcloud Server's command line look like? Here's how it looks for me from a test instance:

www-data@ef86c134f021:/tmp$ curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.9M  100 13.9M    0     0  6545k      0  0:00:02  0:00:02 --:--:-- 6548k
www-data@ef86c134f021:/tmp$ 

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 903254 out of 6408674 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

#40082 has a manual approach for doing so. Also it's tracking the feature request.

Thanks for the info

Here is mine.

Meanwhile, the ping result is pretty good to that address.

[joshtrichards]

Meanwhile, the ping result is pretty good to that address.

Something does not add up. Your throughput is incredibly slow relative to your latency. You're getting like 6558 B/s (!).

Can you sanity check that curl performance by downloading something from elsewhere:

Maybe curl https://github.com/nextcloud/server/archive/refs/tags/v28.0.5.zip -o /dev/null

P.S. Is your Server @ Hetzner?

[hammeractual]

Meanwhile, the ping result is pretty good to that address.

Something does not add up. Your throughput is incredibly slow relative to your latency. You're getting like 6558 B/s (!).

Can you sanity check that curl performance by downloading something from elsewhere:

Maybe curl https://github.com/nextcloud/server/archive/refs/tags/v28.0.5.zip -o /dev/null

P.S. Is your Server @ Hetzner?

Here is mine

Quite fast while curl to another server/address.

[hammeractual]

Any updates guys? :'(

I've tried to increase timeout value in Fetcher.php to 1800 seconds, but no success.

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 1800000 milliseconds with 3661824 out of 6421307 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json GET /settings/apps from x.x.x.x by e2cd37b6-6ebf-103d-84c2-addc165358a8 at 7 Mei 2024, 17.14.29

[joshtrichards]

P.S. Is your Server @ Hetzner?

You may have missed this question. I asked because your ping times are ~1-2 ms.

At the time I was taking a shot in the dark that maybe there is something unusual about Hetzner's internal client-to-client connectivity. Today I ran across a note in the Portainer Knowledge Base while trying to help out a user on the forums that suggests Hetzner's private networking uses an MTU of 1450. Got me thinking about this again.