Function getAppsInfo()as used by serverinfo provokes IP-ban (Connection refused) by apps.nextcloud.com

[ernolf]

I reported this already in the appstore issue-tracker: nextcloud/appstore#1137
I'm posting this here, because I observed this behavior only when the AppstoreFetcher is called by serverinfo.

Steps to reproduce

1. make sure apps.nextcloud.com is accessible:

serverprompt:~# echo -e "\nResponse-Code: $(curl -Lo /dev/null -w "%{http_code}" apps.nextcloud.com)\n==================\n"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   320  100   320    0     0   8396      0 --:--:-- --:--:-- --:--:--  8421
100  301k  100  301k    0     0   271k      0  0:00:01  0:00:01 --:--:--  319k

Response-Code: 200
==================

2. tail -f your logfile: tail -F $(nextcloud logfile) | jq and keep it open
3. open serverinfo in a browser tab
4. whait for log message as below.
5. close serverinfo tab, since the message repeates every 5 minutes.
6. recheck accessibility as under 1:

serverprompt:~# echo -e "\nResponse-Code: $(curl -Lo /dev/null -w "%{http_code}" apps.nextcloud.com)\n==================\n"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to apps.nextcloud.com port 80 after 22 ms: Connection refused

Response-Code: 000
==================

Expected behaviour

no impairments in the connection to apps.nextcloud.com

Actual behaviour

The affected IP get banned/blocked by apps.nextcloud.com.
Since it is my home network, I can easily fetch a new IP but every time a serverinfo tab is opened (no matter what instance, even newly setup in a virtual machine) it get banned again.
Because in my case it's the WAN Ip of my home network, i can not reach the appstore in my browser eather.

Server configuration

Operating system:
Ubuntu 22.04.3 LTS
Web server:
Apache2
Database:
MySQL and PostgeSQL
PHP version:
8.1.0 and 8.2.10
Nextcloud version: (see Nextcloud admin page)
27+

Client configuration

Browser:
Chrome, Brave, Opera
Operating system:
Windows, Linux, Mac

Nextcloud log (data/owncloud.log)

{
  "reqId": "5snl2JvDWBJ4eLJYKjN7",
  "level": 2,
  "time": "2023-09-14T08:46:28+02:00",
  "remoteAddr": "192.168.188.36",
  "user": "ernolf",
  "app": "appstoreFetcher",
  "method": "GET",
  "url": "/apps/serverinfo/update",
  "message": "Could not connect to appstore: cURL error 7: Failed to connect to apps.nextcloud.com port 443 after 41 ms: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json",
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36",
  "version": "27.1.0.5",
  "data": {
    "app": "appstoreFetcher"
  }
}

My workaround:

coment this line:

serverinfo/lib/SystemStatistics.php

Line 74 in 544bacf

'apps' => $this->getAppsInfo()

//			'apps' => $this->getAppsInfo()

I hope this helps.

[joshtrichards]

I suspect this is a longstanding bug, though I'm not sure why it hasn't come up more often (maybe it has but no one realized this was the cause). See #145 (comment)

We can probably address both matters at once.

[bigcat88]

cool, I got ban from the appstore server, when installed serverinfo app and visit it page 😂😂😂

[kesselb]

I had a brief look yesterday and the issue is more tricky than I thought 😒

1. I can confirm that the serverinfo endpoint triggers a http request to the apps store. While there is a lot of caching involved, we still send a request with the etag to validate that the local version of the apps.json is up-to-date. This information, apps.json is up-to-date, is not cached unfortunately and therefore, using the apps' manager from the serverinfo endpoint indeed unfavorable.

2. The serverinfo endpoint is actually a big mess. We mixed short and long living data. It makes sense to fetch the load or free memory reguallary, but the updater situtation is more stable and doesn't need a refresh every few seconds. We really need to split those. However, I cannot just change it because that would break existing setups.

3. Sadly, we are not even using the information if app updates are available from the serverinfo view 😞

As a short-term solution, I will implement an additional get parameter for the monitoring endpoint to skip the updater check and let our frontend use. This should relax the situation a bit.

The long-term solution should be an endpoint v2 with different endpoints for short- and long-living data together with some of the ideas from #284.

[kesselb]

I also looked at moving the update server and update apps information to an own background job, like we do for the storage statistics.

That would work, but a second cache for updates brings on the question of when to clear the cache. Let's assume we run such a background job every 30 minutes. You update and at worst it takes 30 minutes for the monitoring to notice the status change. A repair job to clear the update cache can solve this problem. But then we don't have any information for at worst the next 30 minutes until the job runs. That can be solved by fetching the data right away in the repair job, but that adds another dependency for the critical update process. Or we let the monitoring endpoint refresh the data when nothing is cached, but then the situation is not much better than right now. You visit the page and within a few seconds we trigger a couple of http requests.

That adds a lot of complexity for a feature/metric that, from my perspective, not even belong here.

[ernolf]

for a feature/metric that, from my perspective, not even belong here.

Exactly, I fully agree!
In my understanding the server monitoring app is intended to monitor the underlying layer of the hosting server. The apps aren't part of that layer at all, so strictly speaking, no one should expect them there.
It only produces unneccesary server load.