last_modified header support + optimisation

[bigcat88]

This is a dev PR, after testing it will be adjusted if everything is ok.

The reason for this PR:
Most requests to the AppStore do not send ETag and this leads to a very high load.

This should help with this: the binary result of the request /api/v1/apps.json will be stored in Redis, and the next time if the request does not have an ETag in the header, but the response for this Etag is already in Redis, the result will be directly returned from Redis.

Changes in PR:

1. Added Last Modified header in addition to ETag, feel free to use any of this headers you like.
2. We GZip content of api/v1/apps.json and return it to Nginx, so Nginx will not gzip it on his own for every request and will just return it from cache.

By what percentage this will reduce the load it will soon become clear; the PR will be edited with these data.

Local benchmarks(apps.nextcloud.com is local url in this case):

    start = perf_counter()
    for i in range(100):
        a = requests.get("https://apps.nextcloud.com/api/v1/apps.json")
    print(perf_counter() - start)
    exit()

Original code:
Cold/Hot run: 27.58 / 24.42

With this PR:
Cold/Hot run: 12.56 / 8.82

Expected load reduction in a production environment on the server: 2 to 4 times.

Still will be very high load during app release deletion or uploading, but it is another complex topic.

[officialdanielamani]

I come from #761 about Appstore not show in the Nextcloud (just first time install using Docker)

On my Nextcloud log there are many cURL error 28 for https://apps.nextcloud.com/api/v1/apps.json

{"reqId":"IzICrWBKJiXVWLTDkP9z","level":2,"time":"2023-09-26T12:50:51+00:00","remoteAddr":"10.100.42.2","user":"XXXXX","app":"appstoreFetcher","method":"GET","url":"/settings/apps","message":"Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 688128 out of 5567101 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36","version":"27.1.1.0","data":{"app":"appstoreFetcher"}}

So I try speed test on my browser and load the JSON, it took about 7-15 minute to load the JSON file

My average speed is between 5-30 mbps but the JSON is painfully slow. Is this cause cURL 28 error and cause Appstore not showing on Nextcloud panel. Is there alternative to load Nextcload Appstore, like example replace https://apps.nextcloud.com/api/v1/apps.json with a local file apps.json

[bigcat88]

curl https://apps.nextcloud.com/api/v1/apps.json > tmp.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12.1M  100 12.1M    0     0  4216k      0  0:00:02  0:00:02 --:--:-- 4222k

@officialdanielamani can you retry?

[officialdanielamani]

I try running the command on docker container, this are the current result

root@fcdd0f7aa6b1:/#curl https://apps.nextcloud.com/api/v1/apps.json > tmp.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 12 12.1M   12 1522k    0     0  10850      0  0:19:31  0:02:23  0:17:08  3845

Is this cause by CDN or anther, as I can access https://apps.nextcloud.com/ normally and
https://apps.nextcloud.com/api/v1/categories.json can load without problem

root@fcdd0f7aa6b1:/#curl https://apps.nextcloud.com/api/v1/categories.json > tmp.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  126k  100  126k    0     0  31276      0  0:00:04  0:00:04 --:--:-- 31278

[andrey18106]

I try running the command on docker container, this are the current result

Tested locally and on remote server - apps.json downloading successfully by less then 1-2sec (75-100Mbit/s), so I would say that it's a slow internet connection issue.

[officialdanielamani]

If I have apps.json file, can I add to Nextcloud file so I no need to lookup every time I need load. The connection speed I cannot change because it's my internet speed plan.

[andrey18106]

If I have apps.json file, can I add to Nextcloud file so I no need to lookup every time I need load. The connection speed I cannot change because it's my internet speed plan.

There is already should be local apps.json file located in data/appdata_instanceid/appstore folder.

[officialdanielamani]

Is I just add apps.json and categories.json inside the folder and restart the Nextcloud to load the Appstore?
I check inside the folder there is empty.

[bigcat88]

What Update channel(settings/admin/overview) value do you have?
It should be stable

If it is stable NC stores app.json locally automatically

[officialdanielamani]

I try reinstall the Nextcloud maybe it will fixed the problem. Thank you for support and guide.

[officialdanielamani]

Edit: After waiting for hours suddenly it working. Speed test get about same result, so don't know why

The only clue is the apps.json on my Nextcloud the size is only 2.37MB while downloading direct from https://apps.nextcloud.com/api/v1/categories.json the size is over 12.1MB. On log file there is many duplicate if from direct website (and log file grew to 45MB).

I try on my friend better internet speed still have problem.

When I reinstall, there is option skip suggestion and install suggestion, but no preview of app. I click install but it just redirect to dashboard. The Appstore still missing and folder data/appdata_instanceid/appstore also blank. I also tried using this suggestion to increase timeout but still no luck.

https://help.nextcloud.com/t/cant-install-apps-curl-error-28-ssl-connection-timeout-solved/149920

Here is my docker compose

version: '2'

services:
  nextcloud:
    image: nextcloud:stable
    container_name: nextcloud
    restart: unless-stopped
    ports:
      - XXXXX:80
    volumes:
      - /portainer/Files/AppData/Config/Nextcloud/html:/var/www/html
    environment:
      - PUID=1000
      - PGID=1000
      - MYSQL_HOST=nextcloud-db
      - MYSQL_PASSWORD=XXXXX
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - REDIS_HOST=nextcloud-redis
      - REDIS_HOST_PASSWORD=XXXXX
      - REDIS_HOST_PORT=6379
    links:
      - nextcloud-db
      - nextcloud-redis
  nextcloud-db:
    image: mariadb:10.6
    container_name: nextcloud-db
    restart: unless-stopped
    command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
    ports:
      - XXXXX:3306
    volumes:
      - /portainer/Files/AppData/Config/Nextcloud/DB:/var/lib/mysql
    environment:
      - PUID=1000
      - PGID=1000
      - MYSQL_ROOT_PASSWORD=XXXXX
      - MYSQL_PASSWORD=XXXXX
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
  nextcloud-redis:
    image: redis:alpine
    container_name: nextcloud-redis
    restart: unless-stopped
    command: redis-server --save 20 1 --loglevel warning --requirepass XXXXX
    ports:
      - XXXXX:6379
    environment:
      - PUID=1000
      - PGID=1000
    volumes:
      - /portainer/Files/AppData/Config/Nextcloud/Redis:/data

[bigcat88]

I really don't know how I can help here. :(
The load on the app store server is normal, there have been no complaints/problems so far (except for this one).
All I can do is monitor the situation, and if something else comes up, maybe it will help identify a possible cause and I try to fix it, if there is anything to fix.

[bigcat88]

The only clue is the apps.json on my Nextcloud the size is only 2.37MB while downloading direct from https://apps.nextcloud.com/api/v1/categories.json the size is over 12.1MB. On log file there is many duplicate if from direct website (and log file grew to 45MB).

This is fine, 12.1 Mb size with all applications releases info, and NC stores only releases information for it's version.