Slowed down instance due to *.nextcloud.com outages?

[adroslice]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Yesterday, my own Nextcloud instance (manual install on Debian container) becane extremely unresponsive. Each site / authenticated route would take 5-10 minutes to load, with the only exceptions being the updater when directly navigated to rather than using the settings "Open Updater", and the login screen for unauthenticated browser sessions.
I attempted to backup and update to perhaps alleviate the issue, however found that docs.nextcloud.com and download.nextcloud.com were down, taking a very long time to time out, almost similar to my own instance, rendering me unable to update, but providing a possible line of reasoning.

The only explanation I could come up with was that perhaps the built-in update notification app was attempting to fetch the newest version from its endpoint, which I later dug out of the code to be updates.nextcloud.com/updater_server, but experiencing the same issue i had with docs and download, causing the loading to be delayed until the request finally either timed out or succeeded.

Steps to reproduce

This reproduction is untested.

1. Create a mock webserver that delays timing out requests as long as possible, possibly using retry headers
2. Add a hosts entry to the server hosting Nextcloud to point to that webserver
3. Add your test user to a group that receives update notifications
4. Attempt to log in with that user and load the files app

If my theory is correct, then it should be loading for a long time.

Expected behavior

I expected to load the site in a reasonable amount of time given my setup, despite nextcloud services being down. As a fix, if I've correctly identified the issue, I would propose adding a manual timeout to the fetching of updates, perhaps 1-5 seconds. Low enough not to be a major inconvenience, but high enough to not prematurely terminate would-be-successful requests.

Installation method

None

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "REDACTED PUBLIC DOMAIN",
            "REDACTED LOCAL IP"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "27.0.2.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "overwrite.cli.url": "REDACTED PUBLIC URL",
        "htaccess.RewriteBase": "\/",
        "default_phone_region": "DE",
        "overwritehost": "REDACTED PUBLIC DOMAIN",
        "overwriteprotocol": "https",
        "filelocking.enabled": true,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "allow_local_remote_servers": true,
        "twofactor_enforced": "false",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "maintenance": false,
        "theme": "",
        "loglevel": "1",
        "simpleSignUpLink.shown": false,
        "updater.release.channel": "stable",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "enable_previews": true,
        "preview_max_x": 1080,
        "preview_max_y": 1080,
        "preview_max_memory": 512,
        "enabledPreviewProviders": [
            "OC\\Preview\\BMP",
            "OC\\Preview\\GIF",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\MarkDown",
            "OC\\Preview\\MP3",
            "OC\\Preview\\PNG",
            "OC\\Preview\\TXT",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\OpenDocument",
            "OC\\Preview\\Krita",
            "OC\\Preview\\Movie"
        ],
        "memories.exiftool": "\/var\/www\/html\/nextcloud\/apps\/memories\/exiftool-bin\/exiftool-amd64-glibc",
        "memories.vod.path": "\/var\/www\/html\/nextcloud\/apps\/memories\/exiftool-bin\/go-vod-amd64",
        "memories.vod.ffmpeg": "\/usr\/bin\/ffmpeg",
        "memories.vod.ffprobe": "\/usr\/bin\/ffprobe",
        "app_install_overwrite": [],
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - admin_audit: 1.17.0
  - calendar: 4.4.4
  - cfg_share_links: 4.1.0
  - cloud_federation_api: 1.10.0
  - contacts: 5.3.2
  - dav: 1.27.0
  - external: 5.2.0
  - federatedfilesharing: 1.17.0
  - files: 1.22.0
  - files_pdfviewer: 2.8.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - forms: 3.3.1
  - impersonate: 1.14.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - memories: 5.4.1
  - metadata: 0.19.0
  - music: 1.8.4
  - notes: 4.8.1
  - oauth2: 1.15.1
  - provisioning_api: 1.17.0
  - related_resources: 1.2.0
  - richdocuments: 8.1.1
  - settings: 1.9.0
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - updatenotification: 1.17.0
  - viewer: 2.1.0
  - workflowengine: 2.9.0
Disabled:
  - activity: 2.19.0 (installed 2.14.3)
  - bruteforcesettings: 2.7.0 (installed 2.1.0)
  - circles: 27.0.1 (installed 22.1.1)
  - comments: 1.17.0 (installed 1.11.0)
  - contactsinteraction: 1.8.0 (installed 1.2.0)
  - dashboard: 7.7.0 (installed 7.1.0)
  - encryption: 2.15.0
  - federation: 1.17.0 (installed 1.11.0)
  - files_external: 1.19.0 (installed 1.13.0)
  - firstrunwizard: 2.16.0 (installed 2.10.0)
  - nextcloud_announcements: 1.16.0 (installed 1.10.0)
  - notifications: 2.15.0 (installed 2.9.0)
  - onlyoffice: 8.2.0 (installed 8.2.0)
  - password_policy: 1.17.0 (installed 1.11.0)
  - photos: 2.3.0 (installed 1.6.0)
  - privacy: 1.11.0 (installed 1.5.0)
  - recommendations: 1.6.0 (installed 1.0.0)
  - registration: 2.2.0 (installed 2.2.0)
  - serverinfo: 1.17.0 (installed 1.11.0)
  - sharebymail: 1.17.0 (installed 1.11.0)
  - support: 1.10.0 (installed 1.4.0)
  - survey_client: 1.15.0 (installed 1.9.0)
  - suspicious_login: 5.0.0
  - systemtags: 1.17.0 (installed 1.14.0)
  - twofactor_totp: 9.0.0
  - user_ldap: 1.17.0
  - user_status: 1.7.0 (installed 1.1.1)
  - weather_status: 1.7.0 (installed 1.1.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Excerpt of warnings and errors:
- This one I fixed by removing dead DB entries, which dind't help: {"reqId":"YqbwlDzCeeCqqOQ9kXh6","level":2,"time":"2023-08-28T09:11:56+00:00","remoteAddr":"","user":"--","app":"files","method":"","url":"--","message":"User REDACTED still has unscanned files after running background scan, background scan might be stopped prematurely","userAgent":"--","version":"27.0.1.2","data":{"app":"files"},"id":"64edaab62c8fd"}

Apart from that, there were no errors or warnings, though at the info level there were multiple messages of Memories completing indexing and richdocuments fetching capabilities from my collabora instance. Another info message that struck me as odd stated "Could not resolve OCA\Circles\CirclesManager! Class "OCA\Circles\CirclesManager" does not exist", caused by related_resources.

Additional info

• Hosted on a Proxmox Hypervisor
• uname -a: Linux nextcloud 6.2.16-3-pve Name the repository "server" #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) x86_64 GNU/Linux
• Sits behind a Caddy reverse proxy