Potential issue with Caddy terminating HTTP Put requests when uploaded content exceeds 1MB limit

[Anduin2017]

Hi, Caddy team.

We deployed a Nextcloud server with caddy as reverse proxy.

Our configuration is:

{
	email [email protected]
	log {
		format transform `{request>remote_ip}	{request>method}		{request>host}			{status}	{resp_headers->Server} 	in:{duration}	{request>uri}` {
            
        	}
		output file /var/log/caddy/caddy.log {
			roll_size 1gb
			roll_uncompressed
		}
		level info
	}
	servers :443 {
		metrics
		listener_wrappers {
			proxy_protocol {
				timeout 2s
				allow 127.0.0.1/24
			}
			http_redirect
			tls
		}
	}
}

# Protected by 2FA.
nextcloud.aiursoft.cn {
	#tls {
 	#   alpn http/1.1
	#}
	log
	@hacker {
		not remote_ip 10.0.0.0/10 172.16.50.0/24
		path /index.php/settings/admin*
	}
	encode gzip

	handle @hacker {
		header Content-Type text/html
		respond "<h1>You are not the admin! Don't try to admin my Nextcloud server :)</h1>" 401
	}

	reverse_proxy http://nextcloud {
	}

	redir /.well-known/carddav /remote.php/carddav 301
	redir /.well-known/caldav /remote.php/caldav 301
}

And we built our own caddy with script:

echo "Building caddy..."
xcaddy build  --with github.com/caddyserver/transform-encoder --output /tmp/caddy
sudo systemctl stop caddy.service
sudo cp /tmp/caddy /usr/bin/

Today there came an incident indicating all uploads with file larger than 1M was failed in Nextcloud.

Log from Nextcloud:

{
    "reqId": "05EtXnNGDRndxVbZizqT",
    "level": 3,
    "time": "2023-10-15T17:30:17+00:00",
    "remoteAddr": "10.0.0.184",
    "user": "anduin",
    "app": "no app in context",
    "method": "PUT",
    "url": "/remote.php/dav/uploads/anduin/8929679637737091d80ac726efcff350/000001",
    "message": "Expected filesize of 19292077 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 1040384 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.",
    "userAgent": "Mozilla/5.0 (Android) Nextcloud-android/3.26.0",
    "version": "27.1.0.7",
    "exception": {
        "Exception": "Sabre\\DAV\\Exception\\BadRequest",
        "Message": "Expected filesize of 19292077 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 1040384 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.",
        "Code": 0,
        "Trace": [
            {
                "file": "/var/www/html/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php",
                "line": 149,
                "function": "put",
                "class": "OCA\\DAV\\Connector\\Sabre\\File",
                "type": "->"
            },
            {
                "file": "/var/www/html/nextcloud/apps/dav/lib/Upload/UploadFolder.php",
                "line": 50,
                "function": "createFile",
                "class": "OCA\\DAV\\Connector\\Sabre\\Directory",
                "type": "->",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
                "line": 1098,
                "function": "createFile",
                "class": "OCA\\DAV\\Upload\\UploadFolder",
                "type": "->",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",
                "line": 504,
                "function": "createFile",
                "class": "Sabre\\DAV\\Server",
                "type": "->",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",
                "line": 89,
                "function": "httpPut",
                "class": "Sabre\\DAV\\CorePlugin",
                "type": "->"
            },
            {
                "file": "/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
                "line": 472,
                "function": "emit",
                "class": "Sabre\\DAV\\Server",
                "type": "->"
            },
            {
                "file": "/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
                "line": 253,
                "function": "invokeMethod",
                "class": "Sabre\\DAV\\Server",
                "type": "->"
            },
            {
                "file": "/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
                "line": 321,
                "function": "start",
                "class": "Sabre\\DAV\\Server",
                "type": "->"
            },
            {
                "file": "/var/www/html/nextcloud/apps/dav/lib/Server.php",
                "line": 365,
                "function": "exec",
                "class": "Sabre\\DAV\\Server",
                "type": "->"
            },
            {
                "file": "/var/www/html/nextcloud/apps/dav/appinfo/v2/remote.php",
                "line": 35,
                "function": "exec",
                "class": "OCA\\DAV\\Server",
                "type": "->"
            },
            {
                "file": "/var/www/html/nextcloud/remote.php",
                "line": 172,
                "args": [
                    "/var/www/html/nextcloud/apps/dav/appinfo/v2/remote.php"
                ],
                "function": "require_once"
            }
        ],
        "File": "/var/www/html/nextcloud/apps/dav/lib/Connector/Sabre/File.php",
        "Line": 297,
        "message": "Expected filesize of 19292077 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 1040384 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.",
        "exception": [],
        "CustomMessage": "Expected filesize of 19292077 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 1040384 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side."
    },
    "id": "652c23ffc87a3"
}

All clients are failing with the same issue.

After downgrading to v2.7.4, the problem was mitigated. And uploads was successful.

So I believe this issue has no relationship with Nextcloud. Might because 2.7.5 may terminate an HTTP put request?

I tried to use caddy to reverse proxy other services, but all works fine.

I tried to disable HTTP2, and doesn't help.

[mohammed90]

If you know your way around git, could be bisect before, on, and after this commit: 58ab3a0 ?

Also, is the client configured with http: or https: URL?

[francislavoie]

I see that you're using the http_redirect listener wrapper. Can you try without it?

[WeidiDeng]

Yeah, the problem is with that commit. I configured the read limit to another large size and the problem is gone.

Caddy should lift the limit once the first request is determined to be not an http request. For now, you can configure the read limit to be an arbitrary large number (it's only configurable through json though) or use v2.7.4 until a fix is found.

[WeidiDeng]

@Anduin2017 Can you try build caddy with xcaddy build fix-http-redirect-listener and see if the problem is fixed.

[antage]

I have the same problem with pushing big docker images in private docker registry behind caddy.
git bisect shows the problem was introduced by 58ab3a0 commit.
I tried the branch fix-http-redirect-listener and it has no problem.