placeholders not support {http.request.host}

[yulewang]

Hi,

I have a generic domain name and would like to use the replace-response to reach the replacement of the real domain name requested by the user. But always replace by text:{http.request.host}

As in the following configuration file:

*.example.com:443 {
  replace {
    example.com {http.request.host}
  }

Is there any way to make it happen? Thanks!

[francislavoie]

Please share your full config. Show an example request with curl -v. Show your logs.

[yulewang]

Actually, I have a site deployed to cloudflare.
I'm hoping to use Caddy to enable a generic domain name address proxy the "cloudflare domain" for escape some blocking.
And using replace-response to replace the body contents with the domain name of the user's real request.

Caddy --version:

v2.7.6

Caddyfile:

{
  order replace after encode
}

*.Bexample.com:443 {
  tls {
    dns cloudflare xxxxxxxxxx
  }
  log {
    output file /var/log/caddy/Bexample.com.log
  }
  encode gzip
  replace {
    Aexample.com {http.request.host} //Here's the part that won't work: the get is the text: {http.request.host}, not the user request "*.Bexample.com"
  }
  reverse_proxy * https://Aexample.com {
    header_up Host Aexample.com
    header_up Accept-Encoding identity

    header_up X-Real-IP {http.request.remote.host}
    header_up X-Forwarded-For {http.request.remote.host}
    header_up REMOTE-HOST {http.request.remote.host}
    header_down Set-Cookie Aexample.com Bexample.com
    header_up Referer {http.request.tls.server_name}
  }
}

[francislavoie]

We also need your logs (enable the debug global option) and an example request with curl -v.

It's important. Whether it works or not depends on how your upstream responds.

[yulewang]

sure, I created a test.html and content is 'Aexample.com' for show the "curl -v"

curl -v https://testsub.Bexample.com/clients/test.html

*   Trying ip:443...
* Connected to testsub.Bexample.com (ip) port 443 
(#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.Bexample.com
*  start date: Feb  5 08:53:39 2024 GMT
*  expire date: May  5 08:53:38 2024 GMT
*  subjectAltName: host "testsub.Bexample.com" matched cert's
 "*.Bexample.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after u
pgrade: len=0
* Using Stream ID: 1 (easy handle 0x559800709b20)
> GET /clients/test.html HTTP/2
> Host: testsub.Bexample.com
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< accept-ranges: bytes
< alt-svc: h3=":443"; ma=2592000
< cf-cache-status: DYNAMIC
< cf-ray: 852a5477fbf72abb-LAX
< content-type: text/html
< date: Fri, 09 Feb 2024 07:16:58 GMT
< last-modified: Fri, 09 Feb 2024 06:43:04 GMT
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/
report\/v3?s=z%2B6oC5GpNhlhCMLeZJPGZivKJUEM0NLPhddh%2FWpaoRNaY8TH9FOlz2
dtKqXhOj1SAiyjWS%hddhdh%2FnmN9wokIP2lhbwH4ai%2FpjSb8neNl
9jD%2Bl0lEAsHD5Zj3fjfjfJVRaG0Ck3mmSdDik%3D"}],"group":"cf-nel","max_
age":604800}
< server: Caddy
< server: cloudflare
< strict-transport-security: max-age=31536000
< 
* Connection #0 to host testsub.Bexample.com left intact
{http.request.host}

the last line is: {http.request.host}

[mholt]

Is your backend compressing the response (does it support gzip compression)? If so, the replacer can't operate on compressed payloads...

[eth-limo]

@mholt @francislavoie I was able to confirm this issue as well using a simple NodeJS backend for testing:

const http = require("http");

const server = http.createServer((req, res, next) => {
  console.log(req.url);
  console.log(req.rawHeaders);
  res.writeHead(200);
  res.write("This is a test", "utf8");
  res.end();
});

server.listen(process.argv[2]);

$ node index.js 8181

Using the following Caddyfile:

{
	admin off
	auto_https off

	local_certs

	log {
		level DEBUG
		format console
	}

	order replace after encode
}

:8443 {
	log {
		level INFO
		format console
	}

	bind 0.0.0.0

	tls internal {
		on_demand
	}

	reverse_proxy http://localhost:8181

	replace {
		"test" "I was replaced"
	}
}

Responses are rewritten when using a string with the replacement handler: "I was replaced":

curl https://localhost:8443 -v
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none

< HTTP/2 200 
< alt-svc: h3=":8443"; ma=2592000
< date: Tue, 16 Apr 2024 19:08:36 GMT
< server: Caddy
< 

This is a I was replaced%                        

However when using placeholder {host} or {http.request.host} for the replacement value, {http.request.host} is returned instead of being properly interpolated.

Caddyfile:

{
	admin off
	auto_https off

	local_certs

	log {
		level DEBUG
		format console
	}

	order replace after encode
}

:8443 {
	log {
		level INFO
		format console
	}

	bind 0.0.0.0

	tls internal {
		on_demand
	}

	reverse_proxy http://localhost:8181

	replace {
		"test" {host}
	}
}

Response:

curl https://localhost:8443 -v        
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none

< HTTP/2 200 
< alt-svc: h3=":8443"; ma=2592000
< date: Tue, 16 Apr 2024 19:09:57 GMT
< server: Caddy
< 

This is a {http.request.host}%  

[mholt]

Oh, you know, this is because, for efficiency reasons, we allocate the transformer chain at Provision-time instead of Request-time. So there is no {http.request.host} at the time the transformer chain is created.

I don't know for sure, but @icholy's excellent replace package does have methods like RegexpStringFunc() which might(?) allow us to allocate the transformers once which end up calling our function at replace-time, thus allowing us to evaluate placeholders at request-time.

There is no StringFunc() method though. @icholy Am I on the right track, for enabling dynamic evaluation of what the replacement values should be?

[eth-limo]

Oh, you know, this is because, for efficiency reasons, we allocate the transformer chain at Provision-time instead of Request-time. So there is no {http.request.host} at the time the transformer chain is created.

I don't know for sure, but @icholy's excellent replace package does have methods like RegexpStringFunc() which might(?) allow us to allocate the transformers once which end up calling our function at replace-time, thus allowing us to evaluate placeholders at request-time.

There is no StringFunc() method though. @icholy Am I on the right track, for enabling dynamic evaluation of what the replacement values should be?

Great! Looking forward to seeing what can be done. In our use case, we don't always know what the replacement value should be since we support many vhosts, so being able to dynamically replace would be amazing.

[Jonathazn]

Hello,

I'm trying to replace ngcspnonce in an angular app with this module as well as https://github.com/luludotdev/caddy-requestid so that it's a different nonce on every request. In the header the caddy-requestid module is correctly setting the nonce, but when I try to replace it using this module, it's replacing the string with the string literal. Am I correct in assuming this is because of the same issue here?

My Caddyfile:

https://www.subdomain.domain.tld {
    reverse_proxy example:80

    # Define the request ID module
    request_id {
    }

    # Define the replace-response module
    replace {
        nonce {request_id} # nonce is the value for ngcspnonce in the angular app, nonce becomes {request_id}
    }
}

[floverfelt]

@Jonathazn can confirm, I came here for the same issue - I was attempting to add nonce support to an SPA served using Caddy.

There doesn't appear to have a way to add nonce support by combining the uuid or other placeholders and this plugin due to this bug.

[mholt]

I'd be open to contributions to help address this if anyone has some time :)

[jshenguru]

I'd be open to contributions to help address this if anyone has some time :)

@mholt I don't see any way we can solve this without modifying the icholy/replace package.

I did the following and the performance is great (even faster than direct access to the upstream web site):

1. I modified the RegexpString() function in the replace.go file of the icholy/replace package into the following (introducing a function as the 3rd argument)

func RegexpString(re *regexp.Regexp, template string, replace func(string) string) *RegexpTransformer {
	return RegexpIndexFunc(re, func(src []byte, index []int) []byte {
		return re.Expand(nil, []byte(replace(template)), src, index)
	})
}

2. I added a member to the type Handler in the handler.go file of the caddyserver/replace-response package:

repl *caddy.Replacer

3. I added a member method of type Handler in the handler.go file of the caddyserver/replace-response package:

func (h *Handler) ReplaceRegexp(dst string) string {
	return h.repl.ReplaceKnown(dst, "")
}

4. I modified the call on replace.RegexpString() in the Provision function in the handler.go file of the caddyserver/replace-response package:

tr := replace.RegexpString(repl.re, finalReplace, h.ReplaceRegexp)

5. I added the following to the beginning of the ServerHTTP function in the handler.go file of the caddyserver/replace-response package:

	repl := r.Context().Value(caddy.ReplacerCtxKey).(*caddy.Replacer)
	h.repl = repl

In my use case, I have 3 blocks of replace directive each block has it's own matcher and up to 4 regexp replacements and each replacement is using some placeholder. The upstream server/website is in Europe. The average COMContentLoaded time is 4 seconds using direct access and 3 seconds using Caddy plus the modified replace-response module.

I am very happy with the result.

Please share your comments and thoughts on this.

I am more than happy to create a PR on this if you think my solution is acceptable.

[mholt]

@jshenguru That looks promising! Could you open a PR here for review? Then if all looks good we can figure out how to update the replace package upstream.

[jshenguru]

@jshenguru That looks promising! Could you open a PR here for review? Then if all looks good we can figure out how to update the replace package upstream.

@mholt Done. Please review