proxytest: proxy HTTPS request using MITM

[AndersonQ]

What does this PR do?

It extends proxytest to support proxying HTTPS requests.

• The proxy can now act as a MITM (man-in-the-middle) proxy for HTTPS, generating certificates on the fly for the target host.
• The proxy assumes it should proxy an HTTPS request if it receives a CONNECT requests.

Why is it important?

It'll allow us to create integration tests for mTLS scenarios

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Disruptive User Impact

• None

How to test this PR locally

Run TestRunHTTPSProxy passing the example build tag and follow the instructions to perform a cRUL request using the proxytest:

• run the test: go test -tags example -v -run TestRunHTTPSProxy$ ./testing/proxytest
• run the provided curl command

Example:

$ go test -tags example -v -run TestRunHTTPSProxy$ ./testing/proxytest
=== RUN   TestRunHTTPSProxy
    proxytest_test.go:451: target server running on https://127.0.0.1:34355
    proxytest.go:100: [proxy-https] time=2024-10-16T11:14:33.982+02:00 level=INFO source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:236 msg="running on https://127.0.0.1:40241 -> https://localhost:40241"
    proxytest_test.go:473: proxy running on https://localhost:40241
    proxytest_test.go:479: make request to "https://not-a-server.co" using proxy "https://localhost:40241"
    proxytest_test.go:481: curl \
        --proxy-cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy-cert /tmp/TestRunHTTPSProxy3155179289/001/clientCert.crt \
        --proxy-key /tmp/TestRunHTTPSProxy3155179289/001/clientCert.key \
        --cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy https://127.0.0.1:40241 \
        https://not-a-server.co
    proxytest_test.go:496: CTRL+C to stop
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=INFO source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:183 msg="STARTING - CONNECT '//not-a-server.co:443' HTTP/1.1 127.0.0.1:40496"
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:22 msg="handling CONNECT" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:34 msg="hijacked request" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.092+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:282 msg="original URL: https://not-a-server.co:443/, new URL: https://127.0.0.1:34355/"
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.097+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:129 msg="EOF reached, finishing HTTPS handler" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] [8a3b5c0f-50c0-40d1-84af-59412ed8bdc3] DONE 0 - CONNECT //not-a-server.co:443 HTTP/1.1 127.0.0.1:40496
^Csignal: interrupt

$ curl \
        --proxy-cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy-cert /tmp/TestRunHTTPSProxy3155179289/001/clientCert.crt \
        --proxy-key /tmp/TestRunHTTPSProxy3155179289/001/clientCert.key \
        --cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy https://127.0.0.1:40241 \
        https://not-a-server.co
It works!

Related issues

• Closes Improve test proxy/mock fleet server to proxy HTTPS connections #4903

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @AndersonQ? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[AndersonQ]

once elastic/elastic-agent-libs#240 is merged it will be updated to v0.12.2

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[swiatekm]

Looks correct overall, had some questions about writing HTTP requests by hand.

[swiatekm]

This would look better if we instead created a http.Response and written it to the buffer.

[AndersonQ]

@swiatekm I've finally made the changes you requested. Sorry I ended up force pushing them

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube