Properly authenticate at proxy server

[danielmitterdorfer]

With this commit Rally will set the proper authentication header for
proxies that require it.

Closes #636

[dliappis]

I am documenting how I tested this, in case we want to add an integration test later on for similar stuff:

cat >squid.conf <<EOF
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/squidpasswords
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED
http_access allow authenticated
http_port 3128
EOF

Created a squidpasswords file using htpasswd -c squidpasswords testuser username testuser and password testuser with the following content:

cat >squidpasswords <<EOF
testuser:$apr1$WQMVypS2$vUzNTPn6tKhTrRZP16/li.
EOF

and run:

docker run --rm --name squid -d -v $PWD/squidpasswords:/etc/squid/squidpasswords -v $PWD/squid.conf:/etc/squid/squid.conf -p 3128:3128 datadog/squid

and to see access logs:

docker exec -ti squid tail -f /var/log/squid/access.log

On the Rally side I first set:

export http_proxy=http://testuser:[email protected]:3128

Using current Rally master commit:

esrally --distribution-version=6.6.0 --track="geonames" --track-params="ingest_percentage:15" --challenge="append-no-conflicts-index-only"

shows on the squid log:

1550851149.662      0 172.17.0.1 TCP_DENIED/407 3871 CONNECT github.com:443 - HIER_NONE/- text/html
1550851149.665      0 172.17.0.1 TCP_DENIED/407 3871 CONNECT github.com:443 - HIER_NONE/- text/html
1550851149.667      0 172.17.0.1 TCP_DENIED/407 3871 CONNECT github.com:443 - HIER_NONE/- text/html
1550851149.668      0 172.17.0.1 TCP_DENIED/407 3871 CONNECT github.com:443 - HIER_NONE/- text/html

plus of course Rally itself reports:

[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.

Using this PR with the same Rally command, squid log shows:

1550851566.691  60860 172.17.0.1 TCP_TUNNEL/200 86897 CONNECT github.com:443 testuser HIER_DIRECT/192.30.253.112 -

and the warning is gone from the Rally output.

[dliappis]

LGTM also based on #652 (comment)

[danielmitterdorfer]

@dliappis great writeup! I have added integration tests now in 421f07c based on your comment. Can you please have another look?

[danielmitterdorfer]

For Docker on MacOS this requires to (manually) add /var/folders to the list of allowed paths (otherwise Docker will complain that it can't bind-mount the directory). /var/folders is actually symlinked to /private/var/folders and /private is already configured (recursively) for Docker but not /var/folders is not. I think we should add this to our developer docs. Wdyt?

[danielmitterdorfer]

As discussed via another channel we'll instead use an already existing directory tree that we use for integration tests anyway. I'll push a new commit addressing this.

[dliappis]

LGTM, thanks for adding the integration test!