Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix download from s3/gs by moving http query parameter downstream #1293

Merged
merged 7 commits into from
Jun 28, 2021
Merged

fix download from s3/gs by moving http query parameter downstream #1293

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7877ae9
fix download from s3/gs by moving http query parameter downstream
hendrikmuhs Jun 24, 2021
99fa08f
reformat
hendrikmuhs Jun 24, 2021
79f9071
pylint
hendrikmuhs Jun 24, 2021
b3571b5
Revert "pylint"
hendrikmuhs Jun 24, 2021
a64ac42
Revert "reformat"
hendrikmuhs Jun 24, 2021
aa2f3b4
Revert "fix download from s3/gs by moving http query parameter downst…
hendrikmuhs Jun 24, 2021
30432d8
fix download from s3/gs by only adding http query parameter for http
hendrikmuhs Jun 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion esrally/utils/net.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,8 +193,13 @@ def download_http(url, local_path, expected_size_in_bytes=None, progress_indicat
progress_indicator(bytes_read, size_from_content_header)
return expected_size_in_bytes


def add_url_param_elastic_no_kpi(url):
return _add_url_param(url, {"x-elastic-no-kpi": "true"})
scheme = urllib3.util.parse_url(url).scheme
if scheme.startswith("http"):
return _add_url_param(url, {"x-elastic-no-kpi": "true"})
else:
return url


def _add_url_param(url, params):
Expand Down
5 changes: 5 additions & 0 deletions tests/utils/net_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,11 @@ def test_add_url_param_elastic_no_kpi(self):
assert net.add_url_param_elastic_no_kpi(url) == \
"https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.2.0.tar.gz?x-elastic-no-kpi=true"

def test_add_url_param_elastic_no_kpi_no_http(self):
url = "gs://my_bucket/builds/elasticsearch/elasticsearch-8.0.0-SNAPSHOT.tar.gz"
assert net.add_url_param_elastic_no_kpi(url) == \
"gs://my_bucket/builds/elasticsearch/elasticsearch-8.0.0-SNAPSHOT.tar.gz"

def test_add_url_param_encoding_and_update(self):
url = "https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.2.0.tar.gz?flag1=true"
params = {"flag1": "test me", "flag2": "test@me"}
Expand Down