Fix issue with fetching test-mode -1k document corpora.

[drawlerr]

In test-mode, we download a document corpus with "-1k" added to the filename, that is not specifically described in track.json. This is problematic because, the "expected bytes" of the file is None in this situation, breaking download progress reporting when we divide by None.
To work around this, I've made the net.Progress callback resilient to this situation and added a test.

[danielmitterdorfer]

I have to admit I was puzzled at first why this does not happen all the time but I understand now why we (usually) don't have this issue: this only happens if we can't derive the expected size via other means. An alternative (or amendment) to this PR could be to ensure we determine the expected size also for S3 downloads (worked fine in a local test):

diff --git a/esrally/utils/net.py b/esrally/utils/net.py
index 5ae6168..445bd9b 100644
--- a/esrally/utils/net.py
+++ b/esrally/utils/net.py
@@ -82,10 +82,13 @@ def _download_from_s3_bucket(bucket_name, bucket_path, local_path, expected_size
             self._progress(self._bytes_read, self._expected_size_in_bytes)

     s3 = boto3.resource("s3")
+    bucket = s3.Bucket(bucket_name)
+    if expected_size_in_bytes is None:
+        expected_size_in_bytes = bucket.Object(bucket_path).content_length
     progress_callback = S3ProgressAdapter(expected_size_in_bytes, progress_indicator) if progress_indicator else None
-    s3.Bucket(bucket_name).download_file(bucket_path, local_path,
-                                         Callback=progress_callback,
-                                         Config=boto3.s3.transfer.TransferConfig(use_threads=False))
+    bucket.download_file(bucket_path, local_path,
+                         Callback=progress_callback,
+                         Config=boto3.s3.transfer.TransferConfig(use_threads=False))

What do you think?

[danielmitterdorfer]

Thanks for iterating! LGTM