Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broken HTTP access to polytope in 0.3.2 #21

Closed
sandorkertesz opened this issue Nov 8, 2024 · 0 comments · Fixed by #22
Closed

Broken HTTP access to polytope in 0.3.2 #21

sandorkertesz opened this issue Nov 8, 2024 · 0 comments · Fixed by #22
Assignees
Labels
bug Something isn't working

Comments

@sandorkertesz
Copy link
Contributor

sandorkertesz commented Nov 8, 2024

What happened?

It seems #15 breaks the http access. The attached example works with 0.3.1 but broken in 0.3.2.

The problem comes from this method in http.py:

def title(self):
        headers = self.headers()
        if "content-disposition" in headers:
            value, params = parse_separated_header(headers["content-disposition"])
            if "filename" in params:
                return params["filename"]
        return super().title()

Here parse_separated_header() returns a dict so the return value does not seem to be treated correctly.

What are the steps to reproduce the bug?

Use this code with earthkit.data and multiurl=0.3.2:

import earthkit.data

request = {
   "class": "od",
   "stream" : "enfo",
   "type" : "pf",
   "date" : "20241107",
   "time" : "0000",
   "levtype" : "sfc",
   "expver" : "0079",
   "domain" : "g",
   "param" : "164/167/169",
   "number" : "1/to/50",
   "feature" : {
       "type" : "timeseries",
       "points": [[-9.10, 38.78]],
       "axis": "step",
       "range" : {
           "start" : 0,
           "end" : 360,
       }
   },
   "format": "covjson",
}

ds = earthkit.data.from_source("polytope", "ecmwf-mars", request, stream=False, address='polytope-test.ecmwf.int')

This will generate the following error:

string indices must be integers
Traceback (most recent call last):
  File "[/Users/cgr/git/earthkit-data/src/earthkit/data/utils/lazy.py", line 32](http://localhost:8888/lab/tree/_dev/writers/src/earthkit/data/utils/lazy.py#line=31), in source
    self._source = from_source(
  File "[/Users/cgr/git/earthkit-data/src/earthkit/data/sources/__init__.py", line 150](http://localhost:8888/lab/tree/_dev/writers/src/earthkit/data/sources/__init__.py#line=149), in from_source
    src = get_source(name, *args, **kwargs)
  File "[/Users/cgr/git/earthkit-data/src/earthkit/data/sources/__init__.py", line 131](http://localhost:8888/lab/tree/_dev/writers/src/earthkit/data/sources/__init__.py#line=130), in __call__
    source = klass(*args, **kwargs)
  File "[/Users/cgr/git/earthkit-data/src/earthkit/data/core/__init__.py", line 22](http://localhost:8888/lab/tree/_dev/writers/src/earthkit/data/core/__init__.py#line=21), in __call__
    obj.__init__(*args, **kwargs)
  File "[/Users/cgr/git/earthkit-data/src/earthkit/data/sources/url.py", line 217](http://localhost:8888/lab/tree/_dev/writers/src/earthkit/data/sources/url.py#line=216), in __init__
    self._download()
  File "[/Users/cgr/git/earthkit-data/src/earthkit/data/sources/url.py", line 272](http://localhost:8888/lab/tree/_dev/writers/src/earthkit/data/sources/url.py#line=271), in _download
    self.path = self.cache_file(
  File "[/Users/cgr/git/earthkit-data/src/earthkit/data/sources/__init__.py", line 68](http://localhost:8888/lab/tree/_dev/writers/src/earthkit/data/sources/__init__.py#line=67), in cache_file
    return cache_file(owner, create, args, **kwargs)
  File "[/Users/cgr/git/earthkit-data/src/earthkit/data/core/caching.py", line 1022](http://localhost:8888/lab/tree/_dev/writers/src/earthkit/data/core/caching.py#line=1021), in cache_file
    owner_data = create(path + ".tmp", args)
  File "[/Users/cgr/git/earthkit-data/src/earthkit/data/sources/url.py", line 269](http://localhost:8888/lab/tree/_dev/writers/src/earthkit/data/sources/url.py#line=268), in download
    self.downloader.download(target)
  File "[/Users/cgr/git/multiurl/multiurl/base.py", line 126](http://localhost:8888/Users/cgr/git/multiurl/multiurl/base.py#line=125), in download
    desc=self.title(),
  File "[/Users/cgr/git/multiurl/multiurl/http.py", line 130](http://localhost:8888/Users/cgr/git/multiurl/multiurl/http.py#line=129), in title
    return params["filename"]
TypeError: string indices must be integers

Version

0.3.2

Platform (OS and architecture)

any

Relevant log output

No response

Accompanying data

No response

Organisation

ECMWF

@sandorkertesz sandorkertesz added the bug Something isn't working label Nov 8, 2024
andir added a commit to andir/multiurl that referenced this issue Nov 9, 2024
This should resolve ecmwf#21 which should also solve
ecmwf/ecmwf-opendata#54.

The parsing function returns a dictionary, not a tuple. If we
tuple-unpack a dictionary, that will give us the keys of the *first
two* dictionary entries. If there are more than two entries it would
raise a ValueError (too many values to unpack).
TomHodson pushed a commit that referenced this issue Feb 14, 2025
This should resolve #21 which should also solve
ecmwf/ecmwf-opendata#54.

The parsing function returns a dictionary, not a tuple. If we
tuple-unpack a dictionary, that will give us the keys of the *first
two* dictionary entries. If there are more than two entries it would
raise a ValueError (too many values to unpack).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants