KeyError with kraken list/get/show
#550
Comments
|
I believe it is an issue with zenodo. They upgraded the platform and it didn't go smoothly. There's more information here. Dog willing, it should work again once they've got everything sorted out on their end. |
|
Thanks. Lets hope they can sort everything out. |
|
@mittagessen i just looked into it a little bit. this problem does not seem (entirely) related to the zenodo upgrade. it seems like the response data provided by the api changed (even tho i couldn't find that mentioned anywhere). with the following patch everything seems to work: diff --git a/kraken/repo.py b/kraken/repo.py
index 69282f3..c50dc9c 100644
--- a/kraken/repo.py
+++ b/kraken/repo.py
@@ -131,10 +131,14 @@ def get_model(model_id: str, path: str, callback: Callable[[int, int], Any] = la
logger.error(f'Found {resp["hits"]["total"]} models when querying for id \'{model_id}\'')
raise KrakenRepoException(f'Found {resp["hits"]["total"]} models when querying for id \'{model_id}\'')
- metadata = resp['hits']['hits'][0]
- model_url = [x['links']['self'] for x in metadata['files'] if x['type'] == 'mlmodel'][0]
+ record = resp['hits']['hits'][0]
+ metadata_url = [x['links']['download'] for x in record['files'] if x['filename'] == 'metadata.json'][0]
+ r = requests.get(metadata_url)
+ r.raise_for_status()
+ resp = r.json()
# callable model identifier
- nat_id = os.path.basename(urllib.parse.urlparse(model_url).path)
+ nat_id = resp['name']
+ model_url = [x['links']['download'] for x in record['files'] if x['filename'] == nat_id][0]
spath = os.path.join(path, nat_id)
logger.debug(f'downloading model file {model_url} to {spath}')
with closing(requests.get(model_url, stream=True)) as r:
@@ -177,9 +181,9 @@ def get_description(model_id: str, callback: Callable[..., Any] = lambda: None)
raise KrakenRepoException(msg)
meta_json = None
for file in record['files']:
- if file['key'] == 'metadata.json':
+ if file['filename'] == 'metadata.json':
callback()
- r = requests.get(file['links']['self'])
+ r = requests.get(file['links']['download'])
r.raise_for_status()
callback()
try:
@@ -225,7 +229,7 @@ def get_listing(callback: Callable[[int, int], Any] = lambda total, advance: Non
total = resp['hits']['total']
callback(total, 0)
records.extend(resp['hits']['hits'])
- while 'next' in resp['links']:
+ while 'next' in resp.get('links', []):
logger.debug('Fetching next page')
r = requests.get(resp['links']['next'])
r.raise_for_status()
@@ -242,9 +246,9 @@ def get_listing(callback: Callable[[int, int], Any] = lambda total, advance: Non
if not model_type:
continue
for file in record['files']:
- if file['key'] == 'metadata.json':
+ if file['filename'] == 'metadata.json':
callback(total, 1)
- r = requests.get(file['links']['self'])
+ r = requests.get(file['links']['download'])
r.raise_for_status()
try:
metadata = r.json()
|
|
I just applied the patch of betaboon, but it seems that zendo has already reverted their json response again as I get KeyError 'filename' which is now back to 'key', and links also has not 'download' key, but the old 'self' key. |
|
This can be solved like: diff --git a/kraken/repo.py b/kraken/repo.py
index 69282f3..212447d 100644
--- a/kraken/repo.py
+++ b/kraken/repo.py
@@ -225,13 +225,14 @@ def get_listing(callback: Callable[[int, int], Any] = lambda total, advance: Non
total = resp['hits']['total']
callback(total, 0)
records.extend(resp['hits']['hits'])
- while 'next' in resp['links']:
- logger.debug('Fetching next page')
- r = requests.get(resp['links']['next'])
- r.raise_for_status()
- resp = r.json()
- logger.debug('Found {} new records'.format(len(resp['hits']['hits'])))
- records.extend(resp['hits']['hits'])
+ if 'links' in resp and 'next' in resp['links']:
+ while 'next' in resp['links']:
+ logger.debug('Fetching next page')
+ r = requests.get(resp['links']['next'])
+ r.raise_for_status()
+ resp = r.json()
+ logger.debug('Found {} new records'.format(len(resp['hits']['hits'])))
+ records.extend(resp['hits']['hits'])
logger.debug('Retrieving model metadata')
models = {}
# fetch metadata.jsn for each model |
|
They're obviously fiddling around with the API right now. I'll wait for
them to announce stabilization before merging any patches. It sucks
(especially as older kraken versions will be broken because they don't
version the API) but can't be changed for now.
…On 23/11/04 02:09PM, Maria Levchenko wrote:
This can be solved like:
```python
diff --git a/kraken/repo.py b/kraken/repo.py
index 69282f3..212447d 100644
--- a/kraken/repo.py
+++ b/kraken/repo.py
@@ -225,13 +225,14 @@ def get_listing(callback: Callable[[int, int], Any] = lambda total, advance: Non
total = resp['hits']['total']
callback(total, 0)
records.extend(resp['hits']['hits'])
- while 'next' in resp['links']:
- logger.debug('Fetching next page')
- r = requests.get(resp['links']['next'])
- r.raise_for_status()
- resp = r.json()
- logger.debug('Found {} new records'.format(len(resp['hits']['hits'])))
- records.extend(resp['hits']['hits'])
+ if 'links' in resp and 'next' in resp['links']:
+ while 'next' in resp['links']:
+ logger.debug('Fetching next page')
+ r = requests.get(resp['links']['next'])
+ r.raise_for_status()
+ resp = r.json()
+ logger.debug('Found {} new records'.format(len(resp['hits']['hits'])))
+ records.extend(resp['hits']['hits'])
logger.debug('Retrieving model metadata')
models = {}
# fetch metadata.jsn for each model
```
--
Reply to this email directly or view it on GitHub:
#550 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
|
In the meantime, where can I put Kraken models that I download manually? I tried putting them under the kraken/kraken, but it seems that's the wrong spot. I searched the documentation but couldn't find where they are supposed to be. Would greatly appreciate it if someone could tell me the right spot. Thanks in advance. |
|
On 23/11/05 07:47AM, ihholmes-p wrote:
In the meantime, where can I put Kraken models that I download
manually? I tried putting them under the kraken/kraken, but it seems
that's the wrong spot. I searched the documentation but couldn't find
where they are supposed to be. Would greatly appreciate it if someone
could tell me the right spot. Thanks in advance.
You can just point kraken to the wherever you put them. The order of
resolution for the `-i` argument is first the path and then
$XDG_DIR/kraken (which is where `kraken get` puts the models).
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Getting
KeyError: link,KeyError: typeandKeyError: keyon new setups and long running systems:For
list:For
get:For
show:I first saw this problem when setting up a new Docker instance. While investigating this issue I went down from Kraken v4.3.13 to v4.3.08 with the same result. Then I tested an instance running for two months with Kraken v4.3.12 and got the same errors. They were not there a few months ago.
Am I doing something wrong all of a sudden? Is there an issue with zenodo?
The text was updated successfully, but these errors were encountered: