Update finch to v0.5.1

[huard]

No description provided.

[tlvu]

Problem found with this new Finch: Ouranosinc/PAVICS-e2e-workflow-tests#37 (comment)

[tlvu]

http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/install-birdy-from-source/8/console (Latest Finch 0.4.1 and latest Jupyter 200312)

Is it expected there is a new ref in new Finch response?

20:07:03  _________ pavics-sdi-master/docs/source/notebooks/finch.ipynb::Cell 3 __________
20:07:03  Notebook cell execution failed
20:07:03  Cell 3: Cell outputs differ
20:07:03  
20:07:03  Input:
20:07:03  print(resp.status, '\n')
20:07:03  out = resp.get()
20:07:03  resp.getOutput('/tmp/out.nc')
20:07:03  
20:07:03  print(out)
20:07:03  
20:07:03  
20:07:03  Traceback:
20:07:03   mismatch 'stdout'
20:07:03  
20:07:03   assert reference_output == test_output failed:
20:07:03  
20:07:03    "ProcessSucce...log.txt'\n)\n" == "ProcessSucce...t.meta4'\n)\n"
20:07:03    Skipping 105 identical leading characters in diff, use -v to show
20:07:03    - ATUS_FILE/out.nc',
20:07:03    + ATUS_FILE/frost-days_SRES A2 experiment_20460101-20650101.nc',
20:07:03    -     output_log='https://lvupavics-lvu.pagekite.me/wpsoutputs/STATUS_FILE/log.txt'
20:07:03    +     output_log='https://lvupavics-lvu.pagekite.me/wpsoutputs/STATUS_FILE/log.txt',
20:07:03    ?                                                                                  +
20:07:03    +     ref='https://lvupavics-lvu.pagekite.me/wpsoutputs/STATUS_FILE/input.meta4'
20:07:03      )

Which means to fix this line is to add ref as well to the tuple unpacking? ds, log, ref = resp.get(asobj=True). Then anything useful we can do to showcase this new ref in the notebook?

20:07:03  _________ pavics-sdi-master/docs/source/notebooks/finch.ipynb::Cell 5 __________
20:07:03  Notebook cell execution failed
20:07:03  Cell 5: Cell execution caused an exception
20:07:03  
20:07:03  Input:
20:07:03  ds, log = resp.get(asobj=True)
20:07:03  
20:07:03  Traceback:
20:07:03  
20:07:03  ---------------------------------------------------------------------------
20:07:03  ValueError                                Traceback (most recent call last)
20:07:03  <ipython-input-6-ac7b9d69e307> in <module>
20:07:03  ----> 1 ds, log = resp.get(asobj=True)
20:07:03  
20:07:03  ValueError: too many values to unpack (expected 2)

[huard]

Ouranosinc/pavics-sdi#156

[davidcaron]

@tlvu yes, there is an added metalink output to support computing indices on multiple files.

So the cell getting the outputs should be: ds, log, metalink = resp.get(asobj=True)

[huard]

When I run finch using docker, it seems to use pywps 4.2.2 instead of 4.2.4. @tlvu could you confirm ?
Also, @davidcaron the file name for ds has spaces in it:

NetCDF dataset:  http://localhost:5000/outputs/cd3bda60-67a9-11ea-b0e2-0242ac110002/frost-days_SRES A2 experiment_20460101-20650101.nc
Process log:  http://localhost:5000/outputs/cd3bda60-67a9-11ea-b0e2-0242ac110002/log.txt
Metalink file:  http://localhost:5000/outputs/cd3bda60-67a9-11ea-b0e2-0242ac110002/input.meta4

[huard]

Also, the current running instance of finch seems to have reached the "maximum number of parallel processes".

[huard]

I think one issue is that in the docker image, finch is installed with conda, instead of from the source. How it's done in the cookie-cutter: https://github.com/bird-house/cookiecutter-birdhouse/blob/37c609bc4942227c2a64e6386cf38a9207da9534/%7B%7Bcookiecutter.project_repo_name%7D%7D/Dockerfile#L23

[tlvu]

Also, the current running instance of finch seems to have reached the "maximum number of parallel processes".

Cleared but likely to come up again on next Jenkins run. Seems to have network issues between Jenkins host and PAVICS host so the jobs started by Jenkins never finished properly.

[tlvu]

When I run finch using docker, it seems to use pywps 4.2.2 instead of 4.2.4. @tlvu could you confirm ?

I think there is a mismatch between the version installed and what pywps.__version__ reports.

$ docker run -it --rm --name test-finch --entrypoint sh birdhouse/finch:version-0.4.1 
# conda init bash
no change     /opt/conda/condabin/conda
no change     /opt/conda/bin/conda
no change     /opt/conda/bin/conda-env
no change     /opt/conda/bin/activate
no change     /opt/conda/bin/deactivate
no change     /opt/conda/etc/profile.d/conda.sh
no change     /opt/conda/etc/fish/conf.d/conda.fish
no change     /opt/conda/shell/condabin/Conda.psm1
no change     /opt/conda/shell/condabin/conda-hook.ps1
no change     /opt/conda/lib/python3.7/site-packages/xontrib/conda.xsh
no change     /opt/conda/etc/profile.d/conda.csh
modified      /root/.bashrc

==> For changes to take effect, close and re-open your current shell. <==

# bash
(base) root@4f1e2ebe1a7f:/code# conda env export -n finch |grep pywps
  - pywps=4.2.4=py37_0
(base) root@4f1e2ebe1a7f:/code# conda activate finch
(finch) root@4f1e2ebe1a7f:/code# python -c 'import pywps; print(pywps.__version__)'
4.2.2

[huard]

geopython/pywps#525

[davidcaron]

Also, @davidcaron the file name for ds has spaces in it:

Could you check if there are spaces in the source data's model_id, driving_model_id, experiment_id or driving_experiment_id? If that's the case, what should finch do? Replace them with underscores?

[huard]

The experiment_id has spaces in the original:
https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/flyingpigeon/cmip3/tasmin.sresa2.miub_echo_g.run1.atm.da.nc.html

We can think of two fixes here: a data aware fix fixing attributes, or just a sanitizing function in pywps that escapes strings. We can do the first one quick to get finch out of the door, and then work on the upstream fix.

You can replace spaces by any character that won't break the drs. Aren't underscores used to split components already ?

[tlvu]

Finch notebook fixed in PR bird-house/finch#112

[tlvu]

Jenkins pass with Finch 0.5.1 finally http://jenkins.ouranos.ca/job/ouranos-staging/job/lvupavics-lvu.pagekite.me/18/console

[tlvu]

Tagged 1.8.7.

[tlvu]

Auto-deployed on prod Boreas:

triggerdeploy finished START_TIME=2020-03-20T05:07:01-0400
triggerdeploy finished   END_TIME=2020-03-20T05:11:00-0400