Add workflow for testing dandischema against latest release of dandi-cli #49

jwodder · 2021-06-28T13:33:06Z

Closes #21.

codecov · 2021-06-28T13:33:41Z

Codecov Report

Merging #49 (7541bde) into master (c982460) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #49   +/-   ##
=======================================
  Coverage   95.78%   95.78%           
=======================================
  Files          11       11           
  Lines        1021     1021           
=======================================
  Hits          978      978           
  Misses         43       43

Flag	Coverage Δ
unittests	`95.78% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c982460...7541bde. Read the comment docs.

.github/workflows/test-dandi-cli.yml

yarikoptic · 2021-06-28T14:29:36Z

.github/workflows/test-dandi-cli.yml

+      - name: Run dandi-cli tests
+        run: |
+          python -m pytest -s -v \
+            "$pythonLocation/lib/python${{ matrix.python }}/site-packages/dandi"


why not simpler

python -m pytest -s -v --pyargs dandi

IMHO worth a comment... also would it work on windows? not sure if it is a contributing factor to windows fails like

=================================== ERRORS ==================================== ________________________ ERROR collecting test session ________________________ C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\_pytest\runner.py:311: in from_call result: Optional[TResult] = func() C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\_pytest\runner.py:341: in <lambda> call = CallInfo.from_call(lambda: list(collector.collect()), "collect") C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\_pytest\main.py:690: in collect for direntry in visit(str(argpath), self._recurse): C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\_pytest\pathlib.py:606: in visit yield from visit(entry.path, recurse) C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\_pytest\pathlib.py:591: in visit for entry in os.scandir(path): E PermissionError: [WinError 5] Access is denied: 'D:\\System Volume Information' =========================== short test summary info =========================== ERROR ..\..\.. - PermissionError: [WinError 5] Access is denied: 'D:\\System ... !!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! ============================== 1 error in 0.20s ===============================

--pyargs dandi doesn't pick up conftest.py; see pytest-dev/pytest#1596. I don't know what's going on with Windows.

eh, odd. But do we need that conftest.py for this purpose if we aren't using --dandi-api option here?

Let's skip windows here for now, and create a dedicated issue to be fixed later

Using --pyargs dandi works, even on Windows.

Co-authored-by: Yaroslav Halchenko <[email protected]>

jwodder · 2021-06-28T14:45:26Z

@yarikoptic Note that, because this tests the latest release of dandi-cli instead of installing from source, no changes can be made to dandi-cli's dandischema requirement, so if dandi-cli requires a version of dandischema that is incompatible with the current version, the locally-installed dandischema will be replaced with a compatible version.

yarikoptic · 2021-06-28T19:48:17Z

if dandi-cli requires a version of dandischema that is incompatible with the current version, the locally-installed dandischema will be replaced with a compatible version.

what if pip install -e . on dandischema happens after dandi-cli is installed in case of not-compatible version of dandischema?

The point of the testing against dandi-cli is not to stall development in dandischema but rather just to inform if dandischema in its current devel version compatible with dandi-cli. For that I guess we also ideally test not only against released but also against master of dandi-cli. This way we could see that we might have fixed compatibility issue in dandi-cli master and just need to release it.

Letting pip install dandi to possibly downgrade dandischema to the compatible version makes such test run not useful since we already do test dandi-cli in such fashion.

yarikoptic · 2021-06-28T22:21:40Z

one of the jobs failed with redirector giving us 400 (at dandi-cli) ... I wonder if we should be persistent and retry there... meanwhile restarted

2021-06-28T20:04:54.5984607Z =================================== FAILURES ===================================
2021-06-28T20:04:54.5985857Z _________________ test_download_000027_resume[draft-<lambda>1] _________________
2021-06-28T20:04:54.5986344Z 
2021-06-28T20:04:54.5987246Z tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/test_download_000027_resume_dr1')
2021-06-28T20:04:54.5988353Z resizer = <function <lambda> at 0x7fdb01ee39e0>, version = 'draft'
2021-06-28T20:04:54.5988877Z 
2021-06-28T20:04:54.5989585Z     @pytest.mark.parametrize("resizer", [lambda sz: 0, lambda sz: sz // 2, lambda sz: sz])
2021-06-28T20:04:54.5990431Z     @pytest.mark.parametrize(
2021-06-28T20:04:54.5991024Z         "version",
2021-06-28T20:04:54.5991442Z         [
2021-06-28T20:04:54.5991934Z             pytest.param(
2021-06-28T20:04:54.5992415Z                 "0.200721.2222",
2021-06-28T20:04:54.5993413Z                 marks=pytest.mark.xfail(reason="publish.dandiarchive.org is gone"),
2021-06-28T20:04:54.5994157Z             ),
2021-06-28T20:04:54.5994568Z             "draft",
2021-06-28T20:04:54.5994989Z         ],
2021-06-28T20:04:54.5995370Z     )
2021-06-28T20:04:54.5995949Z     def test_download_000027_resume(tmp_path, resizer, version):
2021-06-28T20:04:54.5996682Z         from ..support.digests import Digester
2021-06-28T20:04:54.5997218Z     
2021-06-28T20:04:54.5997903Z         url = f"https://dandiarchive.org/dandiset/000027/{version}"
2021-06-28T20:04:54.5998629Z         digester = Digester()
2021-06-28T20:04:54.5999249Z         download(url, tmp_path, get_metadata=False)
2021-06-28T20:04:54.5999921Z         dsdir = tmp_path / "000027"
2021-06-28T20:04:54.6000755Z         nwb = dsdir / "sub-RAT123" / "sub-RAT123.nwb"
2021-06-28T20:04:54.6001508Z         digests = digester(str(nwb))
2021-06-28T20:04:54.6002169Z         dldir = nwb.with_name(nwb.name + ".dandidownload")
2021-06-28T20:04:54.6002785Z         dldir.mkdir()
2021-06-28T20:04:54.6003298Z         dlfile = dldir / "file"
2021-06-28T20:04:54.6003827Z         nwb.rename(dlfile)
2021-06-28T20:04:54.6008868Z         size = dlfile.stat().st_size
2021-06-28T20:04:54.6049750Z /opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/pynwb/icephys.py:332: UserWarning: Unit 'A' for VoltageClampSeries 'vcs' is ignored and will be set to 'amperes' as per NWB 2.1.0.
2021-06-28T20:04:54.6050857Z         os.truncate(dlfile, resizer(size))
2021-06-28T20:04:54.6052285Z   % (current_unit, self.__class__.__name__, name, unit, nwb_version))
2021-06-28T20:04:54.6052874Z         with (dldir / "checksum").open("w") as fp:
2021-06-28T20:04:54.6055124Z /opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/pynwb/icephys.py:332: UserWarning: Unit 'A' for CurrentClampSeries 'ccs' is ignored and will be set to 'volts' as per NWB 2.1.0.
2021-06-28T20:04:54.6056175Z             json.dump(digests, fp)
2021-06-28T20:04:54.6056855Z >       download(url, tmp_path, get_metadata=False)
2021-06-28T20:04:54.6057586Z   % (current_unit, self.__class__.__name__, name, unit, nwb_version))
2021-06-28T20:04:54.6061667Z /opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/pynwb/icephys.py:169: UserWarning: Stimulus description 'NA' for IZeroClampSeries 'izcs' is ignored and will be set to 'N/A' as per NWB 2.3.0.
2021-06-28T20:04:54.6062578Z 
2021-06-28T20:04:54.6063962Z   % (current_stim_desc, self.__class__.__name__, name, stim_desc, nwb_version))
2021-06-28T20:04:54.6065053Z /opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/dandi/tests/test_download.py:119: 
2021-06-28T20:04:54.6299648Z /opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/pynwb/icephys.py:332: UserWarning: Unit 'A' for IZeroClampSeries 'izcs' is ignored and will be set to 'volts' as per NWB 2.1.0.
2021-06-28T20:04:54.6300711Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2021-06-28T20:04:54.6303176Z   % (current_unit, self.__class__.__name__, name, unit, nwb_version))
2021-06-28T20:04:54.6304565Z /opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/dandi/download.py:59: in download
2021-06-28T20:04:54.6305334Z     parsed_url = parse_dandi_url(urls[0])
2021-06-28T20:04:54.6306869Z /opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/pynwb/icephys.py:332: UserWarning: Unit 'A' for CurrentClampStimulusSeries 'ccss' is ignored and will be set to 'amperes' as per NWB 2.1.0.
2021-06-28T20:04:54.6308089Z   % (current_unit, self.__class__.__name__, name, unit, nwb_version))
2021-06-28T20:04:54.6309608Z /opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/pynwb/icephys.py:332: UserWarning: Unit 'A' for VoltageClampStimulusSeries 'vcss' is ignored and will be set to 'volts' as per NWB 2.1.0.
2021-06-28T20:04:54.6310825Z   % (current_unit, self.__class__.__name__, name, unit, nwb_version))
2021-06-28T20:04:54.6311862Z /opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/dandi/dandiarchive.py:381: in parse
2021-06-28T20:04:54.6312876Z     new_url = cls.follow_redirect(url)
2021-06-28T20:04:54.6313313Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2021-06-28T20:04:54.6313574Z 
2021-06-28T20:04:54.6314267Z url = 'https://dandiarchive.org/dandiset/000027/draft'
2021-06-28T20:04:54.6314720Z 
2021-06-28T20:04:54.6315059Z     @staticmethod
2021-06-28T20:04:54.6315466Z     def follow_redirect(url):
2021-06-28T20:04:54.6315999Z         r = requests.head(url, allow_redirects=True)
2021-06-28T20:04:54.6316493Z         if r.status_code == 404:
2021-06-28T20:04:54.6316940Z             raise NotFoundError(url)
2021-06-28T20:04:54.6317398Z         elif r.status_code != 200:
2021-06-28T20:04:54.6317908Z             raise FailedToConnectError(
2021-06-28T20:04:54.6318686Z >               f"Response for getting {url} to redirect returned {r.status_code}."
2021-06-28T20:04:54.6319402Z                 f" Please verify that it is a URL related to dandiarchive and"
2021-06-28T20:04:54.6320007Z                 f" supported by dandi client"
2021-06-28T20:04:54.6321429Z E               dandi.exceptions.FailedToConnectError: Response for getting https://dandiarchive.org/dandiset/000027/draft to redirect returned 400. Please verify that it is a URL related to dandiarchive and supported by dandi client
2021-06-28T20:04:54.6322655Z 
2021-06-28T20:04:54.6323652Z /opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/dandi/dandiarchive.py:485: FailedToConnectError
2021-06-28T20:04:54.6324784Z ------------------------------ Captured log call -------------------------------
2021-06-28T20:04:54.6325675Z DEBUG    dandi:dandiarchive.py:362 Parsing url https://dandiarchive.org/dandiset/000027/draft
2021-06-28T20:04:54.6326806Z DEBUG    dandi:dandiarchive.py:362 Parsing url https://gui.dandiarchive.org/#/dandiset/000027/draft
2021-06-28T20:04:54.6330013Z DEBUG    dandi:dandiarchive.py:475 Parsed into DandisetURL(api_url=AnyHttpUrl('https://api.dandiarchive.org/api', scheme='https', host='api.dandiarchive.org', tld='org', host_type='domain', path='/api'), dandiset_id='000027', version_id='draft')
2021-06-28T20:04:54.6331601Z DEBUG    dandi:dandiapi.py:109 GET https://api.dandiarchive.org/api/dandisets/000027/
2021-06-28T20:04:54.6332398Z DEBUG    dandi:dandiapi.py:135 Response: 200
2021-06-28T20:04:54.6333242Z DEBUG    dandi:dandiapi.py:109 GET https://api.dandiarchive.org/api/dandisets/000027/
2021-06-28T20:04:54.6334052Z DEBUG    dandi:dandiapi.py:135 Response: 200
2021-06-28T20:04:54.6334925Z DEBUG    dandi:dandiapi.py:109 GET https://api.dandiarchive.org/api/dandisets/000027/versions/draft/assets/
2021-06-28T20:04:54.6335813Z DEBUG    dandi:dandiapi.py:135 Response: 200
2021-06-28T20:04:54.6337141Z DEBUG    dandi:dandiapi.py:109 GET https://api.dandiarchive.org/api/dandisets/000027/versions/draft/assets/ff453f4c-a435-4a5d-a48b-128abca5ec47/
2021-06-28T20:04:54.6338217Z DEBUG    dandi:dandiapi.py:135 Response: 200
2021-06-28T20:04:54.6338884Z DEBUG    dandi:download.py:654 Starting new download in new download directory
2021-06-28T20:04:54.6340428Z DEBUG    dandi:dandiapi.py:768 Starting download from https://api.dandiarchive.org/api/dandisets/000027/versions/draft/assets/ff453f4c-a435-4a5d-a48b-128abca5ec47/download/
2021-06-28T20:04:54.6342044Z INFO     dandi:dandiapi.py:779 Asset ff453f4c-a435-4a5d-a48b-128abca5ec47 successfully downloaded
2021-06-28T20:04:54.6343751Z DEBUG    dandi:download.py:593 Verified that /tmp/pytest-of-runner/pytest-0/test_download_000027_resume_dr1/000027/sub-RAT123/sub-RAT123.nwb has correct dandi-etag 6ec084ca9d3be17ec194a8f700d65344-1
2021-06-28T20:04:54.6345634Z DEBUG    dandi.support.digests:digests.py:64 Estimating digests for /tmp/pytest-of-runner/pytest-0/test_download_000027_resume_dr1/000027/sub-RAT123/sub-RAT123.nwb
2021-06-28T20:04:54.6346943Z DEBUG    dandi:dandiarchive.py:362 Parsing url https://dandiarchive.org/dandiset/000027/draft
2021-06-28T20:04:54.6347718Z =============================== warnings summary ===============================

yarikoptic · 2021-07-06T19:28:35Z

I guess let's just proceed and see where testing added in this PR would bring us! Thank you @jwodder !

Add workflow for testing dandischema against latest release of dandi-cli

b3f1026

jwodder added the tests Add or improve existing tests label Jun 28, 2021

yarikoptic requested changes Jun 28, 2021

View reviewed changes

Update .github/workflows/test-dandi-cli.yml

adfb9f1

Co-authored-by: Yaroslav Halchenko <[email protected]>

jwodder force-pushed the gh-21 branch from 387ca0b to adfb9f1 Compare June 28, 2021 14:35

Try running dandi-cli tests with --pyargs dandi

dd1fe3d

Install dandischema after dandi-cli; also test against dandi-cli master

7541bde

jwodder force-pushed the gh-21 branch from 985ba85 to 7541bde Compare June 28, 2021 20:01

yarikoptic merged commit 3608577 into master Jul 6, 2021

yarikoptic deleted the gh-21 branch July 6, 2021 19:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add workflow for testing dandischema against latest release of dandi-cli #49

Add workflow for testing dandischema against latest release of dandi-cli #49

jwodder commented Jun 28, 2021

codecov bot commented Jun 28, 2021 •

edited

Loading

yarikoptic Jun 28, 2021

jwodder Jun 28, 2021

yarikoptic Jun 28, 2021

jwodder Jun 28, 2021

jwodder commented Jun 28, 2021

yarikoptic commented Jun 28, 2021

yarikoptic commented Jun 28, 2021

yarikoptic commented Jul 6, 2021

Add workflow for testing dandischema against latest release of dandi-cli #49

Add workflow for testing dandischema against latest release of dandi-cli #49

Conversation

jwodder commented Jun 28, 2021

codecov bot commented Jun 28, 2021 • edited Loading

Codecov Report

yarikoptic Jun 28, 2021

Choose a reason for hiding this comment

jwodder Jun 28, 2021

Choose a reason for hiding this comment

yarikoptic Jun 28, 2021

Choose a reason for hiding this comment

jwodder Jun 28, 2021

Choose a reason for hiding this comment

jwodder commented Jun 28, 2021

yarikoptic commented Jun 28, 2021

yarikoptic commented Jun 28, 2021

yarikoptic commented Jul 6, 2021

codecov bot commented Jun 28, 2021 •

edited

Loading