jsonschema cannot find path specified \\1.0.0\\def.json

[amh-137]

Summary

When I try to install pyhf in a venv and then create a simple model, it gives me a n error

<urlopen error [WinError 3] The system cannot find the path specified: '\\1.0.0\\defs.json'>

When i check the path to the schema with

>>> pyhf.schema.path
WindowsPath('C:/Users/alexh/Downloads/pyhf_test/pyhf_venv/Lib/site-packages/pyhf/schemas')

Following this path then into the 1.0.0 folder, defs.json exists.

OS / Environment

# Windows 11
# Python 3.11.6

Steps to Reproduce

$ python -m venv venv_pyhf
$ venv_pyhf/Scripts/activate.ps1
$ pip install pyhf
$ python
>>> import pyhf
>>> model = pyhf.simplemodels.uncorrelated_background(signal=[1,2,3], bkg=[4,5,6], bkg_uncertainty=[2,3,4])
_RefResolutionError: <urlopen error [WinError 3] The system cannot find the path specified: '\\1.0.0\\defs.json'>

File Upload (optional)

No response

Expected Results

Expected to get a pyhf.pdf.Model.

Actual Results

Traceback (most recent call last):
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\jsonschema\validators.py", line 1097, in resolve_from_url
    document = self.store[url]
               ~~~~~~~~~~^^^^^
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\jsonschema\_utils.py", line 20, in __getitem__
    return self.store[self.normalize(uri)]
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
KeyError: 'file://C:\\Users\\alexh\\Downloads\\pyhf_test\\pyhf_venv\\Lib\\site-packages\\pyhf\\schemas/1.0.0/defs.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\urllib\request.py", line 1505, in open_local_file
    stats = os.stat(localfile)
            ^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 3] The system cannot find the path specified: '\\1.0.0\\defs.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\jsonschema\validators.py", line 1100, in resolve_from_url
    document = self.resolve_remote(url)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\jsonschema\validators.py", line 1204, in resolve_remote
    with urlopen(uri) as url:
         ^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\urllib\request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\urllib\request.py", line 519, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\urllib\request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\urllib\request.py", line 1483, in file_open
    return self.open_local_file(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\urllib\request.py", line 1522, in open_local_file
    raise URLError(exp)
urllib.error.URLError: <urlopen error [WinError 3] The system cannot find the path specified: '\\1.0.0\\defs.json'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\pyhf\simplemodels.py", line 149, in uncorrelated_background
    return Model(spec, batch_size=batch_size, validate=validate)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\pyhf\pdf.py", line 769, in __init__
    schema.validate(self.spec, self.schema, version=self.version)
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\pyhf\schema\validator.py", line 93, in validate
    return validator.validate(spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\jsonschema\validators.py", line 434, in validate
    for error in self.iter_errors(*args, **kwargs):
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\jsonschema\validators.py", line 368, in iter_errors
    for error in errors:
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\jsonschema\_keywords.py", line 284, in ref
    yield from validator._validate_reference(ref=ref, instance=instance)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\jsonschema\validators.py", line 461, in _validate_reference
    scope, resolved = resolve(ref)
                      ^^^^^^^^^^^^
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\jsonschema\validators.py", line 1086, in resolve
    return url, self._remote_cache(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexh\Downloads\pyhf_test\pyhf_venv\Lib\site-packages\jsonschema\validators.py", line 1102, in resolve_from_url
    raise exceptions._RefResolutionError(exc)
jsonschema.exceptions._RefResolutionError: <urlopen error [WinError 3] The system cannot find the path specified: '\\1.0.0\\defs.json'>

pyhf Version

pyhf, version 0.7.4

Code of Conduct

• I agree to follow the Code of Conduct

[kratsg]

This is going to be interesting to debug as I do not have a Windows machine to test some things. We're relying primarily on pathlib for almost everything so it's not abundantly clear to me at first what is going on. Is there a way for you to print out what the following gives you upon import?

>>> pyhf.schema.variables.SCHEMA_CACHE.keys()
dict_keys(['https://scikit-hep.org/pyhf/schemas/1.0.0/defs.json'])
>>> pyhf.schema.variables.schemas
PosixPath('/Users/kratsg/pyhf/src/pyhf/schemas')

I'm wondering what happens in the latter. The former is fine, but I'm realizing Windows filepaths are slightly different from web addresses...

[amh-137]

Looks similar to me on the first, but the second seems different.

(pyhf_venv) PS C:\Users\alexh\Downloads\pyhf_test> python
Python 3.11.6 
>>> import pyhf
>>> import pyhf.schema
>>> pyhf.schema.variables.SCHEMA_CACHE.keys()
dict_keys(['https://scikit-hep.org/pyhf/schemas/1.0.0/defs.json'])
>>> pyhf.schema.variables.schemas
WindowsPath('C:/Users/alexh/Downloads/pyhf_test/pyhf_venv/Lib/site-packages/pyhf/schemas')

Let me know if there is anything else i can help with testing on windows :-)

[matthewfeickert]

We technically don't support Windows at the moment, but I will admit that I find this an interesting problem. The difference in pyhf.schema.variables.schemas seems like it should be benign given that they both are properly pointing to an installation location with a pyhf/schemas directory present. I don't have any experience on how jsonschema resolves this on Windows, though, so I think we would need to get access to dev machines or Windows VMs to properly debug.

[kratsg]

I suspect I've figured out the issue and will submit a PR shortly.

[matthewfeickert]

@SuchAgoodDoge Can you please verify that if you install from main with

python -m pip install --upgrade 'pyhf@git+https://github.com/scikit-hep/pyhf.git'

that things work for you now? If so, we'll make a v0.7.5 patch release.

[amh-137]

Thank you, this works :-)

[matthewfeickert]

@SuchAgoodDoge pyhf v0.7.5 is out on PyPI now and will be available on conda-forge in the next 24 hours. So you should be good to go with a

python -m pip install --upgrade pyhf

Let us know if you have any questions. 👍