Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"invalid projection" error when saving datasets with composite using C02 channel with either sunz_corrected or rayleigh_corrected modifier #1114

Open
ve2fxl opened this issue Mar 15, 2020 · 2 comments
Assignees
Labels

Comments

@ve2fxl
Copy link

ve2fxl commented Mar 15, 2020

Describe the bug
"invalid projection" error when saving datasets with composite using C02 channel with either sunz_corrected or rayleigh_corrected modifier with ABI NetCDF. Doesn't happen always. Doing OK without modifier on C02 channel. Doing OK with those modifiers (1 of them or both) if applied to other channel than C02. Happens more often and quickly with CONUS sector than Mesos. Same error using Scene and Multiscene.

To Reproduce

import os
os.environ['PPP_CONFIG_DIR']='/GRB/Python-scripts/satpy'
if __name__ == '__main__':
  from satpy import Scene
  from glob import glob
  from satpy import MultiScene
  filenames = glob('/GRB/NOAACLASS/20200120/*.nc')
  len(filenames)
  mscn = MultiScene.from_files(filenames, reader='abi_l1b')
  mscn.load(['true_color_night'])
  new_mscn = mscn.resample(resampler='native')
  new_mscn.save_datasets(writer='simple_image',filename='{name}_CONUS_{start_time:%Y%m%d_%H%M%S}.png',base_dir='/GRB/Python-scripts/true_color/goes-imgs/20200120-conus-true-color-night',overlay={'coast_dir': './gshhg-shp-2.3.7/', 'color': (0, 0, 0, 50), 'width': 1, 'level_coast': 1, 'level_borders': 2})

Expected behavior
Script generating true color day/night blending of GOES-16 CONUS images without error

Actual results

Traceback (most recent call last):
  File "Github-issue1-Satpy-ve2fxl.py", line 14, in <module>
    new_mscn.save_datasets(writer='simple_image',filename='{name}_CONUS_{start_time:%Y%m%d_%H%M%S}.png',base_dir='/GRB/Python-scripts/true_color/goes-imgs/20200120-conus-true-color-night',overlay={'coast_dir': './gshhg-shp-2.3.7/', 'color': (0, 0, 0, 50), 'width': 1, 'level_coast': 1, 'level_borders': 2})
  File "C:\ProgramData\Miniconda3\lib\site-packages\satpy\multiscene.py", line 347, in save_datasets
    self._simple_save_datasets(scenes, **kwargs)
  File "C:\ProgramData\Miniconda3\lib\site-packages\satpy\multiscene.py", line 309, in _simple_save_datasets
    scn.save_datasets(**kwargs)
  File "C:\ProgramData\Miniconda3\lib\site-packages\satpy\scene.py", line 1364, in save_datasets
    return writer.save_datasets(datasets, compute=compute, **save_kwargs)
  File "C:\ProgramData\Miniconda3\lib\site-packages\satpy\writers\__init__.py", line 701, in save_datasets
    return compute_writer_results([results])
  File "C:\ProgramData\Miniconda3\lib\site-packages\satpy\writers\__init__.py", line 553, in compute_writer_results
    da.compute(delayeds)
  File "C:\ProgramData\Miniconda3\lib\site-packages\dask\base.py", line 437, in compute
    results = schedule(dsk, keys, **kwargs)
  File "C:\ProgramData\Miniconda3\lib\site-packages\dask\threaded.py", line 84, in get
    **kwargs
  File "C:\ProgramData\Miniconda3\lib\site-packages\dask\local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "C:\ProgramData\Miniconda3\lib\site-packages\dask\local.py", line 316, in reraise
    raise exc
  File "C:\ProgramData\Miniconda3\lib\site-packages\dask\local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "C:\ProgramData\Miniconda3\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\ProgramData\Miniconda3\lib\site-packages\dask\optimization.py", line 982, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "C:\ProgramData\Miniconda3\lib\site-packages\dask\core.py", line 151, in get
    result = _execute_task(task, cache)
  File "C:\ProgramData\Miniconda3\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\ProgramData\Miniconda3\lib\site-packages\dask\utils.py", line 30, in apply
    return func(*args, **kwargs)
  File "C:\ProgramData\Miniconda3\lib\site-packages\pyresample\geometry.py", line 1012, in invproj
    target_proj = Proj(proj_dict)
  File "C:\ProgramData\Miniconda3\lib\site-packages\pyresample\_spatial_mp.py", line 121, in __init__
    **kwargs)
  File "C:\ProgramData\Miniconda3\lib\site-packages\pyproj\proj.py", line 171, in __init__
    super().__init__(cstrencode(projstring.strip()))
  File "pyproj/_proj.pyx", line 30, in pyproj._proj.Proj.__init__
pyproj.exceptions.ProjError: Invalid projection b'C'.: (Internal Proj Error: proj_create: unrecognized format / unknown name)

Screenshots
Not applicable

Environment Info:

  • OS: Windows 10 1909 English, Miniconda3, conda-forge
  • Satpy Version: 0.20.0 build py_1
  • PyResample Version: 1.14.0 build py37h67a5e14_2
  • PyProj Version: 2.5.0 build py37h26f50eb_1
  • Proj Version: 6.3.1 build ha7a8c7b_1
  • Readers and writers dependencies (when relevant): I don't know of any dependencies

Additional context
I just upgraded to Windows 10 a few days ago. Was worse under Windows 7 as I had a lot of "UnicodeDecodeError: 'utf8' codec can't decode byte ..." along this current error (UTF-8 error was happening more often that the invalid projection error), so please update to Windows 10 for at least correcting the UTF-8 BUT I once and only once had until now the above UTF-8 error in Windows 10 but not in my system yet, but at my work computer. Finally, works great (no error) when running under python 2.7/Satpy 0.19 environment

@djhoese djhoese added the bug label Mar 15, 2020
@djhoese djhoese self-assigned this Apr 1, 2020
@djhoese
Copy link
Member

djhoese commented Apr 1, 2020

So I really thought this was just you, but it happened to me yesterday too.

    return [_execute_task(a, cache) for a in arg]
  File "/work/geo2grid/lib/python3.7/site-packages/dask/core.py", line 122, in _execute_task
    elif arg in cache:
  File "/work/geo2grid/lib/python3.7/site-packages/pyproj/crs/crs.py", line 869, in __hash__
    return hash(self.to_wkt())
  File "pyproj/_crs.pyx", line 451, in pyproj._crs.Base.to_wkt
  File "pyproj/_crs.pyx", line 120, in pyproj._crs._to_wkt
  File "pyproj/_crs.pyx", line 24, in pyproj._crs.cstrdecode
  File "/work/geo2grid/lib/python3.7/site-packages/pyproj/compat.py", line 21, in pystrdecode
    return cstr.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 0: invalid continuation byte
Command exited with non-zero status 1

This was running my Geo2Grid software in a CentOS 7 docker container (CentOS docker host). @ve2fxl, you said this happened on your Ubuntu system too, right? What user were you running this as?

Regardless, I'm running geo2grid to generate a true_color composite in a loop of 100 on my local Ubuntu machine but running it in the same CentOS 7 docker image. We'll see if it breaks.

@ve2fxl If you have any more of these errors, please paste them here so we can collect all the variations. This one I got seems slightly different than the one you had posted to slack.

@djhoese
Copy link
Member

djhoese commented Apr 1, 2020

Here is one of the ones you posted to slack:

File "C:\ProgramData\Miniconda3\lib\site-packages\dask\core.py", line 120, in _execute_task
elif not ishashable(arg):
File "C:\ProgramData\Miniconda3\lib\site-packages\dask\core.py", line 20, in ishashable
hash(x)
File "C:\ProgramData\Miniconda3\lib\site-packages\pyproj\crs\crs.py", line 869, in __hash__
return hash(self.to_wkt())
File "pyproj/_crs.pyx", line 451, in pyproj._crs.Base.to_wkt
File "pyproj/_crs.pyx", line 120, in pyproj._crs._to_wkt
File "pyproj/_crs.pyx", line 24, in pyproj._crs.cstrdecode
File "C:\ProgramData\Miniconda3\lib\site-packages\pyproj\compat.py", line 21, in pystrdecode
return cstr.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

Looks like the same thing (different byte but same bit of code).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants