Cannot import gdal in circle CI testing

[yunjunz]

Since last week, the Circle CI testing has been reporting the following error of "Can not import gdal", which is strange, as nothing has changed in this part of the code for a while now. And gdal is installed during the installation process. However, local testings with the same python and gdal versions (3.10 and 3.9) passed without issue. Any clue or suggestion on how to fix this on Circle CI is appreciated!

@jhkennedy

...
******************** step - load_data ********************

load_data.py --template /root/data/SanFranSenDT42/mintpy/smallbaselineApp.cfg /root/tools/MintPy/tests/configs/SanFranSenDT42.txt --project SanFranSenDT42
processor : aria
SAR platform/sensor : Sen
use auto path defined in mintpy.defaults.auto_path for options in auto
    mintpy.load.unwFile        : auto --> ../stack/unwrapStack.vrt
    mintpy.load.corFile        : auto --> ../stack/cohStack.vrt
    mintpy.load.connCompFile   : auto --> ../stack/connCompStack.vrt
    mintpy.load.demFile        : auto --> ../DEM/*.dem
    mintpy.load.incAngleFile   : auto --> ../incidenceAngle/*.vrt
    mintpy.load.azAngleFile    : auto --> ../azimuthAngle/*.vrt
    mintpy.load.waterMaskFile  : auto --> ../mask/watermask.msk
--------------------------------------------------
prepare metadata files for aria products
prep_aria.py --template /root/data/SanFranSenDT42/mintpy/smallbaselineApp.cfg --stack-dir ../stack --unwrap-stack-name unwrapStack.vrt --coherence-stack-name cohStack.vrt --conn-comp-stack-name connCompStack.vrt --dem "../DEM/*.dem" --incidence-angle "../incidenceAngle/*.vrt" --azimuth-angle "../azimuthAngle/*.vrt" --water-mask ../mask/watermask.msk --update
read options from template file: smallbaselineApp.cfg
multilook x/ystep: 2/2
multilook method : nearest
search input data file info:
unwFile       : /root/data/SanFranSenDT42/stack/unwrapStack.vrt
corFile       : /root/data/SanFranSenDT42/stack/cohStack.vrt
connCompFile  : /root/data/SanFranSenDT42/stack/connCompStack.vrt
demFile       : ../DEM/SRTM_3arcsec.dem
incAngleFile  : ../incidenceAngle/20160131_20160107.vrt
azAngleFile   : ../azimuthAngle/20160131_20160107.vrt
waterMaskFile : ../mask/watermask.msk
Traceback (most recent call last):
  File "/root/tools/miniforge/lib/python3.10/site-packages/osgeo/__init__.py", line 30, in swig_import_helper
    return importlib.import_module(mname)
  File "/root/tools/miniforge/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1176, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: /root/tools/miniforge/lib/python3.10/site-packages/osgeo/../../../libgdal.so.35: undefined symbol: sqlite3_total_changes64

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/tools/miniforge/lib/python3.10/site-packages/mintpy/prep_aria.py", line 15, in <module>
    from osgeo import gdal
  File "/root/tools/miniforge/lib/python3.10/site-packages/osgeo/__init__.py", line 35, in <module>
    _gdal = swig_import_helper()
  File "/root/tools/miniforge/lib/python3.10/site-packages/osgeo/__init__.py", line 32, in swig_import_helper
    return importlib.import_module('_gdal')
  File "/root/tools/miniforge/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_gdal'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/tools/miniforge/bin/smallbaselineApp.py", line 8, in <module>
    sys.exit(main())
  File "/root/tools/miniforge/lib/python3.10/site-packages/mintpy/cli/smallbaselineApp.py", line 209, in main
    run_smallbaselineApp(inps)
  File "/root/tools/miniforge/lib/python3.10/site-packages/mintpy/smallbaselineApp.py", line 1155, in run_smallbaselineApp
    app.run(steps=inps.runSteps)
  File "/root/tools/miniforge/lib/python3.10/site-packages/mintpy/smallbaselineApp.py", line 908, in run
    self.run_load_data(sname)
  File "/root/tools/miniforge/lib/python3.10/site-packages/mintpy/smallbaselineApp.py", line 176, in run_load_data
    mintpy.cli.load_data.main(iargs)
  File "/root/tools/miniforge/lib/python3.10/site-packages/mintpy/cli/load_data.py", line 123, in main
    load_data(inps)
  File "/root/tools/miniforge/lib/python3.10/site-packages/mintpy/load_data.py", line 808, in load_data
    prepare_metadata(iDict)
  File "/root/tools/miniforge/lib/python3.10/site-packages/mintpy/load_data.py", line 759, in prepare_metadata
    prep_module.main(iargs)
  File "/root/tools/miniforge/lib/python3.10/site-packages/mintpy/cli/prep_aria.py", line 213, in main
    from mintpy.prep_aria import load_aria
  File "/root/tools/miniforge/lib/python3.10/site-packages/mintpy/prep_aria.py", line 17, in <module>
    raise ImportError('Can not import gdal [version>=3.0]!')
ImportError: Can not import gdal [version>=3.0]!
Traceback (most recent call last):
  File "/root/tools/MintPy/tests/smallbaselineApp.py", line 216, in <module>
    main(sys.argv[1:])
  File "/root/tools/MintPy/tests/smallbaselineApp.py", line 193, in main
    test_smallbaselineApp(dset_name,
  File "/root/tools/MintPy/tests/smallbaselineApp.py", line 160, in test_smallbaselineApp
    raise RuntimeError(f'Test failed for example dataset {dset_name}')
RuntimeError: Test failed for example dataset SanFranSenDT42

[yunjunz]

Seems to be related to this: OSGeo/gdal#9021.

• gdal-3.6: works (circleci)
• gdal-3.7: works (circleci)
• gdal-3.8: works (circleci)
• gdal-3.9.0: do NOT work (circleci)