Dependency and build/CI setup

[dopplershift]

Right now our dependencies in setup.cfg feel kinda overdone

dev = ipython[all]>=3.1
doc = sphinx>=1.8; sphinx-gallery>=0.4; doc8; m2r; netCDF4
examples = cartopy>=0.13.1; matplotlib>=2.2.0; pyproj>=1.9.4,!=2.0.0
test = pytest>=2.4; pytest-runner; pytest-mpl; pytest-flake8; cartopy>=0.16.0; flake8>3.2.0; flake8-builtins!=1.4.0; flake8-comprehensions; flake8-copyright; flake8-docstrings; flake8-import-order; flake8-mutable; flake8-pep3101; flake8-print; flake8-quotes; flake8-rst-docstrings; pep8-naming; netCDF4; pyproj>=1.9.4,!=2.0.0

The goal is to make it easy to set up the environments, including on Travis, AppVeyor, etc. Then there are things like this in .travis.yml:

    - EXTRA_PACKAGES="shapely<1.5.17.post1 coverage<4.4 docutils<0.15"
...
      export EXTRA_PACKAGES="$EXTRA_PACKAGES pillow sphinx_rtd_theme==0.2.5b1.post1 pytest";
...
        export EXTRA_PACKAGES="$EXTRA_PACKAGES flake8-bugbear";

        export EXTRA_PACKAGES="$EXTRA_PACKAGES pytest-cov";

and of course our environment.yml.

What I think I'd like to get to is:

• requirements.txt file(s) to define our build/test environment with pinned versions
• Use Dependabot to maintain the requirements.txt, issuing PRs to update stuff--then version updates for those environments only break in PRs

Some of this might get easier in moving to GitHub actions, like running flake8 (and the dozen plugins) by itself and also maybe having a Docker container for the build. Questions that come to mind:

• Should we pin versions of all dependencies in the test environment and only update via PR (again Dependabot) ?
• Can we do something here to eliminate our private S3 cache of wheels?

It really feels like we have a complicated set of things that get installed around various tasks for the project--almost like spaghetti code, but for dependencies. I'm also getting pretty tired of random ecosystem updates causing CI to fail for days on end; I'd like to have a bit more control of the churn.

[zbruick]

If we have the extra Travis build that #1144 includes to test again dev versions of dependencies so we know if issues are coming, then I think we could pin everything in the build/test environment, and check against an automated PR for dependency updates. That would hopefully keep CI cleaner for outside contributors when dependencies break us.

As far as reducing complexity elsewhere, that's a little outside my knowledge base at the moment, but I agree that things seem complex and not understandable right now, especially on Travis. I would support any cleanup there.

[jthielen]

@dopplershift Not sure if this is useful or not, but I vaguely remember from last telecon that you were having issues with Dependabot and updating pinned environment files? I stumbled across this GitHub actions workflow for updating versions: https://github.com/dask/dask-docker/actions/runs/73860791/workflow

[dopplershift]

I think this has been resolved by #1406, #1409, and #1410.