Refactor infrastructure as described in astropy-helpers removal APE

[astrofrog]

This PR removes astropy-helpers in the core package and update the infrastructure as described in the APE A roadmap for package infrastructure without astropy-helpers. I am putting this up ahead of the coordination meeting in Socorro, to make it easier to see what the changes would be and discuss specific changes.

The main things this does is:

• Remove astropy-helpers as a submodule, along with the ah_bootstrap.py
• Add a new pyproject.toml file describing the build-time dependencies (see PEP 517 and PEP 518)
• Uses an experimental trimmed down fork of astropy-helpers called extension-helpers (developed in a branch of astropy-helpers) which just does the task of collecting extensions
• Move declarations of package data from setup_package.py files to setup.cfg
• Switch to using environment variables to control whether to compile against external libraries
• Switch to using ASTROPY_DEBUG as a way to control whether to compile extensions in debug mode
• Use setuptools_scm to generate the version numbers
• Add a tox configuration to make it easy to automate some of the actions (reviving [WIP] Using Tox #8248)
• Update the CI configuration to use tox whenever possible

I've split out some of the changes into individual self-contained PRs to make the reviewing process a little easier. Specifically, the other PRs that should be reviewed first are:

• Declare all package data in setup.cfg instead of in setup_package.py files [Declare all package data in setup.cfg instead of in setup_package.py files #9729]
• Use environment variables to select system libraries [Use environment variables to select system libraries #9730]
• Use environment variable instead of --debug flag to control extension warnings [Get --debug flag from sys.argv instead of using an astropy helpers routine #9731]

I've also pushed a 4.1.dev tag to make sure that setuptools_scm works properly (how this should be done will be detailed in the packaging docs). For anyone reviewing this, I've tried to keep the commits as tidy as possible, and would recommend looking at individual commits in turn.

• Update the developer documentation to reflect the changes here
• Release a new version of pytest-astropy-header (Catch case where the astropy version is 'unknown' pytest-astropy-header#11)
• Make sure the CI configuration is equivalent to before as much as possible
• Add back coverage testing and reporting
• Add changelog entry
• Add help messages to setup.py about new way to run tests and build docs
• Update release docs for version number
• Wait until the APE is discussed and approved
• Update the Travis configuration to re-enable cron builds
• Rebase once Declare all package data in setup.cfg instead of in setup_package.py files #9729, Use environment variables to select system libraries #9730 and Get --debug flag from sys.argv instead of using an astropy helpers routine #9731 are merged
• Finalize Add more plugins and refactor infrastructure pytest-astropy#29 and release pytest-astropy
• Update CI config to take into account new pytest-astropy release

Things that I won't do as part of this PR and can be done in follow-ups instead:

• Switch to Azure Pipelines or GitHub actions
• Re-implement the equivalent to the --readonly option (Implement equivalent of --readonly option for testing with new infrastructure #9852)

EDIT: Close #9871

[mhvk]

@astrofrog - this is very nice indeed! While I like seeing everything together (and especially that almost everything leads to code reduction rather than increase!!), I do wonder if it isn't best to actually merge roughly following the order you have your commits, so we can discuss smaller pieces in one place. For instance, it would seem that 69df655, which moves packaging information to setup.cfg is super-uncontroversial and could immediately get merged.

As another example, the --debug option, which I will admit I had never looked into, seems rather mis-named - in that it seems to change only whether warnings are visibly generated rather than set up the compiled code to be parsed by a debugger. I may well be wrong about this, but think it is easier to discuss other names if we treat it as a separate commit.

[astrofrog]

@mhvk - I agree that some things here can be broken out into standalone PRs - some of them have to be done together though, e.g. pyproject.toml and tox. I'll take a look tomorrow at what could be split out and merged first.

[bsipocz]

Switch to Azure Pipelines

I would rather switch straight to Actions?

[astrofrog]

I would rather switch straight to Actions?

Yes we should discuss options next week! But in any case I want to not mix that up with this PR.

[astrofrog]

@pllim - done!

[mhvk]

My sense would be to get this in. Any objections?

[Cadair]

Do it! I have two follow-on branches already 😛 🚢

[taldcroft]

@bsipocz @eteq - are you each OK with this PR?

[pllim]

My vote is to merge now. I think @bsipocz already said she got no problem with merging this. And as for @eteq , he can open follow-up PR if he is unhappy with the content when he is available again. 😉

[taldcroft]

OK then, I'm good with merging!

@astrofrog had previously asked the coordination committee about process for approval of this PR, so I'm speaking now with my CoCo hat on.

[pllim]

Merged. Thank you, @astrofrog and everyone involved! 🎉

[Cadair]

🚀 🚀 🎉 🎆 👯‍♀️ 👯‍♂️

[mhvk]

Thanks, indeed, @astrofrog and everyone else!

[astrofrog]

Thanks for merging this! And please don't hesitate to ping me or @Cadair if you run into any issues locally or in future pull requests 🙏

[namurphy]

Thank you to everyone who made this PR happen! I'd also like to thank everyone who has contributed to astropy-helpers over the last several years, as it has been really helpful to the community during the time that it has been around.

[mhvk]

Seconded! I've been convinced over this PR how great it is to get rid of astropy helpers, but it made things much easier when these new nifty tools weren't around!

[eteq]

I had been planning to review this but it's fine that it got merged - @pllim is definitely right, I can always do my own PR with any changes, because big-picture I was already fine with it!