Drop appveyor in favor of azure

[xavfernandez]

We're currently running windows tests on Appveyor & Azure:

On Appveyor, we only get a single worker for all builds that takes roughly 30/31 minutes per build to run:

• the whole test suite on Python27 & Python36-x64
• unit test only on Python27-x64, Python35, Python35-x64 & Python36.

While on the azure pipelines we appear to have several workers that take ~ 41 min per build to run:

• the whole test suite on Python27-x64 & Python36-x64
• unit test only on Python27-x86, Python35-x86, Python35-x64, Python36-x86, Python37-x64 & Python37-x86

Since tests have started to fail on Appveyor for some unknown reason:
https://ci.appveyor.com/project/pypa/pip/builds/26142589 vs https://ci.appveyor.com/project/pypa/pip/builds/26132882 for the same commit (& https://dev.azure.com/pypa/pip/_build/results?buildId=10128 or https://dev.azure.com/pypa/pip/_build/results?buildId=10183 for the same commit on Azure), it might be the good time to drop the testing on Appveyor ?

[pradyunsg]

I'm in.

I also want us to eventually clean up the mess that is the 20+ jobs from Azure on CI.

[pfmoore]

I'm fine with this, but we'd need to make the Azure jobs mandatory (so we don't lose the constraint that Windows test runs must pass) and I don't know if we can specify that only some of the Azure builds are mandatory - I don't recall why we originally wanted the Azure builds to be optional, but we should check that those reasons no longer apply.

[xavfernandez]

I've made the "Windows" (and MacOS) Azure checks mandatory while making Appveyor "optional", we'll see in the next PR how it behaves.

[pfmoore]

Cool!

[xavfernandez]

Ok, it seemed to work on #6203 :)

But given that Windows Azure build takes 40+ minutes (which is close to what travis needs with its 30 minute pypy3 builds), I'm wondering if we should not keep Appveyor and try to take advantage of the various CI platforms to better distribute our build times ?
(This would require Appveyor build to be fixed though...)

[pfmoore]

Yeah, Windows test runs are really annoyingly slow. A longer term fix is likely to need reducing the level to which we spawn subprocesses in the test suite (process creation is much more costly on Windows than on Unix, and anecdotally my impression is that this is where the Windows tests take up a lot of their time - but someone needs to do some actual numbers).

Short term, spreading the load is likely the best approach.

[xavfernandez]

Short term, spreading the load is likely the best approach.

Which would mean keeping Appveyor: any idea on how to fix its build ?

[pfmoore]

any idea on how to fix its build

Not really, I'm afraid. An instinctive guess would be some warning output going to stderr rather than stdout, or a stray exit code being 1 rather than 0 (weren't there relatively recent "rationalise logging" commits - some sort of generally harmless difference there causing Windows to have a fit?). But as to why it's intermittent, no idea.

Unfortunately, I doubt I'll have time to look at this soon. So I guess the really short term solution is to use Azure and live with the long test times :-(

[chrahunt]

#6769 should hopefully fix the CI runs.

[pfmoore]

Nice catch!

[pradyunsg]

I've gone ahead abs merged the CI fix! That's a nice find @chrahunt! :)

[pradyunsg]

I'm also going to take the liberty of closing this issue since Appveyor should be fixed by the PR and switching to Azure is also done.