Rebuild for python310 (redux) #175

h-vetinari · 2021-11-09T06:53:56Z

Based on #174, but cleaning up the left over test skips, invocation & CPU settings from drone/travis.

Closes #174

…nda-forge-pinning 2021.11.09.00.35.45

conda-forge-linter · 2021-11-09T06:54:00Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

h-vetinari · 2021-11-09T07:47:59Z

PPC looks like it's going to be an utter mess of failures again (presumably the same kind of QEMU bugs that are responsible for the scipy test suite failing on PPC-via-QEMU; scipy is currently built without running the test suite, as people were opposed to dropping it completely, and preferred untested binaries to nothing, see discussion in conda-forge/scipy-feedstock#191).

An excerpt:

...........FF.......s...EEEEEEEEEEEEE..EEEE............................s [  6%]
sssssssssssssssssssssssssssss.....................................FFF... [  6%]
......F....................FFFF.....F...FFF...FFF..F..........F.F..FFFFF [  7%]
..F...F..F........................................F.....FF.........FF... [  7%]
......FFFFFF..FFFFFFFFFF...FFFFFFFFFFF.............F......F..........F.. [  7%]
......F.........F.....F..........F...........F..FF.F..F..F..F...F.F.FF.F [  8%]
F.FF.FF.F.FFFF.F..FFFFF....................................F............ [  8%]
..F............F.F............FF...........F..FEEEFEEEFEEEFEEFEEE..FEEE. [  8%]
FF..F.FF..F.......F........F.F......................ssssssssssssssssFsss [  9%]
sssssssssss...........F.......F......F.....F..F.........F.FFF.FFF.FFF... [  9%]
...........F..F.....FFFFFF.F...FF...FF..............F...F.F.FF..F.F.FFFF [  9%]
FFF.F...FFF..FFF.F..FFFF...F....F....FF..FF..FFFFF...FF..FF......FF..... [ 10%]
........FF.F....F..F.........................F.......................F.F [ 10%]
F.F...FF.F.FFF.FF...F....FF..................F.........FFFFFFF..FFFFFFFF [ 10%]
FF...FFFFFFFFFFFFFFFFFF.........F.F.....FFFFF.FFF.FFF.F........FF.FF.... [ 11%]
.........FFFF........................................................... [ 11%]
.................F.F........x.....................................FF.FF. [ 11%]
FF.FF.....F......F..FF....FF...F...FF......................FF..F........ [ 12%]
............F..F..F..........F.........F.F..FFFFFF............F...FFFFF. [ 12%]
.........F...F......FsF...F.........F......F.....F.....F......F..FF.FFF. [ 12%]
FF...F....F...F....F.................................................... [ 13%]

glemaitre · 2021-11-09T10:47:17Z

So conda-forge switched all the builds to Azure?

adrinjalali · 2021-11-09T10:57:03Z

I'd be okay with untested binary for ppc

glemaitre · 2021-11-09T10:59:25Z

I'd be okay with untested binary for PPC

+1 on my side for this release build

h-vetinari · 2021-11-09T12:13:00Z

So conda-forge switched all the builds to Azure?

Involuntarily. Both Travis and Drone became more or less unusable over the last 1-2 months.

h-vetinari · 2021-11-09T12:57:26Z

I'd be okay with untested binary for ppc

The following is hardly a picture of confidence (~1450 failures+errors), but I'm happy to go along with the more pragmatic approach here.

= 1319 failed, 18656 passed, 1445 skipped, 60 xfailed, 39 xpassed, 3359 warnings, 133 errors in 13833.57s (3:50:33) =

h-vetinari · 2021-11-09T20:44:23Z

@isuruf @jakirkham
Just some visibility that - per this PR - the test suite is not being run for PPC anymore. Otherwise there's about 1450 failures + errors. This is how I interpreted the discussion regarding dropping PPC on the scipy-feedstock, but decisions might be different case-by-case?

there hasn't yet been a pypy run completed in QEMU on azure, so we take test timings of a cpython run that hit a slower agent (all added tests ran >80 sec there)

h-vetinari · 2021-11-10T11:30:03Z

@conda-forge/scikit-learn
In addition to the point about PPC, this is now skipping a bunch of tests on aarch (& especially pypy), but at least it's green. 😅
It's possible that aarch+pypy only passes on the faster azure agents, but that's nothing an occasional restart can't fix (and this PR is about cpython anyway).

adrinjalali · 2021-11-10T13:15:51Z

yeah those are quite slow. Thanks for skipping them.

adrinjalali

I think this makes sense.

recipe/meta.yaml

Co-authored-by: Isuru Fernando <[email protected]>

h-vetinari · 2021-11-11T02:56:19Z

win + pypy "only" ran into numpy.core._exceptions._ArrayMemoryError: Unable to allocate <x> MiB - I think this can be considered spurious (considering that it passed before).

If desired, I can try something like setting the pagefile size explicitly (c.f. e.g. conda-forge/scipy-feedstock@fcf4f63) - not sure if that'll help though, as the error is usually more explicitly about running out of pagefile.

conda-forge-linter · 2021-11-11T05:44:25Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

Failed to even lint the recipe, probably because of a conda-smithy bug 😢. This likely indicates a problem in your meta.yaml, though. To get a traceback to help figure out what's going on, install conda-smithy and run conda smithy recipe-lint . from the recipe directory.

conda-forge-linter · 2021-11-11T05:55:31Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

h-vetinari · 2021-11-11T12:45:57Z

Someone feel like pressing the merge button? 🙃

rth

Minor comment otherwise LGTM. Thanks for working on this @h-vetinari !

rth · 2021-11-11T13:13:38Z

recipe/meta.yaml

    - export OMP_NUM_THREADS=1                 # [aarch64 or ppc64le]
-    - pytest --pyargs sklearn -k "not ({{ tests_to_skip }})" -n {{ cpus }} --durations=50 {{ extra_pytest_args }}
+    - pytest --pyargs sklearn -k "not ({{ tests_to_skip }})" -nauto --timeout=1200 --durations=50  # [not ppc64le]


The nauto is only reliable when psutils is installed. e.g. when running inside docker with limited CPU resources https://github.com/pytest-dev/pytest-xdist/blob/2bac85562b996059b152b3491d73ddbbc413318a/src/xdist/plugin.py#L30 is this the case in the CI? Otherwise it would often return say 16 CPU while CI can only use 2.

Hm, interesting point. On azure this seems to work reliably, i.e. there are only ever 2-3 workers per image/run. It's IMO an improvement over explicitly setting a number of processors (as previously), which might then not all be available. Though even in that case, the CI passed, just slower due to contention.

h-vetinari · 2021-11-12T21:15:02Z

Last commit is just a comment update; I believe this PR is ready for an automerge-label 🙃

rth · 2021-11-12T21:17:54Z

I believe this PR is ready for an automerge-label upside_down_face

It's done. Thanks again for your work @h-vetinari !

github-actions · 2021-11-13T00:43:03Z

Hi! This is the friendly conda-forge automerge bot!

I considered the following status checks when analyzing this PR:

linter: passed
azure: passed

Thus the PR was passing and merged! Have a great day!

h-vetinari · 2021-11-13T01:09:44Z

The win+pypy failure is spurious:

E       numpy.core._exceptions._ArrayMemoryError: Unable to allocate 22.9 MiB for an array with shape (300, 5000) and data type complex128

Can someone please press merge? I can also join as a maintainer if people want - that way I wouldn't have to ping as often 😋

regro-cf-autotick-bot and others added 3 commits November 9, 2021 04:53

Rebuild for python310

a77474e

MNT: Re-rendered with conda-build 3.21.4, conda-smithy 3.14.3, and co…

402b96f

…nda-forge-pinning 2021.11.09.00.35.45

remove timout-related test skips now that drone/travis are gone

a728ec8

h-vetinari requested review from adrinjalali, amueller, astaric, glemaitre, jakirkham, jeremiedbb, jnothman, lesteve, ocefpaf, ogrisel and rth as code owners November 9, 2021 06:53

h-vetinari added 3 commits November 9, 2021 23:26

skip test suite on ppc

9b19959

add timeout for individual tests

44c8bfa

skip longest-running tests on aarch

08df65a

h-vetinari added 2 commits November 10, 2021 07:55

skip some more tests on aarch + pypy

bcab1d2

more skips for aarch+pypy due to runtime

3ca04a4

there hasn't yet been a pypy run completed in QEMU on azure, so we take test timings of a cpython run that hit a slower agent (all added tests ran >80 sec there)

h-vetinari force-pushed the 3.10 branch from 250fb2e to 3ca04a4 Compare November 10, 2021 03:38

h-vetinari added 2 commits November 10, 2021 15:24

readd OMP_NUM_THREADS=1 to hopefully reduce runtime on aarch

7ac8a1d

add dev_url

a28efb5

adrinjalali approved these changes Nov 10, 2021

View reviewed changes

isuruf reviewed Nov 10, 2021

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

isuruf reviewed Nov 11, 2021

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

Only skip aarch-tests when running in CI

d008216

Co-authored-by: Isuru Fernando <[email protected]>

h-vetinari force-pushed the 3.10 branch from e45d529 to d008216 Compare November 11, 2021 00:54

one last round of skip updates

a018265

no comments in jinja-ops

0cc5319

rth approved these changes Nov 11, 2021

View reviewed changes

add comment about skipped test suite on PPC

cff036b

rth added the automerge Merge the PR when CI passes label Nov 12, 2021

h-vetinari closed this Nov 13, 2021

h-vetinari reopened this Nov 13, 2021

github-actions bot merged commit d674415 into conda-forge:master Nov 13, 2021

h-vetinari deleted the 3.10 branch November 13, 2021 07:57

xhochy mentioned this pull request Dec 14, 2021

Rebuild for python310 #173

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rebuild for python310 (redux) #175

Rebuild for python310 (redux) #175

h-vetinari commented Nov 9, 2021 •

edited

Loading

conda-forge-linter commented Nov 9, 2021

h-vetinari commented Nov 9, 2021

glemaitre commented Nov 9, 2021

adrinjalali commented Nov 9, 2021

glemaitre commented Nov 9, 2021

h-vetinari commented Nov 9, 2021

h-vetinari commented Nov 9, 2021

h-vetinari commented Nov 9, 2021

h-vetinari commented Nov 10, 2021

adrinjalali commented Nov 10, 2021

adrinjalali left a comment

h-vetinari commented Nov 11, 2021

conda-forge-linter commented Nov 11, 2021

conda-forge-linter commented Nov 11, 2021

h-vetinari commented Nov 11, 2021

rth left a comment

rth Nov 11, 2021 •

edited

Loading

h-vetinari Nov 11, 2021

h-vetinari commented Nov 12, 2021

rth commented Nov 12, 2021

github-actions bot commented Nov 13, 2021 •

edited

Loading

h-vetinari commented Nov 13, 2021

Rebuild for python310 (redux) #175

Rebuild for python310 (redux) #175

Conversation

h-vetinari commented Nov 9, 2021 • edited Loading

conda-forge-linter commented Nov 9, 2021

h-vetinari commented Nov 9, 2021

glemaitre commented Nov 9, 2021

adrinjalali commented Nov 9, 2021

glemaitre commented Nov 9, 2021

h-vetinari commented Nov 9, 2021

h-vetinari commented Nov 9, 2021

h-vetinari commented Nov 9, 2021

h-vetinari commented Nov 10, 2021

adrinjalali commented Nov 10, 2021

adrinjalali left a comment

Choose a reason for hiding this comment

h-vetinari commented Nov 11, 2021

conda-forge-linter commented Nov 11, 2021

conda-forge-linter commented Nov 11, 2021

h-vetinari commented Nov 11, 2021

rth left a comment

Choose a reason for hiding this comment

rth Nov 11, 2021 • edited Loading

Choose a reason for hiding this comment

h-vetinari Nov 11, 2021

Choose a reason for hiding this comment

h-vetinari commented Nov 12, 2021

rth commented Nov 12, 2021

github-actions bot commented Nov 13, 2021 • edited Loading

h-vetinari commented Nov 13, 2021

h-vetinari commented Nov 9, 2021 •

edited

Loading

rth Nov 11, 2021 •

edited

Loading

github-actions bot commented Nov 13, 2021 •

edited

Loading