Make recipes beautiful again #2972

ocefpaf · 2018-06-15T18:28:11Z

@msarahan hopefully I did not break anything. The goal is to:

enforce sha256;
change the script section to use pip;
simplify the recipe by reducing the excess of jinja variables.

I'm sure there are more places we can optimize and simplify the code but I'm not too familiar with conda-skeleton to go any further.

Also, I'd like to remove the single quotes from the name and version values in the package section, to keep a consistent style with the rest of the recipe, but I'm not sure how to do that in an ruamel_yaml.comments.CommentedMap object (and that is not really that important anyway.)

fixes #3000

msarahan · 2018-06-25T17:51:44Z

conda_build/skeletons/pypi.py

-            if any(re.match(r'^setuptools(?:\s|$)', req) for req in d['build_depends']):
-                ordered_recipe['build']['script'] += ('--single-version-externally-managed '
-                                                      '--record=record.txt')
+            ordered_recipe['build']['script'] = 'python -m pip install . --no-deps --ignore-installed'


If we are forcing pip by default, we also need to disable the hermetic build stuff, since that duplicates conda-build's intent and can also break what conda-build does.

https://github.com/pypa/pip/pull/5053/files#diff-1457c307ea972916e3ade69c27f0c41aR456

Now the thing that really stinks is that that flag is only available on new pip, so it'll break old builds, or ecosystems that don't have pip 10 yet.

I'm OK with the rest of the changes in this PR, but I don't think we're ready for this one yet.

I think there is a environment variable we could set which would turn off build isolation and still work with pip 9

great, that would be better. I'll see if we can find it.

Can we add --no-cache-dir to the pip arguments as well (to force it to re-create/re-download the wheel and not rely on local cache, if present)? I was bit by this quite a few times.

PIP_NO_BUILD_ISOLATION=False disables build isolation. My expectation was that this should be set to True but pypa/pip#5229 and my own testing confirms that False is the correct setting. This variable has no effect on older pip versions.

I would also suggest adding -vvv to increase verbosity which makes debugging failed builds much easier.

msarahan · 2018-07-15T17:43:45Z

I rebased onto master and added a bit. I hope that's OK.

nehaljwani · 2018-07-15T18:44:16Z

conda_build/skeletons/pypi.py

-            if any(re.match(r'^setuptools(?:\s|$)', req) for req in d['build_depends']):
-                ordered_recipe['build']['script'] += ('--single-version-externally-managed '
-                                                      '--record=record.txt')
+            ordered_recipe['build']['script'] = ('python -m pip install . --no-deps '


Should this be $PYTHON instead or is that not required?

not strictly. python should always be on PATH always in the env. If you make it $PYTHON, that's more definite, but it's not strictly necessary. I'll go ahead and change it, though.

Hm.. although I'm usually in favor of generalizing and assuring correctness, I'd rather have just python instead of "${PYTHON}"/"%PYTHON%" here. See comments from conda-forge/conda-forge.github.io#551 (comment) onward.
Just python is also the prevalent thing on aggregate and feedstocks.

There was some reason why Intel had this pattern. I can't remember. It did have some advantage over depending on PATH.

The current code uses jinja2 instead of env vars, and should work. Errr... it will, when I can get to a better internet connection that doesn't give me git errors.

ocefpaf · 2018-07-19T12:49:42Z

Thanks for the review here. I'll try to address them ASAP, just got caught on a storm here at the day job.

msarahan · 2018-07-19T13:15:40Z

Don't worry about it. I'll fix the CI crap and it'll be ready to go. Appveyor is really misbehaving, and I don't understand why yet.

msarahan · 2018-07-20T02:17:49Z

finally fixed. Having PYTHON set in the Appveyor matrix seemed to be overriding conda-build, which meant that the {{ PYTHON }} was something dumb, like C:\Miniconda2-x64

ocefpaf · 2018-07-20T11:16:00Z

Thanks @msarahan for taking care of this!

jakirkham · 2018-07-27T18:23:10Z

Thanks @ocefpaf and @msarahan. This is very good to have. :)

github-actions · 2022-04-19T06:07:05Z

Hi there, thank you for your contribution!

This pull request has been automatically locked because it has not had recent activity after being closed.

Please open a new issue or pull request if needed.

Thanks!

msarahan reviewed Jun 25, 2018

View reviewed changes

msarahan force-pushed the make_recipes_beautiful_again branch from 844701e to 25b6a85 Compare July 15, 2018 17:43

msarahan force-pushed the make_recipes_beautiful_again branch from 25b6a85 to 6d16307 Compare July 15, 2018 17:44

nehaljwani reviewed Jul 15, 2018

View reviewed changes

ocefpaf and others added 4 commits July 16, 2018 13:00

remove fn

b062eda

fix typo

9183976

simplify the recipe

e22170f

add additional pip args; set env var to disable pip env isolation

0679c5f

msarahan force-pushed the make_recipes_beautiful_again branch from 3dee69b to 6c1ef40 Compare July 17, 2018 14:08

improve environment marker support in pypi skeleton

e6cbfac

msarahan force-pushed the make_recipes_beautiful_again branch 5 times, most recently from 573edba to 4f4d45c Compare July 18, 2018 00:21

msarahan force-pushed the make_recipes_beautiful_again branch 2 times, most recently from df52fc3 to b82b367 Compare July 19, 2018 16:44

remove + and - from implicit resolver list (removed by pyyaml?)

e6dc777

msarahan force-pushed the make_recipes_beautiful_again branch 5 times, most recently from a3e0b69 to 7480c9a Compare July 20, 2018 01:06

fix slash handling with {{ PYTHON }} on win; add pip dep to pypi skel

e64bfcf

msarahan force-pushed the make_recipes_beautiful_again branch from 7480c9a to e64bfcf Compare July 20, 2018 02:05

msarahan merged commit 9660d13 into conda:master Jul 20, 2018

ocefpaf deleted the make_recipes_beautiful_again branch July 20, 2018 11:15

jakirkham mentioned this pull request Jul 27, 2018

Configure pip during build to avoid dependencies #2748

Closed

nicoddemus mentioned this pull request Aug 4, 2018

Use pip to install packages in recipes generated by PyPI skeleton #2812

Closed

mbargull mentioned this pull request Sep 1, 2018

Disabling pip's cache directory results in .egg-info directories #3094

Closed

github-actions bot added the locked [bot] locked due to inactivity label Apr 19, 2022

github-actions bot locked as resolved and limited conversation to collaborators Apr 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make recipes beautiful again #2972

Make recipes beautiful again #2972

ocefpaf commented Jun 15, 2018 •

edited by msarahan

Loading

msarahan Jun 25, 2018

jjhelmus Jun 25, 2018

msarahan Jun 25, 2018

nehaljwani Jun 26, 2018

jjhelmus Jun 26, 2018

msarahan commented Jul 15, 2018

nehaljwani Jul 15, 2018

msarahan Jul 16, 2018

mbargull Jul 16, 2018

msarahan Jul 16, 2018

ocefpaf commented Jul 19, 2018

msarahan commented Jul 19, 2018

msarahan commented Jul 20, 2018

ocefpaf commented Jul 20, 2018

jakirkham commented Jul 27, 2018

github-actions bot commented Apr 19, 2022

Make recipes beautiful again #2972

Make recipes beautiful again #2972

Conversation

ocefpaf commented Jun 15, 2018 • edited by msarahan Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msarahan commented Jul 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocefpaf commented Jul 19, 2018

msarahan commented Jul 19, 2018

msarahan commented Jul 20, 2018

ocefpaf commented Jul 20, 2018

jakirkham commented Jul 27, 2018

github-actions bot commented Apr 19, 2022

ocefpaf commented Jun 15, 2018 •

edited by msarahan

Loading