Better discovery of macos system version to determine proper platform tag #314

Czaki · 2019-09-24T16:29:14Z

In this moment wheel package create python wheel with same platform tag like python was compiled. But wheel may contain libraries which needs higher macos version. In this patch I add code which scan all files added to wheel and if their suffix suggest that it is library (*.so or *.dynlib) and then try to determine macos version by scan Mach-O header. This code is pure python and do not depends on external files. The struct in comments are copied from Apple heade files.

To manually check library needs use otool on file. and look on LC_BUILD_VERSION or LC_VERSION_MIN_MACOSX section.

To set macosx minimum version set environment variable MACOSX_DEPLOYMENT_TARGET before compile. If compilation is launched with python setup.py ... then the version from variable need to be higher or equal than version returned by distutils.util.get_platform()

travis ci fail is connected with python 2.7.3 and occurs also in original data. It is not connected with my changes.

letmaik · 2019-09-24T17:00:00Z

wheel/lib_file_analyse.py

+        pos = lib_file.tell()
+        segment_base = read_data(SegmentBase, lib_file)
+        lib_file.seek(pos)
+        if segment_base.cmd == LC_VERSION_MIN_MACOSX:


Is it guaranteed that LC_VERSION_MIN_MACOSX always appears before LC_BUILD_VERSION?

They are disjoint. form macosx version 10.8 to 10.13 there is LC_VERSION_MIN_MACOSX section and since 10.14 (current) there is LC_BUILD_VERSION section.

As you can see both contains return statement so if function found one of them then it finish.

codecov · 2019-09-25T21:20:18Z

Codecov Report

Merging #314 into master will increase coverage by 4.92%.
The diff coverage is 91.89%.

@@            Coverage Diff             @@
##           master     #314      +/-   ##
==========================================
+ Coverage   59.71%   64.64%   +4.92%     
==========================================
  Files          12       13       +1     
  Lines         844      987     +143     
==========================================
+ Hits          504      638     +134     
- Misses        340      349       +9

Impacted Files	Coverage Δ
wheel/bdist_wheel.py	`51.61% <0%> (-0.49%)`	⬇️
wheel/lib_file_analyse.py	`93.51% <93.51%> (ø)`
wheel/pep425tags.py	`47.58% <94.59%> (+15.44%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b227ddd...9144297. Read the comment docs.

…on and description on begin of Mach-O parser

letmaik · 2019-09-25T23:17:17Z

wheel/pep425tags.py

@@ -126,8 +141,23 @@ def get_platform(archive_root):
                    if version is not None:
                        base_version = max(base_version, version)
        if base_version[-1] == 0:
-            base_version = base_version[:-1]
-        base_version = "_".join([str(x) for x in base_version])
+            fin_base_version = base_version[:-1]


Looks like fin_base_version is undefined if the if-condition is not true. This would break the code below, right? Is there any reason why you try to support x.y.z instead of just x.y?

I use x.y.z because this is format used inside libraries header. This is 190 line of file lib_file_analyse.py

uint32_t version; /* X.Y.Z is encoded in nibbles xxxx.yy.zz */

I push fix for undefined variable.

codecov · 2019-09-26T05:49:20Z

Codecov Report

Merging #314 into master will increase coverage by 5.17%.
The diff coverage is 91.77%.

@@            Coverage Diff             @@
##           master     #314      +/-   ##
==========================================
+ Coverage   59.71%   64.89%   +5.17%     
==========================================
  Files          12       13       +1     
  Lines         844      997     +153     
==========================================
+ Hits          504      647     +143     
- Misses        340      350      +10

Impacted Files	Coverage Δ
wheel/bdist_wheel.py	`51.61% <0%> (-0.49%)`	⬇️
wheel/macosx_libfile.py	`92.72% <92.72%> (ø)`
wheel/pep425tags.py	`50.32% <95.55%> (+18.18%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b227ddd...b1dd2c4. Read the comment docs.

Czaki · 2019-09-26T09:02:53Z

All improvements comes from request from the discussion in matthew-brett/delocate#56

agronholm

Code style changes required. As for the actual logic, I am not qualified to review that.

setup.py

wheel/pep425tags.py

tests/test_lib_file_analyse.py

wheel/bdist_wheel.py

agronholm · 2019-09-26T13:15:48Z

tests/test_lib_file_analyse.py

+    dirname = os.path.dirname(__file__)
+    dylib_dir = os.path.join(dirname, "testdata",
+                             "macos_minimal_system_version")
+    with mocker.patch("distutils.util.get_platform", return_value="macosx-10.14-x86_64"):


If you're using mocks in with blocks, what's the point of using mocker in the first place?

I'm learning how to use mock. If I good understand in python 2.7 there is preference to use mock library and in python 3.4+ use unittests.mock.

And I do not have idea how test this functions without mock.

How to fix it in best way. change dependency on mock. Do some if depends on python version?

Python 2.6 did not have unittest.mock. But Python 2.7 does. The mocker fixture, on the other hand, works like pytest's built-in monkeypatch fixture in that it automatically undoes all the mocks that it sets up during the test request.

I rewrite tests to base on monkeypatch. First I think that it is not enough, but maybe I better understand it.

wheel/bdist_wheel.py

wheel/lib_file_analyse.py

agronholm · 2019-09-26T15:18:37Z

Close enough. Now I'd like to find at least one qualified reviewer for the actual business logic.

Czaki · 2019-09-26T15:26:14Z

A moment ago I realized that I'm not sure if my change do not break plat-name flag. I'm not sure if I proper understand control flow in distutils.

My changes in bdist_wheel:get_tag.

Could you check it?

agronholm · 2019-09-26T15:28:37Z

Okay, I'll check.

agronholm · 2019-09-26T16:05:43Z

The way I see it is that if someone has explicitly set a platform tag starting with macosx, it will get overridden by the detected tag. Couldn't this simply have been set with plat_name = self.plat_name or get_platform(self.bdist_dir)?

Czaki · 2019-09-26T17:52:02Z

It not fix the poroblem. From the comment on begin of get_tag
# bdist sets self.plat_name if unset, we should only use it for purepy

And I check that on begin of get_tag the name is set.

Maybe finally some patch should be applied to python, but this take very long time (waiting on updates on CI servers, distribution repos etc.) and fixes of wheel package can be used earlier.

… test

Czaki · 2019-09-29T18:24:24Z

@agronholm I found that if user set platform name by cli argument then plat_name_supplied is set to true (in function finalize_optionswhich is called before `get_tag) so first if is satisfied and there is no overwrite of user choose.

        if self.plat_name_supplied:
            plat_name = self.plat_name

Czaki · 2019-10-10T15:55:13Z

Did I need introduce any changes?

agronholm · 2019-10-13T13:28:42Z

No, I've been on holiday for some days and I will review this shortly. Would be nice to also get a statement from someone who understands macOS binary compatibility.

Czaki · 2019-10-13T17:20:16Z

I can send you macos headers (*.h) on which I base during writing.

agronholm · 2019-10-14T09:10:38Z

I honestly don't think they would help as I have no expertise on macOS specific development. I'll ask on the distutils-sig mailing list and see if I can get someone with that expertise to review this. Other than that I have no objections to merging this.

wheel/pep425tags.py

agronholm · 2019-10-17T07:06:30Z

I am also reluctant to add a dependency for wheel, or to vendor such a large project.

Czaki · 2019-10-23T11:50:01Z

@agronholm There are any news? Someone declared to review the code?

agronholm · 2019-10-23T13:08:57Z

Nope, so I'm going to go ahead and merge this soon.

agronholm · 2019-10-23T14:25:33Z

Thank you! Now let's just hope that the (mac) world doesn't burn because of this 😄

Czaki · 2019-10-24T08:43:11Z

Could you inform me when new release was created? I wait for this to update pypa/cibuildwheel#156

agronholm · 2019-10-24T09:27:25Z

I'll let you know once I've made a new release.

YannickJadoul · 2019-12-26T16:41:57Z

Just a question: any idea or plan when this new version of wheel would be released?
I definitely don't want to push/rush anything here, but just an estimate would be nice for further planning of our next cibuildwheel release.

agronholm · 2019-12-26T17:12:00Z

I'm waiting for PR #317 to be finished so I can merge it. I don't want to ship this feature in its current state.

YannickJadoul · 2019-12-26T17:16:24Z

Thanks! I had no clue that PR wasn't ready yet. But good to know this is in @Czaki's hands, then; he's also the author of the PR on the cibuildwheel side, so he can coordinate further, then :-)

Czaki · 2019-12-26T21:27:52Z

@YannickJadoul Could you try to review all changes? (both PR). I testing it in my projects and found two bugs (fixed in PR #317) but maybe it is better to someone with more macos experience take a look on it.

@agronholm In current time I do not see any problem in PR #317 but I learn MacOs on example, so sometimes something surprise me. In this moment I do not have plans for next commits, but I use modified version of wheel in projects to find any problems.

agronholm · 2019-12-26T21:48:21Z

In current time I do not see any problem in PR #317

Your last message ("I cannot create proper test file but I found such in delocate package. I add missed test case.") indicates that you plan to add a test. Is this not correct?

Czaki · 2019-12-26T22:40:09Z

@agronholm I add it in this commit 0dd9f3e
(before I write cited comment)

agronholm · 2019-12-26T22:55:45Z

I see. This is why the correct tense of the verb is important! I will review your PR tomorrow.

YannickJadoul · 2019-12-26T23:36:15Z

@YannickJadoul Could you try to review all changes? (both PR). I testing it in my projects and found two bugs (fixed in PR #317) but maybe it is better to someone with more macos experience take a look on it.

I'm happy to take a look. Which PRs do you mean?
But I really don't have more macOS experience/knowledge than you, I'm afraid. I learned a lot about different versions of macOS to be targeted through your PRs. Though, still happy to take a look, if you want.

Czaki · 2019-12-27T01:29:52Z

@YannickJadoul In both (This and #317) In file https://github.com/Czaki/wheel/blob/master/wheel/pep425tags.py#L112
I introduce function calculate_macosx_platform_tag function.
All tests are here: https://github.com/Czaki/wheel/blob/master/tests/test_macosx_libfile.py

If you have time to check if logic is proper and all most important cases are covered.

@agronholm Sorry. I need to remember to check text before accept comment.

YannickJadoul

@Czaki If you wanted some comments on this, here you go. I don't know enough about these mac headers to say a lot about that part, and for the rest, these are mainly some suggestions and thoughts.

The only two things I'm sure of could be changed are some details in the two messages.

EDIT: If you prefer for me to do it, I can also just create a PR that fixes the typos?

YannickJadoul · 2019-12-28T22:34:26Z

wheel/macosx_libfile.py

+    return (((x << 24) & 0xFF000000) |
+            ((x << 8) & 0x00FF0000) |
+            ((x >> 8) & 0x0000FF00) |
+            ((x >> 24) & 0x000000FF))


Using struct might be shorter/easier? It's also closely related to what you're trying to do (fix endianess). (cfr. https://stackoverflow.com/a/27506692)

YannickJadoul · 2019-12-28T22:42:10Z

wheel/macosx_libfile.py

+  and if apple introduce some changes both implementation will need to be updated.
+"""
+
+import ctypes


Might not be worth changing everything, but isn't there also the struct built-in library that makes readying binary data easy? (See https://stackoverflow.com/a/52035410 for a discussion on struct vs. ctypes.) So struct.unpack(...) might be slightly shorter to use. But again, probably not worth changing.

I decide to use ctypes because I can use types like uint8. Struct library has only standard sizes (long, etc), not fixed sizes. And apple use fixed sizes in header.

Hmm, OK. But it seems that these standard sizes can be enforced:

Note

By default, the result of packing a given C struct includes pad bytes in order to maintain proper alignment for the C types involved; similarly, alignment is taken into account when unpacking. This behavior is chosen so that the bytes of a packed struct correspond exactly to the layout in memory of the corresponding C struct. To handle platform-independent data formats or omit implicit pad bytes, use standard size and alignment instead of native size and alignment: see Byte Order, Size, and Alignment for details.

https://docs.python.org/3/library/struct.html (and even already on Python 2 https://docs.python.org/2/library/struct.html).

But having uint8 might indeed be cleaner. Just thought I'd mention it :-)

YannickJadoul · 2019-12-28T22:43:04Z

wheel/pep425tags.py

+        problematic_files = "\n".join(problematic_files)
+        error_message = \
+            "[WARNING] This wheel needs higher macosx version than {} " \
+            "is set in MACOSX_DEPLOYMENT_TARGET variable. " \


Forgot to remove this line, after adding the format, later?

I do not understand. I use this text in format.

Depending on if "MACOSX_DEPLOYMENT_TARGET" in os.environ, it can be that "is set in MACOSX_DEPLOYMENT_TARGET variable." will be twice in the string, since you format it later and it is already in the format string.

wheel/pep425tags.py

wheel/macosx_libfile.py

YannickJadoul · 2019-12-28T22:58:50Z

wheel/macosx_libfile.py

+
+
+def parse_version(version):
+    zz = version & 2**9-1


Same here; máybe this is easier with struct?

I try to do this with struct but it is not intuitive for me. You have some suggestion how such code should look?

Yeah, OK, I forgot the thing was already converted into an int, and not bytes anymore.
This works, but it's not better, I think: struct.unpack('>HBB', struct.pack('>I', 0x000a0e01)). Not worth using if struct is not used anywhere else, I'd say :-)

Still thinking, though, because the current code confuses me a bit. Is this cleaner? I don't know, up to you, and just close if you like your solution better:

def parse_version(version): x = (version & 0xffff0000) >> 16 y = (version & 0x0000ff00) >> 8 z = (version & 0x000000ff) return x, y, z

It is much cleaner. Thanks.

wheel/pep425tags.py

Czaki added 2 commits September 24, 2019 16:19

read minimal macos version from from mach-O header

99a8bbf

add check macos version to code

a3da68d

Czaki mentioned this pull request Sep 24, 2019

MacOS Platform tag when MACOSX_DEPLOYMENT_TARGET is set #312

Closed

letmaik reviewed Sep 24, 2019

View reviewed changes

letmaik mentioned this pull request Sep 24, 2019

Should MACOSX_DEPLOYMENT_TARGET/LC_VERSION_MIN_MACOSX be verified? matthew-brett/delocate#56

Closed

Czaki mentioned this pull request Sep 25, 2019

upgrade macos version for support c++11 pypa/cibuildwheel#156

Merged

Add tests for get_platform, warning on update of macosx minimal versi…

93f1809

…on and description on begin of Mach-O parser

Czaki force-pushed the master branch from 63f3462 to 93f1809 Compare September 25, 2019 21:23

letmaik reviewed Sep 25, 2019

View reviewed changes

fix error for undefined fin_base_version varaible

9144297

add more verbose output to inform which files forced to bump version

d4ad3d9

agronholm requested changes Sep 26, 2019

View reviewed changes

Fix errors pointed in review.

306cde9

agronholm approved these changes Sep 26, 2019

View reviewed changes

Czaki and others added 2 commits September 26, 2019 23:06

change tests to use monkeypatching instead pytest-mock

be741f7

fix typo in testdata sub folder. Refactor tests to be clear what they…

1cde26c

… test

agronholm reviewed Oct 15, 2019

View reviewed changes

wheel/pep425tags.py Outdated Show resolved Hide resolved

agronholm merged commit b5733b8 into pypa:master Oct 23, 2019

agronholm added a commit that referenced this pull request Oct 25, 2019

Added news item for PR #314

552a230

YannickJadoul mentioned this pull request Nov 18, 2019

Adding PyPy support pypa/cibuildwheel#185

Merged

YannickJadoul mentioned this pull request Dec 26, 2019

Only build x86_64 / Intel 10.9+ on macOS pypa/cibuildwheel#220

Merged

YannickJadoul reviewed Dec 28, 2019

View reviewed changes

wheel/pep425tags.py Show resolved Hide resolved

YannickJadoul reviewed Dec 28, 2019

View reviewed changes

wheel/pep425tags.py Show resolved Hide resolved

YannickJadoul reviewed Dec 28, 2019

View reviewed changes

wheel/pep425tags.py Show resolved Hide resolved

YannickJadoul reviewed Dec 28, 2019

View reviewed changes

wheel/pep425tags.py Show resolved Hide resolved

tgaddair mentioned this pull request Mar 13, 2020

Fixed call to get supported tags when building GPU Dockerfile horovod/horovod#1792

Merged

Czaki mentioned this pull request Mar 31, 2020

replace pep425tags with packaging #346

Merged

pradyunsg mentioned this pull request Apr 1, 2020

tags: MacOS base_version via dynlib files pypa/packaging#288

Open

This was referenced Apr 7, 2020

Weel name fix parse header matthew-brett/delocate#68

Closed

Fix weel name using otool matthew-brett/delocate#61

Closed

QuLogic mentioned this pull request Nov 2, 2023

Limit macOS platform tag by compiled deployment target? mesonbuild/meson-python#524

Closed

Better discovery of macos system version to determine proper platform tag #314

Better discovery of macos system version to determine proper platform tag #314

Conversation

Czaki commented Sep 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 25, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 26, 2019 • edited Loading

Codecov Report

Czaki commented Sep 26, 2019

agronholm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agronholm commented Sep 26, 2019

Czaki commented Sep 26, 2019 • edited Loading

agronholm commented Sep 26, 2019

agronholm commented Sep 26, 2019

Czaki commented Sep 26, 2019

Czaki commented Sep 29, 2019

Czaki commented Oct 10, 2019

agronholm commented Oct 13, 2019

Czaki commented Oct 13, 2019

agronholm commented Oct 14, 2019

agronholm commented Oct 17, 2019

Czaki commented Oct 23, 2019

agronholm commented Oct 23, 2019

agronholm commented Oct 23, 2019

Czaki commented Oct 24, 2019

agronholm commented Oct 24, 2019

YannickJadoul commented Dec 26, 2019

agronholm commented Dec 26, 2019

YannickJadoul commented Dec 26, 2019

Czaki commented Dec 26, 2019

agronholm commented Dec 26, 2019

Czaki commented Dec 26, 2019

agronholm commented Dec 26, 2019

YannickJadoul commented Dec 26, 2019

Czaki commented Dec 27, 2019

YannickJadoul left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YannickJadoul Jan 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Czaki commented Sep 24, 2019 •

edited

Loading

codecov bot commented Sep 25, 2019 •

edited

Loading

codecov bot commented Sep 26, 2019 •

edited

Loading

Czaki commented Sep 26, 2019 •

edited

Loading

YannickJadoul left a comment •

edited

Loading

YannickJadoul Jan 3, 2020 •

edited

Loading