colocate __main__.py with the central directory record of the zipapp

[cosmicexplorer]

When the Python interpreter executes a zipapp, it will import __main__.py, while __pex__/__init__.py will be loaded if the pex file is on the PYTHONPATH. This file is loaded from the zip file handle with zipimport, and the PEX bootstrap script at that location will then import some of the code in .bootstrap/ before the zipapp is unpacked and compiled. Because a zip file's central directory records are located at the end of the file, placing this code at the end of the zip file list will reduce the amount of seeking the Python interpreter needs to perform in order to execute the bootstrap script. This was previously discussed in #2175 (comment) and #2175 (comment).

Changes involved:

1. Make Chroot#zip() respect the relative ordering of labels when collecting from self.filesets.
2. Put "main" and "importhook" labels at the end of the zipapp.

[jsirois]

Ok, this seems fine - but boy oh boy does it seem incredibly complicated vs just ordering the labels correctly in the existing tuple on the LHS and writing a good block comment just above.

[cosmicexplorer]

You're right, and this change could be reduced quite a bit by providing the labels literally (for example, we could avoid changing this method at all). I thought it was useful to document the meanings of each label, since they're not really documented elsewhere unless you grep for the literal strings "executable", "importhook", etc. I was also thinking that naming each section with a classmethod would make it easier for someone to subclass PEXBuilder and insert their own files into the appropriate section while retaining the performance benefit for zipapps. If that's not a supported use case, we could very well reduce the size of this change by a lot.

[jsirois]

@cosmicexplorer I'm a huge non-fan of Problem / Solution formulas for writing commit messages. We're adults and I think you can just write a good message. Quoting people - not helpful. Just explaining the change and its reason should stand on its own.

[cosmicexplorer]

I'm a huge non-fan of Problem / Solution formulas for writing commit messages.

Ok, I will keep that in mind for the future.

Quoting people - not helpful. Just explaining the change and its reason should stand on its own.

In this case it's not actually documented anywhere else that the python interpreter accesses __main__.py or __pex__/__init__.py from the same zip file handle (which I confirmed by looking through the CPython source), so I thought the quote was precisely the rationale needed for this change. However, given this feedback, I have rewritten the OP to be much more concise while simply linking to instead of directly quoting the prior discussions. Is that closer to what you're looking for?

[jsirois]

Thanks @cosmicexplorer, yes. The facts remain the same regardless of who pointed them out. Much more sane to read in the git log.

FWIW:

When the Python interpreter executes a zipapp, it will import main.py or pex/init.py

Is not true. When the Python interpreter executes a zipapp (or a directory) it only looks for __main__.py. The __pex__ package must be imported by the user, Python knows nothing about that at all.

[cosmicexplorer]

You're right; I have fixed the wording to clarify that __pex__/__init__.py is loaded if the pex file is on the PYTHONPATH.