Implement platform specific requirements.txt lock file support. #1401

jsirois · 2021-08-17T17:39:27Z

Pex should be able to generate a lock file for all the concrete Pythons and Platforms passed to a resolve in requirements.txt --hash format that can be used to create a new PEX with the same 3rdparty content via pex -r requirements.locked.txt requirements.pex on the same machine.

The text was updated successfully, but these errors were encountered:

Lock file generation will need to download and hash dists when their hashes are not presented in PEP 503 link fragments and we'll also need to download dists directly if we optimize the resolving from a lock file case when the lock file contains urls ala PEP-665. Work towards pex-tool#1401.

Lock file generation will need to download and hash dists when their hashes are not presented in PEP 503 link fragments and we'll also need to download dists directly if we optimize the resolving from a lock file case when the lock file contains urls ala PEP-665. Work towards #1401.

jsirois · 2021-08-21T23:55:13Z

Ok, non-strict (platform-applicable dists not actually resolved are included) mode platform specific resolve looks good to go. For the resolve below w/o constraints, pyyaml 5.4.1 is picked, but with:

$ cat constraints.txt 
pyyaml==5.4.0

And a find_links repo mixed in that has the wheels extracted from a prior pants.pex resolve, we get:

$ python -mpex -f find-links --constraints constraints.txt pantsbuild.pants -opants.pex
ansicolors==1.1.8 # ansicolors==1.1.8 via -> pantsbuild.pants \
    --hash:sha256=47926ab39ca3a88cc191bc07c50a63bab24f515f4ededdc9ceb46eec2602637a # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/ansicolors-1.1.8-py2.py3-none-any.whl \
    --hash:sha256=00d2dde5a675579325902536738dd27e4fac1fd68f773fe36c21044eb559e187 # https://files.pythonhosted.org/packages/53/18/a56e2fe47b259bb52201093a3a9d4a32014f9d85071ad07e9d60600890ca/ansicolors-1.1.8-py2.py3-none-any.whl \
    --hash:sha256=99f94f5e3348a0bcd43c82e5fc4414013ccc19d70bd939ad71e0133ce9c372e0 # https://files.pythonhosted.org/packages/76/31/7faed52088732704523c259e24c26ce6f2f33fbeff2ff59274560c27628e/ansicolors-1.1.8.zip
certifi==2021.5.30 # certifi>=2017.4.17 via -> requests via -> humbug==0.2.6 via -> pantsbuild.pants \
    --hash:sha256=24b049496cc826bc483682c45a59ea82b8832326fc35166048e3c6113ccc7551 # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/certifi-2021.5.30-py2.py3-none-any.whl \
    --hash:sha256=50b1e4f8446b06f41be7dd6338db18e0990601dce795c2b1686458aa7e8fa7d8 # https://files.pythonhosted.org/packages/05/1b/0a0dece0e8aa492a6ec9e4ad2fe366b511558cdc73fd3abc82ba7348e875/certifi-2021.5.30-py2.py3-none-any.whl \
    --hash:sha256=2bbf76fd432960138b3ef6dda3dde0544f27cbf8546c458e60baf371917ba9ee # https://files.pythonhosted.org/packages/6d/78/f8db8d57f520a54f0b8a438319c342c61c22759d8f9a1cd2e2180b5e5ea9/certifi-2021.5.30.tar.gz
charset-normalizer==2.0.4 # charset-normalizer~=2.0.0; python_version >= "3" via -> requests via -> humbug==0.2.6 via -> pantsbuild.pants \
    --hash:sha256=6d6460d4e270e59b1069e12f2222edb05501d1808512dc976fcbd1ed20b5024d # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/charset_normalizer-2.0.4-py3-none-any.whl \
    --hash:sha256=f23667ebe1084be45f6ae0538e4a5a865206544097e4e8bbcacf42cd02a348f3 # https://files.pythonhosted.org/packages/e7/4e/2af0238001648ded297fb54ceb425ca26faa15b341b4fac5371d3938666e/charset-normalizer-2.0.4.tar.gz \
    --hash:sha256=0c8911edd15d19223366a194a513099a302055a962bca2cec0f54b8b63175d8b # https://files.pythonhosted.org/packages/33/53/b7f6126a2b9fd878b025fe3c40266cfaad696f312165008ce045bffa3fe7/charset_normalizer-2.0.4-py3-none-any.whl
fasteners==0.16 # fasteners==0.16 via -> pantsbuild.pants \
    --hash:sha256=4360d005f68d6f52009f855d272f0ed7ef42ec3544d1085b1922a259bbc437ec # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/fasteners-0.16-py2.py3-none-any.whl \
    --hash:sha256=74b6847e0a6bb3b56c8511af8e24c40e4cf7a774dfff5b251c260ed338096a4b # https://files.pythonhosted.org/packages/78/20/c862d765287e9e8b29f826749ebae8775bdca50b2cb2ca079346d5fbfd76/fasteners-0.16-py2.py3-none-any.whl \
    --hash:sha256=c995d8c26b017c5d6a6de9ad29a0f9cdd57de61ae1113d28fac26622b06a0933 # https://files.pythonhosted.org/packages/d1/8f/a6c06f9bce5691a40283e52b92ec1522d6863951e738a31b109bf6bf2002/fasteners-0.16.tar.gz
humbug==0.2.6 # humbug==0.2.6 via -> pantsbuild.pants \
    --hash:sha256=df7ffdb44fef315297e1b95d24e61989e71b3dbf222ecfef250639da351b4282 # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/humbug-0.2.6-py3-none-any.whl \
    --hash:sha256=beae9dbd069ca04eb3fa9d85a535ed35195bfd86579024bf92151a74c7789579 # https://files.pythonhosted.org/packages/be/f7/c03c0ebd9bfcaaaf28786f6103615040f688848c406116c549eb01799668/humbug-0.2.6-py3-none-any.whl \
    --hash:sha256=f9d6b0d34177298a87464d797400083fc93eb6abe05a32e944218b92151c7914 # https://files.pythonhosted.org/packages/34/bb/e177459bbc2629b306b4759333b220e4624ef74a115350647c7167072e95/humbug-0.2.6.tar.gz
idna==3.2 # idna<4,>=2.5; python_version >= "3" via -> requests via -> humbug==0.2.6 via -> pantsbuild.pants \
    --hash:sha256=b5724057c178f5a44456333fc4116d7af309e906e6bcb8c8f65328cb0e2c24e4 # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/idna-3.2-py3-none-any.whl \
    --hash:sha256=14475042e284991034cb48e06f6851428fb14c4dc953acd9be9a5e95c7b6dd7a # https://files.pythonhosted.org/packages/d7/77/ff688d1504cdc4db2a938e2b7b9adee5dd52e34efbd2431051efc9984de9/idna-3.2-py3-none-any.whl \
    --hash:sha256=467fbad99067910785144ce333826c71fb0e63a425657295239737f7ecd125f3 # https://files.pythonhosted.org/packages/cb/38/4c4d00ddfa48abe616d7e572e02a04273603db446975ab46bbcd36552005/idna-3.2.tar.gz
packaging==20.9 # packaging==20.9 via -> pantsbuild.pants \
    --hash:sha256=f30dc989763ec00160e33f0d0d6366ef79f44018f52433b4ea401845226bd0cf # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/packaging-20.9-py2.py3-none-any.whl \
    --hash:sha256=67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a # https://files.pythonhosted.org/packages/3e/89/7ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af/packaging-20.9-py2.py3-none-any.whl \
    --hash:sha256=5b327ac1320dc863dca72f4514ecc086f31186744b84a230374cc1fd776feae5 # https://files.pythonhosted.org/packages/86/3c/bcd09ec5df7123abcf695009221a52f90438d877a2f1499453c6938f5728/packaging-20.9.tar.gz
pantsbuild-pants==2.6 # pantsbuild.pants \
    --hash:sha256=9d98372c4b02ded0c8b97587eeb15676e2abd59ff18d09af67e279c1ebd3a4bc # https://files.pythonhosted.org/packages/5d/c3/78a3aa1c161b123a2b9188459cfbd7e9076190df723293a882e68725f6b4/pantsbuild.pants-2.6.0-cp39-cp39-manylinux2014_x86_64.whl \
    --hash:sha256=a75773b51727f6538d6d8202755fd4cb1b26a4a94716a6ed2a10cfc411f47821 # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/pantsbuild.pants-2.6.0-cp39-cp39-linux_x86_64.whl
pex==2.1.42 # pex==2.1.42 via -> pantsbuild.pants \
    --hash:sha256=7c5a6cc086d89973b281827da2d2e0399de43187241c377ef7595517a7bf80c0 # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/pex-2.1.42-py2.py3-none-any.whl \
    --hash:sha256=f27acb6e9b9907a75344dddae3197887ef85d7983d5554601863cd68fd527ab9 # https://files.pythonhosted.org/packages/c1/05/f568a9937cda5af03f9d6930439e4dea59de3ae7b0b710ae7f72e014ea81/pex-2.1.42-py2.py3-none-any.whl \
    --hash:sha256=2deb088a3943891d07f9871e47409407e6308fbff3ee9514a0238791dc8da99f # https://files.pythonhosted.org/packages/26/d5/9ba5639365c8e5a400f5baa81fa2cc4e7546ea7b9d73741629b59f68814a/pex-2.1.42.tar.gz
psutil==5.8 # psutil==5.8.0 via -> pantsbuild.pants \
    --hash:sha256=23b872dbb41b7ba65a78950e40e3865d80590da3d6fd3230b26db3a427742e44 # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl \
    --hash:sha256=90d4091c2d30ddd0a03e0b97e6a33a48628469b99585e2ad6bf21f17423b112b # https://files.pythonhosted.org/packages/91/4d/033cc02ae3a47197d0ced818814e4bb8d9d29ebed4f1eb55badedec160f7/psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl \
    --hash:sha256=0c9ccb99ab76025f2f0bbecf341d4656e9c1351db8cc8a03ccd62e318ab4b5c6 # https://files.pythonhosted.org/packages/e1/b0/7276de53321c12981717490516b7e612364f2cb372ee8901bd4a66a000d7/psutil-5.8.0.tar.gz
pyparsing==2.4.7 # pyparsing>=2.0.2 via -> packaging==20.9 via -> pantsbuild.pants \
    --hash:sha256=faeb2cf4dabd0ce86dac3cd44b8f30b9a6913bbefbb0002046bc1a61242a26e4 # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/pyparsing-2.4.7-py2.py3-none-any.whl \
    --hash:sha256=ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b # https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl \
    --hash:sha256=c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1 # https://files.pythonhosted.org/packages/c1/47/dfc9c342c9842bbe0036c7f763d2d6686bcf5eb1808ba3e170afdb282210/pyparsing-2.4.7.tar.gz
pyyaml==5.4 # pyyaml==5.4.0 via -> -c constraints.txt (line 1) \
    --hash:sha256=02c78d77281d8f8d07a255e57abdbf43b02257f59f50cc6b636937d68efa5dd0 # https://files.pythonhosted.org/packages/43/e8/31007862b01580c507e24b88aeedb71bb81d6125a71c651a26370e6e0648/PyYAML-5.4-cp39-cp39-manylinux1_x86_64.whl \
    --hash:sha256=3c49e39ac034fd64fd576d63bb4db53cda89b362768a67f07749d55f128ac18a # https://files.pythonhosted.org/packages/b5/fd/15638de2da0a5aa91c095718444624aa565f766fc178249ca6faa372f71a/PyYAML-5.4.tar.gz
requests==2.26 # requests via -> humbug==0.2.6 via -> pantsbuild.pants \
    --hash:sha256=b397088800d78fe0b08371c946b1aaee50e59481da2593934b6115dc33c05afb # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/requests-2.26.0-py2.py3-none-any.whl \
    --hash:sha256=6c1246513ecd5ecd4528a0906f910e8f0f9c6b8ec72030dc9fd154dc1a6efd24 # https://files.pythonhosted.org/packages/92/96/144f70b972a9c0eabbd4391ef93ccd49d0f2747f4f6a2a2738e99e5adc65/requests-2.26.0-py2.py3-none-any.whl \
    --hash:sha256=b8aa58f8cf793ffd8782d3d8cb19e66ef36f7aba4353eec859e74678b01b07a7 # https://files.pythonhosted.org/packages/e7/01/3569e0b535fb2e4a6c384bdbed00c55b9d78b5084e0fb7f4d0bf523d7670/requests-2.26.0.tar.gz
setproctitle==1.2.2 # setproctitle==1.2.2 via -> pantsbuild.pants \
    --hash:sha256=df93e5e47ccfa1fac6b088d787aa1425d8885222af834127ba530ecbdb04f29c # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/setproctitle-1.2.2-cp39-cp39-manylinux1_x86_64.whl \
    --hash:sha256=970798d948f0c90a3eb0f8750f08cb215b89dcbee1b55ffb353ad62d9361daeb # https://files.pythonhosted.org/packages/5e/59/f9fef4d0682ff03a392365322d160d8ca5257a0a782b93cea7cb7658e53e/setproctitle-1.2.2-cp39-cp39-manylinux1_x86_64.whl \
    --hash:sha256=7dfb472c8852403d34007e01d6e3c68c57eb66433fb8a5c77b13b89a160d97df # https://files.pythonhosted.org/packages/a1/7f/a1d4f4c7b66f0fc02f35dc5c85f45a8b4e4a7988357a29e61c14e725ef86/setproctitle-1.2.2.tar.gz
setuptools==56.2 # setuptools<57.0,>=56.0.0 via -> pantsbuild.pants \
    --hash:sha256=d0ada9054591b78d23c26335f36d305f8b3f48213c9a6f353abe4044ad9fc5ba # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/setuptools-56.2.0-py3-none-any.whl \
    --hash:sha256=bc30153eec47d82f20c6f5e1a13d4ee725c6deb7013a67557f89bfe5d25235c4 # https://files.pythonhosted.org/packages/d0/15/5041473f5d142ee93bf1593deb8f932e27a078f6f04e2020cf44044f72c5/setuptools-56.2.0-py3-none-any.whl \
    --hash:sha256=7bb5652625e94e73b9358b7ed8c6431b732e80cf31f4e0972294c64f0e5b849e # https://files.pythonhosted.org/packages/fc/0a/b486efab52f8ad03c3eca0c998dd3deafba0c39b29e0c49c68a7152c8b2d/setuptools-56.2.0.tar.gz
six==1.16 # six via -> fasteners==0.16 via -> pantsbuild.pants \
    --hash:sha256=6232081f5f841e9328c0a3277d9263f4d232c21686fcc5d8564137469c52a025 # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/six-1.16.0-py2.py3-none-any.whl \
    --hash:sha256=8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254 # https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl \
    --hash:sha256=1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926 # https://files.pythonhosted.org/packages/71/39/171f1c67cd00715f190ba0b100d606d440a28c93c7714febeca8b79af85e/six-1.16.0.tar.gz
toml==0.10.2 # toml==0.10.2 via -> pantsbuild.pants \
    --hash:sha256=4414d506bb25b99bae5f8d454081681b557ae21adc464a672961efc589bd74de # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/toml-0.10.2-py2.py3-none-any.whl \
    --hash:sha256=806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b # https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl \
    --hash:sha256=b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f # https://files.pythonhosted.org/packages/be/ba/1f744cdc819428fc6b5084ec34d9b30660f6f9daaf70eead706e3203ec3c/toml-0.10.2.tar.gz
types-pyyaml==5.4.3 # types-PyYAML==5.4.3 via -> pantsbuild.pants \
    --hash:sha256=6067e642504dccbc0ff117ad3fcb0e05fd173dcde1dced1aa8da747ff2e4f612 # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/types_PyYAML-5.4.3-py2.py3-none-any.whl \
    --hash:sha256=2e7b81b2b7af751634425107b986086c6ba7cb61270a43a5c290c58be8cdbc3a # https://files.pythonhosted.org/packages/c1/6a/76f047f9289ed40b8a6460ae9900deec9aa85b9e1276da24112ce8fc9a7b/types-PyYAML-5.4.3.tar.gz \
    --hash:sha256=bca83cbfc0be48600a8abf1e3d87fb762a91e6d35d724029a3321dd2dce2ceb1 # https://files.pythonhosted.org/packages/b8/86/b957c28571005b1f13b92703400453c6a3a7fddd3f75c4c83bd2e7dc4d47/types_PyYAML-5.4.3-py2.py3-none-any.whl
types-setuptools==57 # types-setuptools==57.0.0 via -> pantsbuild.pants \
    --hash:sha256=6a84e526074436e8f4f8c2119447336d2dc173f7398cf3c9c6429ffb344f2a3a # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/types_setuptools-57.0.0-py3-none-any.whl \
    --hash:sha256=b3ada82b21dcb8e0cafd7658d8a16018a000e55bdb7f6f3cec033223360563ce # https://files.pythonhosted.org/packages/a4/ed/2d921fde85491f7c99fa9cd4a0ee79930e4e40ca45831181186377dbec24/types-setuptools-57.0.0.tar.gz \
    --hash:sha256=71ed0f4c71d8fb5f3026a90ae82d163c13749b110e157d82126725ac8f714360 # https://files.pythonhosted.org/packages/0d/14/3a722babb1c472f91b088c2622ba4e547fa668a58d892892278091390862/types_setuptools-57.0.0-py3-none-any.whl
types-toml==0.1.3 # types-toml==0.1.3 via -> pantsbuild.pants \
    --hash:sha256=25d50f2ace3b40fd30234c7b3940810818fc7b1f8535d376f9f83a297245a7c2 # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/types_toml-0.1.3-py2.py3-none-any.whl \
    --hash:sha256=33ebe67bebaec55a123ecbaa2bd98fe588335d8d8dda2c7ac53502ef5a81a79a # https://files.pythonhosted.org/packages/d8/77/561ad81396093af160995a8abd7125821f3fd738606ca4a8ff9b304c1c49/types-toml-0.1.3.tar.gz \
    --hash:sha256=d4add39a90993173d49ff0b069edd122c66ad4cf5c01082b590e380ca670ee1a # https://files.pythonhosted.org/packages/e8/9b/97adb9cf0b76b2f9f8a93d6f0e850bcfb1af1c358ec9f4188d3b83ea6755/types_toml-0.1.3-py2.py3-none-any.whl
typing-extensions==3.7.4.3 # typing-extensions==3.7.4.3 via -> pantsbuild.pants \
    --hash:sha256=bb517f22907e2209a7f0f3810ef590e0d105fed22b0ecc0dcf2888106549c818 # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/typing_extensions-3.7.4.3-py3-none-any.whl \
    --hash:sha256=7cb407020f00f7bfc3cb3e7881628838e69d8f3fcab2f64742a5e76b2f841918 # https://files.pythonhosted.org/packages/60/7a/e881b5abb54db0e6e671ab088d079c57ce54e8a01a3ca443f561ccadb37e/typing_extensions-3.7.4.3-py3-none-any.whl \
    --hash:sha256=99d4073b617d30288f569d3f13d2bd7548c3a7e4c8de87db09a9d29bb3a4a60c # https://files.pythonhosted.org/packages/16/06/0f7367eafb692f73158e5c5cbca1aec798cdf78be5167f6415dd4205fa32/typing_extensions-3.7.4.3.tar.gz
urllib3==1.26.6 # urllib3<1.27,>=1.21.1 via -> requests via -> humbug==0.2.6 via -> pantsbuild.pants \
    --hash:sha256=fb525207c1f8e3a9206a68e7a974b8e14f265a51376c7790c2df28f7994c44ed # file:///home/jsirois/dev/pantsbuild/jsirois-pex/find-links/urllib3-1.26.6-py2.py3-none-any.whl \
    --hash:sha256=39fb8672126159acb139a7718dd10806104dec1e2f0f6c88aab05d17df10c8d4 # https://files.pythonhosted.org/packages/5f/64/43575537846896abac0b15c3e5ac678d787a4021e906703f1766bfb8ea11/urllib3-1.26.6-py2.py3-none-any.whl \
    --hash:sha256=f57b4c16c62fa2760b7e3d97c35b255512fb6b59a259730f36ba32ce9f8e342f # https://files.pythonhosted.org/packages/4f/5a/597ef5911cb8919efe4d86206aa8b2658616d676a7088f0825ca08bd7cb8/urllib3-1.26.6.tar.gz

stuhood · 2021-08-23T17:40:12Z

Nice!

How does the via comment line work when a dep is depended on by multiple consumers?

jsirois · 2021-08-23T18:13:48Z

No clue. The dependency graph data, when Pex needs it for real (to implement PEP-665 for example), will come from walking the resolved dist metadata and not the Pip log scrape where this data comes from in the sample output above.

Kill ~unused resolve and rename resolve_multi to resolve. Eliminate ResolvedDistribution alias in favor of InstalledDistribution which better reflects the shap of the distribution (not a wheel, but an installed wheel). Fix test-use fallout. Work towards pex-tool#1401. Closes pex-tool#969

Kill ~unused resolve and rename resolve_multi to resolve. Eliminate ResolvedDistribution alias in favor of InstalledDistribution which better reflects the shap of the distribution (not a wheel, but an installed wheel). Fix test-use fallout. Work towards #1401. Closes #969

This allows collecting results from a Job's side-effects and will be used to convert resolve information from Pip verbose logs into lock file data. Work towards pex-tool#1401.

Previously only erroring jobs partook in log analysis. To support lock file generation, we'll need to support analysis for successful downloads. Works towards pex-tool#1401.

This allows collecting results from a Job's side-effects and will be used to convert resolve information from Pip verbose logs into lock file data. Work towards #1401.

Previously only erroring jobs partook in log analysis. To support lock file generation, we'll need to support analysis for successful downloads. Works towards #1401.

For now this data is only used in test_locked_resolve, but it forms the basis for adding lock generation support to Pex. This is the core of the work for pex-tool#1401. The remaining work is wiring a CLI option to generate a lock file (needs to be able to store multiple platform specific locks since a single resolve can generate many) and consume the lock file.

For now this data is only used in test_locked_resolve, but it forms the basis for adding lock generation support to Pex. This is the core of the work for #1401. The remaining work is wiring a CLI option to generate a lock file (needs to be able to store multiple platform specific locks since a single resolve can generate many) and consume the lock file.

This lifts Pex global configuration in pex.commands for re-use in the `pex3` CLI. Work towards #1401.

This provides a basic platform specific lock generation facility. For now, the lock is only useful when a single platform or interpreter is selected by the target options. A follow-up will introduce a lock format that can store multiple locks allowing use and export of these as individual units. Work towards pex-tool#1401.

This provides a basic platform specific lock generation facility. For now, the lock is only useful when a single platform or interpreter is selected by the target options. A follow-up will introduce a lock format that can store multiple locks allowing use and export of these as individual units. Work towards #1401.

This moves `pex.locked_resolve` to the `pex.resolve package` and also consolidates requirements and constraints parsing to `pex.resolver.requirement_configuration` for re-use by upcoming `pex3 lock {create,export}` work. Work towards pex-tool#1401 and pex-tool#1414.

This moves `pex.locked_resolve` to the `pex.resolve package` and also consolidates requirements and constraints parsing to `pex.resolver.requirement_configuration` for re-use by upcoming `pex3 lock {create,export}` work. Work towards #1401 and #1414.

These will be needed for `pex3 lock {create,export}` testing. Work towards pex-tool#1401 and pex-tool#1414.

These will be needed for `pex3 lock {create,export}` testing. Also address some PEX `.bootstrap/` sloppiness and elide testing support files that live in the production tree only for importability by many tests. Work towards pex-tool#1401 and pex-tool#1414.

These will be needed for `pex3 lock {create,export}` testing. Also address some PEX `.bootstrap/` sloppiness and elide testing support files that live in the production tree only for importability by many tests. Work towards #1401 and #1414.

This will be used for structures that must be sorted to be correct in Pex lockfile work. Work towards pex-tool#1401 and pex-tool#1414.

This brings in support for custom field sort function which will be helpful for lockfile work. Work towards #1401 and #1414.

This will be used for structures that must be sorted to be correct in Pex lockfile work. Work towards #1401 and #1414.

This will be used for error messages when introducing sortability to `Enum` and in lock file parsing. Work towards pex-tool#1401 and pex-tool#1414.

Upcoming lock file work will benefit from this. Instead of just sorting by the string value of an `Enum.Value`, introduce an ordinal to sort by and take the opportunity to dry up the values method. Work towards pex-tool#1401 and pex-tool#1414.

This will be used for error messages when introducing sortability to `Enum` and in lock file parsing. Work towards #1401 and #1414.

Upcoming lock file work will benefit from this. Instead of just sorting by the string value of an `Enum.Value`, introduce an ordinal to sort by and take the opportunity to dry up the values method. Work towards #1401 and #1414.

This switches `lock create` to store lock files in a Pex-specific format that retains all information needed to later use portions of the lock. The `export` subcommand then uses this facility to select an appropriate lock and then translate it to Pip requirements format for export. Closes pex-tool#1401 Closes pex-tool#1414

This switches `lock create` to store lock files in a Pex-specific format that retains all information needed to later use portions of the lock. The `export` subcommand then uses this facility to select an appropriate lock and then translate it to Pip requirements format for export. Closes #1401 Closes #1414

This was referenced Aug 17, 2021

Implement platform agnostic requirements.txt lock file support. #1402

Closed

Implement lock file support (Umbrella). #1400

Closed

jsirois added feature request resolver labels Aug 17, 2021

jsirois mentioned this issue Aug 17, 2021

Consider implementing PEP-665 lock file format support. #1403

Closed

jsirois self-assigned this Aug 17, 2021

jsirois added the in progress label Aug 17, 2021

jsirois mentioned this issue Aug 19, 2021

Extract URLFetcher for reuse with lock files. #1406

Merged

This was referenced Aug 22, 2021

Support lock file updates #1412

Closed

Canonicalize file:// URLs in locks #1413

Closed

Support selecting the appropriate lock for a resolve. #1414

Closed

jsirois mentioned this issue Aug 23, 2021

Clean up resolver internal APIs. #1420

Merged

jsirois added a commit to jsirois/pex that referenced this issue Sep 1, 2021

Add SpawnedJob.and_then API.

4fcfd76

This allows collecting results from a Job's side-effects and will be used to convert resolve information from Pip verbose logs into lock file data. Work towards pex-tool#1401.

jsirois mentioned this issue Sep 1, 2021

Add SpawnedJob.and_then API. #1435

Merged

jsirois mentioned this issue Sep 1, 2021

Allow Pip log analysis for successful jobs. #1436

Merged

jsirois added a commit that referenced this issue Sep 1, 2021

Add SpawnedJob.and_then API. (#1435)

af39f1d

This allows collecting results from a Job's side-effects and will be used to convert resolve information from Pip verbose logs into lock file data. Work towards #1401.

jsirois added a commit that referenced this issue Sep 1, 2021

Allow Pip log analysis for successful jobs. (#1436)

19c797f

Previously only erroring jobs partook in log analysis. To support lock file generation, we'll need to support analysis for successful downloads. Works towards #1401.

jsirois mentioned this issue Sep 1, 2021

Enrich resolve and download with lock data. #1437

Merged

jsirois added a commit that referenced this issue Sep 21, 2021

Extract global configuration. (#1463)

0ab02be

This lifts Pex global configuration in pex.commands for re-use in the `pex3` CLI. Work towards #1401.

jsirois mentioned this issue Sep 21, 2021

Introduce the pex3 lock create command. #1464

Merged

jsirois mentioned this issue Sep 23, 2021

Refactor resolve helpers. #1465

Merged

jsirois added a commit to jsirois/pex that referenced this issue Sep 23, 2021

Extract LockedResolve test helpers.

17fcb35

These will be needed for `pex3 lock {create,export}` testing. Work towards pex-tool#1401 and pex-tool#1414.

jsirois mentioned this issue Sep 23, 2021

Extract LockedResolve test helpers. #1466

Merged

jsirois mentioned this issue Sep 25, 2021

Upgrade attrs. #1467

Merged

jsirois added a commit to jsirois/pex that referenced this issue Sep 25, 2021

Introduce SortedTuple.

329c0bf

This will be used for structures that must be sorted to be correct in Pex lockfile work. Work towards pex-tool#1401 and pex-tool#1414.

jsirois mentioned this issue Sep 25, 2021

Introduce SortedTuple. #1468

Merged

jsirois added a commit that referenced this issue Sep 25, 2021

Upgrade attrs. (#1467)

0a173b8

This brings in support for custom field sort function which will be helpful for lockfile work. Work towards #1401 and #1414.

jsirois added a commit that referenced this issue Sep 25, 2021

Introduce SortedTuple. (#1468)

85a4a7d

This will be used for structures that must be sorted to be correct in Pex lockfile work. Work towards #1401 and #1414.

jsirois added a commit to jsirois/pex that referenced this issue Sep 25, 2021

Introduce a qualified_name utility.

57645f9

This will be used for error messages when introducing sortability to `Enum` and in lock file parsing. Work towards pex-tool#1401 and pex-tool#1414.

jsirois mentioned this issue Sep 25, 2021

Introduce a qualified_name utility. #1469

Merged

jsirois mentioned this issue Sep 25, 2021

Make Enums sortable. #1470

Merged

jsirois added a commit that referenced this issue Sep 25, 2021

Introduce a qualified_name utility. (#1469)

2cf1ec6

This will be used for error messages when introducing sortability to `Enum` and in lock file parsing. Work towards #1401 and #1414.

jsirois mentioned this issue Sep 26, 2021

Introduce the pex3 lock export command. #1471

Merged

jsirois closed this as completed in #1471 Sep 28, 2021

jsirois removed the in progress label Sep 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement platform specific requirements.txt lock file support. #1401

Implement platform specific requirements.txt lock file support. #1401

jsirois commented Aug 17, 2021 •

edited

Loading

jsirois commented Aug 21, 2021 •

edited

Loading

stuhood commented Aug 23, 2021

jsirois commented Aug 23, 2021 •

edited

Loading

Implement platform specific requirements.txt lock file support. #1401

Implement platform specific requirements.txt lock file support. #1401

Comments

jsirois commented Aug 17, 2021 • edited Loading

jsirois commented Aug 21, 2021 • edited Loading

stuhood commented Aug 23, 2021

jsirois commented Aug 23, 2021 • edited Loading

jsirois commented Aug 17, 2021 •

edited

Loading

jsirois commented Aug 21, 2021 •

edited

Loading

jsirois commented Aug 23, 2021 •

edited

Loading