-
-
Notifications
You must be signed in to change notification settings - Fork 919
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix up checks in Makefile and make them portable #1661
Fix up checks in Makefile and make them portable #1661
Conversation
This fixes "fatal: ambiguous argument 'head'", which occurs on some systems, inclding GNU/Linux systems, with "git rev-parse head".
This sorts numerically for each of major, minor, and patch, rather than, e.g., rating 2.1.15 as a higher version than 2.1.2. It also rates things like X-beta and X-rc as lower versions than X, but X-patched (not SemVer, but present in this project) as higher versions than X.
The avoids showing the message when the build command was already run in a virtual environment. It also keeps the command failing, so the subsequent twine command is not attempted. (Just adding "|| echo ..." caused the command to succeed, because "echo ..." itself succeeds except in the rare case it cannot write to standard output.)
This changes the build command to run with "python" when in a virtual environment, since all virtual environments support this even when "python" outside it is absent or refers to the wrong version. On Windows, virtual environments don't contain a python3 command, but a global python3 command may be present, so the errors are confusing. This fixes that by avoiding such errors altogether.
This fixes how init-tests-after-clone.sh appears in .gitattributes so it gets LF (Unix-style) line endings on all systems as intended, and adds Makefile to be treated the same way.
As documented in the release instructions in README.md.
cc202cc put an end to the problem where, when run outside a virtual environment, it would always "succeed", because all "|| echo ..." required to succeed was for the echo command reporting the error to succeed. Unfortunately, that commit created the oppposite problem, causing that case to always fail! This commit fixes it, so it fails when there is an error, and succeeds when there is no error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot! I didn't even read all explanation yet as I have a single change request before doing so: Could you place the script section of the release
section into an actual shell script? If you see fit, it could take an argument that allows to force the release.
With that, it will be way easier to produce a script that is free of unnecessary noise. It's my recommendation to use set -eu -o pipefail
to be able to get rid of &&
as well.
Thank you.
As a follow up, I recommend setting up automated deploys to PyPI from GitHub Actions using the new "Trusted Publishers", to increase security and make releasing easier: |
This extracts the check logic from the release target in Makefile to a new script, check-version.sh. The code is also modified, mainly to account for different ways output is displayed and errors are reported and treated in a Makefile versus a standalone shell script. (The .sh suffix is for consistency with the naming of init-tests-after-clone.sh and is *not* intended to suggest sourcing the script; this script should be executed, not sourced.)
This moves the conditional build dependency installation logic and build logic from the force_release Makefile target to a shell script build-release.sh, which force_release calls. The code is changed to clean it up, and also to account for differences between how output is displayed and errors reported in Makefiles and shell scripts. (As in check-version.sh, the .sh suffix does not signify anything about how the script is to be used: like the other shell scripts in the project, this should be executed, no sourced.)
@Byron Good idea. That has a number of advantages, including that shell scripts in proper files can be checked for bugs using static analyzers. I have now made such a change, or something like it--this may not be quite what you want and it can definitely be changed further. I extracted everything in the In both scripts, I have gotten rid of most Because Because not all failures--especially with Maybe the scripts should be moved and/or renamed. I put them in the root of the repository, but it may be best to reduce clutter there. I named them with a |
This changes the hashbangs in Makefile helper scripts to be static. Often, "#!/usr/bin/env bash" is a better hashbang for bash scripts than "#!/bin/bash", because it automatically works on Unix-like systems that are not GNU/Linux and do not have bash in /bin, but on which it has been installed in another $PATH directory, such as /usr/local/bin. (It can also be helpful on macOS, where /bin/bash is usually an old version of bash, while a package manager such as brew may have been used to install a newer version elsewhere.) Windows systems with WSL installed often have a deprecated bash.exe in the System32 directory that runs commands and scripts inside an installed WSL system. (wsl.exe should be used instead.) Anytime that bash is used due to a "#!/usr/bin/env bash" hashbang, it is wrong, because that only happens if the caller is some Unix-style script running natively or otherwise outside WSL. Normally this is not a reason to prefer a "#!/bin/bash" hashbang, because normally any environment in which one can run a script in a way that determines its interpreter from its hashbang is an environment in which a native (or otherwise appropriate) bash precedes the System32 bash in a PATH search. However, MinGW make, a popular make implementation used on Windows, is an exception. The goal of this change is not to sacrifice support for some Unix-like systems to better support Windows, which wouldn't necessarily be justified. Rather, this is to avoid extremely confusing wrong behavior that in some cases would have bizarre effects that are very hard to detect. I discovered this problem because the VIRTUAL_ENV variable was not inheried by the bash interpreter (because it was, fortunately, not passed through to WSL). But if "python3 -m build" finds a global "build" package, things might get much further before any problem is noticed.
This also makes a correct but confusing comment clearer.
Like ".venv" and "venv", ".env" and "env" are common, plus "env" appears in the example command shown for making a virtual environment for the purpose of building a release, under some circumstances when "make release" or "make force_release" fail.
4b7d702
to
729372f
Compare
That way shell scripts will be handled correctly by default, anywhere.
- use `echo` where feasible to avoid explicit newlines - use `function` syntax out of habit - deduplicate release invocation - make `venv` based invocation less verbose - make help-text in non-venv more prominent
A huge thanks for this massive improvement! I learned a lot when studying the shell script - absolutely stunning what tricks As always, the attention to detail of this PR and the description are nothing a mere mortal could have produced, and your PRs inspire me to do better myself in the 'explain your reasoning' department, if not for the world, then for future me. I still have the problem that most of that information that I would leave in commit messages is lost the next time the file moves, as I have made a few adjustments in case you want to take a look. Something you might find interesting is the
I also thought that maybe there would be a better place for them in some directory, but then I could also appreciate that some automation is so easily discoverable. It's probably OK to leave them as is for now. As for the Thanks for this awesome PR, it makes making releases so much safer. And of course I will use |
@hugovk Thanks for the hint! I imagine that this could be a viable next step. The workflow could be that the release is prepared to the point where When thinking about it, at this point most of the work was already done and running However, I truly think this is the way forward once multiple people should be able to make releases without entrusting them with write access to the python packages on pypi. |
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [GitPython](https://togithub.com/gitpython-developers/GitPython) | `==3.1.36` -> `==3.1.37` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/GitPython/3.1.37?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/GitPython/3.1.37?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/GitPython/3.1.36/3.1.37?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/GitPython/3.1.36/3.1.37?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>gitpython-developers/GitPython (GitPython)</summary> ### [`v3.1.37`](https://togithub.com/gitpython-developers/GitPython/releases/tag/3.1.37): - a proper fix CVE-2023-41040 [Compare Source](https://togithub.com/gitpython-developers/GitPython/compare/3.1.36...3.1.37) #### What's Changed - Improve Python version and OS compatibility, fixing deprecations by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1654](https://togithub.com/gitpython-developers/GitPython/pull/1654) - Better document env_case test/fixture and cwd by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1657](https://togithub.com/gitpython-developers/GitPython/pull/1657) - Remove spurious executable permissions by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1658](https://togithub.com/gitpython-developers/GitPython/pull/1658) - Fix up checks in Makefile and make them portable by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1661](https://togithub.com/gitpython-developers/GitPython/pull/1661) - Fix URLs that were redirecting to another license by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1662](https://togithub.com/gitpython-developers/GitPython/pull/1662) - Assorted small fixes/improvements to root dir docs by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1663](https://togithub.com/gitpython-developers/GitPython/pull/1663) - Use venv instead of virtualenv in test_installation by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1664](https://togithub.com/gitpython-developers/GitPython/pull/1664) - Omit py_modules in setup by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1665](https://togithub.com/gitpython-developers/GitPython/pull/1665) - Don't track code coverage temporary files by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1666](https://togithub.com/gitpython-developers/GitPython/pull/1666) - Configure tox by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1667](https://togithub.com/gitpython-developers/GitPython/pull/1667) - Format tests with black and auto-exclude untracked paths by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1668](https://togithub.com/gitpython-developers/GitPython/pull/1668) - Upgrade and broaden flake8, fixing style problems and bugs by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1673](https://togithub.com/gitpython-developers/GitPython/pull/1673) - Fix rollback bug in SymbolicReference.set_reference by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1675](https://togithub.com/gitpython-developers/GitPython/pull/1675) - Remove `@NoEffect` annotations by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1677](https://togithub.com/gitpython-developers/GitPython/pull/1677) - Add more checks for the validity of refnames by [@​facutuesca](https://togithub.com/facutuesca) in [https://github.com/gitpython-developers/GitPython/pull/1672](https://togithub.com/gitpython-developers/GitPython/pull/1672) **Full Changelog**: gitpython-developers/GitPython@3.1.36...3.1.37 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/allenporter/flux-local). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi45Ny4xIiwidXBkYXRlZEluVmVyIjoiMzYuOTcuMSIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
In #1693 I took this further in what I would regard to be that direction, using two -echo $'\nThe VERSION must be the same in all locations, and so must the HEAD and tag SHA'
+echo
+echo 'The VERSION must be the same in all locations, and so must the HEAD and tag SHA' However, you might prefer the way with Another way to write this, which personally I do not think is as nice as either echo -e '\nThe VERSION must...' The reason I think the other approaches (including echo -e "2 loaves of bread\n3 eggs\n$var juice\na sponge" In contrast, POSIX permits For these and other reasons,
I think we could document the
This actually works remarkably better even than one might expect. The permissions system in Windows is very different from Unix-like systems, and nothing that really corresponds to Unix-style executable bits exists. But when one uses a Unix-style shell on Windows, such as the Git Bash shell, the available Unix-style utilities like (This is separate from WSL: a system in WSL fully supports Unix-style permissions, though like any such operating system, it will not support them when using a Windows filesystem. Windows filesystems being used in a Unix-like system are typically--both when accessing files outside WSL from inside it, and when mounting an NTFS drive on an independent Unix-like operating system--mounted in such a way as to treat every file as having executable bits set. Then, the output of commands like
Publishing with GitHub Actions could potentially replace some of the logic in these scripts. For example, a workflow could be set up that builds and published to PyPI when a release is made on GitHub, or when a push creates a tag, comes from a PR with a particular label, etc. (If one of those latter triggers is used, the workflow could even create the GitHub release as well, populating it with the default description or with a custom-generated one.) Here's an example of building and publishing to PyPI when a release is made on GitHub (though that project uses the dependency manager
I think this is strictly speaking true: any user who has the power to make arbitrary changes to the repository can use that power to cause a release to be published to PyPI, when trusted publishing is used. However, no PyPI secret has to be stored in the repository--changes are validated as having originated from the GitHub repository (and more specifically as from a CI job produced from a particular workflow whose file name has been put in via the PyPI web interface while setting it up), without involving separate stored credentials. |
Thanks for illuminating what happens to executable bits on Windows under various conditions - I see that it's definitely not portable. I think I am slowly starting to be less ignorant towards Windows, once again. Regarding publishing through GitHub actions, I see as main benefit that it would allow other maintainers to create valid publishes, and this is what I would want to use it for as well. |
When using Windows, I usually use PowerShell rather than a Unix-style shell (as my interactive shell), so
Yes, I agree. However, it may be that the process of building and publishing releases could be simplified in other ways, if and when this is done. |
Bump gitpython from 3.1.35 to 3.1.37 Bumps gitpython from 3.1.35 to 3.1.37. Release notes Sourced from gitpython's releases. 3.1.37 - a proper fix CVE-2023-41040 What's Changed Improve Python version and OS compatibility, fixing deprecations by @EliahKagan in gitpython-developers/GitPython#1654 Better document env_case test/fixture and cwd by @EliahKagan in gitpython-developers/GitPython#1657 Remove spurious executable permissions by @EliahKagan in gitpython-developers/GitPython#1658 Fix up checks in Makefile and make them portable by @EliahKagan in gitpython-developers/GitPython#1661 Fix URLs that were redirecting to another license by @EliahKagan in gitpython-developers/GitPython#1662 Assorted small fixes/improvements to root dir docs by @EliahKagan in gitpython-developers/GitPython#1663 Use venv instead of virtualenv in test_installation by @EliahKagan in gitpython-developers/GitPython#1664 Omit py_modules in setup by @EliahKagan in gitpython-developers/GitPython#1665 Don't track code coverage temporary files by @EliahKagan in gitpython-developers/GitPython#1666 Configure tox by @EliahKagan in gitpython-developers/GitPython#1667 Format tests with black and auto-exclude untracked paths by @EliahKagan in gitpython-developers/GitPython#1668 Upgrade and broaden flake8, fixing style problems and bugs by @EliahKagan in gitpython-developers/GitPython#1673 Fix rollback bug in SymbolicReference.set_reference by @EliahKagan in gitpython-developers/GitPython#1675 Remove @NoEffect annotations by @EliahKagan in gitpython-developers/GitPython#1677 Add more checks for the validity of refnames by @facutuesca in gitpython-developers/GitPython#1672 Full Changelog: gitpython-developers/[email protected] Commits b27a89f fix makefile to compare commit hashes only 0bd2890 prepare next release 832b6ee remove unnecessary list comprehension to fix CI e98f57b Merge pull request #1672 from trail-of-forks/robust-refname-checks 1774f1e Merge pull request #1677 from EliahKagan/no-noeffect a4701a0 Remove @NoEffect annotations d40320b Merge pull request #1675 from EliahKagan/rollback d1c1f31 Merge pull request #1673 from EliahKagan/flake8 e480985 Tweak rollback logic in log.to_file ff84b26 Refactor try-finally cleanup in git/ Additional commits viewable in compare view Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase. Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: @dependabot rebase will rebase this PR @dependabot recreate will recreate this PR, overwriting any edits that have been made to it @dependabot merge will merge this PR after your CI passes on it @dependabot squash and merge will squash and merge this PR after your CI passes on it @dependabot cancel merge will cancel a previously requested merge and block automerging @dependabot reopen will reopen this PR if it is closed @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page. Reviewed-by: Vladimir Vshivkov
Fixes #1660
In
.gitattributes
:init-tests-after-clone.sh
with LF line endings on all systems (this was broken).Makefile
to so it is also checked out with LF line endings on all systems.(Windows ports of
make
will usually tolerate CRLF line endings, but\r
or other representations of CR may, and often do, appear in output, giving an appearance of a problem that is hard to dismiss. So it's worth keeping it LF.)For the
force_release
target:python
, notpython3
, for compatibility with Windows.(Windows may have a
python3
command installed outside the virtual environment, which is the most confusing case, since it gets used even from inside, so it feels like installing packages in the venv had no effect. In contrast, some Debian and derived systems, and some macOS systems, have nopython
command outside a venv, or it's Python 2. So keep usingpython3
when no venv is detected.)For the
release
target:HEAD
rather thanhead
.HEAD
is more portable, becausegit
does not treat names case-insensitively by default on all systems.head
works on Windows and I think macOS, but not GNU/Linux or most other Unix-like systems.git tag --sort=-v:refname
with an appropriateversionsort.suffix
to order the version tags in the specific SemVer-inspired way this project uses, and a glob to omit the non-version tags. This avoids the correctness problem with the| sort -nr
approach, is portable, and is more precise for this project's versions than even nonportablesort -V
tweaked for SemVer, because this project has SemVer suffixes like-pre
whereN-pre
sorts beforeN
, but also non-SemVer-patched
whereN-patched
sorts afterN
.(Thanks to @hugovk for investigating what
sort -nr
was doing on macOS!)git
commands and pipelines.$(
)
fails (by extracting them to assignments), so that if agit
command whose output is needed fails, this doesn't cause a check to wrongly succeed.VERSION
andchanges.rst
, against each other and also the latest tag name.The above includes checks for indications that each of the actions listed in the
README.md
release instructions was done, and is intended to make it somake release
is correct, fully usable, and preferable tomake force_release
in most situations.I have tested this on Ubuntu and Windows. When testing, I pressed Ctrl+C when
twine
prompted for credentials. I tested both therelease
andforce_release
targets. Some of the manual tests of therelease
target on Ubuntu can be seen in this original gist from before that review, and this newer gist testingbuild-release.sh
. Although not shown in those gists, I have also tested that the Makefile targets use the scripts correctly since extracting commands to them.Here's an example of output from
make release
showing various checks and the new version/tag/ref table and failing with an error due to an inconsistency that should block the release:Here's an example of a failure where it doesn't get anywhere near that far: