-
-
Notifications
You must be signed in to change notification settings - Fork 908
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix Git.execute shell use and reporting bugs #1687
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
That test is not testing whether or not a shell is used (nor does it need to test that), but just whether the library actually runs git, passes an argument to it, and returns text from its stdout.
In the Popen calls in Git.execute, for all combinations of allowed values for Git.USE_SHELL and the shell= keyword argument.
The log message shows "Popen(...)", not "execute(...)". So it should faithfully report what is about to be passed to Popen in cases where a user reading the log would otherwise be misled into thinking this is what has happened. Reporting the actual "shell=" argument passed to Popen is also more useful to log than the argument passed to Git.execute (at least if only one of them is to be logged).
Both new shell-related tests were causing the code under test to split "git" into ["g", "i", "t"] and thus causing the code under test to attempt to execute a "g" command. This passes the command as a one-element list of strings rather than as a string, which avoids this on all operating systems regardless of whether the code under test has a bug being tested for. This would not occur on Windows, which does not iterate commands of type str character-by-character even when the command is run without a shell. But it did happen on other systems. Most of the time, the benefit of using a command that actually runs "git" rather than "g" is the avoidance of confusion in the error message. But this is also important because it is possible for the user who runs the tests to have a "g" command, in which case it could be very inconvenient, or even unsafe, to run "g". This should be avoided even when the code under test has a bug that causes a shell to be used when it shouldn't or not used when it should, so having separate commands (list and str) per test case parameters would not be a sufficient solution: it would only guard against running "g" when a bug in the code under test were absent.
This also helps mock Popen over a smaller scope, which may be beneficial (especially if it is mocked in the subprocess module, rather than the git.cmd module, in the future).
Because mock.call.kwargs, i.e. the ability to examine m.call_args.kwargs where m is a Mock or MagicMock, was introduced in Python 3.8. Currently it is only in test/test_git.py that any use of mocks requires this, so I've put the conditional import logic to import mock (the top-level package) rather than unittest.mock only there. The mock library is added as a development (testing) dependency only when the Python version is lower than 3.8, so it is not installed when not needed. This fixes a problem in the new tests of whether a shell is used, and reported as used, in the Popen call in Git.execute. Those just-introduced tests need this, to be able to use mock_popen.call_args.kwargs on Python 3.7.
This updates the shell variable itself, only when it is None, from self.USE_SHELL. (That attribute is usually Git.USE_SHELL rather than an instance attribute. But although people probably shouldn't set it on instances, people will expect that this is permitted.) Now: - USE_SHELL is used as a fallback only. When shell=False is passed, USE_SHELL is no longer consuled. Thus shell=False always keeps a shell from being used, even in the non-default case where the USE_SHELL attribue has been set to True. - The debug message printed to the log shows the actual value that is being passed to Popen, because the updated shell variable is used both to produce that message and in the Popen call.
This was referenced Oct 2, 2023
Byron
approved these changes
Oct 2, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks so much for the fix which brings the behaviour in line with the documentation, and for adding regression protection.
This PR is also a wonderful example on what's possible with the unittest
and mock
libraries and generally what's possible with python testing.
🙏
renovate bot
referenced
this pull request
in allenporter/flux-local
Oct 20, 2023
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [GitPython](https://togithub.com/gitpython-developers/GitPython) | `==3.1.37` -> `==3.1.40` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/GitPython/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/GitPython/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/GitPython/3.1.37/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/GitPython/3.1.37/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>gitpython-developers/GitPython (GitPython)</summary> ### [`v3.1.40`](https://togithub.com/gitpython-developers/GitPython/compare/3.1.38...3.1.40) [Compare Source](https://togithub.com/gitpython-developers/GitPython/compare/3.1.38...3.1.40) ### [`v3.1.38`](https://togithub.com/gitpython-developers/GitPython/releases/tag/3.1.38) [Compare Source](https://togithub.com/gitpython-developers/GitPython/compare/3.1.37...3.1.38) #### What's Changed - Add missing assert keywords by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1678](https://togithub.com/gitpython-developers/GitPython/pull/1678) - Make clear every test's status in every CI run by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1679](https://togithub.com/gitpython-developers/GitPython/pull/1679) - Fix new link to license in readme by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1680](https://togithub.com/gitpython-developers/GitPython/pull/1680) - Drop unneeded flake8 suppressions by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1681](https://togithub.com/gitpython-developers/GitPython/pull/1681) - Update instructions and test helpers for git-daemon by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1684](https://togithub.com/gitpython-developers/GitPython/pull/1684) - Fix Git.execute shell use and reporting bugs by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1687](https://togithub.com/gitpython-developers/GitPython/pull/1687) - No longer allow CI to select a prerelease for 3.12 by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1689](https://togithub.com/gitpython-developers/GitPython/pull/1689) - Clarify Git.execute and Popen arguments by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1688](https://togithub.com/gitpython-developers/GitPython/pull/1688) - Ask git where its daemon is and use that by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1697](https://togithub.com/gitpython-developers/GitPython/pull/1697) - Fix bugs affecting exception wrapping in rmtree callback by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1700](https://togithub.com/gitpython-developers/GitPython/pull/1700) - Fix dynamically-set **all** variable by [@​DeflateAwning](https://togithub.com/DeflateAwning) in [https://github.com/gitpython-developers/GitPython/pull/1659](https://togithub.com/gitpython-developers/GitPython/pull/1659) - Fix small [#​1662](https://togithub.com/gitpython-developers/GitPython/issues/1662) regression due to [#​1659](https://togithub.com/gitpython-developers/GitPython/issues/1659) by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1701](https://togithub.com/gitpython-developers/GitPython/pull/1701) - Drop obsolete info on yanking from security policy by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1703](https://togithub.com/gitpython-developers/GitPython/pull/1703) - Have Dependabot offer submodule updates by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1702](https://togithub.com/gitpython-developers/GitPython/pull/1702) - Bump git/ext/gitdb from `49c3178` to `8ec2390` by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/gitpython-developers/GitPython/pull/1704](https://togithub.com/gitpython-developers/GitPython/pull/1704) - Bump git/ext/gitdb from `8ec2390` to `6a22706` by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/gitpython-developers/GitPython/pull/1705](https://togithub.com/gitpython-developers/GitPython/pull/1705) - Update readme for milestone-less releasing by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1707](https://togithub.com/gitpython-developers/GitPython/pull/1707) - Run Cygwin CI workflow commands in login shells by [@​EliahKagan](https://togithub.com/EliahKagan) in [https://github.com/gitpython-developers/GitPython/pull/1709](https://togithub.com/gitpython-developers/GitPython/pull/1709) #### New Contributors - [@​DeflateAwning](https://togithub.com/DeflateAwning) made their first contribution in [https://github.com/gitpython-developers/GitPython/pull/1659](https://togithub.com/gitpython-developers/GitPython/pull/1659) **Full Changelog**: gitpython-developers/GitPython@3.1.37...3.1.38 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/allenporter/flux-local). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xOS4yIiwidXBkYXRlZEluVmVyIjoiMzcuMTkuMiIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1685
Fixes #1686
Changes to
git
module codeThis updates the
shell
variable itself, only when it isNone
, fromself.USE_SHELL
.(That attribute usually gets its value from
Git.USE_SHELL
rather than being separately an instance attribute. Butself.USE_SHELL
is an idiomatic way to access it. Furthermore, although people probably shouldn't set it on instances, people may expect that this is permitted.)Now:
USE_SHELL
is used as a fallback only. Whenshell=False
is passed,USE_SHELL
is no longer consulted. Thusshell=False
always keeps a shell from being used, even in the non-default case where theUSE_SHELL
attribute has been set toTrue
.Popen
, because the updated shell variable is used both to produce that message and in thePopen
call.New tests
This also (actually, first) adds tests, some to check that the correct value is always passed as
shell=
whenGit.execute
callsPopen
, and others to check that whatever it passes is the same as it claims to be passing in the log. I've made them two separate test methods rather than a test method with multiple assertions, because I think the results of these particular tests are easier to understand that way (even compared to the option of usingself.subTest
). But the major shared logic is extracted to a helper function, where its important subtleties are commented. I have verified that the tests fail, in the expected way, before the bugs are fixed, and then pass, and that these are both the case even on native Windows where we don't (yet) have CI jobs.The tests cover the six combinations of valid values of the
shell=
argument toGit.execute
(None
,False
, andTrue
) andGit.USE_SHELL
(False
andTrue
). I usedddt
for this, so there are only two test methods, even though there are six test cases. The tests are simple in principle, with the main source of complexity being the question of how to deal with how the command argument toPopen
typically differs, including by type, based on theshell
argument, yet the tests must not misbehave, nor induce extraneous misbehavior, even when the code under test has a bug that would ordinarily prevent such agreement.On Python 3.7 only, this adds
mock
as a development dependency (in thetest
extra, not as a regular dependency).mock
is a backport ofunittest.mock
, and a couple of my assertions use a handy feature that was introduced to the mock library in Python 3.8. Besides being harder to write without it, I believe it would be less clearly expressed. (Details are commented with the import logic.)While adding the new tests, I also renamed
test_it_executes_git_to_shell_and_returns_result
totest_it_executes_git_and_returns_result
, because that test case does not test that a shell is used to rungit
(and a shell is not used in that test).