Type Hints #1095

mdantonio · 2020-12-21T07:43:06Z

Hello,

I was looking for any type hints for GitPython, but I found nothing neither in the code nor in independent stub repositories.

Is there any plan to introduce type hinting?

Byron · 2020-12-22T04:50:24Z

I am absolutely open to receiving PRs adding type hints, and GitPython would be happy to host them directly in source.

Any amount of types would help and is appreciated, thus there is no need to add type hints everywhere at once.

Yobmod · 2021-02-18T20:05:10Z

I looked at starting to contribute type hints, but wanted to ask would it be feasible to drop support for python 3.4 (EOL 2 years ago)?

With 3.5+ type hints can be added directly to the code, but for 3.4 they would need to be in .pyi files, doubling the number of files and requiring they are kept in sync (AFAIK, 3.4 will give errors rather than ignore most inline type hints even though annotation syntax is possible)

Byron · 2021-02-19T15:26:13Z

@Yobmod Thanks for bringing this to my attention, and I would be glad to see type hints in code. Please feel free to submit a draft PR that starts out by dropping PY3.4 support to allow adding the first type hints in code. That one wouldn't have to be complete at all, but could lay the foundation for more PRs adding types in future.

Yobmod · 2021-02-24T17:21:53Z

I've put in a PR to drop python 3.4.
Planning to use Monkeytype to add initial types to the most user-facing files and see how that goes.

Yobmod · 2021-02-27T14:17:11Z

mdantonio · 2021-02-28T08:19:19Z

Thank for all the effort towards type hints! I am sorry I have not been able to contribute so far, I hope to be able to put some effort after the great initial start of @Yobmod

Yobmod · 2021-02-28T18:59:44Z

Thank for all the effort towards type hints! I am sorry I have not been able to contribute so far, I hope to be able to put some effort after the great initial start of @Yobmod

Thanks, Any help is always good :)

Branch for adding types is here: https://github.com/Yobmod/GitPython/tree/addtypes

Mypy-stubgen just typed everythin as Any, and Monkeytype has not been as useful as i hoped. If any wants to run it, commands are:

monkeytype run test/tstrunner.py
monkeytype list > mtype_filelist.txt
monkeytype stub git.cmd > git/cmd.pyi

But i'm mostly doing them by hand.

kasium · 2021-06-30T11:28:50Z

Hey, it seems that most of the code is typed, right? Would it be possible to include a "py.typed" file into the repo so that mypy can pick the types up?

Yobmod · 2021-06-30T12:16:13Z

Hey, it seems that most of the code is typed, right? Would it be possible to include a "py.typed" file into the repo so that mypy can pick the types up?

Hi, there already is one. Are you on the newest version?
Annotations showing up for me.

kasium · 2021-06-30T12:22:18Z

Yes I use the latest version. The py.typed file is not present in the tar.gz nor in the wheel - maybe that's the issue.
Mypy (also latest version) complains that no stub could be found

Yobmod · 2021-06-30T16:27:59Z

I've read up on some setuptools options, it seems that the package_data and include_package_data arguments are mutually exclusive. So setup.py is picking up the package_data only from the MANIFEST.INI and not from within the setup.include_package_data list.

I'll add py.typed to the manifest and hopefully that fixes it.

kasium · 2021-06-30T18:04:51Z

Great. thanks a lot

Yobmod · 2021-07-09T22:54:23Z

I've installed from git and it puts the py.typed where it should be now.
If anyone wants the typehints before the next release, can do:

pip install git+https://github.com/gitpython-developers/GitPython.git -U

kasium · 2021-07-21T05:52:34Z

@Yobmod thanks for the work here. Are there any plans when 3.1.19 will be released?

Byron · 2021-07-21T06:02:59Z

@kasium Judging from the checklist further up all work seems to be done waiting for everything to be released. @Yobmod - do you think a release is in order?

Yobmod · 2021-07-21T10:53:30Z

Yep, we could do a release now.
I avoided any api changes, but lots of small runtime changes (None checks, isinstance(), etc), so would be good to find out if they trip up on any edge cases, which I think only a release can do.

The next step for typing is to check the runtime types to fix the remaining unknowns. Unfortunately I cant find a library to do it (Typeguard etc. will report if using the wrong type, but not if the type is too broad), so it will take a while.

Byron · 2021-07-23T00:13:52Z

And here is the new release.

Fingers crossed!

adeandrade · 2021-07-23T03:45:03Z

    import git
  File "/opt/python/lib/python3.9/site-packages/git/__init__.py", line 8, in <module>
    from git.exc import *                       # @NoMove @IgnorePep8
  File "/opt/python/lib/python3.9/site-packages/git/exc.py", line 10, in <module>
    from git.compat import safe_decode
  File "/opt/python/lib/python3.9/site-packages/git/compat.py", line 32, in <module>
    from git.types import TBD
  File "/opt/python/lib/python3.9/site-packages/git/types.py", line 18, in <module>
    from typing_extensions import TypeGuard  # noqa: F401
ImportError: cannot import name 'TypeGuard' from 'typing_extensions' (/opt/python/lib/python3.9/site-packages/typing_extensions.py)

@Byron Just got this with 3.1.19 and Python 3.9.6. Maybe we should yank it?

Yobmod · 2021-07-23T09:16:56Z

It looks like typing_extensions has to be updated (again) to 3.10.0.0. That fixed it on my machine, so I've push the commit for requirements.txt. Typehints still seem to be changing every few months, even after 7 years! Dominic

…

On Fri, 23 Jul 2021, 04:45 Anderson de Andrade, ***@***.***> wrote: import git File "/opt/python/lib/python3.9/site-packages/git/__init__.py", line 8, in <module> from git.exc import * # @nomove @IgnorePep8 File "/opt/python/lib/python3.9/site-packages/git/exc.py", line 10, in <module> from git.compat import safe_decode File "/opt/python/lib/python3.9/site-packages/git/compat.py", line 32, in <module> from git.types import TBD File "/opt/python/lib/python3.9/site-packages/git/types.py", line 18, in <module> from typing_extensions import TypeGuard # noqa: F401 ImportError: cannot import name 'TypeGuard' from 'typing_extensions' (/opt/python/lib/python3.9/site-packages/typing_extensions.py) @Byron <https://github.com/Byron> Just got this with 3.1.19 and Python 3.9.6. Maybe we should yank it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1095 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFJUSQ3XHNB6XABCFOWJSUDTZDQUVANCNFSM4VD23UVQ> .

mdantonio · 2021-07-23T10:24:41Z

Hello,
installed v3.1.19 (and upgraded typing_extensions to 3.10.0.0) but type hints are still ignored by mypy because py.typed seems to be yet not included in the release

Here the content of the installation folder:

$ ls /usr/local/lib/python3.9/dist-packages/git/
cmd.py  compat.py  config.py  db.py  diff.py  exc.py  index  __init__.py  objects  __pycache__  refs  remote.py  repo  types.py  util.py

$ grep "3.1.19" /usr/local/lib/python3.9/dist-packages/git/*
grep: /usr/local/lib/python3.9/dist-packages/git/index: Is a directory
/usr/local/lib/python3.9/dist-packages/git/__init__.py:__version__ = '3.1.19'

type hints started to work after a touch /usr/local/lib/python3.9/dist-packages/git/py.typed

Byron · 2021-07-23T12:00:20Z

Interesting this wasn’t caught by CI. Yanked the release.

…

Sent from my iPhone

On Jul 23, 2021, at 5:17 PM, Dominic ***@***.***> wrote: It looks like typing_extensions has to be updated (again) to 3.10.0.0. That fixed it on my machine, so I've push the commit for requirements.txt. Typehints still seem to be changing every few months, even after 7 years! Dominic On Fri, 23 Jul 2021, 04:45 Anderson de Andrade, ***@***.***> wrote: > import git > File "/opt/python/lib/python3.9/site-packages/git/__init__.py", line 8, in <module> > from git.exc import * # @nomove @IgnorePep8 > File "/opt/python/lib/python3.9/site-packages/git/exc.py", line 10, in <module> > from git.compat import safe_decode > File "/opt/python/lib/python3.9/site-packages/git/compat.py", line 32, in <module> > from git.types import TBD > File "/opt/python/lib/python3.9/site-packages/git/types.py", line 18, in <module> > from typing_extensions import TypeGuard # noqa: F401 > ImportError: cannot import name 'TypeGuard' from 'typing_extensions' (/opt/python/lib/python3.9/site-packages/typing_extensions.py) > > @Byron <https://github.com/Byron> Just got this with 3.1.19 and Python > 3.9.6. Maybe we should yank it? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#1095 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFJUSQ3XHNB6XABCFOWJSUDTZDQUVANCNFSM4VD23UVQ> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Yobmod · 2021-07-23T12:02:37Z

😭
I've looked at loads of example libraries and instructions, and they're all different.

Encode/starlette have py.typed listed in setup.py package_data, but not in manifest.in, and that works.
That what mypy instructions say but it didn't work last release.

Encode/httpx (issue linked above) had same issue, so added py.typed to manifest.in.
So we did that, and it worked for installs from github, but it seems not from pypi.

I've now added it to both setup.py and manifest.in (DRY is so 1980s) to see if that fixes it (based on https://stackoverflow.com/questions/3596979/manifest-in-ignored-on-python-setup-py-install-no-data-files-installed)

If that doesn't work, the only thing I can think of is something to do with the installed lib having a different name? (git vs gitpython).

Maybe should release a beta with the fixes I commited this morn? Then I can install from pypi. I can't think of how else to test this?
Or I'll release as a clone package and test that one.

Yobmod · 2021-07-23T12:11:44Z

Yeah, CI doesn't error because it only happens with pypi install into already existing environments, but CI is making fresh venvs with the newest available libs each time

mdantonio · 2021-07-23T12:16:01Z

mypy docs (see the very last paragraph in https://mypy.readthedocs.io/en/latest/installed_packages.html#creating-pep-561-compatible-packages) suggests:

global-include *.typed

in MANIFEST.in

But i think that include py.typed should work as well...

(However, none of the files in the MANIFEST.in are included in the package... it seems that the MANIFEST is ignored at all)

This release includes the unfinished addition of types to the library. For more information see: gitpython-developers/GitPython#1095

Yobmod · 2021-07-28T15:09:58Z

I think the more version specific guard is the correct fix, mypy just complaining because it can't cope with the same variable being defined 3 different ways for different pythons.

Yobmod · 2021-07-28T15:27:08Z

Type checked all my functions that are using gitpython (removed several Any and cast, I'm super happy!) and this is the only problematic case that I found:

I have a function that can return gitobj.active_branch.name, but mypy complains:
Returning Any from function declared to return "Optional[str]"
At runtime i got a str from active_branch.name so probably it is still un-annotated

What is the gitobj?
The only active_branch.name in the code base is an attribute of Repo, but that is typed (as Union[str, PathLike[str], which ive changed to just str).
Or if it was a SymbolicReference of some type, that has a name property that was missing a type, now added.

For some of the Git comand wrappers, they are too dynamic to add types (so default to Any), until/unless someone adds specific methods/overloads for each command.

mdantonio · 2021-07-28T16:27:55Z

gitobj is an instance of Repo

Here a minimal test case to reproduce the issue:

from git import Repo

def get_active_branch(gitobj: Repo) -> str:
    return gitobj.active_branch.name

gitobj = Repo(".")
print(get_active_branch(gitobj))

$ mypy --warn-return-any active_branch_test.py 
active_branch_test.py:4: error: Returning Any from function declared to return "str"
Found 1 error in 1 file (checked 1 source file)

Yobmod · 2021-07-28T16:47:58Z

Ah, no error for me.

What does mypy show for reveal_type(gitobj.active_branch)?
For me it is 'Head' from your snippet
and reveal_type(gitobj.active_branch.name) is str.

So maybe adding type hint to symbolicReference.name has fixed it in main? (Head <- Reference <- SymbolicReference)

mdantonio · 2021-07-28T17:59:11Z

$ mypy --warn-return-any active_branch_test.py 
active_branch_test.py:5: note: Revealed type is "git.repo.base.Repo"
active_branch_test.py:6: note: Revealed type is "git.refs.symbolic.SymbolicReference"
active_branch_test.py:7: note: Revealed type is "Any"
active_branch_test.py:8: error: Returning Any from function declared to return "str"
Found 1 error in 1 file (checked 1 source file)

Yobmod · 2021-07-28T18:19:51Z

Thx for checking. Then it is fixed for next release :)

Byron · 2021-07-29T03:11:41Z

Thanks everyone for bearing with us. Yesterday after the second issue was filed I yanked 3.1.20, so it looks like that the issues around older python versions and ordered dict is all there is for now.

@Yobmod What do you think about setting up CI to test patch releases as well, to have more probes when determining that the type related changes are actually working for everyone? Typing seems to be a minefield when trying to support as many python versions as we do, and I have a feeling we have to beef up our testing game in order to avoid all these hassles in the future.
My suggestion is to setup CI to test the oldest patch level on each minor python version. If it's not supported by GitHub workflows, we might use pyenv or something similar and implement that ourselves. Maybe there are python actions that do that already though.
Just some thoughts of mine, I am looking forward to learning about yours.

kasium · 2021-07-29T06:32:21Z

Hi all,

just a short idea: typing imports could be wrapped into if TYPE_CHECKING and then all types must be wrapped into quotes. With this you could avoid these kind of issues like with python 3.7.1.

In addition there is also a checker for flake8: https://github.com/asottile/flake8-typing-imports

foo.py

from typing import OrderedDict

Execute: flake8 --select TYP --min-python-version 3.7.0 foo.py

foo.py:1:1: TYP001 guard import by `if TYPE_CHECKING:`: OrderedDict (not in 3.7.0, 3.7.1)

EDIT: I'm happy to file a PR to add flake8-typing-imports if you think this makes sense

gaborbernat · 2021-08-02T11:59:14Z

When running in strict mode one gets failures for the repository not exporting explicitly its properties:

Module "git" does not explicitly export attribute "Git"; implicit reexport disabled

See python/mypy#10198 (comment) for more details.

Yobmod · 2021-08-02T12:37:04Z

Hi all,

just a short idea: typing imports could be wrapped into if TYPE_CHECKING and then all types must be wrapped into quotes. With this you could avoid these kind of issues like with python 3.7.1.

In addition there is also a checker for flake8: https://github.com/asottile/flake8-typing-imports

foo.py
from typing import OrderedDict
Execute: flake8 --select TYP --min-python-version 3.7.0 foo.py
foo.py:1:1: TYP001 guard import by `if TYPE_CHECKING:`: OrderedDict (not in 3.7.0, 3.7.1)
EDIT: I'm happy to file a PR to add flake8-typing-imports if you think this makes sense

We already have a similar library in requirements-dev.txt (flake8-type-checking), but it has too many false positives to use in CI.
So feel free to swap it for flake8-typing-imports and if it works (and once all errors are fixed/ignored) it can be added to test-requirements.txt.

The issue for Orderdict though is that we are subclassing it, so one import has to be runtime available. Maybe we can put the typed versions in the if TYPE_CHECKING block after the runtime import? It will still need py version guards for mypy though? it just wont break at runtime.

Yobmod · 2021-08-02T12:41:46Z

When running in strict mode one gets failures for the repository not exporting explicitly its properties:
Module "git" does not explicitly export attribute "Git"; implicit reexport disabled
See python/mypy#10198 (comment) for more details.

Yep, we have a lot of * imports that rely on implicit-reexport for mypy. For end-users this should not cause problems, but its a known mypy bug: python/mypy#8754.

If someone wants to change all the * imports to match __all__ for each module it will fix it, but otherwise have to wait for the bug to be fixed. Current workaround is if using --strict, also use --implicit-reexport.

gaborbernat · 2021-08-02T13:06:05Z

I personally think for reasons explained in python/mypy#10198 (comment) this is not a mypy bug 🤔 and I don't think due to that will be fixed 🤔 At best I'd imagine in py.typed you'd have some information about the package using implicit re-export but you cannot assume all packages will use that. Implicit re-export is a convenient but dangerous feature, and should be IMHO an opt-in not an always on thing.

kasium · 2021-08-02T13:23:15Z

The issue for Orderdict though is that we are subclassing it, so one import has to be runtime available. Maybe we can put the typed versions in the if TYPE_CHECKING block after the runtime import? It will still need py version guards for mypy though? it just wont break at runtime.

Yes, I guess it will only prevent runtime issues but during type checking time, the imports must work. However, normal users might get more stable releases

So feel free to swap it for flake8-typing-imports and if it works (and once all errors are fixed/ignored) it can be added to test-requirements.txt.

Let me check and open a PR if possible. Please note, that these libraries are doing different things. The current one is more style related, while flake8-typing-imports checks import from typing based on the current python version

kasium · 2021-08-02T13:48:10Z

So, I opened #1309.
Please note that gitpython says it's compatible with >= 3.6.0 but it's only compatible with > 3.6.1! See the PR for details.

Yobmod · 2021-08-02T14:13:21Z

Seems useful already!

So we have to decide:

Drop 3.6 now,
or i'll pull out NamedTuple defaults (there are only a couple of uses) for now.

(NoReturn and Deque are available in typing_extensions, so that can be easily fixed.)

I was planning to wait until 3.10 released to drop 3.6.
But I see it had it's final release this June and i have noticed big packages already dropping it (numpy and pandas did last year and therefore most of scientific packages).

Byron · 2021-08-02T14:15:40Z

Dropping py 3.6 seems like the path of least resistance.
I suggest to go with that.

gaborbernat · 2021-08-02T15:17:49Z

If someone wants to change all the * imports to match __all__ for each module it will fix it, but otherwise have to wait for the bug to be fixed. Current workaround is if using --strict, also use --implicit-reexport.

See python/mypy#8754 (comment), seems this is not a bug in mypy, but a feature and the issue lies in this package, not the type checkers (both mypy and pyright seem to assume that implicit re-export is off).

Yobmod · 2021-08-02T15:37:58Z

If someone wants to change all the * imports to match __all__ for each module it will fix it, but otherwise have to wait for the bug to be fixed. Current workaround is if using --strict, also use --implicit-reexport.

See python/mypy#8754 (comment), seems this is not a bug in mypy, but a feature and the issue lies in this package, not the type checkers (both mypy and pyright seem to assume that implicit re-export is off).

Its fixable here, but there are 156 implicit-reexports errors if its activated in the codebase atm, so it wont be done for the next few releases unless someone volunteers.
I've added implicit_reexport = true to our settings, but I don't know if that will be overriden downstream?

kasium · 2021-08-11T19:27:57Z

Is there any update how you want to proceed with this?

Yobmod · 2021-09-09T19:47:12Z

@Byron

I think i've done everything I planned for the next release.
3.6 is dropped, and all reported errors in typing fixed and tested against an expanded test matrix. (Github actions only goes as early as 3.7.5 though).

Testpypi uploads fine (using 3.7.3), so ready to go!

Byron · 2021-09-10T02:16:15Z

Thanks a lot! Thanks to a 'hickup' this was ultimately released as https://pypi.org/project/GitPython/3.1.23/ .

Byron added enhancement help wanted labels Dec 22, 2020

Yobmod mentioned this issue Feb 24, 2021

Dropping python 3.4 #1131

Closed

Byron added this to the v3.1.19 - Bugfixes milestone Jul 23, 2021

kajarenc mentioned this issue Jul 23, 2021

Pin GitPython version not equal to 3.1.19 to avoid nightly builds crush. streamlit/streamlit#3599

Merged

Yobmod mentioned this issue Jul 28, 2021

Hopefully fixes Orderedict error #1301

Closed

miffels mentioned this issue Aug 2, 2021

Set fixed dependency version for GitPython timvink/mkdocs-git-revision-date-localized-plugin#56

Closed

kasium mentioned this issue Aug 2, 2021

Add flake8-typing-imports #1309

Merged

Byron modified the milestones: v3.1.19 - Bugfixes, v3.1.21 - Bugfixes Aug 2, 2021

Byron closed this as completed Sep 10, 2021

tucked mentioned this issue Sep 24, 2021

mypy is failing on from git import Blob #1349

Closed

Byron unpinned this issue Jan 7, 2022

dd-ssc mentioned this issue Jul 27, 2023

pyright still reports PathLike as unknown #1620

Closed

Type Hints #1095

Type Hints #1095

Comments

mdantonio commented Dec 21, 2020

Byron commented Dec 22, 2020

Yobmod commented Feb 18, 2021 • edited Loading

Byron commented Feb 19, 2021

Yobmod commented Feb 24, 2021

Yobmod commented Feb 27, 2021 • edited Loading

mdantonio commented Feb 28, 2021

Yobmod commented Feb 28, 2021 • edited Loading

kasium commented Jun 30, 2021

Yobmod commented Jun 30, 2021

kasium commented Jun 30, 2021

Yobmod commented Jun 30, 2021

kasium commented Jun 30, 2021

Yobmod commented Jul 9, 2021 • edited Loading

kasium commented Jul 21, 2021

Byron commented Jul 21, 2021

Yobmod commented Jul 21, 2021

Byron commented Jul 23, 2021

adeandrade commented Jul 23, 2021

Yobmod commented Jul 23, 2021 via email

mdantonio commented Jul 23, 2021 • edited Loading

Byron commented Jul 23, 2021 via email

Yobmod commented Jul 23, 2021 • edited Loading

Yobmod commented Jul 23, 2021 • edited Loading

mdantonio commented Jul 23, 2021 • edited Loading

Yobmod commented Jul 28, 2021

Yobmod commented Jul 28, 2021 • edited Loading

mdantonio commented Jul 28, 2021

Yobmod commented Jul 28, 2021 • edited Loading

mdantonio commented Jul 28, 2021

Yobmod commented Jul 28, 2021

Byron commented Jul 29, 2021

kasium commented Jul 29, 2021 • edited Loading

gaborbernat commented Aug 2, 2021

Yobmod commented Aug 2, 2021 • edited Loading

Yobmod commented Aug 2, 2021 • edited Loading

gaborbernat commented Aug 2, 2021 • edited Loading

kasium commented Aug 2, 2021 • edited Loading

kasium commented Aug 2, 2021

Yobmod commented Aug 2, 2021

Byron commented Aug 2, 2021

gaborbernat commented Aug 2, 2021

Yobmod commented Aug 2, 2021 • edited Loading

kasium commented Aug 11, 2021

Yobmod commented Sep 9, 2021

Byron commented Sep 10, 2021

Yobmod commented Feb 18, 2021 •

edited

Loading

Yobmod commented Feb 27, 2021 •

edited

Loading

Yobmod commented Feb 28, 2021 •

edited

Loading

Yobmod commented Jul 9, 2021 •

edited

Loading

mdantonio commented Jul 23, 2021 •

edited

Loading

Yobmod commented Jul 23, 2021 •

edited

Loading

Yobmod commented Jul 23, 2021 •

edited

Loading

mdantonio commented Jul 23, 2021 •

edited

Loading

Yobmod commented Jul 28, 2021 •

edited

Loading

Yobmod commented Jul 28, 2021 •

edited

Loading

kasium commented Jul 29, 2021 •

edited

Loading

Yobmod commented Aug 2, 2021 •

edited

Loading

Yobmod commented Aug 2, 2021 •

edited

Loading

gaborbernat commented Aug 2, 2021 •

edited

Loading

kasium commented Aug 2, 2021 •

edited

Loading

Yobmod commented Aug 2, 2021 •

edited

Loading