[BEAM-7746] Add python type hints (part 1) #9915

chadrik · 2019-10-29T03:16:14Z

This is part 1 of #9056

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Gearpump	Samza	Spark
Go		---	---	---	---
Java
Python		---		---	---
XLang	---	---	---	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

chadrik · 2019-10-29T04:32:06Z

R: @robertwb

chadrik · 2019-10-29T15:07:23Z

Run Python PreCommit

robertwb

I've looked at everything up to apache_beam/runners/common.py.

sdks/python/apache_beam/coders/coder_impl.py

sdks/python/apache_beam/coders/coders.py

sdks/python/apache_beam/io/iobase.py

sdks/python/apache_beam/io/textio.py

sdks/python/apache_beam/options/value_provider.py

chadrik · 2019-11-12T17:28:30Z

Just a friendly reminder to nudge this along.

robertwb

Reviewed up through runners/portability.

sdks/python/apache_beam/runners/direct/transform_evaluator.py

sdks/python/apache_beam/runners/portability/fn_api_runner.py

chadrik · 2019-11-18T18:55:11Z

Reviewed up through runners/portability.

You're almost 2/3 of the way through. Almost there :)

robertwb

Reviewe everything up through transforms/ptransform.py

sdks/python/apache_beam/runners/runner.py

sdks/python/apache_beam/runners/worker/bundle_processor.py

sdks/python/apache_beam/transforms/core.py

chadrik · 2019-11-22T00:26:16Z

We're so close. Just a few more. Thanks for your hard work getting through all of this!

robertwb

Thanks. Just a couple of comments, but it all LGTM. Unsurprisingly there's some merge conflicts, but hopefully they should be easy to resolve.

sdks/python/apache_beam/transforms/window.py

robertwb · 2019-11-22T19:26:23Z

sdks/python/apache_beam/transforms/window.py

@@ -505,7 +527,8 @@ def get_window_coder(self):
    return coders.IntervalWindowCoder()

  def merge(self, merge_context):
-    to_merge = []
+    # type: (WindowFn.MergeContext) -> None
+    to_merge = []  # type: List[Union[IntervalWindow, GlobalWindow]]


List[BoundedWindow]

Changing this and the other use of Union[IntervalWindow, GlobalWindow] to BoundedWindow results in these errors:

apache_beam/transforms/window.py:541: error: "BoundedWindow" has no attribute "start" [attr-defined] apache_beam/transforms/window.py:543: error: "BoundedWindow" has no attribute "start" [attr-defined] apache_beam/transforms/window.py:550: error: "BoundedWindow" has no attribute "start" [attr-defined] apache_beam/transforms/window.py:557: error: "BoundedWindow" has no attribute "start" [attr-defined]

The docs for BoundedWindow say that it's for timestamps in the range (-infinity, end).

How about we add a start property to BoundedWindow:

class BoundedWindow(object): """A window for timestamps in range (-infinity, end). Attributes: end: End of window. """ def __init__(self, end): # type: (Union[int, float, Timestamp]) -> None self._end = Timestamp.of(end) @property def start(self): # type: () -> Timestamp return MAX_TIMESTAMP

sdks/python/apache_beam/utils/timestamp.py

robertwb · 2019-11-22T19:32:17Z

sdks/python/apache_beam/utils/timestamp.py

@@ -136,14 +140,17 @@ def to_rfc3339(self):
    return self.to_utc_datetime().isoformat() + 'Z'

  def __float__(self):
+    # type: () -> float


I'm surprised mypy doesn't know these.

It may, but I didn't test it.

sdks/python/apache_beam/utils/windowed_value.py

sdks/python/mypy.ini

sdks/python/setup.py

chadrik · 2019-11-25T07:06:09Z

Run Python PreCommit

chadrik · 2019-11-25T07:45:52Z

@udim pytest is causing failures in the setup of the portable runner tests. It has something to do with the use of setup_requires to install pytest-runner in setup.py. This feature of setuptools has been abandoned for a long time, and the pytest-runner PyPI page says that it's deprecated and should not be used.

I've created a PR to try to modernize and simplify the python build process, especially as it relates to build requirements: #10038. I'm still ironing out some details. Let me know what you think.

In the meantime I think we should remove the use of pytest_runner and setup_requires.

chadrik · 2019-11-25T15:56:26Z

Run Python PreCommit

chadrik · 2019-11-25T19:59:24Z

Run Python PreCommit

chadrik · 2019-11-26T16:02:50Z

Run Python PreCommit

chadrik · 2019-12-02T18:31:19Z

Run Python PreCommit

chadrik · 2019-12-02T23:54:53Z

Run Java PreCommit

chadrik · 2019-12-03T01:40:14Z

Run Python PreCommit

chadrik · 2019-12-03T06:41:01Z

all tests are passing and there are no conflicts with master!

chadrik · 2019-12-03T06:42:07Z

@robertwb I addressed all of your notes in a final commit so that you can compare them against your notes and ensure they are resolved to your liking.

chadrik · 2019-12-04T01:12:37Z

merge conflicts again....

Many of these (those marked with [misc]) are due to mypy taking issue with: - the use of invalid type objects (typehints not typing) - the use of type objects at runtime

`None` is only safe to pass as the context to Coder.to_runner_api() when we know that the Coder class does not have components, otherwise the context is used to find and convert them. We could remove the need for special case `ignores` by reorganizing our Coder class hierarchy to add a CompoundCoder base class or mixin. Right now this is the only place in the code that we need to ignore this problem, so it's not worth tackling now.

This problem will be fixed when the next version of mypy is released

The test runs but error status is ignored.

This resolves a pip conflict that was happening between setupVirtualenv task and the tox task by ensuring that everything agrees on the same set of versions.

There are no meaningful runtime changes in this commit.

pylint-dev/pylint#3217

This seems to be fine to do on an instance as long as the types match, but not on a class.

chadrik · 2019-12-10T00:31:04Z

Run Java PreCommit

chadrik · 2019-12-10T17:12:54Z

Run Python PreCommit

chadrik · 2019-12-10T23:52:22Z

Run Python PreCommit

udim · 2019-12-13T19:31:01Z

I believe this PR broke :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
Opened https://issues.apache.org/jira/browse/BEAM-8966

chadrik · 2019-12-13T21:13:44Z

Is that not part of the precommit suite? The tests for this PR definitely ran and passed. The traceback seemed to indicate that google.protobuf is not installed...

…

On Fri, Dec 13, 2019 at 11:31 AM Udi Meiri ***@***.***> wrote: I believe this PR broke :sdks:python:test-suites:direct:py37:hdfsIntegrationTest Opened https://issues.apache.org/jira/browse/BEAM-8966 — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#9915?email_source=notifications&email_token=AAAPOE3TOM63EE2P72YWOTTQYPPHNA5CNFSM4JGC3N42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG3AO5Y#issuecomment-565577591>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAPOE3HEVMZEZOXV2MHHM3QYPPHNANCNFSM4JGC3N4Q> .

udim · 2019-12-16T18:30:14Z

Is that not part of the precommit suite? The tests for this PR definitely ran and passed. The traceback seemed to indicate that google.protobuf is not installed...
…
On Fri, Dec 13, 2019 at 11:31 AM Udi Meiri @.***> wrote: I believe this PR broke :sdks:python:test-suites:direct:py37:hdfsIntegrationTest Opened https://issues.apache.org/jira/browse/BEAM-8966 — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#9915?email_source=notifications&email_token=AAAPOE3TOM63EE2P72YWOTTQYPPHNA5CNFSM4JGC3N42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG3AO5Y#issuecomment-565577591>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPOE3HEVMZEZOXV2MHHM3QYPPHNANCNFSM4JGC3N4Q .

Not it is not part of precommits.

tvalentyn · 2020-01-06T23:24:07Z

sdks/python/.pylintrc

@@ -178,6 +179,7 @@ indent-after-paren=4
 ignore-long-lines=(?x)
  (^\s*(import|from)\s
   |^\s*(\#\ )?<?(https?|ftp):\/\/[^\s\/$.?#].[^\s]*>?$
+   |# type:


This line disabled line-too-long errors across the board: https://issues.apache.org/jira/browse/BEAM-9058.

chadrik mentioned this pull request Oct 29, 2019

[BEAM-7746] Add python type hints #9056

Closed

robertwb reviewed Nov 6, 2019

View reviewed changes

robertwb reviewed Nov 13, 2019

View reviewed changes

sdks/python/apache_beam/runners/direct/transform_evaluator.py Show resolved Hide resolved

sdks/python/apache_beam/runners/portability/fn_api_runner.py Outdated Show resolved Hide resolved

robertwb reviewed Nov 19, 2019

View reviewed changes

robertwb approved these changes Nov 22, 2019

View reviewed changes

chadrik force-pushed the python-static-typing-part1 branch 6 times, most recently from 5657f26 to d7963d4 Compare November 24, 2019 17:08

chadrik mentioned this pull request Nov 27, 2019

[BEAM-8840][BEAM-3713] Remove setup_requires, tests_require from setup.py #10227

Merged

3 tasks

chadrik force-pushed the python-static-typing-part1 branch 2 times, most recently from c1a02d1 to 7a40e82 Compare December 2, 2019 22:25

chadrik added 17 commits December 9, 2019 14:35

Add tests for Timestamp/Duration comparisons

a348e6e

Ignore some conditional imports in tests

effe225

Ignore functions missing from typeshed

00c6ff7

Ignore some objects missing from typeshed

f8eaf35

Ignore some monkey-patching for tests

60dac7b

Ignore an issue regarding asymmetrical property getter/setter

7814c08

Ignore unavoidable errors

95d7618

Many of these (those marked with [misc]) are due to mypy taking issue with: - the use of invalid type objects (typehints not typing) - the use of type objects at runtime

Address issues with a few missing modules/objects

70f06ed

Ignore errors about dynamic base classes

c697caf

This problem will be fixed when the next version of mypy is released

Add a mypy lint job

58df66f

The test runs but error status is ignored.

Consolidate python build requirements

79744db

This resolves a pip conflict that was happening between setupVirtualenv task and the tox task by ensuring that everything agrees on the same set of versions.

Add type annotations to bulk of python codebase

bfd04a1

There are no meaningful runtime changes in this commit.

Workaround a bug in pylint

92bf5f6

pylint-dev/pylint#3217

Silence errors about overriding methods.

18578d8

This seems to be fine to do on an instance as long as the types match, but not on a class.

Address review notes

6d2a801

Rebase fixup

5c48091

chadrik force-pushed the python-static-typing-part1 branch from c32cf2e to 5c48091 Compare December 9, 2019 22:37

pabloem merged commit df37616 into apache:master Dec 11, 2019

chadrik mentioned this pull request Dec 12, 2019

[BEAM-7746] Add python type hints (part 2) #10367

Merged

3 tasks

kamilwu mentioned this pull request Dec 17, 2019

[BEAM-8979] Remove mypy-protobuf dependency #10400

Merged

3 tasks

tvalentyn reviewed Jan 6, 2020

View reviewed changes

cghawthorne mentioned this pull request Feb 10, 2020

Install typing package only for Python < 3.5 #10821

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-7746] Add python type hints (part 1) #9915

[BEAM-7746] Add python type hints (part 1) #9915

chadrik commented Oct 29, 2019

chadrik commented Oct 29, 2019

chadrik commented Oct 29, 2019

robertwb left a comment

chadrik commented Nov 12, 2019

robertwb left a comment

chadrik commented Nov 18, 2019

robertwb left a comment

chadrik commented Nov 22, 2019

robertwb left a comment

robertwb Nov 22, 2019

chadrik Nov 22, 2019

robertwb Nov 22, 2019

chadrik Nov 22, 2019

chadrik commented Nov 25, 2019

chadrik commented Nov 25, 2019 •

edited

Loading

chadrik commented Nov 25, 2019

chadrik commented Nov 25, 2019

chadrik commented Nov 26, 2019

chadrik commented Dec 2, 2019

chadrik commented Dec 2, 2019

chadrik commented Dec 3, 2019

chadrik commented Dec 3, 2019

chadrik commented Dec 3, 2019

chadrik commented Dec 4, 2019

chadrik commented Dec 10, 2019

chadrik commented Dec 10, 2019

chadrik commented Dec 10, 2019

udim commented Dec 13, 2019

chadrik commented Dec 13, 2019 via email

udim commented Dec 16, 2019

tvalentyn Jan 6, 2020

[BEAM-7746] Add python type hints (part 1) #9915

[BEAM-7746] Add python type hints (part 1) #9915

Conversation

chadrik commented Oct 29, 2019

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

chadrik commented Oct 29, 2019

chadrik commented Oct 29, 2019

robertwb left a comment

Choose a reason for hiding this comment

chadrik commented Nov 12, 2019

robertwb left a comment

Choose a reason for hiding this comment

chadrik commented Nov 18, 2019

robertwb left a comment

Choose a reason for hiding this comment

chadrik commented Nov 22, 2019

robertwb left a comment

Choose a reason for hiding this comment

robertwb Nov 22, 2019

Choose a reason for hiding this comment

chadrik Nov 22, 2019

Choose a reason for hiding this comment

robertwb Nov 22, 2019

Choose a reason for hiding this comment

chadrik Nov 22, 2019

Choose a reason for hiding this comment

chadrik commented Nov 25, 2019

chadrik commented Nov 25, 2019 • edited Loading

chadrik commented Nov 25, 2019

chadrik commented Nov 25, 2019

chadrik commented Nov 26, 2019

chadrik commented Dec 2, 2019

chadrik commented Dec 2, 2019

chadrik commented Dec 3, 2019

chadrik commented Dec 3, 2019

chadrik commented Dec 3, 2019

chadrik commented Dec 4, 2019

chadrik commented Dec 10, 2019

chadrik commented Dec 10, 2019

chadrik commented Dec 10, 2019

udim commented Dec 13, 2019

chadrik commented Dec 13, 2019 via email

udim commented Dec 16, 2019

tvalentyn Jan 6, 2020

Choose a reason for hiding this comment

chadrik commented Nov 25, 2019 •

edited

Loading