[BEAM-11607] Add word count tasks #14467

isidroamv · 2021-04-07T23:48:11Z

This PR adds a task to check the environment setup and tasks for wordCount example for Java, Python, and Go.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	ULR	Samza	Twister2
Go	---	---	---
Java
Python	---	---	---
XLang	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2021-04-08T00:10:57Z

Codecov Report

Merging #14467 (d3a338f) into master (bb7c427) will increase coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #14467      +/-   ##
==========================================
  Coverage   83.47%   83.47%              
==========================================
  Files         450      898     +448     
  Lines       58806   117956   +59150     
==========================================
+ Hits        49088    98464   +49376     
- Misses       9718    19492    +9774

Impacted Files	Coverage Δ
...rcs/sdks/python/apache_beam/testing/test_stream.py
...hon/apache_beam/runners/dataflow/internal/names.py
...mples/snippets/transforms/elementwise/partition.py
...uild/srcs/sdks/python/apache_beam/metrics/cells.py
...n/apache_beam/runners/dataflow/native_io/iobase.py
...cp/internal/clients/storage/storage_v1_messages.py
.../srcs/sdks/python/apache_beam/io/range_trackers.py
...les/complete/juliaset/juliaset/juliaset_test_it.py
...beam/runners/interactive/background_caching_job.py
.../srcs/sdks/python/apache_beam/utils/annotations.py
... and 1338 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb7c427...d3a338f. Read the comment docs.

tysonjh · 2021-04-08T16:11:11Z

It would be helpful to also see the documentation updates to go along with this to make sure the task name makes sense.

examples/java/build.gradle

isidroamv · 2021-04-08T19:10:22Z

It would be helpful to also see the documentation updates to go along with this to make sure the task name makes sense.

This PR will update the documentation based on the work of BEAM-11607 and BEAM-11608

tysonjh · 2021-04-09T17:02:16Z

When I ran this locally, it failed with the following error:


> Task :sdks:python:setupVirtualenv FAILED
created virtual environment CPython3.8.7.final.0-64 in 334ms
  creator CPython3Posix(dest=/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/usr/local/google/home/tysonjh/.local/share/virtualenv)
    added seed packages: pip==21.0.1, pkg_resources==0.0.0, setuptools==54.1.1, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Traceback (most recent call last):
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 8, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_internal/cli/cmdoptions.py", line 20, in <module>
    from pip._vendor.packaging.utils import canonicalize_name
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_vendor/packaging/utils.py", line 9, in <module>
    from .tags import Tag, parse_tag
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_vendor/packaging/tags.py", line 7, in <module>
    import distutils.util
ModuleNotFoundError: No module named 'distutils.util'

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> Process 'command 'sh'' finished with non-zero exit value 1

isidroamv · 2021-04-12T16:11:57Z

When I ran this locally, it failed with the following error:


> Task :sdks:python:setupVirtualenv FAILED
created virtual environment CPython3.8.7.final.0-64 in 334ms
  creator CPython3Posix(dest=/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/usr/local/google/home/tysonjh/.local/share/virtualenv)
    added seed packages: pip==21.0.1, pkg_resources==0.0.0, setuptools==54.1.1, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Traceback (most recent call last):
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 8, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_internal/cli/cmdoptions.py", line 20, in <module>
    from pip._vendor.packaging.utils import canonicalize_name
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_vendor/packaging/utils.py", line 9, in <module>
    from .tags import Tag, parse_tag
  File "/usr/local/google/home/tysonjh/Development/beam/build/gradleenv/1922375555/lib/python3.8/site-packages/pip/_vendor/packaging/tags.py", line 7, in <module>
    import distutils.util
ModuleNotFoundError: No module named 'distutils.util'

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> Process 'command 'sh'' finished with non-zero exit value 1

I already solved this issue. Try again @tysonjh

tysonjh

I'm not able to verify the python tests for some reason due to my local setup. I'm going to ask for someone else to try and merge. Otherwise LGTM.

build.gradle.kts

tysonjh · 2021-04-13T17:48:20Z

R: @udim

udim · 2021-04-14T00:37:15Z

Runs for me successfully

udim · 2021-04-14T00:37:24Z

Run PythonLint PreCommit

udim · 2021-04-14T00:37:32Z

Run Python PreCommit

udim

LGTM, waiting for tests to pass

pabloem · 2021-04-15T18:37:12Z

I successfully ran this in my environment. I'll merge if that's okay : )

[BEAM-11607] Add word count tasks

628b3d3

Remove white space

0bbb917

tysonjh reviewed Apr 8, 2021

View reviewed changes

examples/java/build.gradle Show resolved Hide resolved

Add task description

485ecb3

Add gradle description

391d775

Fix dependency in wordcount task

a519b3a

tysonjh approved these changes Apr 13, 2021

View reviewed changes

build.gradle.kts Outdated Show resolved Hide resolved

Fix task name

d3a338f

udim approved these changes Apr 14, 2021

View reviewed changes

pabloem merged commit 0d8f22d into apache:master Apr 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-11607] Add word count tasks #14467

[BEAM-11607] Add word count tasks #14467

isidroamv commented Apr 7, 2021 •

edited

Loading

codecov bot commented Apr 8, 2021 •

edited

Loading

tysonjh commented Apr 8, 2021

isidroamv commented Apr 8, 2021 •

edited

Loading

tysonjh commented Apr 9, 2021

isidroamv commented Apr 12, 2021

tysonjh left a comment

tysonjh commented Apr 13, 2021

udim commented Apr 14, 2021

udim commented Apr 14, 2021

udim commented Apr 14, 2021

udim left a comment

pabloem commented Apr 15, 2021

[BEAM-11607] Add word count tasks #14467

[BEAM-11607] Add word count tasks #14467

Conversation

isidroamv commented Apr 7, 2021 • edited Loading

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

codecov bot commented Apr 8, 2021 • edited Loading

Codecov Report

tysonjh commented Apr 8, 2021

isidroamv commented Apr 8, 2021 • edited Loading

tysonjh commented Apr 9, 2021

isidroamv commented Apr 12, 2021

tysonjh left a comment

Choose a reason for hiding this comment

tysonjh commented Apr 13, 2021

udim commented Apr 14, 2021

udim commented Apr 14, 2021

udim commented Apr 14, 2021

udim left a comment

Choose a reason for hiding this comment

pabloem commented Apr 15, 2021

isidroamv commented Apr 7, 2021 •

edited

Loading

codecov bot commented Apr 8, 2021 •

edited

Loading

isidroamv commented Apr 8, 2021 •

edited

Loading