[BEAM-12266] Adds a connector that streams data from twitter #14531

usingh83 · 2021-04-14T03:27:13Z

using a Standard Twitter app.

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ @pabloem ] Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	ULR	Samza	Twister2
Go	---	---	---
Java
Python	---	---	---
XLang	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

pabloem

I think it may be best if you move your connector into the examples/java directory, so that you get all of the build configurations provided by the examples dir (see: https://github.com/apache/beam/blob/master/examples/java/build.gradle)

the RAT Precommit failure is because every file needs to include an Apache License header (see for example: https://github.com/apache/beam/blob/master/examples/java/build.gradle#L1-L17)

could you create a JIRA issue for it and assign it to yourself?

Also, I think your classes have Tweeter but the product/company is called Twitter so maybe rename those? : )

examples/twitter-beam-java/src/main/java/ReadFromTweeterDoFn.java

examples/twitter-beam-java/src/main/java/TweeterStream.java

examples/twitter-beam-java/src/main/java/ReadFromTweeterDoFn.java

aaltay · 2021-04-29T22:13:23Z

What is the next step for this PR?

usingh83 · 2021-05-02T09:13:43Z

I think it may be best if you move your connector into the examples/java directory, so that you get all of the build configurations provided by the examples dir (see: https://github.com/apache/beam/blob/master/examples/java/build.gradle)

the RAT Precommit failure is because every file needs to include an Apache License header (see for example: https://github.com/apache/beam/blob/master/examples/java/build.gradle#L1-L17)

could you create a JIRA issue for it and assign it to yourself?

Also, I think your classes have Tweeter but the product/company is called Twitter so maybe rename those? : )

Hi Pablo, I created the issue, but I am not able to assign it to myself - https://issues.apache.org/jira/browse/BEAM-12266! Do you know what might be wrong?

I will take care of the rest.

pabloem · 2021-05-03T20:16:16Z

you needed to request contributor permissions in the Beam dev@ list - I've granted thejm to you and assigned the but to you

usingh83 · 2021-05-04T00:29:07Z

What is the next step for this PR?

Working on it :)

...s/java/src/main/java/org/apache/beam/examples/complete/TwitterStreamGenerator/TwitterIO.java

examples/twitter-beam-java/build.gradle

examples/java/build.gradle

# This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e44 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: Added changes for making the implementation more streamlined and understandable # This is the commit message apache#3: Lint the files. # This is the commit message apache#4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message apache#5: Linting the project and making some stuff private # This is the commit message apache#6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message apache#7: delete the unused folder # This is the commit message apache#8: reorganizing pipeline # This is the commit message apache#9: Spotless check # This is the commit message apache#10: Spotless check # This is the commit message apache#11: build failure corrected # This is the commit message apache#12: Java_Examples_Dataflow PreCommit fix # This is the commit message apache#13: Java_Examples_Dataflow PreCommit refix # This is the commit message apache#14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#15: Java_Examples_Dataflow PreCommit assign nullable correctly

pabloem

this is looking great. Thanks Uday! just added a couple comments

...s/java/src/main/java/org/apache/beam/examples/complete/twitterstreamgenerator/TwitterIO.java

examples/java/src/main/java/org/apache/beam/examples/complete/twitterstreamgenerator/README.md

...t/java/org/apache/beam/examples/complete/twitterstreamgenerator/ReadFromTwitterDoFnTest.java

# This is the 1st commit message: # This is a combination of 2 commits. # This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e44 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: Added changes for making the implementation more streamlined and understandable # This is the commit message apache#3: Lint the files. # This is the commit message apache#4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message apache#5: Linting the project and making some stuff private # This is the commit message apache#6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message apache#7: delete the unused folder # This is the commit message apache#8: reorganizing pipeline # This is the commit message apache#9: Spotless check # This is the commit message apache#10: Spotless check # This is the commit message apache#11: build failure corrected # This is the commit message apache#12: Java_Examples_Dataflow PreCommit fix # This is the commit message apache#13: Java_Examples_Dataflow PreCommit refix # This is the commit message apache#14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#2: # This is a combination of 3 commits. # This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e44 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: Added changes for making the implementation more streamlined and understandable # This is the commit message apache#3: Lint the files. # This is the commit message apache#4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message apache#5: Linting the project and making some stuff private # This is the commit message apache#6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message apache#7: delete the unused folder # This is the commit message apache#8: reorganizing pipeline # This is the commit message apache#9: Spotless check # This is the commit message apache#10: Spotless check # This is the commit message apache#11: build failure corrected # This is the commit message apache#12: Java_Examples_Dataflow PreCommit fix # This is the commit message apache#13: Java_Examples_Dataflow PreCommit refix # This is the commit message apache#14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#3: # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: Added changes for making the implementation more streamlined and understandable # This is the commit message apache#3: Lint the files. # This is the commit message apache#4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message apache#5: Linting the project and making some stuff private # This is the commit message apache#6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message apache#7: delete the unused folder # This is the commit message apache#8: reorganizing pipeline # This is the commit message apache#9: Spotless check # This is the commit message apache#10: Spotless check # This is the commit message apache#11: build failure corrected # This is the commit message apache#12: Java_Examples_Dataflow PreCommit fix # This is the commit message apache#13: Java_Examples_Dataflow PreCommit refix # This is the commit message apache#14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: Added changes for making the implementation more streamlined and understandable # This is the commit message apache#3: Lint the files. # This is the commit message apache#4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message apache#5: Linting the project and making some stuff private # This is the commit message apache#6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message apache#7: delete the unused folder # This is the commit message apache#8: reorganizing pipeline # This is the commit message apache#9: Spotless check # This is the commit message apache#10: Spotless check # This is the commit message apache#11: build failure corrected # This is the commit message apache#12: Java_Examples_Dataflow PreCommit fix # This is the commit message apache#13: Java_Examples_Dataflow PreCommit refix # This is the commit message apache#14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix # This is a combination of 3 commits. # This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e44 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: Added changes for making the implementation more streamlined and understandable # This is the commit message apache#3: Lint the files. # This is the commit message apache#4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message apache#5: Linting the project and making some stuff private # This is the commit message apache#6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message apache#7: delete the unused folder # This is the commit message apache#8: reorganizing pipeline # This is the commit message apache#9: Spotless check # This is the commit message apache#10: Spotless check # This is the commit message apache#11: build failure corrected # This is the commit message apache#12: Java_Examples_Dataflow PreCommit fix # This is the commit message apache#13: Java_Examples_Dataflow PreCommit refix # This is the commit message apache#14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#3: # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: Added changes for making the implementation more streamlined and understandable # This is the commit message apache#3: Lint the files. # This is the commit message apache#4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message apache#5: Linting the project and making some stuff private # This is the commit message apache#6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message apache#7: delete the unused folder # This is the commit message apache#8: reorganizing pipeline # This is the commit message apache#9: Spotless check # This is the commit message apache#10: Spotless check # This is the commit message apache#11: build failure corrected # This is the commit message apache#12: Java_Examples_Dataflow PreCommit fix # This is the commit message apache#13: Java_Examples_Dataflow PreCommit refix # This is the commit message apache#14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message apache#2: Added changes for making the implementation more streamlined and understandable # This is the commit message apache#3: Lint the files. # This is the commit message apache#4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message apache#5: Linting the project and making some stuff private # This is the commit message apache#6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message apache#7: delete the unused folder # This is the commit message apache#8: reorganizing pipeline # This is the commit message apache#9: Spotless check # This is the commit message apache#10: Spotless check # This is the commit message apache#11: build failure corrected # This is the commit message apache#12: Java_Examples_Dataflow PreCommit fix # This is the commit message apache#13: Java_Examples_Dataflow PreCommit refix # This is the commit message apache#14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message apache#16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix Final Commit with all changes Added unit test adding examples for usage usage for TwitterIO added and Java PreCommit failure fix Spotless PreCommit failure fix

.../main/java/org/apache/beam/examples/complete/twitterstreamgenerator/ReadFromTwitterDoFn.java

...s/java/src/main/java/org/apache/beam/examples/complete/twitterstreamgenerator/TwitterIO.java

pabloem · 2021-05-18T17:10:05Z

thanks @usingh83 ! This is great. If you'd like, it may be cool to add this change to CHANGES.md in another PR so other users know that they can use it for demos / playing around with Beam.

pabloem reviewed Apr 14, 2021

View reviewed changes

pabloem self-requested a review April 14, 2021 23:29

pabloem reviewed May 4, 2021

View reviewed changes

usingh83 changed the title ~~Added a connector that streams data from twitter using a Standard Twi…~~ [BEAM-12266] Added a connector that streams data from twitter using a Standard Twi… May 5, 2021

usingh83 force-pushed the std_twitter_conn branch from 955a386 to 7e46773 Compare May 7, 2021 08:34

pabloem reviewed May 7, 2021

View reviewed changes

usingh83 changed the title ~~[BEAM-12266] Added a connector that streams data from twitter using a Standard Twi…~~ [BEAM-12266] Adds a connector that streams data from twitter using a Standard Twi… May 11, 2021

pabloem reviewed May 11, 2021

View reviewed changes

Unit test for multiple config added, and beautification

0056c3d

usingh83 force-pushed the std_twitter_conn branch from 8ac682a to 0056c3d Compare May 13, 2021 22:42

Spotless apply fixed

0e5a504

usingh83 changed the title ~~[BEAM-12266] Adds a connector that streams data from twitter using a Standard Twi…~~ [BEAM-12266] Adds a connector that streams data from twitter May 14, 2021

usingh83 added 3 commits May 13, 2021 18:18

Removing redundant comments

2d78ad9

Removing newly added test

fab6391

adding newly added test back

e1bded8

pabloem merged commit 4fcf7be into apache:master May 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-12266] Adds a connector that streams data from twitter #14531

[BEAM-12266] Adds a connector that streams data from twitter #14531

usingh83 commented Apr 14, 2021 •

edited

Loading

pabloem left a comment

aaltay commented Apr 29, 2021

usingh83 commented May 2, 2021

pabloem commented May 3, 2021

usingh83 commented May 4, 2021

pabloem left a comment

pabloem commented May 18, 2021

[BEAM-12266] Adds a connector that streams data from twitter #14531

[BEAM-12266] Adds a connector that streams data from twitter #14531

Conversation

usingh83 commented Apr 14, 2021 • edited Loading

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

pabloem left a comment

Choose a reason for hiding this comment

aaltay commented Apr 29, 2021

usingh83 commented May 2, 2021

pabloem commented May 3, 2021

usingh83 commented May 4, 2021

pabloem left a comment

Choose a reason for hiding this comment

pabloem commented May 18, 2021

usingh83 commented Apr 14, 2021 •

edited

Loading