🎉Source GitHub: Add more streams #5757

yevhenii-ldv · 2021-08-31T13:01:04Z

What

clones #5483.

How

Describe the solution

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions
Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

…-org-stream

yevhenii-ldv · 2021-08-31T13:01:55Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1186220113
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1186220113

avida · 2021-08-31T13:34:22Z

airbyte-integrations/connectors/source-github/source_github/source.py

@@ -98,11 +104,14 @@ def check_connection(self, logger: AirbyteLogger, config: Mapping[str, Any]) ->
    def streams(self, config: Mapping[str, Any]) -> List[Stream]:
        authenticator = self._get_authenticator(config["access_token"])
        repositories = self._generate_repositories(config=config, authenticator=authenticator)
+        organizations = list(set([org.split("/")[0] for org in repositories]))


list({org.split("/")[0] for org in repositories})

airbyte-integrations/connectors/source-github/source_github/streams.py

yevhenii-ldv · 2021-08-31T16:49:52Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1187001360
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1187001360

airbyte-integrations/connectors/source-github/source_github/source.py

sherifnada

@tovbinm thanks for the contribution! LGTM with some questions about minimization

sherifnada · 2021-08-31T22:28:53Z

airbyte-integrations/connectors/source-github/source_github/streams.py

+    API docs: https://docs.github.com/en/rest/reference/repos#list-organization-repositories
+    """
+
+    fields_to_minimize = ("owner",)


@tovbinm could we keep this un-minimized? or is it redundant with something else? IMO it's useful to keep this info here since the owner might not be a contributor present on any other stream in the user field

Yea we can keep it un-minimized.

I don't have enough access to push to this branch but one of you here can just update the schemas to refer to the shared user object schema.

sherifnada · 2021-08-31T22:30:39Z

airbyte-integrations/connectors/source-github/source_github/streams.py


-    def path(self, stream_slice: Mapping[str, Any] = None, **kwargs) -> str:
-        return f"orgs/{stream_slice['organization']}/repos"
+    fields_to_minimize = ("merged_by",)


same question as below about minimization

Yea we can keep it un-minimized.

yevhenii-ldv · 2021-09-01T11:53:13Z

/publish connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1190040159
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1190040159

cjwooo and others added 15 commits August 17, 2021 16:41

Add Organizations stream to Github Source

02f76cb

Add Repositories stream

2dea2a9

Add Repositories stream spec

417fcba

Add Branches stream

5141625

Add Tags stream

5ba0e11

Add ReviewComments stream

2569049

Merge branch 'master' into cwu/github-org-stream

1837043

Update with latest config changes

612c088

Add users stream

c8014a0

Add pull request stats stream

6ffb39d

Merge branch 'master' of github.com:airbytehq/airbyte into cwu/github…

c88bd5d

…-org-stream

Specify date-time fields

ef28e6e

ykurochkin updates

4b115c3

Merge origin/master into branch

1994a29

a little update code

bedc88e

github-actions bot added the area/connectors Connector related issues label Aug 31, 2021

jrhizor temporarily deployed to more-secrets August 31, 2021 13:04 Inactive

yevhenii-ldv mentioned this pull request Aug 31, 2021

Source GitHub: Add more streams #5483

Closed

36 tasks

yevhenii-ldv assigned yevhenii-ldv and unassigned yevhenii-ldv Aug 31, 2021

yevhenii-ldv requested review from Zirochkaa, avida and htrueman August 31, 2021 13:19

avida reviewed Aug 31, 2021

View reviewed changes

avida approved these changes Aug 31, 2021

View reviewed changes

htrueman approved these changes Aug 31, 2021

View reviewed changes

Zirochkaa suggested changes Aug 31, 2021

View reviewed changes

airbyte-integrations/connectors/source-github/source_github/streams.py Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-github/source_github/streams.py Show resolved Hide resolved

update code

fb6d2ab

yevhenii-ldv requested a review from Zirochkaa August 31, 2021 16:49

jrhizor temporarily deployed to more-secrets August 31, 2021 16:52 Inactive

Zirochkaa approved these changes Aug 31, 2021

View reviewed changes

yevhenii-ldv requested review from sherifnada, davinchia and tuliren August 31, 2021 19:26

sherifnada reviewed Aug 31, 2021

View reviewed changes

airbyte-integrations/connectors/source-github/source_github/source.py Show resolved Hide resolved

sherifnada approved these changes Aug 31, 2021

View reviewed changes

ykurochkin added 2 commits September 1, 2021 14:49

update changelogs, update code after review

b0ddddd

bump version

6018d17

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 1, 2021

jrhizor temporarily deployed to more-secrets September 1, 2021 11:55 Inactive

yevhenii-ldv merged commit d74c47a into master Sep 1, 2021

yevhenii-ldv deleted the cwu/github-org-stream branch September 1, 2021 12:02

jrhizor mentioned this pull request Sep 3, 2021

Bump Airbyte version from 0.29.13-alpha to 0.29.14-alpha #5831

Merged

igrankova added connectors/sources-api connectors/source/github labels Jan 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉Source GitHub: Add more streams #5757

🎉Source GitHub: Add more streams #5757

yevhenii-ldv commented Aug 31, 2021

yevhenii-ldv commented Aug 31, 2021 •

edited by github-actions bot

Loading

avida Aug 31, 2021

yevhenii-ldv commented Aug 31, 2021 •

edited by github-actions bot

Loading

sherifnada left a comment

sherifnada Aug 31, 2021

cjwooo Aug 31, 2021

cjwooo Aug 31, 2021

sherifnada Aug 31, 2021

cjwooo Aug 31, 2021

yevhenii-ldv commented Sep 1, 2021 •

edited by github-actions bot

Loading

🎉Source GitHub: Add more streams #5757

🎉Source GitHub: Add more streams #5757

Conversation

yevhenii-ldv commented Aug 31, 2021

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

yevhenii-ldv commented Aug 31, 2021 • edited by github-actions bot Loading

avida Aug 31, 2021

Choose a reason for hiding this comment

yevhenii-ldv commented Aug 31, 2021 • edited by github-actions bot Loading

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Aug 31, 2021

Choose a reason for hiding this comment

cjwooo Aug 31, 2021

Choose a reason for hiding this comment

cjwooo Aug 31, 2021

Choose a reason for hiding this comment

sherifnada Aug 31, 2021

Choose a reason for hiding this comment

cjwooo Aug 31, 2021

Choose a reason for hiding this comment

yevhenii-ldv commented Sep 1, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented Aug 31, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Aug 31, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Sep 1, 2021 •

edited by github-actions bot

Loading