Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: qualifiedName support + populating glue ARN #5952

Merged
merged 2 commits into from
Sep 16, 2022

Conversation

skrydal
Copy link
Collaborator

@skrydal skrydal commented Sep 15, 2022

This PR brings 2 changes:

  1. It seemed that ContainerProperties lacked qualifiedName field when querying GMS GraphQL (even though it was there in .pdl files). This PR adds this field of ContainerProperties to the graphQL definition and ContainerMapper so it can be retrieved for a container.
  2. Glue connector stores ARN for each entity it ingests (table and database) as qualifiedName. ARN for glue resources is not available in AWS REST API response (this is confirmed by AWS documentation), so we build it on the fly according to the methodology described by AWS docs.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX labels Sep 15, 2022
@github-actions
Copy link

Unit Test Results (metadata ingestion)

       8 files  ±0         8 suites  ±0   54m 57s ⏱️ - 1m 17s
   707 tests ±0     704 ✔️ +1  3 💤 ±0  0  - 1 
1 414 runs  ±0  1 408 ✔️ +1  6 💤 ±0  0  - 1 

Results for commit 732edd3. ± Comparison against base commit 66985335.

@github-actions
Copy link

Unit Test Results (build & test)

571 tests  +50   571 ✔️ +56   15m 20s ⏱️ + 1m 7s
141 suites +19       0 💤 ±  0 
141 files   +19       0  -   6 

Results for commit 732edd3. ± Comparison against base commit 66985335.

@maggiehays maggiehays added the community-contribution PR or Issue raised by member(s) of DataHub Community label Sep 15, 2022
Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Python code LGTM

Copy link
Collaborator

@jjoyce0510 jjoyce0510 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - Thank you!

@hsheth2 hsheth2 changed the title QualifiedName support for containers and setting ARN as qualifiedName for glue resources feat: qualifiedName support + populating glue ARN Sep 16, 2022
@hsheth2 hsheth2 merged commit a026c84 into datahub-project:master Sep 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community-contribution PR or Issue raised by member(s) of DataHub Community ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants