feat(domains): Adding backend for Asset Domains (p1) #3952

jjoyce0510 · 2022-01-22T00:43:27Z

Summary
This PR contains the models and GraphQL implementation required for the new asset domains feature. Following this PR will be a PR to add the UI for rendering asset domains.

The domain feature permits the creation and association of business "domains" with a custom set of assets. Domains can be thought of as logical collections spanning different metadata entities. Examples of domains include "finance", "marketing", "engineering", and so on.

Changes

The specific changes in this PR include:

Adding Entity for Domains
Adding "Domains" aspect, attaching to core 5 entities (dataset, chart, dashboard, dataflow, datajob)
Adding GraphQL Endpoints for listing, fetching, creating, and updating Domains.
Adding relevant privileges for managing, editing domains assigned to an entity.
Adding Recommendations candidate source for displaying top domains.

Status
Ready to merge.

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable)

… the board

github-actions · 2022-01-22T01:13:27Z

Unit Test Results (build & test)

  66 files +  6   66 suites +6 13m 16s ⏱️ -35s
581 tests +23 522 ✔️ +23 59 💤 ±0 0 ❌ ±0

Results for commit f32617a. ± Comparison against base commit 35ebca7.

♻️ This comment has been updated with latest results.

github-actions · 2022-01-22T01:14:53Z

Unit Test Results (metadata ingestion)

      5 files       5 suites 34m 59s ⏱️
  274 tests   273 ✔️   0 💤 1 ❌
1 259 runs 1 224 ✔️ 34 💤 1 ❌

For more details on these failures, see this check.

Results for commit f32617a.

♻️ This comment has been updated with latest results.

gabe-lyons · 2022-01-27T03:12:51Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/GmsGraphQLEngine.java

+            .dataFetcher("relationships", new AuthenticatedResolver<>(
+                new EntityRelationshipsResultResolver(graphClient)
+            ))


can you say more about domains having relationships? Is that something that currently exists or is this future proofing?

resolved offline!

gabe-lyons · 2022-01-27T03:14:44Z

...l-core/src/main/java/com/linkedin/datahub/graphql/resolvers/domain/CreateDomainResolver.java

+        throw new AuthorizationException("Unauthorized to perform this action. Please contact your DataHub administrator.");
+      }
+
+      // TODO: Add exists check. Currently this can override previously created domains.


you can take a look at how we do this in the add/create tag modal to prevent someone from creating an existing tag name

I plan to do this (and delete domain) in a followup based on demand

gabe-lyons · 2022-01-27T03:16:17Z

...core/src/main/java/com/linkedin/datahub/graphql/resolvers/domain/DomainEntitiesResolver.java

+      try {
+
+        final Criterion filterCriterion =  new Criterion()
+            .setField(DOMAINS_FIELD_NAME + ".keyword")


what does .keyword here do?

It makes sure the filter is an exact match -- useful for urns, where the default splits them by :

gabe-lyons · 2022-01-27T03:16:56Z

...core/src/main/java/com/linkedin/datahub/graphql/resolvers/domain/DomainEntitiesResolver.java

+    final QueryContext context = environment.getContext();
+    final String urn = ((Domain) environment.getSource()).getUrn();
+
+    final DomainEntitiesInput input = environment.getArgument(INPUT_ARG_NAME) != null
+        ? bindArgument(environment.getArgument(INPUT_ARG_NAME), DomainEntitiesInput.class)
+        : DEFAULT_ENTITIES_INPUT;
+
+    final String query = input.getQuery() != null ? input.getQuery() : DEFAULT_QUERY;
+    final int start = input.getStart() != null ? input.getStart() : DEFAULT_START;
+    final int count = input.getCount() != null ? input.getCount() : DEFAULT_COUNT;


it seems unnecessary to implement this search/pagination logic for every entity- isn't this mostly generic logic just the Criterion being domains specific?

Are you suggesting to not provide start and count?

gabe-lyons · 2022-01-27T03:40:08Z

...e/src/main/java/com/linkedin/datahub/graphql/resolvers/mutate/UpdateDescriptionResolver.java

+    switch (targetUrn.getEntityType()) {
+      case Constants.DATASET_ENTITY_NAME:
+        return updateDatasetDescription(targetUrn, input, environment.getContext());
+      case Constants.DOMAIN_ENTITY_NAME:
+        return updateDomainDescription(targetUrn, input, environment.getContext());


gabe-lyons · 2022-01-27T03:41:02Z

...core/src/main/java/com/linkedin/datahub/graphql/types/dashboard/mappers/DashboardMapper.java

+            } else if (DOMAINS_ASPECT_NAME.equals(name)) {
+                final Domains domains = new Domains(data);
+                // Currently we only take the first domain if it exists.
+                if (domains.getDomains().size() > 0) {
+                    result.setDomain(Domain.builder()
+                        .setType(EntityType.DOMAIN)
+                        .setUrn(domains.getDomains().get(0).toString()).build());
+                }


shouldn't we not need to do this bc the domains aspect isn't in the snapshot?

We are not using snapshots anymore! We're using batchGetV2, which fetches generic aspects. We still fetch in batch, as opposed to having 1 call per aspect

gabe-lyons · 2022-01-27T03:41:27Z

...core/src/main/java/com/linkedin/datahub/graphql/types/dashboard/mappers/DashboardMapper.java

@@ -79,6 +81,14 @@ public Dashboard apply(@Nonnull final EntityResponse entityResponse) {
                result.setInstitutionalMemory(InstitutionalMemoryMapper.map(new InstitutionalMemory(data)));
            } else if (GLOSSARY_TERMS_ASPECT_NAME.equals(name)) {
                result.setGlossaryTerms(GlossaryTermsMapper.map(new GlossaryTerms(data)));
+            } else if (DOMAINS_ASPECT_NAME.equals(name)) {
+                final Domains domains = new Domains(data);
+                // Currently we only take the first domain if it exists.


do we foresee this causing confusion?

Did the same for containers.. I think it causes less confusion when you see it as a document model:

dataset {
....
domains: [ ]
}

gabe-lyons · 2022-01-27T03:42:33Z

datahub-graphql-core/src/main/resources/entity.graphql

+    ownership: Ownership
+
+    """
+    References to internal resources related to the dataset


s/dataset/domain

datahub-graphql-core/src/main/resources/entity.graphql

gabe-lyons · 2022-01-27T03:44:13Z

...ta-models/src/main/pegasus/com/linkedin/metadata/recommendation/RecommendationRenderType.pdl

+  /**
+   * List of domains to search
+   */
+  DOMAIN_SEARCH_LIST


do we hide this in the case there are no domains? Or have a prompt to create a domain?

yup we hide it! in case of 0 domains

jjoyce0510 added 30 commits January 10, 2022 15:35

Adding Domain Models

94b5ddb

Fixing modeling names

a2685d6

Adding domains models

bad72b7

Finishing modeling domains

af10019

Adding domains to graphql

c970d9f

Fixing domains modeling

e71634d

Adding domain operations

8fe178a

fix up

3c85ede

Adding domains resolvers

56a3b7d

Adding domain

d7e3957

Check urn nullability

95b5985

Add domains

29dcb3f

Adding policy

0e89549

Merge branch 'master' into data-domains

26a957c

Merging master and finishing fetch domain capability

b43d76a

Adding institutional memory to domains

e9cdb1e

Adding domains to be searchable on the server

f488185

COMMON: Updating UI and Resolvers with Common Changes required across…

79c2e8c

… the board

COMMON: Migrating UTILS to use MetadataChangeProposal

ce26d19

Adding required things

35225f2

Adding canManageDomains privilege to MeResolver

b86d6f4

Adding list domains resolver

73568fb

Update description resolver changes

d2909b1

Adding missing domains resolver

dd111a5

adding domain recommendations

df1a8b9

Finalize Recommendation for domain

1c0eb51

change domain recs ordering

41aaf8c

Improving DomainEntitiesResolver

07942f7

Merge branch 'master' into data-domains

f8ff59e

Fix set bug

85d1f05

jjoyce0510 added 2 commits January 21, 2022 16:34

Merge branch 'master' into data-domains

87e346a

Removing unnecessary changes

6617b35

jjoyce0510 mentioned this pull request Jan 22, 2022

feat(domains): Adding UI for Asset Domains (P2) #3953

Closed

4 tasks

jjoyce0510 added 10 commits January 24, 2022 09:37

Merge branch 'master' into data-domains

a5ab6bc

Remove unused import

f27b743

Adding resolver + type tests

d621d4b

Require properties to be rendered

d69954c

Require key aspect

c94c8d7

Adding Domain Entities Resolver Test

753053f

Merge branch 'master' into data-domains

5b141c7

Merging latest master

99afc5e

Adding domains test

6beeee4

Removing unnecessary code

203d9e0

gabe-lyons reviewed Jan 27, 2022

View reviewed changes

datahub-graphql-core/src/main/resources/entity.graphql Show resolved Hide resolved

gabe-lyons reviewed Jan 27, 2022

View reviewed changes

gabe-lyons approved these changes Jan 27, 2022

View reviewed changes

Merge branch 'master' into data-domains

f32617a

shirshanka approved these changes Jan 27, 2022

View reviewed changes

shirshanka merged commit 282851e into datahub-project:master Jan 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(domains): Adding backend for Asset Domains (p1) #3952

feat(domains): Adding backend for Asset Domains (p1) #3952

jjoyce0510 commented Jan 22, 2022 •

edited

Loading

github-actions bot commented Jan 22, 2022 •

edited

Loading

github-actions bot commented Jan 22, 2022 •

edited

Loading

gabe-lyons Jan 27, 2022

jjoyce0510 Jan 27, 2022

gabe-lyons Jan 27, 2022

jjoyce0510 Jan 27, 2022

gabe-lyons Jan 27, 2022

jjoyce0510 Jan 27, 2022

gabe-lyons Jan 27, 2022

jjoyce0510 Jan 27, 2022

gabe-lyons Jan 27, 2022

gabe-lyons Jan 27, 2022

jjoyce0510 Jan 27, 2022

gabe-lyons Jan 27, 2022

jjoyce0510 Jan 27, 2022

gabe-lyons Jan 27, 2022

gabe-lyons Jan 27, 2022

jjoyce0510 Jan 27, 2022

feat(domains): Adding backend for Asset Domains (p1) #3952

feat(domains): Adding backend for Asset Domains (p1) #3952

Conversation

jjoyce0510 commented Jan 22, 2022 • edited Loading

Checklist

github-actions bot commented Jan 22, 2022 • edited Loading

Unit Test Results (build & test)

github-actions bot commented Jan 22, 2022 • edited Loading

Unit Test Results (metadata ingestion)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jjoyce0510 commented Jan 22, 2022 •

edited

Loading

github-actions bot commented Jan 22, 2022 •

edited

Loading

github-actions bot commented Jan 22, 2022 •

edited

Loading