fix(tableau): fixes some aspects being emitted multiple times #12258
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
ingestion
datahub-cyborg
bot
added
the
needs-review
sgomezvillamor
requested review from
hsheth2 and
mayurinehate
and removed request for
hsheth2
hsheth2
approved these changes
Jan 2, 2025
| optimized_query = copy.deepcopy(query_filter) | ||
|
|
||
| if query_filter.get(c.ID_WITH_IN): | ||
| optimized_query[c.ID_WITH_IN] = list(set(query_filter[c.ID_WITH_IN])) |
There was a problem hiding this comment.
could we use our OrderedSet type here? mainly to ensure consistent behavior across runs and make testing/mocking easier
There was a problem hiding this comment.
done in commit 877a50a
datahub-cyborg
bot
added
pending-submitter-merge
and removed
needs-review
sgomezvillamor
deleted the
feature/cus-3546-fix-tableau-duplicate-query-ids
branch
brock-acryl
pushed a commit
to brock-acryl/datahub
that referenced
this pull request
Jan 30, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When working with the Tableau API, this fix prevents duplicate values in query filters for certain queries. Duplicates in the query lead to duplicates in the result, causing some aspects to be emitted multiple times.
I've verified this beyond unit tests by running a local Tableau ingestion. On the left, you see the results with the
mastercode, and on the right, results with the fix. Notably, counters likechart.statusremain 57 in both scenarios, but manychartaspects are nearly doubled on the left. The same goes fordashboardanddatasetaspects.I pushed down this optimization as much as possible, to avoid dependency on the callers of
get_connection_objectsneeding to be aware of it.Checklist