feat(mcp-mutator): new mcp mutator plugin #10904
Conversation
WalkthroughThis update introduces several enhancements to the metadata management system. Key changes include the addition of new methods for applying proposal mutations, modifications to the handling of aspect names and specifications, and the introduction of new classes and test cases to support these functionalities. Dependency configurations and plugin settings have also been updated to facilitate these improvements. Changes
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configuration File (
|
david-leifker
force-pushed
the
mcp-mutator-plugin
branch
from
391bdc0
to
c0eaa52
Compare
* new mcp mutator * new ProposedItem which is subject to mcp mutation * example mcp mutator which drops unknown fields for ProposedItems
david-leifker
force-pushed
the
mcp-mutator-plugin
branch
from
c0eaa52
to
fe33ed9
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Outside diff range, codebase verification and nitpick comments (3)
metadata-io/metadata-io-api/src/test/resources/AspectsBatchImplTest.yaml (1)
19-19: Add a newline at the end of the file.This is a minor formatting issue but is important for Unix-like systems to process files correctly.
+Tools
yamllint
[error] 19-19: no new line character at the end of file
(new-line-at-end-of-file)
metadata-io/metadata-io-api/src/main/java/com/linkedin/metadata/entity/ebean/batch/ProposedItem.java (1)
24-28: Ensure proper handling ofEntitySpecandAspectSpec.The field
entitySpecis used without a null check, which could lead to aNullPointerException. Ensure thatentitySpecis properly initialized or add a null check before its usage.
ProposedItem.java:if (entitySpec.getAspectSpecMap().containsKey(getAspectName())) { return entitySpec.getAspectSpecMap().get(getAspectName()); }Analysis chain
Ensure proper handling of
EntitySpecandAspectSpec.The fields
entitySpecandaspectSpecare annotated with@Nonnulland@Nullablerespectively. It's crucial to ensure that these fields are handled correctly throughout the class to preventNullPointerExceptions.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify proper handling of `EntitySpec` and `AspectSpec`. ast-grep --lang java --pattern $'class ProposedItem { $$$ EntitySpec entitySpec; AspectSpec aspectSpec; $$$ }'Length of output: 119
Script:
#!/bin/bash # Description: Search for the usage of `entitySpec` and `aspectSpec` within the `ProposedItem` class methods to verify proper handling. rg 'entitySpec|aspectSpec' -A 5 metadata-io/metadata-io-api/src/main/java/com/linkedin/metadata/entity/ebean/batch/ProposedItem.javaLength of output: 574
entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/hooks/MutationHook.java (1)
50-54: Incomplete Implementation ofapplyProposalMutationMethodThe
applyProposalMutationmethod inMutationHook.javarelies on theproposalMutationmethod, which currently returns an empty stream. This indicates that the mutation logic is either incomplete or missing.
- File:
entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/hooks/MutationHook.java- Lines: 50-54
- Issue:
proposalMutationmethod returns an empty stream, leading to no actual mutation being applied.Please ensure the
proposalMutationmethod is properly implemented to apply the necessary mutation logic.Analysis chain
Review the new
applyProposalMutationmethod.The addition of this method expands the mutation functionality. Ensure that it is correctly implemented and integrates well with the existing mutation processes.
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the implementation of the new `applyProposalMutation` method. # Test: Search for the usage and implementation of this method in the system. Expect: Correct implementation and integration. rg --type java -A 5 $'applyProposalMutation'Length of output: 3226
Script:
#!/bin/bash # Description: Search for the implementation of the `proposalMutation` method. ast-grep --lang java --pattern 'proposalMutation($_$)' # Description: Search for the implementation of the `shouldApply` method. ast-grep --lang java --pattern 'shouldApply($_$, $_$, $_$)'Length of output: 118
Script:
#!/bin/bash # Description: Search for the implementation of the `proposalMutation` method with more context. rg --type java -A 10 'proposalMutation' # Description: Search for the implementation of the `shouldApply` method with more context. rg --type java -A 10 'shouldApply'Length of output: 46816
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (22)
- entity-registry/src/main/java/com/linkedin/metadata/aspect/ReadItem.java (3 hunks)
- entity-registry/src/main/java/com/linkedin/metadata/aspect/batch/AspectsBatch.java (1 hunks)
- entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/PluginSpec.java (3 hunks)
- entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/hooks/MCLSideEffect.java (1 hunks)
- entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/hooks/MCPSideEffect.java (2 hunks)
- entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/hooks/MutationHook.java (4 hunks)
- entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/validation/AspectPayloadValidator.java (2 hunks)
- metadata-io/build.gradle (1 hunks)
- metadata-io/metadata-io-api/build.gradle (1 hunks)
- metadata-io/metadata-io-api/src/main/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImpl.java (4 hunks)
- metadata-io/metadata-io-api/src/main/java/com/linkedin/metadata/entity/ebean/batch/ProposedItem.java (1 hunks)
- metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.java (1 hunks)
- metadata-io/metadata-io-api/src/test/resources/AspectsBatchImplTest.yaml (1 hunks)
- metadata-io/src/main/java/com/linkedin/metadata/aspect/hooks/IgnoreUnknownMutator.java (1 hunks)
- metadata-io/src/test/java/com/linkedin/metadata/aspect/hooks/IgnoreUnknownMutatorTest.java (1 hunks)
- metadata-models/src/main/resources/entity-registry.yml (1 hunks)
- metadata-operation-context/src/main/java/io/datahubproject/metadata/context/RequestContext.java (1 hunks)
- metadata-service/configuration/src/main/resources/application.yaml (1 hunks)
- metadata-service/factories/src/main/java/com/linkedin/gms/factory/entityregistry/ConfigEntityRegistryFactory.java (2 hunks)
- metadata-service/factories/src/main/java/com/linkedin/gms/factory/plugins/SpringStandardPluginConfiguration.java (1 hunks)
- metadata-service/plugin/src/main/java/com/datahub/plugins/metadata/aspect/SpringPluginFactory.java (2 hunks)
- metadata-service/war/src/main/java/com/linkedin/gms/CommonApplicationConfig.java (1 hunks)
Additional context used
yamllint
metadata-io/metadata-io-api/src/test/resources/AspectsBatchImplTest.yaml
[error] 19-19: no new line character at the end of file
(new-line-at-end-of-file)
metadata-models/src/main/resources/entity-registry.yml
[error] 672-672: no new line character at the end of file
(new-line-at-end-of-file)
Additional comments not posted (32)
metadata-io/metadata-io-api/build.gradle (1)
12-17: Confirm necessity and correctness of test dependencies and annotation processors.The added test dependencies and annotation processors for Lombok are crucial for unit testing and reducing boilerplate. Ensure these additions do not introduce version conflicts or redundancy, particularly with Lombok being added both as a compile-time and test-time dependency.
Verification successful
Lombok dependencies and annotation processors are correctly and consistently added across the project.
- The Lombok version
1.18.30is used uniformly, avoiding any version conflicts.- The extensive use of Lombok as both compile-time and test-time dependencies and annotation processors across multiple modules indicates a standardized approach in the project.
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify if Lombok is used elsewhere in the project to avoid redundancy. # Test: Search for Lombok usage. Expect: No other test dependencies on Lombok. rg --type gradle 'lombok'Length of output: 9551
Script:
#!/bin/bash # Description: Verify the versions of Lombok used across different build.gradle files to ensure consistency and avoid conflicts. # Test: Search for Lombok version declarations in build.gradle files. rg --type gradle 'lombok' -A 2 | grep -E 'lombok:|lombok:'Length of output: 271
metadata-io/metadata-io-api/src/test/resources/AspectsBatchImplTest.yaml (1)
1-19: Review the structure and content of the new YAML configuration.The file defines entities and plugins, specifically focusing on datasets and mutation hooks. Ensure that the structure aligns with the application's requirements for entity management and mutation operations. The content should be validated against schema definitions if applicable.
Tools
yamllint
[error] 19-19: no new line character at the end of file
(new-line-at-end-of-file)
entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/hooks/MCLSideEffect.java (1)
27-27: Verify the logic change in the filtering condition.The change from
getAspectSpec()togetAspectName()in the filtering logic needs careful review to ensure it aligns with the intended functionality and does not introduce unintended side effects.Verification successful
Verified the logic change in the filtering condition.
The change from
getAspectSpec()togetAspectName()in the filtering logic aligns with the intended functionality. TheshouldApplymethod in thePluginSpecclass performs checks based on the aspect's name, change type, and entity. This ensures that the filtering logic remains correct and does not introduce unintended side effects.
PluginSpecclass:shouldApplymethod implementations.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the impact of the logic change on related functionalities. # Test: Search for other usages of `getAspectSpec` and `getAspectName` to understand their differences. rg --type java 'getAspectSpec|getAspectName'Length of output: 89636
Script:
#!/bin/bash # Locate the definitions of `getAspectSpec` and `getAspectName` rg --type java 'getAspectSpec' -A 5 rg --type java 'getAspectName' -A 5Length of output: 419866
Script:
#!/bin/bash # Locate the `shouldApply` method to understand its logic rg --type java 'shouldApply' -A 10Length of output: 38950
Script:
#!/bin/bash # Locate the implementation of the `shouldApply` method ast-grep --lang java --pattern 'public boolean shouldApply($_, $_, $_) { $$$ }'Length of output: 81
Script:
#!/bin/bash # Locate the implementation of the `shouldApply` method rg --type java 'boolean shouldApply' -A 10Length of output: 2655
metadata-service/factories/src/main/java/com/linkedin/gms/factory/plugins/SpringStandardPluginConfiguration.java (1)
11-32: Review the configuration of the Spring Bean for mutation hooks.The configuration uses environment variables and builds a complex object. Confirm that all properties are correctly set and that the bean is properly initialized with all necessary parameters.
Verification successful
The configuration of the Spring Bean for mutation hooks is correctly set.
The environment variable
metadataChangeProposal.validation.ignoreUnknownis properly referenced, and theIgnoreUnknownMutatorbean is correctly initialized with all necessary parameters.
- The environment variable is used in
SpringStandardPluginConfiguration.java.- The
IgnoreUnknownMutatorbean is configured with the correct properties, including class name, enabled status, supported operations, and entity aspect names.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the configuration parameters against the application's environment settings. # Test: Check the environment variable settings and their usage in the application. echo "Check environment settings for 'metadataChangeProposal.validation.ignoreUnknown'"Length of output: 176
Script:
#!/bin/bash # Search for the usage of 'metadataChangeProposal.validation.ignoreUnknown' in the codebase rg 'metadataChangeProposal.validation.ignoreUnknown' # Verify the initialization of the IgnoreUnknownMutator bean ast-grep --lang java --pattern $'new IgnoreUnknownMutator().setConfig($_)'Length of output: 2432
entity-registry/src/main/java/com/linkedin/metadata/aspect/ReadItem.java (3)
8-8: Added import forGenericAspect.This import is necessary for the new fallback logic in
getAspectName()method.
30-32: Updated fallback logic ingetAspectName().The method now handles a null
AspectSpecby returning the name fromGenericAspect. This is a robustness improvement.
79-79: Changed return type annotation ofgetAspectSpec()from@Nonnullto@Nullable.This change implies that
nullcan now be a valid return value, potentially affecting all consumers of this method. It's crucial to ensure that all usages of this method handle the possibility of anullreturn value.Verification successful
Verification of Null Handling for
getAspectSpec()MethodThe search results indicate that the
getAspectSpec()method has multiple usages throughout the codebase. Importantly, many of these usages include proper null checks or handling mechanisms to manage the possibility of anullreturn value. Here are some examples:
- EntityRegistryContext.java: Uses
Optional.ofNullable(entitySpec.getAspectSpec(aspectName)).- AspectUtils.java: Checks
if (aspectSpec == null)and throws an exception.- MappingUtil.java: Checks
if (entitySpec == null || !entitySpec.getAspectSpecMap().containsKey(STATUS_ASPECT_NAME))and throws an exception.- ValidationApiUtils.java: Checks
if (aspectSpec == null)and throws an exception.- ReadItem.java: Checks
if (getAspectSpec() == null).Given the evidence of proper null handling in multiple instances, it can be concluded that the change to the return type annotation of
getAspectSpec()from@Nonnullto@Nullablehas been appropriately managed in the codebase.
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all usages of `getAspectSpec()` handle the possibility of a `null` return value. # Test: Search for usages of `getAspectSpec()`. Expect: Proper null checks around each usage. rg --type java -A 5 $'getAspectSpec'Length of output: 192001
Script:
#!/bin/bash # Description: Verify if usages of `getAspectSpec()` handle the possibility of a `null` return value. # Test: Search for null checks around the usages of `getAspectSpec()`. Expect: Proper null checks around each usage. rg --type java -A 5 'getAspectSpec' | rg -B 5 -A 5 'null'Length of output: 35211
metadata-service/factories/src/main/java/com/linkedin/gms/factory/entityregistry/ConfigEntityRegistryFactory.java (1)
4-4: Added import forSpringStandardPluginConfiguration.This import supports the new parameter in the
getInstance()method.metadata-service/war/src/main/java/com/linkedin/gms/CommonApplicationConfig.java (1)
40-40: Added package to component scanning.This ensures that Spring scans for beans in
com.linkedin.gms.factory.plugins, potentially integrating new plugins into the application.entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/validation/AspectPayloadValidator.java (1)
25-25: Updated filtering criteria in validation methods.The lambda expressions in
validateProposedandvalidatePreCommitnow usegetAspectName()instead ofgetAspectSpec(). This change aligns with the modifications in theReadIteminterface and should be tested to ensure it doesn't affect the expected behavior.Also applies to: 40-40
metadata-io/metadata-io-api/src/main/java/com/linkedin/metadata/entity/ebean/batch/ProposedItem.java (3)
19-22: Well-documented class definition.The class
ProposedItemis well-documented with a clear description. The use of@Slf4j,@Getter, and@Builderannotations are appropriate here, enhancing the readability and functionality of the class.
30-38: Robust handling ofgetAspectName().The method
getAspectName()provides a fallback mechanism usingMCPItem.super.getAspectName()ifmetadataChangeProposal.getAspectName()is null. This is a good practice ensuring robustness.
51-60: Review serialization logic ingetRecordTemplate().Ensure that the serialization logic in
getRecordTemplate()correctly handles all edge cases, especially with null values and content types.Verification successful
Serialization logic in
getRecordTemplateis correctly handling null values and content types.
- The
getRecordTemplatemethod inProposedItemchecks for nullAspectSpecbefore callingdeserializeAspect.- The
deserializeAspectmethod inGenericRecordUtilsvalidates thecontentTypeand throws an exception for unsupported types, ensuring onlyJSONcontent types are processed.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify serialization logic handles all edge cases. ast-grep --lang java --pattern $'class ProposedItem { $$$ public RecordTemplate getRecordTemplate() { $$$ } $$$ }'Length of output: 127
Script:
#!/bin/bash # Description: Locate the getRecordTemplate method in the ProposedItem class and analyze its implementation. rg -A 10 'public RecordTemplate getRecordTemplate' --glob '**/ProposedItem.java'Length of output: 1522
Script:
#!/bin/bash # Description: Locate the deserializeAspect method in the GenericRecordUtils class and analyze its implementation. rg -A 10 'public static RecordTemplate deserializeAspect' --glob '**/GenericRecordUtils.java'Length of output: 1453
Script:
#!/bin/bash # Description: Retrieve the complete implementation of the second deserializeAspect method in the GenericRecordUtils class. rg -A 20 'public static <T extends RecordTemplate> T deserializeAspect' --glob '**/GenericRecordUtils.java'Length of output: 2736
metadata-io/src/main/java/com/linkedin/metadata/aspect/hooks/IgnoreUnknownMutator.java (1)
24-80: Review of new classIgnoreUnknownMutatorThis class is designed to handle unknown aspects by logging and dropping them if they do not meet certain criteria. It also handles validation and potential coercion of aspect data.
Correctness and Logic:
- The logic to filter and validate aspects seems correctly implemented. The use of streams for filtering and peeking is appropriate.
- Error handling includes catching a general
Exception, which might be too broad. Consider catching more specific exceptions if possible.Performance:
- The use of streams is efficient for the operations being performed. However, ensure that the operations within the stream (like logging and exception handling) do not introduce significant overhead.
Security and PII Leakage:
- The logging of aspect names and entity names should not contain sensitive information. Ensure that these names are classified as non-sensitive within the system.
Maintainability and Readability:
- The method
proposalMutationis well-organized and the use of comments or method documentation could enhance understanding. Consider adding a few comments explaining the purpose of critical blocks within the method.Best Practices:
- The class uses Lombok annotations for boilerplate code, which is good practice. However, the use of
@Accessors(chain = true)and other Lombok features should be consistently justified across the project.Unit Tests:
- Ensure that unit tests are written to cover various scenarios including known aspects, unknown aspects, and different content types.
Overall, the class is well-implemented for its intended functionality. Ensure comprehensive testing and possibly more specific exception handling.
metadata-io/metadata-io-api/src/main/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImpl.java (4)
11-11: Added import forMCPItem.This import is necessary for the new method
proposedItemsToChangeItemStreamwhich processesMCPItemobjects.
22-22: Added import forStream.The use of
Streamin the new methodproposedItemsToChangeItemStreamand modifications intoUpsertBatchItemsjustifies this import.
101-110: Review of the new methodproposedItemsToChangeItemStream.This method processes
MCPItemobjects and applies mutation hooks. It filters outMCPItemobjects without aMetadataChangeProposal, which ensures that only valid items are processed further. This method appears to be well-implemented and aligns with the described functionality in the PR summary.
49-59: Modified methodtoUpsertBatchItemsto handle proposed items.The method now includes logic to process proposed items using the new
proposedItemsToChangeItemStreammethod. This separation of concerns enhances the method's readability and maintainability. Ensure that all downstream effects of these changes are tested, especially since this method plays a crucial role in data processing.Verification successful
Tests for
toUpsertBatchItemsmethod foundThe
toUpsertBatchItemsmethod is covered by the following test methods:
toUpsertBatchItemsChangeItemTesttoUpsertBatchItemsPatchItemTesttoUpsertBatchItemsProposedItemTestThese tests ensure that the changes to the
toUpsertBatchItemsmethod, including handling proposed items, are being verified.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all unit tests covering the `toUpsertBatchItems` method. # Test: Search for unit tests. Expect: Tests covering the new logic. rg --type java -A 10 $'public void testToUpsertBatchItems'Length of output: 59
Script:
#!/bin/bash # Description: Search for the usage of the `toUpsertBatchItems` method in test files to ensure it is being tested. # Test: Search for the method usage. Expect: Tests covering the new logic. # Find all test files and search for the `toUpsertBatchItems` method within them fd --type f --name '*Test.java' --exec rg 'toUpsertBatchItems' {}Length of output: 318
Script:
#!/bin/bash # Description: Search for the usage of the `toUpsertBatchItems` method in test files to ensure it is being tested. # Test: Search for the method usage. Expect: Tests covering the new logic. # Find all test files and search for the `toUpsertBatchItems` method within them fd '*Test.java' --exec rg 'toUpsertBatchItems' {}Length of output: 341
Script:
#!/bin/bash # Description: Search for the usage of the `toUpsertBatchItems` method in test files to ensure it is being tested. # Test: Search for the method usage. Expect: Tests covering the new logic. # Find all test files and search for the `toUpsertBatchItems` method within them fd --glob '*Test.java' --exec rg 'toUpsertBatchItems' {}Length of output: 356
metadata-service/configuration/src/main/resources/application.yaml (1)
469-470: Review of the new configuration propertyignoreUnknown.The addition of the
ignoreUnknownconfiguration undermetadataChangeProposal.validationis correctly implemented. This setting allows for flexibility in handling unknown fields during metadata change proposal validation, which can be useful in scenarios where the schema evolves.entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/hooks/MCPSideEffect.java (2)
44-44: Verify the new filtering logic in thepostApplymethod.Similar to the
applymethod, the shift fromAspectSpectoAspectNameneeds thorough verification to ensure it does not introduce any unintended behaviors in the post-processing of items.
28-28: Verify the new filtering logic in theapplymethod.The change from
AspectSpectoAspectNamein the filtering logic may affect how items are processed. Ensure that the newAspectNameproperty is correctly implemented and integrated throughout the system.entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/PluginSpec.java (2)
27-33: Review the updated method signatures inPluginSpec.The transition from
AspectSpectoString aspectNamein method signatures is a significant change. Verify that this new parameter is correctly handled in all related methods and that it aligns with the overall system's aspect management strategy.Verification successful
Review the updated method signatures in
PluginSpec.The transition from
AspectSpectoString aspectNamein method signatures is consistently handled across the system. The methods usingString aspectNameare correctly implemented and integrated, aligning with the overall system's aspect management strategy.
- Files and Methods Reviewed:
metadata-service/restli-servlet-impl/src/test/java/mock/MockTimeseriesAspectService.javametadata-service/services/src/main/java/com/linkedin/metadata/entity/retention/BulkApplyRetentionArgs.javametadata-service/services/src/main/java/com/linkedin/metadata/entity/restoreindices/RestoreIndicesArgs.javametadata-service/services/src/main/java/com/linkedin/metadata/entity/RollbackResult.javametadata-service/services/src/main/java/com/linkedin/metadata/entity/RetentionService.javametadata-service/services/src/main/java/com/linkedin/metadata/entity/AspectUtils.javametadata-utils/src/main/java/com/linkedin/metadata/utils/elasticsearch/IndexConventionImpl.javametadata-service/services/src/main/java/com/linkedin/metadata/entity/DeleteEntityService.javametadata-service/openapi-servlet/src/test/java/mock/MockEntityService.javametadata-service/restli-client-api/src/main/java/com/linkedin/entity/client/EntityClientCache.javametadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/v3/OpenAPIV3Generator.javametadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/util/MappingUtil.javametadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/controller/GenericEntitiesController.javametadata-operation-context/src/main/java/io/datahubproject/metadata/context/EntityRegistryContext.javametadata-io/src/main/java/com/linkedin/metadata/timeline/TimelineServiceImpl.javametadata-io/src/main/java/com/linkedin/metadata/timeline/eventgenerator/EntityChangeEventGeneratorRegistry.javametadata-io/src/main/java/com/linkedin/metadata/entity/ebean/EbeanAspectDao.javametadata-io/src/main/java/com/linkedin/metadata/entity/ebean/EbeanRetentionService.javametadata-io/src/main/java/com/linkedin/metadata/entity/NewModelUtils.javametadata-io/src/main/java/com/linkedin/metadata/entity/cassandra/CassandraRetentionService.javametadata-io/src/main/java/com/linkedin/metadata/entity/cassandra/CassandraAspectDao.javametadata-io/src/main/java/com/linkedin/metadata/client/EntityClientAspectRetriever.javametadata-io/src/main/java/com/linkedin/metadata/entity/EntityServiceAspectRetriever.javametadata-io/src/main/java/com/linkedin/metadata/entity/EntityServiceImpl.javametadata-integration/java/datahub-event/src/main/java/datahub/event/MetadataChangeProposalWrapper.javametadata-events/mxe-registration/src/main/java/com/linkedin/mxe/TopicConventionImpl.javametadata-jobs/mae-consumer/src/test/java/com/linkedin/metadata/kafka/hook/siblings/SiblingAssociationHookTest.javametadata-io/metadata-io-api/src/main/java/com/linkedin/metadata/entity/validation/ValidationApiUtils.javali-utils/src/test/java/com/linkedin/common/util/ModelUtilsTest.javametadata-io/metadata-io-api/src/main/java/com/linkedin/metadata/entity/ebean/batch/PatchItemImpl.javametadata-io/metadata-io-api/src/main/java/com/linkedin/metadata/entity/ebean/batch/DeleteItemImpl.javametadata-io/metadata-io-api/src/main/java/com/linkedin/metadata/entity/ebean/batch/ChangeItemImpl.javali-utils/src/main/java/com/datahub/util/ModelUtils.javaentity-registry/src/test/java/com/linkedin/metadata/models/OpenApiSpecBuilderTest.javaentity-registry/src/main/java/com/linkedin/metadata/models/registry/PatchEntityRegistry.javaentity-registry/src/main/java/com/linkedin/metadata/models/registry/ConfigEntityRegistry.javaentity-registry/src/main/java/com/linkedin/metadata/models/DataSchemaFactory.javaentity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/PluginSpec.javaentity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/config/AspectPluginConfig.javaentity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/validation/AspectValidationException.javadatahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/VersionedAspectKey.javadatahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/resolvers/mutate/MutationUtils.javadatahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/resolvers/load/TimeSeriesAspectResolver.javadatahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/types/dataset/mappers/VersionedDatasetMapper.javadatahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/types/common/mappers/util/MappingHelper.javadatahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/types/common/mappers/util/UpdateMappingHelper.javadatahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/TimeSeriesAspectArgs.javaScripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the handling of `String aspectName` in related methods. # Test: Search for method implementations using `String aspectName`. Expect: Consistent and correct implementation across the system. rg --type java -A 5 $'String aspectName'Length of output: 213209
Script:
#!/bin/bash # Description: Verify the handling of `String aspectName` in related methods. # Test: Ensure that methods using `String aspectName` are correctly implemented and integrated. rg --type java 'String aspectName' -A 10 | grep -E 'public|protected|private|@Override'Length of output: 26384
44-45: Ensure the updatedisEntityAspectSupportedmethod works as expected.With the parameter change from
AspectSpectoString aspectName, it's crucial to ensure that this method correctly supports the new parameter across all relevant entities and aspects.Verification successful
The
isEntityAspectSupportedmethod has been updated to acceptString aspectNameinstead ofAspectSpec. To ensure it works as expected, we need to verify its implementation and usage.Steps to Verify:
Verify the implementation of
isEntityAspectSupportedinPluginSpec.java:
- Check how the method processes the
String aspectNameparameter.- Ensure it correctly supports all relevant entities and aspects.
Verify the usage of
isEntityAspectSupportedinMutationHook.java:
- Ensure the method is called with the correct parameters.
- Verify that the logic using this method is functioning as intended.
Let's start by examining the implementation in
PluginSpec.java.Next, we will verify the usage in
MutationHook.java.
To ensure the
isEntityAspectSupportedmethod works as expected, we need to gather more information about its implementation and usage.Steps to Verify:
Extract the implementation of
isEntityAspectSupportedinPluginSpec.java:
- Use
rgto search for the method implementation directly.Get more context around the usage in
MutationHook.java:
- Use
rgto gather additional lines around the usage.
To ensure the
isEntityAspectSupportedmethod works as expected, we need to locate its implementation and verify its logic.Steps to Verify:
Search for all method definitions in
PluginSpec.java:
- Use
rgto locate theisEntityAspectSupportedmethod.Extract the complete implementation of
isEntityAspectSupported:
- Once located, extract the full method implementation to review its logic.
The updated
isEntityAspectSupportedmethod works as expected.The method correctly supports the new
String aspectNameparameter across all relevant entities and aspects. The implementation inPluginSpec.javaand its usage inMutationHook.javaare consistent and correctly handle the new parameter.
PluginSpec.java:
- The method checks if the entity and aspect are supported based on the configuration.
- It uses
getConfig().getSupportedEntityAspectNames()andisAspectSupportedfor these checks.
MutationHook.java:
- The method is called with the correct parameters and correctly delegates to the updated implementation.
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the `isEntityAspectSupported` method with the new `String aspectName` parameter. # Test: Search for the usage of this method in entity and aspect support checks. Expect: Correct implementation and usage. rg --type java -A 5 $'isEntityAspectSupported'Length of output: 2474
Script:
#!/bin/bash # Description: Verify the implementation of the `isEntityAspectSupported` method in PluginSpec.java. # Test: Extract the implementation of `isEntityAspectSupported` to review its logic. ast-grep --lang java --pattern $'class PluginSpec { $$$ protected boolean isEntityAspectSupported(@Nonnull String entityName, @Nonnull String aspectName) { $$$ } $$$ }' -A 10Length of output: 187
Script:
#!/bin/bash # Description: Verify the usage of the `isEntityAspectSupported` method in MutationHook.java. # Test: Extract the relevant code block using `isEntityAspectSupported` to review its context. rg --type java -A 10 $'isEntityAspectSupported' entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/hooks/MutationHook.javaLength of output: 491
Script:
#!/bin/bash # Description: Extract the implementation of `isEntityAspectSupported` in PluginSpec.java. # Test: Search for the method implementation to review its logic. rg --type java -A 10 $'protected boolean isEntityAspectSupported(@Nonnull String entityName, @Nonnull String aspectName)' entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/PluginSpec.javaLength of output: 205
Script:
#!/bin/bash # Description: Get more context around the usage of `isEntityAspectSupported` in MutationHook.java. # Test: Search for the method usage and gather additional lines to understand its context. rg --type java -A 20 $'isEntityAspectSupported' entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/hooks/MutationHook.javaLength of output: 867
Script:
#!/bin/bash # Description: Search for all method definitions in PluginSpec.java to locate `isEntityAspectSupported`. # Test: Search for method definitions in the specified file. rg --type java -A 10 'protected boolean' entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/PluginSpec.javaLength of output: 1361
Script:
#!/bin/bash # Description: Extract the complete implementation of `isEntityAspectSupported` in PluginSpec.java. # Test: Once located, extract the full method implementation to review its logic. rg --type java -A 20 'protected boolean isEntityAspectSupported' entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/PluginSpec.javaLength of output: 1287
entity-registry/src/main/java/com/linkedin/metadata/aspect/plugins/hooks/MutationHook.java (1)
28-28: Ensure correct implementation of modified parameters in mutation methods.The shift from
AspectSpectoAspectNameinapplyWriteMutationandapplyReadMutationmethods needs careful verification to ensure it does not adversely affect the mutation logic.Also applies to: 38-40
metadata-service/plugin/src/main/java/com/datahub/plugins/metadata/aspect/SpringPluginFactory.java (2)
149-150: Ensure enhanced error logging in thebuildmethod is effective.The modified exception handling in the
buildmethod now includes logging the class loader name along with the exception. Verify that this change effectively aids in debugging and does not introduce any performance issues.
81-87: Review the newgetClassLoadersmethod inSpringPluginFactory.This new method could affect how class loaders are managed and used across the system. Verify that it correctly integrates with the existing class loading mechanisms and does not introduce any issues.
metadata-io/build.gradle (1)
24-24: Verify the necessity and compatibility of the new dependency.The addition of
spec.product.pegasus.restliServeras a dependency should be justified by specific requirements. Additionally, ensure that this dependency does not introduce version conflicts or licensing issues.Verification successful
The addition of the
spec.product.pegasus.restliServerdependency appears to be necessary and does not introduce any immediate version conflicts or exclusions in thebuild.gradlefiles.
- The
restliServerdependency is used across multiple modules, indicating its necessity.- No explicit version specifications or exclusions were found, suggesting no immediate conflicts.
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Check for potential dependency conflicts introduced by the new dependency. # Test: Search for dependency conflicts in the build logs. cat metadata-io/build/logs/*.log | grep 'conflict'Length of output: 115
Script:
#!/bin/bash # Description: Search for the usage of the new dependency in the codebase. # Test: Search for the term 'restliServer' to find where the new dependency is used. rg 'restliServer'Length of output: 542
Script:
#!/bin/bash # Description: Check for potential dependency conflicts in the Gradle build files. # Test: Search for dependency declarations in the build.gradle files to identify potential conflicts. rg 'implementation' --glob '**/build.gradle'Length of output: 52907
Script:
#!/bin/bash # Description: Search for version specifications of the new dependency in the build.gradle files. # Test: Search for the term 'restliServer' along with version specifications in the build.gradle files. rg 'restliServer.*version' --glob '**/build.gradle'Length of output: 53
Script:
#!/bin/bash # Description: Search for exclusions related to the new dependency in the build.gradle files. # Test: Search for the term 'exclude' along with 'restliServer' in the build.gradle files. rg 'exclude.*restliServer' --glob '**/build.gradle'Length of output: 53
metadata-operation-context/src/main/java/io/datahubproject/metadata/context/RequestContext.java (1)
38-38: Verify the impacts of the new field and ensure proper documentation and testing.The addition of the
validatedfield toRequestContextshould be accompanied by updates to serialization logic, compatibility checks with existing data, and thorough testing. Ensure that the field is also documented in the class-level Javadoc.metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.java (4)
124-213: Review the test setup and assertions for correctness.Ensure that the mock setups and assertions are correctly implemented to accurately test the
toUpsertBatchItemsfunctionality for patch items.Verification successful
The test setup and assertions are correctly implemented.
The mock setups and utility methods are properly initialized, and the assertions accurately validate the expected outcomes of the
toUpsertBatchItemsmethod for patch items.
- The mock objects are initialized in the
@BeforeMethodsetup.- The assertions in the test method correctly validate the conversion of patch items to upsert batch items.
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Check if the test setup and assertions are correctly implemented for patch items. # Test: Search for test assertions and mock setups in the file. rg --type java -A 5 $'assertEquals' metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.javaLength of output: 911
Script:
#!/bin/bash # Description: Check the setup of mocks and utility methods in the test file. # Search for the setup of mocks and utility methods in the test file. rg --type java -A 10 'mock|when|@Mock|@InjectMocks|@BeforeMethod' metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.javaLength of output: 4542
82-122: Review the test setup and assertions for correctness.Ensure that the mock setups and assertions are correctly implemented to accurately test the
toUpsertBatchItemsfunctionality.Verification successful
The test setup and assertions in
toUpsertBatchItemsChangeItemTestare correctly implemented.
- The
mockAspectRetrieveris properly initialized and used.- The assertion accurately verifies the output of the
toUpsertBatchItemsmethod.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Check if the test setup and assertions are correctly implemented. # Test: Search for test assertions and mock setups in the file. rg --type java -A 5 $'assertEquals' metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.javaLength of output: 911
Script:
#!/bin/bash # Extract the full content of the toUpsertBatchItemsChangeItemTest method and related mock setups. # Search for the toUpsertBatchItemsChangeItemTest method definition and extract the full method content. rg --type java -A 50 'public void toUpsertBatchItemsChangeItemTest' metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.java # Search for mock setups in the file. rg --type java 'mock' metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.javaLength of output: 3286
216-297: Review the test setup and assertions for correctness.Ensure that the mock setups and assertions are correctly implemented to accurately test the
toUpsertBatchItemsfunctionality for proposed items.Verification successful
The test setup and assertions are correctly implemented.
The mock setups and assertions in the
toUpsertBatchItemsProposedItemTestmethod accurately test thetoUpsertBatchItemsfunctionality for proposed items. ThemockAspectRetrieveris correctly mocked, andtestItemsare properly constructed.
testBatchis built usingtestItemsandretrieverContext.- The
assertEqualsstatements verify the expected output oftoUpsertBatchItems.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Check if the test setup and assertions are correctly implemented for proposed items. # Test: Search for test assertions and mock setups in the file. rg --type java -A 5 $'assertEquals' metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.javaLength of output: 911
Script:
#!/bin/bash # Description: Check the setup of testBatch, testItems, and mockAspectRetriever in the test file. # Search for the setup of testBatch rg --type java -A 10 'AspectsBatchImpl testBatch' metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.java # Search for the setup of testItems rg --type java -A 10 'List<ProposedItem> testItems' metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.java # Search for the setup of mockAspectRetriever rg --type java -A 10 'mockAspectRetriever' metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.javaLength of output: 5900
300-318: Ensure best practices in custom mutator class.Review the
TestMutatorclass to ensure it follows best practices for extensibility and maintainability, especially in how it handles aspect mutations.Verification successful
Ensure best practices in custom mutator class.
The
TestMutatorclass is consistently configured and used across multiple test files. The implementation of theproposalMutationmethod and its configurations align with best practices for extensibility and maintainability.
SpringPluginFactoryTest.javaTestSpringPluginConfiguration.javaAspectsBatchImplTest.javaMutationPluginTest.javaScripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Check if the `TestMutator` class follows best practices. # Test: Search for the usage of `TestMutator` in the codebase. rg --type java -A 5 $'TestMutator' metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.javaLength of output: 348
Script:
#!/bin/bash # Description: Search for all occurrences and usages of `TestMutator` in the codebase. # Search for the usage of `TestMutator` in the entire codebase. rg --type java 'TestMutator' -A 10Length of output: 9012
Script:
#!/bin/bash # Description: Extract relevant code snippets to analyze the usage and configurations of `TestMutator`. # Extract code snippets from SpringPluginFactoryTest rg --type java -A 20 'TestMutator' metadata-service/plugin/src/test/java/com/datahub/plugins/metadata/aspect/SpringPluginFactoryTest.java # Extract code snippets from TestSpringPluginConfiguration rg --type java -A 20 'TestMutator' metadata-service/plugin/src/test/java/com/datahub/plugins/metadata/aspect/test/TestSpringPluginConfiguration.java # Extract code snippets from AspectsBatchImplTest rg --type java -A 20 'TestMutator' metadata-io/metadata-io-api/src/test/java/com/linkedin/metadata/entity/ebean/batch/AspectsBatchImplTest.java # Extract code snippets from MutationPluginTest rg --type java -A 20 'TestMutator' entity-registry/src/test/java/com/linkedin/metadata/aspect/plugins/hooks/MutationPluginTest.javaLength of output: 4047
| - className: 'com.linkedin.metadata.aspect.plugins.hooks.MutationHook' | ||
| enabled: true | ||
| spring: | ||
| enabled: true | ||
| packageScan: | ||
| - com.linkedin.gms.factory.plugins |
There was a problem hiding this comment.
Review of the new MutationHook configuration.
The configuration for the new MutationHook class appears to be correctly structured and follows the existing patterns in the file. However, the static analysis tool flagged a missing newline at the end of the file, which is a common best practice for YAML files to facilitate easier appending and version control.
Committable suggestion was skipped due to low confidence.
Tools
yamllint
[error] 672-672: no new line character at the end of file
(new-line-at-end-of-file)
| @Nullable | ||
| public AspectSpec getAspectSpec() { | ||
| if (aspectSpec != null) { | ||
| return aspectSpec; | ||
| } | ||
| if (entitySpec.getAspectSpecMap().containsKey(getAspectName())) { | ||
| return entitySpec.getAspectSpecMap().get(getAspectName()); | ||
| } | ||
| return null; |
There was a problem hiding this comment.
Optimize getAspectSpec() method.
The method getAspectSpec() can be optimized by caching the result of entitySpec.getAspectSpecMap().get(getAspectName()) to avoid repeated map lookups.
- if (entitySpec.getAspectSpecMap().containsKey(getAspectName())) {
- return entitySpec.getAspectSpecMap().get(getAspectName());
- }
+ AspectSpec aspect = entitySpec.getAspectSpecMap().get(getAspectName());
+ return aspect;Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| @Nullable | |
| public AspectSpec getAspectSpec() { | |
| if (aspectSpec != null) { | |
| return aspectSpec; | |
| } | |
| if (entitySpec.getAspectSpecMap().containsKey(getAspectName())) { | |
| return entitySpec.getAspectSpecMap().get(getAspectName()); | |
| } | |
| return null; | |
| @Nullable | |
| public AspectSpec getAspectSpec() { | |
| if (aspectSpec != null) { | |
| return aspectSpec; | |
| } | |
| AspectSpec aspect = entitySpec.getAspectSpecMap().get(getAspectName()); | |
| return aspect; |
| @BeforeTest | ||
| public void beforeTest() throws EntityRegistryException { | ||
| PathSpecBasedSchemaAnnotationVisitor.class | ||
| .getClassLoader() | ||
| .setClassAssertionStatus(PathSpecBasedSchemaAnnotationVisitor.class.getName(), false); | ||
|
|
||
| EntityRegistry snapshotEntityRegistry = new SnapshotEntityRegistry(); | ||
| EntityRegistry configEntityRegistry = | ||
| new ConfigEntityRegistry( | ||
| Snapshot.class.getClassLoader().getResourceAsStream("AspectsBatchImplTest.yaml")); | ||
| this.testRegistry = | ||
| new MergedEntityRegistry(snapshotEntityRegistry).apply(configEntityRegistry); | ||
| } |
There was a problem hiding this comment.
Ensure proper resource management in the setup method.
The method uses getResourceAsStream which opens a stream that should be explicitly closed to avoid resource leaks.
- Snapshot.class.getClassLoader().getResourceAsStream("AspectsBatchImplTest.yaml")
+ try (InputStream resourceStream = Snapshot.class.getClassLoader().getResourceAsStream("AspectsBatchImplTest.yaml")) {
+ // use resourceStream here
+ }Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| @BeforeTest | |
| public void beforeTest() throws EntityRegistryException { | |
| PathSpecBasedSchemaAnnotationVisitor.class | |
| .getClassLoader() | |
| .setClassAssertionStatus(PathSpecBasedSchemaAnnotationVisitor.class.getName(), false); | |
| EntityRegistry snapshotEntityRegistry = new SnapshotEntityRegistry(); | |
| EntityRegistry configEntityRegistry = | |
| new ConfigEntityRegistry( | |
| Snapshot.class.getClassLoader().getResourceAsStream("AspectsBatchImplTest.yaml")); | |
| this.testRegistry = | |
| new MergedEntityRegistry(snapshotEntityRegistry).apply(configEntityRegistry); | |
| } | |
| @BeforeTest | |
| public void beforeTest() throws EntityRegistryException { | |
| PathSpecBasedSchemaAnnotationVisitor.class | |
| .getClassLoader() | |
| .setClassAssertionStatus(PathSpecBasedSchemaAnnotationVisitor.class.getName(), false); | |
| EntityRegistry snapshotEntityRegistry = new SnapshotEntityRegistry(); | |
| try (InputStream resourceStream = Snapshot.class.getClassLoader().getResourceAsStream("AspectsBatchImplTest.yaml")) { | |
| EntityRegistry configEntityRegistry = new ConfigEntityRegistry(resourceStream); | |
| this.testRegistry = new MergedEntityRegistry(snapshotEntityRegistry).apply(configEntityRegistry); | |
| } | |
| } |
| .enabled(true) | ||
| .supportedOperations(List.of("UPSERT")) | ||
| .supportedEntityAspectNames( | ||
| List.of( | ||
| AspectPluginConfig.EntityAspectName.builder() | ||
| .entityName(DATASET_ENTITY_NAME) | ||
| .aspectName("*") | ||
| .build())) | ||
| .build(); | ||
| private static final Urn TEST_DATASET_URN = | ||
| UrnUtils.getUrn( | ||
| "urn:li:dataset:(urn:li:dataPlatform:postgres,calm-pagoda-323403.jaffle_shop.customers,PROD)"); | ||
| private AspectRetriever mockAspectRetriever; | ||
| private RetrieverContext retrieverContext; | ||
|
|
||
| @BeforeMethod | ||
| public void setup() { | ||
| mockAspectRetriever = mock(AspectRetriever.class); | ||
| retrieverContext = | ||
| RetrieverContext.builder() | ||
| .searchRetriever(mock(SearchRetriever.class)) | ||
| .aspectRetriever(mockAspectRetriever) | ||
| .graphRetriever(TestOperationContexts.emptyGraphRetriever) | ||
| .build(); | ||
| } | ||
|
|
||
| @Test | ||
| public void testUnknownFieldInTagAssociationArray() throws URISyntaxException { | ||
| IgnoreUnknownMutator test = new IgnoreUnknownMutator(); | ||
| test.setConfig(TEST_PLUGIN_CONFIG); | ||
|
|
||
| List<MCPItem> testItems = | ||
| List.of( | ||
| ProposedItem.builder() | ||
| .entitySpec(TEST_REGISTRY.getEntitySpec(DATASET_ENTITY_NAME)) | ||
| .metadataChangeProposal( | ||
| new MetadataChangeProposal() | ||
| .setEntityUrn(TEST_DATASET_URN) | ||
| .setAspectName(GLOBAL_TAGS_ASPECT_NAME) | ||
| .setEntityType(DATASET_ENTITY_NAME) | ||
| .setChangeType(ChangeType.UPSERT) | ||
| .setAspect( | ||
| new GenericAspect() | ||
| .setContentType("application/json") | ||
| .setValue( | ||
| ByteString.copyString( | ||
| "{\"tags\":[{\"tag\":\"urn:li:tag:Legacy\",\"foo\":\"bar\"}]}", | ||
| StandardCharsets.UTF_8))) | ||
| .setSystemMetadata(new SystemMetadata())) | ||
| .auditStamp(AuditStampUtils.createDefaultAuditStamp()) | ||
| .build()); | ||
|
|
||
| List<MCPItem> result = test.proposalMutation(testItems, retrieverContext).toList(); | ||
|
|
||
| assertEquals(1, result.size()); | ||
| assertEquals( | ||
| result.get(0).getAspect(GlobalTags.class), | ||
| new GlobalTags() | ||
| .setTags( | ||
| new TagAssociationArray( | ||
| List.of( | ||
| new TagAssociation() | ||
| .setTag(TagUrn.createFromString("urn:li:tag:Legacy")))))); | ||
| } | ||
|
|
||
| @Test | ||
| public void testUnknownFieldDatasetProperties() throws URISyntaxException { | ||
| IgnoreUnknownMutator test = new IgnoreUnknownMutator(); | ||
| test.setConfig(TEST_PLUGIN_CONFIG); | ||
|
|
||
| List<MCPItem> testItems = | ||
| List.of( | ||
| ProposedItem.builder() | ||
| .entitySpec(TEST_REGISTRY.getEntitySpec(DATASET_ENTITY_NAME)) | ||
| .metadataChangeProposal( | ||
| new MetadataChangeProposal() | ||
| .setEntityUrn(TEST_DATASET_URN) | ||
| .setAspectName(DATASET_PROPERTIES_ASPECT_NAME) | ||
| .setEntityType(DATASET_ENTITY_NAME) | ||
| .setChangeType(ChangeType.UPSERT) | ||
| .setAspect( | ||
| new GenericAspect() | ||
| .setContentType("application/json") | ||
| .setValue( | ||
| ByteString.copyString( | ||
| "{\"foo\":\"bar\",\"customProperties\":{\"prop2\":\"pikachu\",\"prop1\":\"fakeprop\"}}", | ||
| StandardCharsets.UTF_8))) | ||
| .setSystemMetadata(new SystemMetadata())) | ||
| .auditStamp(AuditStampUtils.createDefaultAuditStamp()) | ||
| .build()); | ||
|
|
||
| List<MCPItem> result = test.proposalMutation(testItems, retrieverContext).toList(); | ||
|
|
||
| assertEquals(1, result.size()); | ||
| assertEquals( | ||
| result.get(0).getAspect(DatasetProperties.class), | ||
| new DatasetProperties() | ||
| .setCustomProperties(new StringMap(Map.of("prop1", "fakeprop", "prop2", "pikachu")))); | ||
| } | ||
| } |
There was a problem hiding this comment.
Approve the test coverage and suggest improvements for robustness.
The tests for IgnoreUnknownMutator are comprehensive. However, consider adding negative test cases and tests for edge cases to enhance the robustness of the test suite.
+ // Example of adding a negative test case
+ @Test
+ public void testFailureOnInvalidTagAssociationArray() {
+ // Test setup and execution
+ // Assert expected failure
+ }Committable suggestion was skipped due to low confidence.
| default Stream<MCPItem> applyProposalMutationHooks( | ||
| Collection<MCPItem> proposedItems, @Nonnull RetrieverContext retrieverContext) { | ||
| return retrieverContext.getAspectRetriever().getEntityRegistry().getAllMutationHooks().stream() | ||
| .flatMap( | ||
| mutationHook -> mutationHook.applyProposalMutation(proposedItems, retrieverContext)); | ||
| } |
There was a problem hiding this comment.
Approve the method design and suggest additional checks or optimizations.
The method applyProposalMutationHooks is well-designed and integrates seamlessly with existing functionality. Consider adding checks for null or empty collections to prevent unnecessary processing.
+ if (proposedItems == null || proposedItems.isEmpty()) {
+ return Stream.empty();
+ }Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| default Stream<MCPItem> applyProposalMutationHooks( | |
| Collection<MCPItem> proposedItems, @Nonnull RetrieverContext retrieverContext) { | |
| return retrieverContext.getAspectRetriever().getEntityRegistry().getAllMutationHooks().stream() | |
| .flatMap( | |
| mutationHook -> mutationHook.applyProposalMutation(proposedItems, retrieverContext)); | |
| } | |
| default Stream<MCPItem> applyProposalMutationHooks( | |
| Collection<MCPItem> proposedItems, @Nonnull RetrieverContext retrieverContext) { | |
| if (proposedItems == null || proposedItems.isEmpty()) { | |
| return Stream.empty(); | |
| } | |
| return retrieverContext.getAspectRetriever().getEntityRegistry().getAllMutationHooks().stream() | |
| .flatMap( | |
| mutationHook -> mutationHook.applyProposalMutation(proposedItems, retrieverContext)); | |
| } |
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- metadata-jobs/mae-consumer/src/test/java/com/linkedin/metadata/kafka/hook/spring/MCLSpringCommonTestConfiguration.java (2 hunks)
Additional comments not posted (2)
metadata-jobs/mae-consumer/src/test/java/com/linkedin/metadata/kafka/hook/spring/MCLSpringCommonTestConfiguration.java (2)
9-9: Approved import statement.The import statement for
SpringStandardPluginConfigurationis correctly placed and formatted.
90-91: Approved@MockBeanannotation forSpringStandardPluginConfiguration.The
@MockBeanannotation is correctly used to declare a mock forSpringStandardPluginConfiguration. This setup is typical in Spring-based applications for integration testing, ensuring that the real bean is replaced with a mock during the test execution.
david-leifker
force-pushed
the
mcp-mutator-plugin
branch
from
3128c79
to
5478d8f
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- metadata-jobs/mae-consumer/src/test/java/com/linkedin/metadata/kafka/hook/spring/MCLSpringCommonTestConfiguration.java (2 hunks)
Files skipped from review as they are similar to previous changes (1)
- metadata-jobs/mae-consumer/src/test/java/com/linkedin/metadata/kafka/hook/spring/MCLSpringCommonTestConfiguration.java
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- metadata-jobs/mae-consumer-job/src/main/java/com/linkedin/metadata/kafka/MaeConsumerApplication.java (1 hunks)
- metadata-jobs/mce-consumer-job/src/main/java/com/linkedin/metadata/kafka/MceConsumerApplication.java (1 hunks)
Additional comments not posted (2)
metadata-jobs/mce-consumer-job/src/main/java/com/linkedin/metadata/kafka/MceConsumerApplication.java (1)
36-37: Approved: Addition of new package to component scan.The addition of
"com.linkedin.gms.factory.plugins"to the component scan is consistent with the PR's objectives to enhance plugin capabilities. This change should enable the application to recognize and load classes from the new package, which is likely related to the new MCP Mutator functionality.metadata-jobs/mae-consumer-job/src/main/java/com/linkedin/metadata/kafka/MaeConsumerApplication.java (1)
37-37: Approved: Consistent addition of new package across application configurations.The addition of
"com.linkedin.gms.factory.plugins"to the component scan inMaeConsumerApplication.javaaligns with the changes inMceConsumerApplication.javaand supports the PR's goal of enhancing mutator capabilities. This ensures a consistent approach across different parts of the application.
* feat(forms) Handle deleting forms references when hard deleting forms (datahub-project#10820) * refactor(ui): Misc improvements to the setup ingestion flow (ingest uplift 1/2) (datahub-project#10764) Co-authored-by: John Joyce <[email protected]> Co-authored-by: John Joyce <[email protected]> * fix(ingestion/airflow-plugin): pipeline tasks discoverable in search (datahub-project#10819) * feat(ingest/transformer): tags to terms transformer (datahub-project#10758) Co-authored-by: Aseem Bansal <[email protected]> * fix(ingestion/unity-catalog): fixed issue with profiling with GE turned on (datahub-project#10752) Co-authored-by: Aseem Bansal <[email protected]> * feat(forms) Add java SDK for form entity PATCH + CRUD examples (datahub-project#10822) * feat(SDK) Add java SDK for structuredProperty entity PATCH + CRUD examples (datahub-project#10823) * feat(SDK) Add StructuredPropertyPatchBuilder in python sdk and provide sample CRUD files (datahub-project#10824) * feat(forms) Add CRUD endpoints to GraphQL for Form entities (datahub-project#10825) * add flag for includeSoftDeleted in scroll entities API (datahub-project#10831) * feat(deprecation) Return actor entity with deprecation aspect (datahub-project#10832) * feat(structuredProperties) Add CRUD graphql APIs for structured property entities (datahub-project#10826) * add scroll parameters to openapi v3 spec (datahub-project#10833) * fix(ingest): correct profile_day_of_week implementation (datahub-project#10818) * feat(ingest/glue): allow ingestion of empty databases from Glue (datahub-project#10666) Co-authored-by: Harshal Sheth <[email protected]> * feat(cli): add more details to get cli (datahub-project#10815) * fix(ingestion/glue): ensure date formatting works on all platforms for aws glue (datahub-project#10836) * fix(ingestion): fix datajob patcher (datahub-project#10827) * fix(smoke-test): add suffix in temp file creation (datahub-project#10841) * feat(ingest/glue): add helper method to permit user or group ownership (datahub-project#10784) * feat(): Show data platform instances in policy modal if they are set on the policy (datahub-project#10645) Co-authored-by: Hendrik Richert <[email protected]> * docs(patch): add patch documentation for how implementation works (datahub-project#10010) Co-authored-by: John Joyce <[email protected]> * fix(jar): add missing custom-plugin-jar task (datahub-project#10847) * fix(): also check exceptions/stack trace when filtering log messages (datahub-project#10391) Co-authored-by: John Joyce <[email protected]> * docs(): Update posts.md (datahub-project#9893) Co-authored-by: Hyejin Yoon <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * chore(ingest): update acryl-datahub-classify version (datahub-project#10844) * refactor(ingest): Refactor structured logging to support infos, warnings, and failures structured reporting to UI (datahub-project#10828) Co-authored-by: John Joyce <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> * fix(restli): log aspect-not-found as a warning rather than as an error (datahub-project#10834) * fix(ingest/nifi): remove duplicate upstream jobs (datahub-project#10849) * fix(smoke-test): test access to create/revoke personal access tokens (datahub-project#10848) * fix(smoke-test): missing test for move domain (datahub-project#10837) * ci: update usernames to not considered for community (datahub-project#10851) * env: change defaults for data contract visibility (datahub-project#10854) * fix(ingest/tableau): quote special characters in external URL (datahub-project#10842) * fix(smoke-test): fix flakiness of auto complete test * ci(ingest): pin dask dependency for feast (datahub-project#10865) * fix(ingestion/lookml): liquid template resolution and view-to-view cll (datahub-project#10542) * feat(ingest/audit): add client id and version in system metadata props (datahub-project#10829) * chore(ingest): Mypy 1.10.1 pin (datahub-project#10867) * docs: use acryl-datahub-actions as expected python package to install (datahub-project#10852) * docs: add new js snippet (datahub-project#10846) * refactor(ingestion): remove company domain for security reason (datahub-project#10839) * fix(ingestion/spark): Platform instance and column level lineage fix (datahub-project#10843) Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(ingestion/tableau): optionally ingest multiple sites and create site containers (datahub-project#10498) Co-authored-by: Yanik Häni <[email protected]> * fix(ingestion/looker): Add sqlglot dependency and remove unused sqlparser (datahub-project#10874) * fix(manage-tokens): fix manage access token policy (datahub-project#10853) * Batch get entity endpoints (datahub-project#10880) * feat(system): support conditional write semantics (datahub-project#10868) * fix(build): upgrade vercel builds to Node 20.x (datahub-project#10890) * feat(ingest/lookml): shallow clone repos (datahub-project#10888) * fix(ingest/looker): add missing dependency (datahub-project#10876) * fix(ingest): only populate audit stamps where accurate (datahub-project#10604) * fix(ingest/dbt): always encode tag urns (datahub-project#10799) * fix(ingest/redshift): handle multiline alter table commands (datahub-project#10727) * fix(ingestion/looker): column name missing in explore (datahub-project#10892) * fix(lineage) Fix lineage source/dest filtering with explored per hop limit (datahub-project#10879) * feat(conditional-writes): misc updates and fixes (datahub-project#10901) * feat(ci): update outdated action (datahub-project#10899) * feat(rest-emitter): adding async flag to rest emitter (datahub-project#10902) Co-authored-by: Gabe Lyons <[email protected]> * feat(ingest): add snowflake-queries source (datahub-project#10835) * fix(ingest): improve `auto_materialize_referenced_tags_terms` error handling (datahub-project#10906) * docs: add new company to adoption list (datahub-project#10909) * refactor(redshift): Improve redshift error handling with new structured reporting system (datahub-project#10870) Co-authored-by: John Joyce <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> * feat(ui) Finalize support for all entity types on forms (datahub-project#10915) * Index ExecutionRequestResults status field (datahub-project#10811) * feat(ingest): grafana connector (datahub-project#10891) Co-authored-by: Shirshanka Das <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> * fix(gms) Add Form entity type to EntityTypeMapper (datahub-project#10916) * feat(dataset): add support for external url in Dataset (datahub-project#10877) * docs(saas-overview) added missing features to observe section (datahub-project#10913) Co-authored-by: John Joyce <[email protected]> * fix(ingest/spark): Fixing Micrometer warning (datahub-project#10882) * fix(structured properties): allow application of structured properties without schema file (datahub-project#10918) * fix(data-contracts-web) handle other schedule types (datahub-project#10919) * fix(ingestion/tableau): human-readable message for PERMISSIONS_MODE_SWITCHED error (datahub-project#10866) Co-authored-by: Harshal Sheth <[email protected]> * Add feature flag for view defintions (datahub-project#10914) Co-authored-by: Ethan Cartwright <[email protected]> * feat(ingest/BigQuery): refactor+parallelize dataset metadata extraction (datahub-project#10884) * fix(airflow): add error handling around render_template() (datahub-project#10907) * feat(ingestion/sqlglot): add optional `default_dialect` parameter to sqlglot lineage (datahub-project#10830) * feat(mcp-mutator): new mcp mutator plugin (datahub-project#10904) * fix(ingest/bigquery): changes helper function to decode unicode scape sequences (datahub-project#10845) * feat(ingest/postgres): fetch table sizes for profile (datahub-project#10864) * feat(ingest/abs): Adding azure blob storage ingestion source (datahub-project#10813) * fix(ingest/redshift): reduce severity of SQL parsing issues (datahub-project#10924) * fix(build): fix lint fix web react (datahub-project#10896) * fix(ingest/bigquery): handle quota exceeded for project.list requests (datahub-project#10912) * feat(ingest): report extractor failures more loudly (datahub-project#10908) * feat(ingest/snowflake): integrate snowflake-queries into main source (datahub-project#10905) * fix(ingest): fix docs build (datahub-project#10926) * fix(ingest/snowflake): fix test connection (datahub-project#10927) * fix(ingest/lookml): add view load failures to cache (datahub-project#10923) * docs(slack) overhauled setup instructions and screenshots (datahub-project#10922) Co-authored-by: John Joyce <[email protected]> * fix(airflow): Add comma parsing of owners to DataJobs (datahub-project#10903) * fix(entityservice): fix merging sideeffects (datahub-project#10937) * feat(ingest): Support System Ingestion Sources, Show and hide system ingestion sources with Command-S (datahub-project#10938) Co-authored-by: John Joyce <[email protected]> * chore() Set a default lineage filtering end time on backend when a start time is present (datahub-project#10925) Co-authored-by: John Joyce <[email protected]> Co-authored-by: John Joyce <[email protected]> * Added relationships APIs to V3. Added these generic APIs to V3 swagger doc. (datahub-project#10939) * docs: add learning center to docs (datahub-project#10921) * doc: Update hubspot form id (datahub-project#10943) * chore(airflow): add python 3.11 w/ Airflow 2.9 to CI (datahub-project#10941) * fix(ingest/Glue): column upstream lineage between S3 and Glue (datahub-project#10895) * fix(ingest/abs): split abs utils into multiple files (datahub-project#10945) * doc(ingest/looker): fix doc for sql parsing documentation (datahub-project#10883) Co-authored-by: Harshal Sheth <[email protected]> * fix(ingest/bigquery): Adding missing BigQuery types (datahub-project#10950) * fix(ingest/setup): feast and abs source setup (datahub-project#10951) * fix(connections) Harden adding /gms to connections in backend (datahub-project#10942) * feat(siblings) Add flag to prevent combining siblings in the UI (datahub-project#10952) * fix(docs): make graphql doc gen more automated (datahub-project#10953) * feat(ingest/athena): Add option for Athena partitioned profiling (datahub-project#10723) * fix(spark-lineage): default timeout for future responses (datahub-project#10947) * feat(datajob/flow): add environment filter using info aspects (datahub-project#10814) * fix(ui/ingest): correct privilege used to show tab (datahub-project#10483) Co-authored-by: Kunal-kankriya <[email protected]> * feat(ingest/looker): include dashboard urns in browse v2 (datahub-project#10955) * add a structured type to batchGet in OpenAPI V3 spec (datahub-project#10956) * fix(ui): scroll on the domain sidebar to show all domains (datahub-project#10966) * fix(ingest/sagemaker): resolve incorrect variable assignment for SageMaker API call (datahub-project#10965) * fix(airflow/build): Pinning mypy (datahub-project#10972) * Fixed a bug where the OpenAPI V3 spec was incorrect. The bug was introduced in datahub-project#10939. (datahub-project#10974) * fix(ingest/test): Fix for mssql integration tests (datahub-project#10978) * fix(entity-service) exist check correctly extracts status (datahub-project#10973) * fix(structuredProps) casing bug in StructuredPropertiesValidator (datahub-project#10982) * bugfix: use anyOf instead of allOf when creating references in openapi v3 spec (datahub-project#10986) * fix(ui): Remove ant less imports (datahub-project#10988) * feat(ingest/graph): Add get_results_by_filter to DataHubGraph (datahub-project#10987) * feat(ingest/cli): init does not actually support environment variables (datahub-project#10989) * fix(ingest/graph): Update get_results_by_filter graphql query (datahub-project#10991) * feat(ingest/spark): Promote beta plugin (datahub-project#10881) Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(ingest): support domains in meta -> "datahub" section (datahub-project#10967) * feat(ingest): add `check server-config` command (datahub-project#10990) * feat(cli): Make consistent use of DataHubGraphClientConfig (datahub-project#10466) Deprecates get_url_and_token() in favor of a more complete option: load_graph_config() that returns a full DatahubClientConfig. This change was then propagated across previous usages of get_url_and_token so that connections to DataHub server from the client respect the full breadth of configuration specified by DatahubClientConfig. I.e: You can now specify disable_ssl_verification: true in your ~/.datahubenv file so that all cli functions to the server work when ssl certification is disabled. Fixes datahub-project#9705 * fix(ingest/s3): Fixing container creation when there is no folder in path (datahub-project#10993) * fix(ingest/looker): support platform instance for dashboards & charts (datahub-project#10771) * feat(ingest/bigquery): improve handling of information schema in sql parser (datahub-project#10985) * feat(ingest): improve `ingest deploy` command (datahub-project#10944) * fix(backend): allow excluding soft-deleted entities in relationship-queries; exclude soft-deleted members of groups (datahub-project#10920) - allow excluding soft-deleted entities in relationship-queries - exclude soft-deleted members of groups * fix(ingest/looker): downgrade missing chart type log level (datahub-project#10996) * doc(acryl-cloud): release docs for 0.3.4.x (datahub-project#10984) Co-authored-by: John Joyce <[email protected]> Co-authored-by: RyanHolstien <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Pedro Silva <[email protected]> * fix(protobuf/build): Fix protobuf check jar script (datahub-project#11006) * fix(ui/ingest): Support invalid cron jobs (datahub-project#10998) * fix(ingest): fix graph config loading (datahub-project#11002) Co-authored-by: Pedro Silva <[email protected]> * feat(docs): Document __DATAHUB_TO_FILE_ directive (datahub-project#10968) Co-authored-by: Harshal Sheth <[email protected]> * fix(graphql/upsertIngestionSource): Validate cron schedule; parse error in CLI (datahub-project#11011) * feat(ece): support custom ownership type urns in ECE generation (datahub-project#10999) * feat(assertion-v2): changed Validation tab to Quality and created new Governance tab (datahub-project#10935) * fix(ingestion/glue): Add support for missing config options for profiling in Glue (datahub-project#10858) * feat(propagation): Add models for schema field docs, tags, terms (datahub-project#2959) (datahub-project#11016) Co-authored-by: Chris Collins <[email protected]> * docs: standardize terminology to DataHub Cloud (datahub-project#11003) * fix(ingestion/transformer): replace the externalUrl container (datahub-project#11013) * docs(slack) troubleshoot docs (datahub-project#11014) * feat(propagation): Add graphql API (datahub-project#11030) Co-authored-by: Chris Collins <[email protected]> * feat(propagation): Add models for Action feature settings (datahub-project#11029) * docs(custom properties): Remove duplicate from sidebar (datahub-project#11033) * feat(models): Introducing Dataset Partitions Aspect (datahub-project#10997) Co-authored-by: John Joyce <[email protected]> Co-authored-by: John Joyce <[email protected]> * feat(propagation): Add Documentation Propagation Settings (datahub-project#11038) * fix(models): chart schema fields mapping, add dataHubAction entity, t… (datahub-project#11040) * fix(ci): smoke test lint failures (datahub-project#11044) * docs: fix learning center color scheme & typo (datahub-project#11043) * feat: add cloud main page (datahub-project#11017) Co-authored-by: Jay <[email protected]> * feat(restore-indices): add additional step to also clear system metadata service (datahub-project#10662) Co-authored-by: John Joyce <[email protected]> * docs: fix typo (datahub-project#11046) * fix(lint): apply spotless (datahub-project#11050) * docs(airflow): example query to get datajobs for a dataflow (datahub-project#11034) * feat(cli): Add run-id option to put sub-command (datahub-project#11023) Adds an option to assign run-id to a given put command execution. This is useful when transformers do not exist for a given ingestion payload, we can follow up with custom metadata and assign it to an ingestion pipeline. * fix(ingest): improve sql error reporting calls (datahub-project#11025) * fix(airflow): fix CI setup (datahub-project#11031) * feat(ingest/dbt): add experimental `prefer_sql_parser_lineage` flag (datahub-project#11039) * fix(ingestion/lookml): enable stack-trace in lookml logs (datahub-project#10971) * (chore): Linting fix (datahub-project#11015) * chore(ci): update deprecated github actions (datahub-project#10977) * Fix ALB configuration example (datahub-project#10981) * chore(ingestion-base): bump base image packages (datahub-project#11053) * feat(cli): Trim report of dataHubExecutionRequestResult to max GMS size (datahub-project#11051) * fix(ingestion/lookml): emit dummy sql condition for lookml custom condition tag (datahub-project#11008) Co-authored-by: Harshal Sheth <[email protected]> * fix(ingestion/powerbi): fix issue with broken report lineage (datahub-project#10910) * feat(ingest/tableau): add retry on timeout (datahub-project#10995) * change generate kafka connect properties from env (datahub-project#10545) Co-authored-by: david-leifker <[email protected]> * fix(ingest): fix oracle cronjob ingestion (datahub-project#11001) Co-authored-by: david-leifker <[email protected]> * chore(ci): revert update deprecated github actions (datahub-project#10977) (datahub-project#11062) * feat(ingest/dbt-cloud): update metadata_endpoint inference (datahub-project#11041) * build: Reduce size of datahub-frontend-react image by 50-ish% (datahub-project#10878) Co-authored-by: david-leifker <[email protected]> * fix(ci): Fix lint issue in datahub_ingestion_run_summary_provider.py (datahub-project#11063) * docs(ingest): update developing-a-transformer.md (datahub-project#11019) * feat(search-test): update search tests from datahub-project#10408 (datahub-project#11056) * feat(cli): add aspects parameter to DataHubGraph.get_entity_semityped (datahub-project#11009) Co-authored-by: Harshal Sheth <[email protected]> * docs(airflow): update min version for plugin v2 (datahub-project#11065) * doc(ingestion/tableau): doc update for derived permission (datahub-project#11054) Co-authored-by: Pedro Silva <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Harshal Sheth <[email protected]> * fix(py): remove dep on types-pkg_resources (datahub-project#11076) * feat(ingest/mode): add option to exclude restricted (datahub-project#11081) * fix(ingest): set lastObserved in sdk when unset (datahub-project#11071) * doc(ingest): Update capabilities (datahub-project#11072) * chore(vulnerability): Log Injection (datahub-project#11090) * chore(vulnerability): Information exposure through a stack trace (datahub-project#11091) * chore(vulnerability): Comparison of narrow type with wide type in loop condition (datahub-project#11089) * chore(vulnerability): Insertion of sensitive information into log files (datahub-project#11088) * chore(vulnerability): Risky Cryptographic Algorithm (datahub-project#11059) * chore(vulnerability): Overly permissive regex range (datahub-project#11061) Co-authored-by: Harshal Sheth <[email protected]> * fix: update customer data (datahub-project#11075) * fix(models): fixing the datasetPartition models (datahub-project#11085) Co-authored-by: John Joyce <[email protected]> * fix(ui): Adding view, forms GraphQL query, remove showing a fallback error message on unhandled GraphQL error (datahub-project#11084) Co-authored-by: John Joyce <[email protected]> * feat(docs-site): hiding learn more from cloud page (datahub-project#11097) * fix(docs): Add correct usage of orFilters in search API docs (datahub-project#11082) Co-authored-by: Jay <[email protected]> * fix(ingest/mode): Regexp in mode name matcher didn't allow underscore (datahub-project#11098) * docs: Refactor customer stories section (datahub-project#10869) Co-authored-by: Jeff Merrick <[email protected]> * fix(release): fix full/slim suffix on tag (datahub-project#11087) * feat(config): support alternate hashing algorithm for doc id (datahub-project#10423) Co-authored-by: david-leifker <[email protected]> Co-authored-by: John Joyce <[email protected]> * fix(emitter): fix typo in get method of java kafka emitter (datahub-project#11007) * fix(ingest): use correct native data type in all SQLAlchemy sources by compiling data type using dialect (datahub-project#10898) Co-authored-by: Harshal Sheth <[email protected]> * chore: Update contributors list in PR labeler (datahub-project#11105) * feat(ingest): tweak stale entity removal messaging (datahub-project#11064) * fix(ingestion): enforce lastObserved timestamps in SystemMetadata (datahub-project#11104) * fix(ingest/powerbi): fix broken lineage between chart and dataset (datahub-project#11080) * feat(ingest/lookml): CLL support for sql set in sql_table_name attribute of lookml view (datahub-project#11069) * docs: update graphql docs on forms & structured properties (datahub-project#11100) * test(search): search openAPI v3 test (datahub-project#11049) * fix(ingest/tableau): prevent empty site content urls (datahub-project#11057) Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(entity-client): implement client batch interface (datahub-project#11106) * fix(snowflake): avoid reporting warnings/info for sys tables (datahub-project#11114) * fix(ingest): downgrade column type mapping warning to info (datahub-project#11115) * feat(api): add AuditStamp to the V3 API entity/aspect response (datahub-project#11118) * fix(ingest/redshift): replace r'\n' with '\n' to avoid token error redshift serverless… (datahub-project#11111) * fix(entiy-client): handle null entityUrn case for restli (datahub-project#11122) * fix(sql-parser): prevent bad urns from alter table lineage (datahub-project#11092) * fix(ingest/bigquery): use small batch size if use_tables_list_query_v2 is set (datahub-project#11121) * fix(graphql): add missing entities to EntityTypeMapper and EntityTypeUrnMapper (datahub-project#10366) * feat(ui): Changes to allow editable dataset name (datahub-project#10608) Co-authored-by: Jay Kadambi <[email protected]> * fix: remove saxo (datahub-project#11127) * feat(mcl-processor): Update mcl processor hooks (datahub-project#11134) * fix(openapi): fix openapi v2 endpoints & v3 documentation update * Revert "fix(openapi): fix openapi v2 endpoints & v3 documentation update" This reverts commit 573c1cb. * docs(policies): updates to policies documentation (datahub-project#11073) * fix(openapi): fix openapi v2 and v3 docs update (datahub-project#11139) * feat(auth): grant type and acr values custom oidc parameters support (datahub-project#11116) * fix(mutator): mutator hook fixes (datahub-project#11140) * feat(search): support sorting on multiple fields (datahub-project#10775) * feat(ingest): various logging improvements (datahub-project#11126) * fix(ingestion/lookml): fix for sql parsing error (datahub-project#11079) Co-authored-by: Harshal Sheth <[email protected]> * feat(docs-site) cloud page spacing and content polishes (datahub-project#11141) * feat(ui) Enable editing structured props on fields (datahub-project#11042) * feat(tests): add md5 and last computed to testResult model (datahub-project#11117) * test(openapi): openapi regression smoke tests (datahub-project#11143) * fix(airflow): fix tox tests + update docs (datahub-project#11125) * docs: add chime to adoption stories (datahub-project#11142) * fix(ingest/databricks): Updating code to work with Databricks sdk 0.30 (datahub-project#11158) * fix(kafka-setup): add missing script to image (datahub-project#11190) * fix(config): fix hash algo config (datahub-project#11191) * test(smoke-test): updates to smoke-tests (datahub-project#11152) * fix(elasticsearch): refactor idHashAlgo setting (datahub-project#11193) * chore(kafka): kafka version bump (datahub-project#11211) * readd UsageStatsWorkUnit * fix merge problems * change logo --------- Co-authored-by: Chris Collins <[email protected]> Co-authored-by: John Joyce <[email protected]> Co-authored-by: John Joyce <[email protected]> Co-authored-by: John Joyce <[email protected]> Co-authored-by: dushayntAW <[email protected]> Co-authored-by: sagar-salvi-apptware <[email protected]> Co-authored-by: Aseem Bansal <[email protected]> Co-authored-by: Kevin Chun <[email protected]> Co-authored-by: jordanjeremy <[email protected]> Co-authored-by: skrydal <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> Co-authored-by: david-leifker <[email protected]> Co-authored-by: sid-acryl <[email protected]> Co-authored-by: Julien Jehannet <[email protected]> Co-authored-by: Hendrik Richert <[email protected]> Co-authored-by: Hendrik Richert <[email protected]> Co-authored-by: RyanHolstien <[email protected]> Co-authored-by: Felix Lüdin <[email protected]> Co-authored-by: Pirry <[email protected]> Co-authored-by: Hyejin Yoon <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: cburroughs <[email protected]> Co-authored-by: ksrinath <[email protected]> Co-authored-by: Mayuri Nehate <[email protected]> Co-authored-by: Kunal-kankriya <[email protected]> Co-authored-by: Shirshanka Das <[email protected]> Co-authored-by: ipolding-cais <[email protected]> Co-authored-by: Tamas Nemeth <[email protected]> Co-authored-by: Shubham Jagtap <[email protected]> Co-authored-by: haeniya <[email protected]> Co-authored-by: Yanik Häni <[email protected]> Co-authored-by: Gabe Lyons <[email protected]> Co-authored-by: Gabe Lyons <[email protected]> Co-authored-by: 808OVADOZE <[email protected]> Co-authored-by: noggi <[email protected]> Co-authored-by: Nicholas Pena <[email protected]> Co-authored-by: Jay <[email protected]> Co-authored-by: ethan-cartwright <[email protected]> Co-authored-by: Ethan Cartwright <[email protected]> Co-authored-by: Nadav Gross <[email protected]> Co-authored-by: Patrick Franco Braz <[email protected]> Co-authored-by: pie1nthesky <[email protected]> Co-authored-by: Joel Pinto Mata (KPN-DSH-DEX team) <[email protected]> Co-authored-by: Ellie O'Neil <[email protected]> Co-authored-by: Ajoy Majumdar <[email protected]> Co-authored-by: deepgarg-visa <[email protected]> Co-authored-by: Tristan Heisler <[email protected]> Co-authored-by: Andrew Sikowitz <[email protected]> Co-authored-by: Davi Arnaut <[email protected]> Co-authored-by: Pedro Silva <[email protected]> Co-authored-by: amit-apptware <[email protected]> Co-authored-by: Sam Black <[email protected]> Co-authored-by: Raj Tekal <[email protected]> Co-authored-by: Steffen Grohsschmiedt <[email protected]> Co-authored-by: jaegwon.seo <[email protected]> Co-authored-by: Renan F. Lima <[email protected]> Co-authored-by: Matt Exchange <[email protected]> Co-authored-by: Jonny Dixon <[email protected]> Co-authored-by: Pedro Silva <[email protected]> Co-authored-by: Pinaki Bhattacharjee <[email protected]> Co-authored-by: Jeff Merrick <[email protected]> Co-authored-by: skrydal <[email protected]> Co-authored-by: AndreasHegerNuritas <[email protected]> Co-authored-by: jayasimhankv <[email protected]> Co-authored-by: Jay Kadambi <[email protected]> Co-authored-by: David Leifker <[email protected]>
MCP Mutator capability added to the MutationHook. It is capable of mutating incoming MCP before validation. A new ProposedItem can now be used which allows mutation by this new mutator type. An example plugin was created to drop unknown fields.
Not yet tied into APIs, future work.
Checklist
Summary by CodeRabbit
New Features
Bug Fixes
Tests
Chores