feat(search): Support Boolean OR Filters in Rest.li APIs

[jjoyce0510]

Changes
This filter accomplishes a few things:

1. moves the "filter" model from datahub-gma into datahub repo
2. adds support for OR filters in the Rest.li GMS apis. Previously we only supported ANDs.

Compatibility
This change is backwards incompatible for the Rest.li APIs exposed by GMS. If you are actively using them with the Filter functionality, you will need to update your code to use the new ConjunctiveCriterion model.

Old Filter structure:

"filter": {
        "criteria": [
           {
                "field": "title",
                "value": "Baz Chart 1",
                "condition": "EQUAL"
           }
        ]
 }

and new Filter structure:

"filter": {
        "or": [
           {
                "and": [ 
                  {
                       "field": "title",
                       "value": "Baz Chart 1",
                       "condition": "EQUAL"
                   }
                ]
           }
        ]
 }

As you can see, in adding this API we are going to support Disjunctive Normal Form boolean queries in our API, which makes it much more powerful!

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable)

[dexter-mh-lee]

There is reference to filters in TimeseriesAspectService that you missed

[dexter-mh-lee]

Also seems like we cannot do and of or operations. is this intended?

[dexter-mh-lee]

Can you delete files under metadata-ingestion/src/datahub/metadata ? Should be ignored

[dexter-mh-lee]

LGTM! Did you confirm metadata-io:tests are passing? Since it has been disabled in workflow want to make sure they are passing at least loaclly

[jjoyce0510]

Tested locally, yes!

[shirshanka]

LGTM!

[EnricoMi]

@jjoyce0510 @shirshanka should the classes that have been moved out of datahub-gma be marked as deprecated in a new release of datahub-gma and datahub move to that version? Otherwise it might confuse that datahub and its current datahub-gma version both have identical classes under different packages and it is not clear which to use.

[EnricoMi]

@jjoyce0510 @shirshanka @gabe-lyons Also, Filter in com.linkedin.metadata.query.filter still contains Criteria along with ConjunctiveCriterionArray. Can a filter have set both? Does GraphService have to implement both? If criteria is considered deprecated, it should have been removed from Filter while copying it over from datahub-gma.

Does a GraphService have to implement the full set of possible filter expressions (multiple ORs, multiple ANDs, all of com.linkedin.metadata.query.filter.Condition)?

[shirshanka]

@EnricoMi : we kept the Criteria as we didn't want to break any existing clients of the REST protocol. I think we should translate Criteria based calls to the Conjunctive form higher up the stack so that by the time the query hits the GraphService it has already been translated to the Conjunctive form.
I will let @jjoyce0510 respond to your questions on what expressions the GraphService needs to implement.

[jjoyce0510]

@EnricoMi We are looking to this on our end. Ideally we can just kill that field for good and simplify this so no one has to deal with the case. Until then, you would be expected to deal with the possibility that the criteria field is populated. Update to come soon

[jjoyce0510]

should the classes that have been moved out of datahub-gma be marked as deprecated in a new release of datahub-gma and datahub move to that version? Otherwise it might confuse that datahub and its current datahub-gma version both have identical classes under different packages and it is not clear which to use.

Yes ideally that'd be the case, however to our knowledge LinkedIn is still maintaining a version of datahub-gma in active use, so it's unclear if we'll be able to remove those files. We are currently transitioning away from depending on datahub-gma altogether and once that completes we'll be able to remove all dependencies to make things more comprehensible. But yes the ideal state is not the one we're currently operating in (yet)