feat: add null value filter #1721

sadakchap · 2021-06-03T06:05:41Z

Feature request #1658

Added new filter to filter records if field contains null value.

mtrezza · 2021-06-03T12:06:59Z

Thanks for this PR.

In MongoDB, the value null is sometimes treated the same as if the field does not exist at all. In other words, these two documents are sometimes treated as identical:

{
  a: 1,
  b: null
}

{
  a: 1
}

That means from a MongoDB perspective there may not be a difference between the Parse Dashboard filters "does not exist" and "is null". To unambiguously query for objects with a field of content null we would have to query for the BSON Type 10.

The distinction in MongoDB is:

a) field : { $type: 10 }: returns documents where the field has value null.
b) field : { $exists: false }: returns documents where the field does not exist.
c) field : null: returns documents where the field either has value null or does not exist.

Your video shows a filer for null values, but the filer description isNull is ambiguous - which of the filters above are applied? We would have to look behind the scenes into Parse Server MongoDB adapter to see how the query is ultimately constructed. I suspect it is c), which would in that case overlap with the "does not exist" filter.

Another aspect to consider is how that plays with the Postgres adapter, or whether that is something unique to MongoDB.

sadakchap · 2021-06-03T12:21:44Z

In MongoDB, the value null is sometimes treated the same as if the field does not exist at all. In other words, these two documents are sometimes treated as identical:

True. That's why we are filtering for both exists & equal to null value. If filtering field is age then we are sending a query like this.

where: {
  $and: [
      age: { $exists: true },
      age: null
   ]
}

Your video shows a filer for null values, but the filer description isNull is ambiguous - which of the filters above are applied?

Could you please elaborate? I don't think I understand the question properly.

mtrezza · 2021-06-03T12:28:32Z

That's why we are filtering for both exists & equal to null value.

Do you mean that is how you personally use the filter to query for a)?

It may return the same results, but the correct - because more efficient - query for this is:

where: {
    age: { $type: 10 }
}

sadakchap · 2021-06-03T12:37:34Z

Do you mean that is how you personally use the filter to query for a)?

yes.

It may return the same results, but the correct - because more efficient - query for this is:
where: {
    age: { $type: 10 }
}

Yes, you are correct! But, How do I send a query like this, I searched in the docs but couldn't find anything that would query on type of the field.

mtrezza · 2021-06-03T14:20:13Z

Not sure whether Parse Server supports this yet, you would have to look into the code, it may exist but not be in the docs.
However, both filters (a) and (c) can be useful, so we may well offer both filters in the dashboard at some point in the future.

Therefore I suggest this naming to make them distinguishable:

a) field : { $type: 10 } filter name: is null
c) field: null filter name: does not exist or is null

If later someone adds filter (a), they can name the filter is null and it won't conflict with the filter name in this PR.
I don't know what this means for Postgres or other DBs, but it would be accurate for MongoDB.

sadakchap · 2021-06-04T01:58:12Z

Not sure whether Parse Server supports this yet, you would have to look into the code, it may exist but not be in the docs.

I look into this file, but I couldn't find any thing that would query on type.

Therefore I suggest this naming to make them distinguishable:

a) field : { $type: 10 } filter name: is null
c) field: null filter name: does not exist or is null

If later someone adds filter (a), they can name the filter is null and it won't conflict with the filter name in this PR.

Sounds good 👍, so should I change the current query as well for does not exist or is null option
from

query.exists(field);
let nullQuery = new Parse.Query(className);
nullQuery.equalTo(field, null); 
query = Parse.Query.and(query, nullQuery);

to

query.equalTo(field, null)

as the current query fetches records only with null value.

mtrezza · 2021-06-04T02:14:41Z

should I change the current query as well for does not exist or is null option

Yes I think so. Can one still query only for null values by adding the field exists filter additionally for the same field in the dashboard?

sadakchap · 2021-06-04T02:41:37Z

Can one still query only for null values by adding the field exists filter additionally for the same field in the dashboard?

I think only one filter is allowed on a field at a time. No, I don't think it is possible.

mtrezza · 2021-06-04T14:04:45Z

I think only one filter is allowed on a field at a time. No, I don't think it is possible.

You could additionally add the is null filter if we find out how to do where { age: { $type: 10 } }. I think it should be already possible, maybe with a REST request, if the JS SDK doesn't support this yet.

If it really isn't possible, then you could add this:

query.exists(field);
let nullQuery = new Parse.Query(className);
nullQuery.equalTo(field, null); 
query = Parse.Query.and(query, nullQuery);

with a comment that this is a workaround because where { age: { $type: 10 } } is not possible with the JS SDK.

Then we could have both filters (is null; does not exist or is null) in the dashboard with this PR.

dplewis

@sadakchap Thanks for the PR! I was the one who requested it.

dplewis · 2021-06-04T17:13:43Z

src/lib/queryFromFilters.js

@@ -104,6 +104,9 @@ function addConstraint(query, filter) {
    case 'keyLte':
      addQueryConstraintFromObject(query, filter, 'lessThanOrEqualTo');
      break;
+    case 'dneOrNull':
+      query.equalTo(filter.get('field'), null);


I don't think this query will return null values.

True, it will also include records which does not have filter field even set, that's why the name of filter does not exits or null

That sounds good

I think only one filter is allowed on a field at a time.

You can set composable to true if you to do multiple filters on a field. Similar to how less than does it. (Perhaps we should document these)

Enabling composable is a good idea.

Regarding the commented code here, the difference is the MongoDB performance. The current query is an explicit "does not exist or is null" conditional, without $and. This distinction is also relevant for indexing.

Enabling composable and add this filter query allows for both ways, so that's the most versatile way to go.

I mean document composable and comparable it usually takes a while to figure out what they mean.

It sends where={age:{$exists: true}} if you set a combined filter of (1) and (2)?

It sends where={age:{$exists: true}} if you set a combined filter of (1) and (2)?

yes.

I think that's strange as, for let's say 2 filters less than & exists, we do something like

query.lessThan(field, compareTo); // compareTo is value entered by user to which s(he) wants to compare query.exists(field);

This sends a query like where={age: {$lt: 4, $exists: true}}

Then, why not this

query.equalTo(field, null); query.exists(field);

sends a query like where={age: {$eq: null, $exists: true}}.

I tried searching in thee docs about equalTo query, whether it is composble or not. But, there's nothing about that 😕

@dplewis when you mentioned composable, on what level did you mean that? On a MongoDB level?

I don't think it's a limitation with MongoDB rather it's with JS Sdk.(just an opinion)

As when I explicitly(using cURL) make a request with where={age: {$exists: true, $eq: null }}, it does return records only with null field value.

But, when I'm trying to couple $exists & $eq query, it only sends request with where={age: {$exists: true}}.
I'm trying this way to construct the query

query.equalTo(field, null); query.exists(field);

Maybe I'm missing some point with js sdk. Would really appreciate any help with creating a query with where={age: {$exists: true, $eq: null }}.

I created a new issue for this parse-community/Parse-SDK-JS#1372.

mtrezza · 2021-07-26T12:55:01Z

@sadakchap Do you think we can resolve any open questions here? I think this PR is almost ready for merge, it would be great to have it in the upcoming release of Parse Dashboard.

sadakchap · 2021-07-28T13:15:50Z

@sadakchap Do you think we can resolve any open questions here? I think this PR is almost ready for merge, it would be great to have it in the upcoming release of Parse Dashboard.

This pr was originally to add a feature through which users can filter records with only null value.

But, we decided to add a filter dneOrNull, on which user a add one more filter exists => then user should be getting records with only null value. But, that is not the case, $exists and $eq query seems to overriding each other. Please see #1721 (comment) .

I also raised an issue in Parse-SDK-JS. But, I think I am really not the right person to solve this issue.

For this feature, maybe we can try coupling $exists & $eq with $and. Something like this 38d2f16 , maybe ?

mtrezza · 2021-09-24T00:53:44Z

@sadakchap So maybe we can finalize this PR like this:

Add a filter condition "does not exist or is null" for (c) with this PR
Add a filter condition "is null" for (b) in a later PR once the Parse JS SDK issue is fixed and supports this properly

What do you think?

sadakchap · 2021-09-25T10:05:49Z

Thanks @mtrezza .

I was working on this issue's PR. I was able to to clear all unit test cases. Right now, I think only LiveQuery Integration test cases are failing.

I know that, this is open from long time, but If we can wait just 1 more week ( so that I can try to fix those failing integration test cases). It would be really nice. Or else we can do as you have suggested(incase it takes more than a week).

mtrezza · 2021-09-25T11:20:19Z

Sure. Let's wait as you suggest.

cbaker6 · 2022-01-03T04:15:54Z

Another aspect to consider is how that plays with the Postgres adapter, or whether that is something unique to MongoDB.

For Postgres, the server treats doesNotExist and null the same, so I believe the filters here will produce the same results, which should be fine. I discuss more about this here: https://community.parseplatform.org/t/query-where-field-nil/2317/16?u=cbaker6

sadakchap added 2 commits June 2, 2021 19:15

add isNull filter for all datatypes

b53fd00

filtering null values only

38d2f16

changed filter name & query

761cdc4

dplewis reviewed Jun 4, 2021

View reviewed changes

mtrezza changed the title ~~Add isNull filter~~ feat: add isNull filter Sep 19, 2021

mtrezza changed the title ~~feat: add isNull filter~~ feat: add null value filter Sep 19, 2021

parse-community deleted a comment from parse-github-assistant bot Sep 19, 2021

Merge branch 'master' into add-isnull-filter

dec4980

Merge branch 'master' into add-isnull-filter

2b3211d

mtrezza marked this pull request as draft September 29, 2021 17:41

mtrezza mentioned this pull request Oct 6, 2021

feat: add isNull and isNotNull filters #1838

Closed

5 tasks

sadakchap mentioned this pull request Oct 13, 2021

feat: combine equalTo clauses with clauses parse-community/Parse-SDK-JS#1373

Open

5 tasks

This was referenced Oct 13, 2021

feat: Add support for $eq in LiveQuery parse-community/parse-server#7607

Closed

feat: allow removing a pointer from a field parse-community/parse-server#7518

Closed

Merge branch 'alpha' into add-isnull-filter

936275f

Merge branch 'alpha' into add-isnull-filter

c2f36ec

mtrezza force-pushed the alpha branch from d32a043 to 0f42a34 Compare May 1, 2022 13:11

snyk-bot mentioned this pull request Jun 18, 2022

refactor: upgrade commander from 9.2.0 to 9.3.0 #2173

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add null value filter #1721

feat: add null value filter #1721

sadakchap commented Jun 3, 2021

mtrezza commented Jun 3, 2021 •

edited

Loading

sadakchap commented Jun 3, 2021 •

edited

Loading

mtrezza commented Jun 3, 2021 •

edited

Loading

sadakchap commented Jun 3, 2021 •

edited

Loading

mtrezza commented Jun 3, 2021 •

edited

Loading

sadakchap commented Jun 4, 2021

mtrezza commented Jun 4, 2021

sadakchap commented Jun 4, 2021 •

edited

Loading

mtrezza commented Jun 4, 2021 •

edited

Loading

dplewis left a comment •

edited

Loading

dplewis Jun 4, 2021

sadakchap Jun 4, 2021

dplewis Jun 4, 2021

mtrezza Jun 4, 2021 •

edited

Loading

dplewis Jun 4, 2021

mtrezza Jun 6, 2021 •

edited

Loading

sadakchap Jun 6, 2021 •

edited

Loading

mtrezza Jun 6, 2021

sadakchap Jun 7, 2021

sadakchap Jun 8, 2021

mtrezza commented Jul 26, 2021

sadakchap commented Jul 28, 2021

mtrezza commented Sep 24, 2021 •

edited

Loading

sadakchap commented Sep 25, 2021

mtrezza commented Sep 25, 2021

cbaker6 commented Jan 3, 2022

feat: add null value filter #1721

Are you sure you want to change the base?

feat: add null value filter #1721

Conversation

sadakchap commented Jun 3, 2021

mtrezza commented Jun 3, 2021 • edited Loading

sadakchap commented Jun 3, 2021 • edited Loading

mtrezza commented Jun 3, 2021 • edited Loading

sadakchap commented Jun 3, 2021 • edited Loading

mtrezza commented Jun 3, 2021 • edited Loading

sadakchap commented Jun 4, 2021

mtrezza commented Jun 4, 2021

sadakchap commented Jun 4, 2021 • edited Loading

mtrezza commented Jun 4, 2021 • edited Loading

dplewis left a comment • edited Loading

Choose a reason for hiding this comment

dplewis Jun 4, 2021

Choose a reason for hiding this comment

sadakchap Jun 4, 2021

Choose a reason for hiding this comment

dplewis Jun 4, 2021

Choose a reason for hiding this comment

mtrezza Jun 4, 2021 • edited Loading

Choose a reason for hiding this comment

dplewis Jun 4, 2021

Choose a reason for hiding this comment

mtrezza Jun 6, 2021 • edited Loading

Choose a reason for hiding this comment

sadakchap Jun 6, 2021 • edited Loading

Choose a reason for hiding this comment

mtrezza Jun 6, 2021

Choose a reason for hiding this comment

sadakchap Jun 7, 2021

Choose a reason for hiding this comment

sadakchap Jun 8, 2021

Choose a reason for hiding this comment

mtrezza commented Jul 26, 2021

sadakchap commented Jul 28, 2021

mtrezza commented Sep 24, 2021 • edited Loading

sadakchap commented Sep 25, 2021

mtrezza commented Sep 25, 2021

cbaker6 commented Jan 3, 2022

mtrezza commented Jun 3, 2021 •

edited

Loading

sadakchap commented Jun 3, 2021 •

edited

Loading

mtrezza commented Jun 3, 2021 •

edited

Loading

sadakchap commented Jun 3, 2021 •

edited

Loading

mtrezza commented Jun 3, 2021 •

edited

Loading

sadakchap commented Jun 4, 2021 •

edited

Loading

mtrezza commented Jun 4, 2021 •

edited

Loading

dplewis left a comment •

edited

Loading

mtrezza Jun 4, 2021 •

edited

Loading

mtrezza Jun 6, 2021 •

edited

Loading

sadakchap Jun 6, 2021 •

edited

Loading

mtrezza commented Sep 24, 2021 •

edited

Loading