feat(graph,graphql): filter by child entity (interfaces) #3677

kamilkisiela · 2022-06-17T16:31:12Z

Continuation of #3184, adds support for interfaces.

GraphQL Schema

type Musician @entity {
  id: ID!
  name: String!
  mainBand: Band
  bands: [Band!]!
  writtenSongs: [Song]! @derivedFrom(field: "writtenBy")
}

type Band @entity {
  id: ID!
  name: String!
  members: [Musician!]! @derivedFrom(field: "bands")
  reviews: [BandReview] @derivedFrom(field: "band")
  originalSongs: [Song!]!
}

type Song @entity {
  id: ID!
  title: String!
  writtenBy: Musician!
  publisher: Publisher!
  band: Band @derivedFrom(field: "originalSongs")
  reviews: [SongReview] @derivedFrom(field: "song")
}

type SongStat @entity {
  id: ID!
  song: Song @derivedFrom(field: "id")
  played: Int!
}

type Publisher {
  id: Bytes!
}

interface Review {
  id: ID!
  body: String!
  author: User!
}

type SongReview implements Review @entity {
  id: ID!
  body: String!
  song: Song
  author: User!
}

type BandReview implements Review @entity {
  id: ID!
  body: String!
  band: Band
  author: User!
}

type User @entity {
  id: ID!
  name: String!
  bandReviews: [BandReview] @derivedFrom(field: "author")
  songReviews: [SongReview] @derivedFrom(field: "author")
  reviews: [Review] @derivedFrom(field: "author")
}

GraphQL Query

query {
  users(first: 5, orderBy: id, where: { reviews_: { body_starts_with:  "Good" } }) {
    name
    reviews {
      body
    }
  }
}

SQL statement

select
  'User' as entity,
  to_jsonb(c.*) as data
from
  (
    select
      *
    from
      "sgd445"."user" c
    where
      c.block_range @> $1
      and (
        (
          exists (
            select
              1
            from
              "sgd445"."song_review" as i
            where
              c."id" = i."author"
              and i.block_range @> $2
              and (i."body" like $3)
          )
          or exists (
            select
              1
            from
              "sgd445"."band_review" as i
            where
              c."id" = i."author"
              and i.block_range @> $4
              and (i."body" like $5)
          )
        )
      )
    order by "id", block_range
    limit 100
  ) c

kamilkisiela · 2022-07-15T09:25:43Z

@azf20 @lutter please let me know what other use cases I should cover in the tests.

lutter

Looks good in general; surprising that it was this easy! I am a little worried about using or in queries, but we can look more into this once we have some examples of queries that use this feature and are slow and try out alternative queries.

The tests should be adapted a little, but other than that, looks good!

lutter · 2022-07-27T11:21:49Z

graphql/tests/query.rs

@@ -154,6 +163,7 @@ fn test_schema(id: DeploymentHash, id_type: IdType) -> Schema {
        id: ID!
        name: String!
        members: [Musician!]! @derivedFrom(field: \"bands\")
+        reviews: [BandReview] @derivedFrom(field: \"band\")


Is there a reason why this can't be [BandReview!]! like the other fields? IIRC, we do not allow arrays with nulls in them anyway. (There's a couple more places here that raise the same question)

Those are now non-nullable

lutter · 2022-07-27T11:23:27Z

graphql/tests/query.rs

+        bandReviews: [BandReview] @derivedFrom(field: \"author\")
+        songReviews: [SongReview] @derivedFrom(field: \"author\")
+        reviews: [Review] @derivedFrom(field: \"author\")
+    }


It looks like all the interface fields in these tests are @derivedFrom. Could some of them be made into direct fields? In general, we want to make sure that all four combinations of derived/direct and single value/array fields work.

I added a few new test scenarios but I don't know exactly how the interfaces are supposed to work, what is possible, and what is not. I'm guessing here a bit. One test is failing but it yells at me about some non-existing operator (operator does not exist: bytea = text). I'm lost at this point :)

That error means that somewhere where a SQL query compares two columns with =, one is of type bytea and the other of type text. That's an error somewhere in query generation, not necessarily connected to child filters. I wanted to try that out myself, but couldn't build this branch since cargo gets unhappy when it tries to check out https://github.com/dotansimha/graphql-tools-rs?branch=clone-validation-error#89085f31 and says

object not found - no match for id (89085f316a71b390fecd9bd4c3f55c3884c33169); class=Odb (9); code=NotFound (-3)

The best way to get to the bottom of this is to turn on logging of all statements in Postgres with alter system set log_statement = 'all'; in psql in your test database. (You might have to restart Postgres after this) With this set, run the failing test and the Postgres logs will have the query that is causing that error. Actually, I just noticed that the error message from the test has the full query, though I still recommend setting this.

kamilkisiela · 2022-08-08T16:52:38Z

I don't know exactly how the interfaces are supposed to work, what is possible, and what is not with and without the @derivedFrom directive. I'm guessing here a bit. Waiting for a few actual use cases to match my implementation against some real and confirmed to be working examples/tests.

lutter · 2022-08-12T23:50:00Z

I just made a big table to get an overview of what combinations of parent and child types are possible and came up with this:

parent intf	child intf	derived	list	parent	child
f	f	f	f	Musician	mainBand
f	f	f	t	Musician	bands
f	f	t	f	Song	band
f	f	t	t	Band	members
f	t	f	f	User	latestReview
f	t	f	t	Song	media
f	t	t	f	---	---
f	t	t	t	User	reviews
t	f	f	f	Review	author
t	f	f	t	---	---
t	f	t	f	Media	song
t	f	t	t	---	---
t	t	f	f	---	---
t	t	f	t	---	---
t	t	t	f	---	---
t	t	t	t	---	---

They all are for a query along the lines of

{
  parents(where: { child_: { someField_filter: "value" } }) { .. stuff .. }
}

The columns mean:

parent intf: should the parent be an interface
child intf: should the child be an interface
derived: should the child be derived
list: should the child be a list/array
parent: the type of the parent to use
child: the name of the child field to use

I tried to fill this in with examples that I see possible from the test schema and marked the ones where I think that's not possible in the schema with --- - since there's so many of them, I think we should try and make sure we cover all the cases that I marked as possible (they might all be covered already, haven't checked against the tests yet) and get them working. Once that's done, let's merge this PR and work on another one to cover more of the possible combinations here.

kamilkisiela · 2022-08-24T08:30:56Z

@lutter the table says that Song type (not an interface) can have media: [Media!]! where Media is an interface.

For this query:

query {
  songs(
    first: 100, 
    orderBy: id, 
    where: { 
      media_: {
        title_starts_with: "Cheesy Tune"
      }
    }
  ) {
    title
    media {
      title
    }
  }
}

This is the generated SQL query:

select 'Song' as entity, to_jsonb(c.*) as data from (select  *
  from "sgd52"."song" c
where c.block_range @> 1 and ((exists (select 1 from "sgd52"."photo" as i where i."id" = any(c."media") and i.block_range @> 1 and (i."title" like '%Cheesy Tune%')) or exists (select 1 from "sgd52"."video" as i where i."id" = any(c."media") and i.block_range @> 1 and (i."title" like '%Cheesy Tune%'))))
order by "id", block_range
limit 100) c

That query is valid and works.

The operator does not exist: bytea = text error comes from another SQL query that is not related to my code.

with matches as (select c.* from unnest($1::bytea[]) as q(id)
 cross join lateral (select 'Photo' as entity, c.id, c.vid, p.id::text as g$parent_id, c.block_range
/* children_type_c */  from rows from (unnest($2), reduce_dim(array[array['0xf1', '0xf2']]::text[][])) as p(id, child_ids) cross join lateral (select  *  from "sgd54"."photo" c where c.block_range @> $3 and q.id = p.id and c.id = any(p.child_ids)) c
union all
select 'Video' as entity, c.id, c.vid, p.id::text as g$parent_id, c.block_range
/* children_type_c */  from rows from (unnest($4), reduce_dim(array[array['0xf1', '0xf2']]::text[][])) as p(id, child_ids) cross join lateral (select  *  from "sgd54"."video" c where c.block_range @> $5 and q.id = p.id and c.id = any(p.child_ids)) c
order by "id", block_range
 limit 100) c)
select m.*, to_jsonb("c".*)|| jsonb_build_object('g$parent_id', m.g$parent_id) as data
  from "sgd54"."photo" c, matches m
 where c.vid = m.vid and m.entity = 'Photo'
union all
select m.*, to_jsonb("c".*)|| jsonb_build_object('g$parent_id', m.g$parent_id) as data
  from "sgd54"."video" c, matches m
 where c.vid = m.vid and m.entity = 'Video'
 order by g$parent_id, "id"

Can you confirm it's not supported by graph-node or it's a bug?

wminshew · 2022-09-01T22:15:46Z

graphql/src/schema/api.rs

@@ -381,11 +364,14 @@ fn field_list_filter_input_values(
                    )
                }
            }
-            TypeDefinition::Interface(_) => {
+            TypeDefinition::Interface(parent) => {


apologies if I'm missing something (relatively new to rust) but this arm can be combined w line 357 into TypeDefinition::Object(parent) | TypeDefinition::Interface(parent) => as in line 231 i think? both of these match expressions (L358-366, L368-375) look the same to me

parent is different in both and I think Rust does not support unions. If I would merge them and try to access parent.name I would get an error.

I see -- so it works in L231 bc you're discarding the object anyway and here the compiler ~can't understand they both have .name independently? and so in this case it would need to use .. some kind of trait?

appreciate the response, thanks!

Nice catch - you can make this work if you destructure further because then name gets bound to a &String in both branches:

TypeDefinition::Object(ObjectType { name, .. }) | TypeDefinition::Interface(InterfaceType { name, .. }) => { if ast::get_derived_from_directive(field).is_some() { (None, Some(name.clone())) } else { (Some(Type::NamedType("String".into())), Some(name.clone())) } }

lutter · 2022-09-07T23:52:35Z

@lutter the table says that Song type (not an interface) can have media: [Media!]! where Media is an interface.
...
Can you confirm it's not supported by graph-node or it's a bug?

Yes, that is indeed a pre-existing bug related to using Bytes for ids, caused by the media { title } part of the GraphQL query. I just added a commit to this PR that fixes that issue. You might want to move that commit before the commit that adds the test, I just didn't want to change too much in your PR.

I am glad that adding these tests actually uncovered that issue!

lutter

Great! Let's merge it!

lutter · 2022-09-15T00:04:38Z

graphql/src/store/query.rs

+                    }))
+                })
+                .collect::<Result<Vec<EntityFilter>, QueryExecutionError>>()?,
+        ))


Wow .. nice. That's a pretty nasty case. We'll see how it works out in practice, but it could be that all those or's end up being slow and we need to change how the query is generated; but it's easier to determine that once we have an example of that.

We did not account for parent ids being `bytea` when generating queries for children type c which only caused trouble when such entities were used in interfaces.

See #3677 (comment) for a list of possible parent/child types and how they are related

kamilkisiela self-assigned this Jul 15, 2022

kamilkisiela force-pushed the kamil-filter-by-child-interfaces branch 2 times, most recently from 3be9131 to 15ffb98 Compare July 15, 2022 09:24

dotansimha requested review from lutter, leoyvens, azf20 and tilacog July 17, 2022 07:47

lutter requested changes Jul 27, 2022

View reviewed changes

kamilkisiela force-pushed the kamil-filter-by-child-interfaces branch from 15ffb98 to fdee4db Compare July 28, 2022 14:02

kamilkisiela force-pushed the kamil-filter-by-child-interfaces branch from fdee4db to 37f8143 Compare August 24, 2022 12:50

dotansimha requested a review from lutter August 25, 2022 13:41

wminshew reviewed Sep 1, 2022

View reviewed changes

kamilkisiela force-pushed the kamil-filter-by-child-interfaces branch from 433614e to e057635 Compare September 13, 2022 09:56

lutter approved these changes Sep 15, 2022

View reviewed changes

kamilkisiela and others added 3 commits September 16, 2022 09:13

graphql,store: child filters on interface-type fields

7f55c4e

store: Handle is type 'Bytes' correctly for children type c

2231e7d

We did not account for parent ids being `bytea` when generating queries for children type c which only caused trouble when such entities were used in interfaces.

graphql: Test and support more kinds of child/parent relations

93395eb

See #3677 (comment) for a list of possible parent/child types and how they are related

lutter force-pushed the kamil-filter-by-child-interfaces branch from e057635 to 93395eb Compare September 16, 2022 16:18

lutter merged commit 93395eb into master Sep 16, 2022

lutter deleted the kamil-filter-by-child-interfaces branch September 16, 2022 16:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(graph,graphql): filter by child entity (interfaces) #3677

feat(graph,graphql): filter by child entity (interfaces) #3677

kamilkisiela commented Jun 17, 2022 •

edited

Loading

kamilkisiela commented Jul 15, 2022

lutter left a comment

lutter Jul 27, 2022

kamilkisiela Jul 28, 2022

lutter Jul 27, 2022

kamilkisiela Jul 28, 2022

lutter Aug 12, 2022

kamilkisiela commented Aug 8, 2022

lutter commented Aug 12, 2022

kamilkisiela commented Aug 24, 2022 •

edited

Loading

wminshew Sep 1, 2022 •

edited

Loading

kamilkisiela Sep 7, 2022

wminshew Sep 7, 2022

lutter Sep 7, 2022 •

edited

Loading

lutter commented Sep 7, 2022

lutter left a comment

lutter Sep 15, 2022

feat(graph,graphql): filter by child entity (interfaces) #3677

feat(graph,graphql): filter by child entity (interfaces) #3677

Conversation

kamilkisiela commented Jun 17, 2022 • edited Loading

kamilkisiela commented Jul 15, 2022

lutter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kamilkisiela commented Aug 8, 2022

lutter commented Aug 12, 2022

kamilkisiela commented Aug 24, 2022 • edited Loading

wminshew Sep 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lutter Sep 7, 2022 • edited Loading

Choose a reason for hiding this comment

lutter commented Sep 7, 2022

lutter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kamilkisiela commented Jun 17, 2022 •

edited

Loading

kamilkisiela commented Aug 24, 2022 •

edited

Loading

wminshew Sep 1, 2022 •

edited

Loading

lutter Sep 7, 2022 •

edited

Loading