Recursive inclusion of related models

[bajtos]

In LB3, it's possible to recursively include models related to included models. Let's implement that feature in LB4 too.

For example, consider the domain model where Author has many Post instances and each Post instance has many Comment instances.

Users can fetch authors with posts and comments using the following query:

userRepo.find({
  include: [
    {
      relation: 'posts',
      scope: {
        include: [{relation: 'comments'}],
      },
    },
  ],
});

LB3 also offer few simpler alternatives how to express the same query:

userRepo.find({include: {posts: 'comments'}});
userRepo.find({include: {posts: {relation: 'comments'}}});

LB3 test suite:
loopback-datasource-juggler/test/include.test.js#L175-L195

See also #3387

Acceptance criteria

TBD - will be filled by the team.

[st119848]

waiting for this feature

[eyasalmamoun]

Plus one

[upscreen]

let me know if we can contribute

[marioestradarosa]

@upscreen , it would be nice to have your contribution.

See our Contributing guide and Submitting a pull request to LoopBack 4 to get started.

[malek0512]

Waiting for this feature also.
Is there any progress ?

[raymondfeng]

@agnes512 Can you clarify?

[agnes512]

@malek0512 @st119848 @eyasalmamoun @upscreen I think this is being done in the issue #3453 and the PR #4263. The documentations can be found in the Query multiple relations section of each relation.

@raymondfeng from the description, I think the feature is already implemented, and I didn't realize until you ping me, thanks. Closing it.

[tcmal]

Is there any way to include nested relationships infinitely/to N degrees using this? For example,

@model()
export class Comment extends Entity {
	// ...

	@hasMany(() => Comment, {keyTo: 'replyTo'})
	replies: Comment[];

My current attempt is:

let scopeRecurse: any = {relation: "replies", scope: {}};
scopeRecurse.scope.include = [scopeRecurse];

const filter: Filter = {
	// ...
	include: [scopeRecurse]
}

const comments: CommentWithRelations[] = await this.postRepository.comments(id).find(filter);

but this only includes the first level.

Ideally there would be some sort of option for recursive relationships, ie {relation: "replies", depth: 10}.

[malek0512]

@tcmal you can actually include multiple nested relations with the same fashion thanks to the scope field.
See example below :
filter = {include:[{relation: "replies", scope:{ include: [{relation: "nestedRelation"}] } }, etc..]
See also documentation here

[tcmal]

Yes, that's what I'm trying to do, except the same relation nested an infinite amount of times.

So comments have many replies, what I'm trying to do is fetch a comment, its replies, its replies' replies, its replies' replies' replies, etc. until there's no more replies.

The code I posted is the same as what you posted, is a trick to essentially get:

let scopeRecurse = {relation: "replies", scope: {include: [scopeRecurse]}

If you try to access each level you get:

scopeRecurse.relation // replies
scopeRecurse.scope.include[0].relation // replies
scopeRecurse.scope.include[0].relation // replies
scopeRecurse.scope.include[0].scope.include[0].relation // replies

However many levels deep you go. I'd expect this to fetch replies for every comment fetched by the query, no matter how many levels deep, but instead it only fetches the first level of replies.

[agnes512]

@tcmal we don't support recursive inclusion in such form {relation: "replies", depth: 10}. But it should be able to traverse nested relations even the relation name is the same as long as the foreign key is set up correctly. I will check on my end to see if it only fetches the first level of replies in your case.

[agnes512]

@tcmal I just tried it on my end, and it seems work.
My model:

  @property({
    type: 'number',
    id: true,
    generated: true,
  })
  id: number;

  @property({
    type: 'string',
  })
  content?: string;

  @hasMany(() => Comment)
  replies: Comment[];

  @property({
    type: 'number',
  })
  commentId?: number;

My repository:

export class CommentRepository extends DefaultCrudRepository<
  Comment,
  typeof Comment.prototype.id,
  CommentRelations
  > {

  public readonly replies: HasManyRepositoryFactory<Comment, typeof Comment.prototype.id>;

  constructor(
    @inject('datasources.db') dataSource: DbDataSource,
  ) {
    super(Comment, dataSource);
    this.replies = this.createHasManyRepositoryFactoryFor('replies', Getter.fromValue(this));
    this.registerInclusionResolver('replies', this.replies.inclusionResolver);
  }
}

With filter:

    const fil = {
      where: {id: 1},
      include: [
        {
          relation: 'replies',
          scope: {
            include: [{relation: 'replies'}],
          },
        },
      ],
    };

I got the expected result.

I am not sure if something's wrong with let scopeRecurse = {relation: "replies", scope: {include: [scopeRecurse]}

[tcmal]

I might not be explaining myself right.

I have data:

{
      "1": "{\"content\":\"a\",\"commentId\":1}",
      "2": "{\"replyTo\":1,\"content\":\"b\",\"commentId\":2}",
      "3": "{\"replyTo\":2,\"content\":\"c\",\"commentId\":3}",
      "4": "{\"replyTo\":3,\"content\":\"d\",\"commentId\":4}",
      "5": "{\"replyTo\":4,\"content\":\"e\",\"commentId\":5}",
      "6": "{\"replyTo\":5,\"content\":\"f\",\"commentId\":6}",
}

In this case I'm expecting it to return all 6 of those comments, but when I try your example it only returns up to c:

{
  "commentId": 1,
  "content": "a",
  "replies": [
    {
      "replyTo": 1,
      "commentId": 2,
      "content": "b",
      "replies": [
        {
          "userName": "tcmal",
          "replyTo": 2,
          "commentId": 3,
          "content": "c",
        }
      ]
    }
  ],
}

[agnes512]

Sorry I didn't make it clear. The example I have above is to check if it works well with nested relations. And it does.

As for recursively querying the related data, we currently don't support such a recursive way due to the design. i.e you will need to write the filter yourself to query a certain levels of relations. The filter above is an example that queries two levels of replies.

For your case, you need to set the scope field to include the next level replies several times. It might look like this eventually:

{
      // where: {id: 1},
      include: [
        {
          relation: 'replies',
          scope: {
            include: [{relation: 'replies',
            scope: {
              include: [{relation: 'replies',
              scope: {
                include: [{relation: 'replies',
                ... include more nested relations ...}],
              },}],
            }}],
          },
        },
      ],
    };

[tcmal]

Would it be possible to have this implemented? At the very least as a helper function that returns the nested struct:

let atEachLevel = {relation: "replies"}
let root = {...atEachLevel};
let currentNode = root;
for (let n = 0; n < maxDepth; n++) {
	currentNode.scope = {include: [{...atEachLevel}]};
	currentNode = currentNode.scope.include[0];
}

[agnes512]

@tcmal I think your use case is valid. I've created a story for it, see #5080. Feel free to join the discussion ( or submit a PR :D ). Thanks!