Concurrency for pagination

[boryn]

Hi!

I was reading the new v3 documentation and actually I am not sure if it were possible to implement asynchronous fetching of all pages for the "divided in parts" response? If we new total items, it could be easer, as we would know how many pages to fetch. If no total given maybe we could try to pool in batches of let's say 5 pages?

Will Saloon later automatically combine all the results into a collection?

[Sammyjo20]

Hey @boryn!

With v3, asynchronous pagination is still supported - you just have to know exactly how many pages there are in the request. Are you suggesting that it would be cool to batch 5 requests and then if the next 5 each find nothing - then stop?

[boryn]

That's great that pagination works asynchronously. Unfortunately I have not found any information/example on how to use it in the docs. How should we fire asynchronous pagination? And finally, we will have all the pages glued together?

Yes, I think that even if API does not give us the total number of pages we could "shoot" in batches until we receive 0 results.

[boryn]

Update, found docs here: https://docs.saloon.dev/installable-plugins/pagination#asynchronous-pagination-pooling, will try it out!

[Sammyjo20]

Awesome! You will probably need to be responsible for joining them together in the right order because there’s no guarantee of the order they come back in, but it’s so powerful!

[boryn]

Hi @Sammyjo20

I hoped it would be that easy to fetch all the results in a collection (like with synchronous calls):

$spotifyConnector = new SpotifyConnector;
$paginator = $spotifyConnector->paginate(new GetLikedSongsRequest);
$paginator->async();

$paginatorCollection = $paginator->collect();

Unfortunately it does not work :( Should we then combine by ourselves the items from responses like:

$paginatorCollection = collect();
$pool->withResponseHandler(function (Response $response) use ($paginatorCollection) {
    $paginatorCollection->push($response->json('items')[0]);
});

it works but it's not elegant in any way...

[Sammyjo20]

Hey @boryn apologies to hear that, it's just a limitation with asynchronous pagination I believe... Unless you can think of a way of linking them together in order, I can't think of a solution. What you have proposed in your second code block is what I was thinking too.

[boryn]

Actually, the order may not be a problem at all. It's just about convenience, so that the asynchronous results are as easily (and "automagically") converted into a collection as the synchronous results. I believe this could be implemented into the library (unfortunately I'm not that fluent to prepare a PR for this) so that using collection or maybe even dtos would be that simple as with "regular" calls.

PS. Is getPageItems() at all used with asynchronous calls? As we define the 'items'/'data' key (e.g. $response->json('items')) in the paginator class, but I cannot figure out how to use it with asynchronous calls. Or with asynchronous pagination we have at this moment only access to each $response?