Enhance pagination with parsed Link header

[AlbertKogan]

What problem are you trying to solve?

Paginating some GitHub API endpoints with extra logic (logging etc) when reaching internal limit of fetched pages.

Describe the feature

Would be nice to have more control on pagination logic. For example, parse-link-header provides well structured format of parsed Link header.

Raw Link header:

<https://<api_url>?page=3&per_page=100>; rel="next",
<https://<api_url>?page=1&per_page=100>; rel="prev"; pet="cat",
<https://<api_url>?page=5&per_page=100>; rel="last"

Parsed result:

{
  next: {
    page: '3',
    per_page: '100',
    rel: 'next',
    url: '<api_url>?page=3&per_page=100',
  },
  prev: {
    page: '1',
    per_page: '100',
    rel: 'prev',
    pet: 'cat',
    url: '<api_url>?page=1&per_page=100',
  },
  last: {
    page: '5',
    per_page: '100',
    rel: 'last',
    url: '<api_url>?page=5&per_page=100',
  },
}

If this data will be accessible, for example from pagination.shouldContinue, it will help to add custom logic based on next or previous pages.

Example

const got = require('got');
const limit = 10;

got.paginate('https://example.com/items', {
  searchParams: {
    limit,
  },
  pagination: {
    shouldContinue: (item, allItems, currentItems, pagination) => {
      if (pagination.next.page > limit) {
        log.warn('Reached page limit');
        return false;
      }
      return true;
    },
  },
});

Checklist

• I have read the documentation and made sure this feature doesn't already exist.

[nrajkum2-uiuc]

@szmarczak I'm interested in doing this. I suppose we'd first have to agree with the metadata of the pagination would look like. I think the schema suggested by @AlbertKogan is a fine start. Although support for something like pagination tokens could be provided as well.

[szmarczak]

I would prefer something like:

[
	{url: URL, rel: 'next'},
	{url: URL, rel: 'prev', pet: 'cat'},
	{url: URL, rel: 'last'}
]

[szmarczak]

#1667 fixes this. Got will export a function calledparseLinkHeader:

p(`<https://one.example.com>; rel="preconnect", <https://two.example.com>; rel="preconnect", <https://three.example.com>; rel="preconnect"`)
[
  {
    reference: 'https://one.example.com',
    parameters: { rel: '"preconnect"' }
  },
  {
    reference: 'https://two.example.com',
    parameters: { rel: '"preconnect"' }
  },
  {
    reference: 'https://three.example.com',
    parameters: { rel: '"preconnect"' }
  }
]