Skip to content

Commit

Permalink
Finalized naming and tests
Browse files Browse the repository at this point in the history
  • Loading branch information
@ddnexus
ddnexus committed Dec 13, 2024
1 parent 6dd14f1 commit 0707a86
Show file tree
Hide file tree
Showing 13 changed files with 372 additions and 147 deletions.
4 changes: 3 additions & 1 deletion .simplecov
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ SimpleCov.start do
add_group 'Keyset', %w[gem/lib/pagy/keyset.rb
gem/lib/pagy/keyset/active_record.rb
gem/lib/pagy/keyset/sequel.rb
gem/lib/pagy/extras/keyset.rb]
gem/lib/pagy/keyset/numeric.rb
gem/lib/pagy/extras/keyset.rb
gem/lib/pagy/extras/keyset_numeric.rb]
add_group 'Tests', %w[test]
end
71 changes: 33 additions & 38 deletions docs/api/keyset.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,15 @@ integrate it with your app.
The "Keyset" pagination, also known as "Cursor Pagination" or "SQL Seek Method" is a technique that avoids the inevitable slowness
of querying pages deep into a collection (i.e. when `offset` is a big number, you're going to get slower queries).

This technique comes with that huge advantage and a set of limitations that makes it particularly useful for APIs and less
It is also accurate: while offset pagination can skip or double-show records after insertion and deletions, keyset is always accurate.

This technique comes with that huge advantages and a set of limitations that makes it particularly useful for APIs and less
convenient for UIs in general.

!!!success UI-Compatible Keyset pagination is also available!
!!!success UI-Compatible Keyset Numeric pagination is also available!

If you want the best of the two worlds, check out the [keyset_numeric extra](/docs/extras/keyset_numeric.md) that supports for the `pagy_*nav` and the other Frontend helpers
If you want the best of the two worlds, check out the [keyset_numeric extra](/docs/extras/keyset_numeric.md) that supports for the
`pagy_*nav` and the other Frontend helpers
!!!

### Glossary
Expand All @@ -48,41 +51,51 @@ If you want the best of the two worlds, check out the [keyset_numeric extra](/do
| `set` | The `uniquely ordered` `ActiveRecord::Relation` or `Sequel::Dataset` collection to paginate. |
| `keyset` | The hash of column/direction pairs. Pagy extracts it from the order of the `set`. |
| `keyset attributes` | The hash of keyset-column/record-value pairs of a record. |
| `cutoff` | The point beyond the last record of a `page`. <br/>(It's the encoded string of the `keyset attributes` of the last record in a page) |
| `cutoff_args` | The hash of `cutoff_args` used to filter the page records beyond the `cutoff` |
| `page` | The current `page`, i.e. the page of records fetched beyond the `cutoff` of the **previous page**. |
| `next` | The next `page`, i.e. the page of records fetched beyond the `cutoff` of the **current page**. |

| `cut` | A point in the `set` that separates the records of two contiguous `page`s. It's the encoded reference to the last record of a `page`. |
| `page` | The current `page`, i.e. the page of records beginning after the `prev_cut`. Also the `:page` variable, which is set to the `prev_cut` |
| `next` | The next `page`, i.e. the page of records beginning after the `next_cut`. Also the `next_cut` value retured by the `next` method. |

### Keyset or Offset pagination?
### Keyset, Numeric or Offset pagination?

+++ Keyset

!!!success Use Keyset pagination with large dataset and API

You will get the fastest pagination, regardless the table size and the relative position of the page
You will get the fastest pagination and accuracy, regardless the table size and the relative position of the page

!!!warning Limited use for UIs

Only useful when you don't need any frontend (e.g. infinite pagination)
!!!

+++ Numeric
!!!success The best of the two worlds!

* The same performance of Keyset
* Most of the Frontend features

!!!warning Advanced usage

It requires more effort and resource to setup
!!!

+++ Offset
!!!success Use Offset pagination with UIs even with large datasets
!!!success Use Offset pagination with UIs and small DBs

- You will get all the frontend features
- You can avoid the slowness on big tables by simply limiting the `:max_pages` pages: the users would not browse thousands of
* You will get all the frontend features
* You can avoid the slowness by simply limiting the `:max_pages` pages: the users would not browse thousands of
records deep into your collection anyway

!!!warning Limited use for APIs

Your server will suffer on big data and your API will be slower for no good reasons
* Your server will suffer on big data and your API will be slower for no good reasons
* Not accurate: It can skip or double-show records after insertion and deletions.
!!!
+++

## Usage

### Constraints for simple Keyset (non-cached) pagination
### Constraints for simple Keyset pagination

!!!success IMPORTANT!

Expand Down Expand Up @@ -143,9 +156,9 @@ If you need a specific order:
### How Pagy::Keyset works

- You pass an `uniquely ordered` `set` and `Pagy::Keyset` pulls the `:limit` of records of the first page.
- It requests the `next` URL by setting its `page` query string param to the `cutoff` of the current page.
- At each request, the new `:page` (i.e. the `cutoff` of the previous page) is decoded into `cutoff_args` that are coupled
with a `where` filter query, and the `:limit` of new records is pulled.
- It requests the `next` URL by setting its `page` query string param to the `next_cut` of the current page.
- At each request, the new `page` is decoded into `cut_args` that are coupled with a `where` filter query, and the `:limit` of new
records is pulled.
- You know that you reached the end of the collection when `pagy.next.nil?`.

## ORMs
Expand All @@ -169,7 +182,7 @@ The constructor takes the `set`, and an optional hash of [variables](#variables)

==- `next`

The next `page`, i.e. the `cutoff` beyond the last record of the **current page**. It is `nil` for the last page.
The next `page`, i.e. the `cut` after the last record of the **current page**. It is `nil` for the last page.

==- `records`

Expand All @@ -181,7 +194,7 @@ The `Array` of fetched records for the current page.

=== `:page`

The current page, i.e. the `cutoff` beyond the last record of the **previous page**. Default `nil` for the first page.
The current page, i.e. the `next_cut` of the **previous page**. Default `nil` for the first page.

=== `:limit`

Expand All @@ -194,24 +207,6 @@ Boolean variable that enables the tuple comparison e.g. `(brand, id) > (:brand,
order, hence it's ignored for mixed order. Check how your DB supports it (your `keyset` should include only `NOT NULL` columns).
Default `nil`.

==- `:filter_records`

**Use this for DB-specific extra optimizations, if you know what you are doing.**

If the `:filter_records` variable is set to a lambda, pagy will call it with the `set`, `cutoff_args` and `keyset` arguments
instead of using its auto-generated query to filter the records. It must return the filtered set. For example:

```ruby
filter_records = lambda do |set, cutoff_args, keyset|
# for ActiveRecord set
set.where(my_beyond_cutoff_sql(keyset), **cutoff_args)
# for Sequel set
# set.where(::Sequel.lit(my_beyond_cutoff_sql(keyset), **cutoff_args))
end

Pagy::Keyset(set, filter_records:)
```

==- `:jsonify_keyset_attributes`

A lambda to override the generic json encoding of the `keyset` attributes. Use it when the generic `to_json` method would lose
Expand Down
50 changes: 33 additions & 17 deletions docs/api/keyset_numeric.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@ A [Pagy::Keyset](keyset.md) subclass with numeric pages supporting `pagy_*nav` a
## Overview

The regular `Pagy::Keyset` uses the fastest technique for pagination, but it cannot work with any Frontend helper because they require numeric
pages. That's why we created `Pagy::Keyset::Numeric`: it uses keyset pagination AND supports `pagy_*navs` and the other Frontend
pages.

That's why we created `Pagy::Keyset::Numeric`: it uses keyset pagination AND supports `pagy_*navs` and the other Frontend
helpers.

!!!
Expand All @@ -27,34 +29,52 @@ You should also familiarize with the [Pagy::Keyset](keyset.md) class.

[!button corners="pill" variant="success" text=":icon-play: Try it now!"](/playground.md#5-keyset-apps)

### Glossary
## Glossary

Integrates the [Keyset Glossary](keyset_numeric.md#glossary)

| Term | Description |
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `keyset numeric pagination` | The pagy exclusive technique to use `keyset pagination` with numeric pages, supporting `pagy_*navs` and the other Frontend helpers.<br/>The best technique for performance AND functionality! |
| `page` | The current page **number** |
| `cutoffs` | The known `cutoff` variables used to keep track of the visited pages. |
| `cache_key` | The key used to locate the `cutoffs` in the cache storage |
| `cuts` | The `cut`s of the pagination known so far, used to keep track of the visited pages. |

## How Pagy::Keyset::Numeric works

### How Pagy::Keyset::Numeric works
### Caching the known cuts

This `Pagy::Keyset` subclass keeps track of the visited numeric pages: as soon as a page is visited, its `cutoff` gets added to the `cutoffs` array (which is cached) hence it can be retrieved in future requests.
`Pagy::Keyset::Numeric` keeps track of the state of the pagination using the `cuts`Array. As soon as a page is visited, its `next_cut` gets added to the `cuts` array (which is cached) hence it can be retrieved by numeric index in future requests.

While the `cutoffs` data must be cached for this class to work, the cache itself is not handled by this class at all: that is a concern of the [keyset_numeric extra](/docs/extras/keyset_numeric). Here is a simplified example of what must happen with the `cutoffs` at each request:
While the `cuts` data must persist between requests for this class to work, this class does not handle the persistency at all: that is a concern of the [keyset_numeric extra](/docs/extras/keyset_numeric). Here is a simplified example of what must happen with the `cuts` at each request:

```ruby
cutoffs = read_from_cache(cache_key)
pagy = Keyset::Numeric.new(set, cutoffs:, **vars)
write_to_cache(cache_key, pagy.cutoffs)
cuts = read_from_cache(cache_key)
pagy = Keyset::Numeric.new(set, cuts:, **vars)
write_to_cache(cache_key, pagy.cuts)
```

!!! Notice

The cache handling is external to the `Pagy::Keyset::Numeric` class for easy overriding. See [Understanding the cache](/docs/extras/keyset_numeric#understanding-the-cache) for more details.
!!!

### Numeric variables for the Frontend Helpers

Keeping track of the state through the `cuts` allows to set the numeric variables that the Frontend helpers require (see [Attribute Readers](#attribute-readers)).

However, it's still keyset pagination, which doesn't know any future page after the `next` page, so we add more pages on the go like we do with `Pagy::Countless`... just better for two reasons:

1. We don't lose the future pages when we jump back because we can count on the cache.
2. Keyset pagination is A LOT faster than offset pagination.

### Accurate queries

`Pagy::Keyset::Numeric` knows all the `cuts` of the current pagination. If the `cuts` don't contain the `next_cut`, then it's a new requested page, and it proceeds exactly like the standard `Pagy::Keyset` class.

However, if the `cuts` contain the `next_cut` then it's already requested page, and from the time of the first request the number of records pulled with LIMIT may have changed for the page.

Querying with the LIMIT again, might cause records to get skipped or to appear twice! So, to get it right, it pulls the records BETWEEN the `prev_cut` and `next_cut` avoiding the LIMIT and inluding the right records regardless.

## Setup

See the [Keyset Setup](keyset.md#setup).
Expand Down Expand Up @@ -90,13 +110,9 @@ The `Array` of fetched records for the current page.

This section ontegrates the [Pagy::Keyset variables](keyset.md#variables):

==- `:cache_key_param`

The name of the cache_key variable. `:cache_key` by default. Pass a different symbol to change it.

==- `:cutoffs`
==- `:cuts`

Mandatory variable that must persist between requests.
Mandatory array that must persist between requests.

==- `:max_pages`

Expand All @@ -110,7 +126,7 @@ Resets the pagination in case of overflow, instead of raising a `Pagy::OverflowE

## Attribute Readers

`cutoffs`, `in`, `last`, `limit`, `next`, `page`, `prev`, `vars`
`cuts`, `in`, `last`, `limit`, `next`, `page`, `prev`, `vars`

## Troubleshooting

Expand Down
18 changes: 14 additions & 4 deletions docs/extras/keyset_numeric.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,10 @@ def pagy_cache_new_key = my_custom_cache.generate_key

## Understanding the cache

This extra uses the `session` object as the cache for `cutoffs` by default, because it's simple and works in any app, at least for
This extra uses the `session` object as the cache for the `cuts` by default, because it's simple and works in any app, at least for
prototyping.

Notice that the `cutoffs` array can potentially grow big if you don't use `:max_pages`, especially if your `keyset` contains
Notice that the `cuts` array can potentially grow big if you don't use `:max_pages`, especially if your `keyset` contains
multiple ordered columns and more if their size is big. You must be aware of it.

!!!danger Do not use the cookie-based session as the cache
Expand Down Expand Up @@ -91,7 +91,17 @@ It might simplify the handling of the cache considerably, but it will require so

## Variables

See the [Pagy::Keyset::Numeric variables](/docs/api/keyset_numeric#variables)
This section integrates the [Pagy::Keyset::Numeric variables](/docs/api/keyset_numeric#variables).

==- `:cache_key`

The key used to locate the `cuts` in the cache storage.

==- `:cache_key_param`

The name of the cache key param. It is `:cache_key` by default. Pass a different symbol to change it.

===

## Methods

Expand All @@ -111,7 +121,7 @@ This method handles cache writing. It uses the `session` cache by default. Custo
==- `pagy_cache_new_key`

This method must generate and return a new cache key. It is called when a new cache entry is needed. It uses a simple algorithm
that allows 1B number shortened to max 5 letters. Customize it adding expiration or other property to the new entry, before
that allows 1B number shortened to max 5 letters. Customize it by adding expiration or other property to the new entry, before
returning the key.

===
8 changes: 4 additions & 4 deletions gem/lib/pagy/extras/keyset_numeric.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,12 @@ module KeysetNumericExtra
def pagy_keyset_numeric(set, **vars)
vars[:page] ||= pagy_get_page(vars) # numeric page
vars[:limit] ||= pagy_get_limit(vars)
vars[:cache_key] ||= key = params[vars[:cache_key_param] || DEFAULT[:cache_key_param]] ||
pagy_cache_new_key
vars[:cutoffs] ||= pagy_cache_read(key)
vars[:cache_key] ||= params[vars[:cache_key_param] || DEFAULT[:cache_key_param]] ||
pagy_cache_new_key
vars[:cuts] ||= pagy_cache_read(vars[:cache_key])

pagy = Keyset::Numeric.new(set, **vars)
pagy_cache_write(key, pagy.cutoffs)
pagy_cache_write(vars[:cache_key], pagy.cuts)
[pagy, pagy.records]
end

Expand Down
Loading

0 comments on commit 0707a86

Please sign in to comment.