diff --git a/.simplecov b/.simplecov index ae18afbbe..bbd8ae895 100644 --- a/.simplecov +++ b/.simplecov @@ -29,6 +29,8 @@ SimpleCov.start do add_group 'Keyset', %w[gem/lib/pagy/keyset.rb gem/lib/pagy/keyset/active_record.rb gem/lib/pagy/keyset/sequel.rb - gem/lib/pagy/extras/keyset.rb] + gem/lib/pagy/keyset/numeric.rb + gem/lib/pagy/extras/keyset.rb + gem/lib/pagy/extras/keyset_numeric.rb] add_group 'Tests', %w[test] end diff --git a/docs/api/keyset.md b/docs/api/keyset.md index 73f9fb4fb..bfcf22983 100644 --- a/docs/api/keyset.md +++ b/docs/api/keyset.md @@ -29,12 +29,15 @@ integrate it with your app. The "Keyset" pagination, also known as "Cursor Pagination" or "SQL Seek Method" is a technique that avoids the inevitable slowness of querying pages deep into a collection (i.e. when `offset` is a big number, you're going to get slower queries). -This technique comes with that huge advantage and a set of limitations that makes it particularly useful for APIs and less +It is also accurate: while offset pagination can skip or double-show records after insertion and deletions, keyset is always accurate. + +This technique comes with that huge advantages and a set of limitations that makes it particularly useful for APIs and less convenient for UIs in general. -!!!success UI-Compatible Keyset pagination is also available! +!!!success UI-Compatible Keyset Numeric pagination is also available! -If you want the best of the two worlds, check out the [keyset_numeric extra](/docs/extras/keyset_numeric.md) that supports for the `pagy_*nav` and the other Frontend helpers +If you want the best of the two worlds, check out the [keyset_numeric extra](/docs/extras/keyset_numeric.md) that supports for the +`pagy_*nav` and the other Frontend helpers !!! ### Glossary @@ -48,41 +51,51 @@ If you want the best of the two worlds, check out the [keyset_numeric extra](/do | `set` | The `uniquely ordered` `ActiveRecord::Relation` or `Sequel::Dataset` collection to paginate. | | `keyset` | The hash of column/direction pairs. Pagy extracts it from the order of the `set`. | | `keyset attributes` | The hash of keyset-column/record-value pairs of a record. | -| `cutoff` | The point beyond the last record of a `page`.
(It's the encoded string of the `keyset attributes` of the last record in a page) | -| `cutoff_args` | The hash of `cutoff_args` used to filter the page records beyond the `cutoff` | -| `page` | The current `page`, i.e. the page of records fetched beyond the `cutoff` of the **previous page**. | -| `next` | The next `page`, i.e. the page of records fetched beyond the `cutoff` of the **current page**. | - +| `cut` | A point in the `set` that separates the records of two contiguous `page`s. It's the encoded reference to the last record of a `page`. | +| `page` | The current `page`, i.e. the page of records beginning after the `prev_cut`. Also the `:page` variable, which is set to the `prev_cut` | +| `next` | The next `page`, i.e. the page of records beginning after the `next_cut`. Also the `next_cut` value retured by the `next` method. | -### Keyset or Offset pagination? +### Keyset, Numeric or Offset pagination? +++ Keyset !!!success Use Keyset pagination with large dataset and API -You will get the fastest pagination, regardless the table size and the relative position of the page +You will get the fastest pagination and accuracy, regardless the table size and the relative position of the page !!!warning Limited use for UIs Only useful when you don't need any frontend (e.g. infinite pagination) !!! + ++++ Numeric +!!!success The best of the two worlds! + +* The same performance of Keyset +* Most of the Frontend features + +!!!warning Advanced usage + +It requires more effort and resource to setup +!!! +++ Offset -!!!success Use Offset pagination with UIs even with large datasets +!!!success Use Offset pagination with UIs and small DBs -- You will get all the frontend features -- You can avoid the slowness on big tables by simply limiting the `:max_pages` pages: the users would not browse thousands of +* You will get all the frontend features +* You can avoid the slowness by simply limiting the `:max_pages` pages: the users would not browse thousands of records deep into your collection anyway !!!warning Limited use for APIs -Your server will suffer on big data and your API will be slower for no good reasons +* Your server will suffer on big data and your API will be slower for no good reasons +* Not accurate: It can skip or double-show records after insertion and deletions. !!! +++ ## Usage -### Constraints for simple Keyset (non-cached) pagination +### Constraints for simple Keyset pagination !!!success IMPORTANT! @@ -143,9 +156,9 @@ If you need a specific order: ### How Pagy::Keyset works - You pass an `uniquely ordered` `set` and `Pagy::Keyset` pulls the `:limit` of records of the first page. -- It requests the `next` URL by setting its `page` query string param to the `cutoff` of the current page. -- At each request, the new `:page` (i.e. the `cutoff` of the previous page) is decoded into `cutoff_args` that are coupled - with a `where` filter query, and the `:limit` of new records is pulled. +- It requests the `next` URL by setting its `page` query string param to the `next_cut` of the current page. +- At each request, the new `page` is decoded into `cut_args` that are coupled with a `where` filter query, and the `:limit` of new + records is pulled. - You know that you reached the end of the collection when `pagy.next.nil?`. ## ORMs @@ -169,7 +182,7 @@ The constructor takes the `set`, and an optional hash of [variables](#variables) ==- `next` -The next `page`, i.e. the `cutoff` beyond the last record of the **current page**. It is `nil` for the last page. +The next `page`, i.e. the `cut` after the last record of the **current page**. It is `nil` for the last page. ==- `records` @@ -181,7 +194,7 @@ The `Array` of fetched records for the current page. === `:page` -The current page, i.e. the `cutoff` beyond the last record of the **previous page**. Default `nil` for the first page. +The current page, i.e. the `next_cut` of the **previous page**. Default `nil` for the first page. === `:limit` @@ -194,24 +207,6 @@ Boolean variable that enables the tuple comparison e.g. `(brand, id) > (:brand, order, hence it's ignored for mixed order. Check how your DB supports it (your `keyset` should include only `NOT NULL` columns). Default `nil`. -==- `:filter_records` - -**Use this for DB-specific extra optimizations, if you know what you are doing.** - -If the `:filter_records` variable is set to a lambda, pagy will call it with the `set`, `cutoff_args` and `keyset` arguments -instead of using its auto-generated query to filter the records. It must return the filtered set. For example: - -```ruby -filter_records = lambda do |set, cutoff_args, keyset| - # for ActiveRecord set - set.where(my_beyond_cutoff_sql(keyset), **cutoff_args) - # for Sequel set - # set.where(::Sequel.lit(my_beyond_cutoff_sql(keyset), **cutoff_args)) -end - -Pagy::Keyset(set, filter_records:) -``` - ==- `:jsonify_keyset_attributes` A lambda to override the generic json encoding of the `keyset` attributes. Use it when the generic `to_json` method would lose diff --git a/docs/api/keyset_numeric.md b/docs/api/keyset_numeric.md index b0f8ecc27..f9327fc72 100644 --- a/docs/api/keyset_numeric.md +++ b/docs/api/keyset_numeric.md @@ -15,7 +15,9 @@ A [Pagy::Keyset](keyset.md) subclass with numeric pages supporting `pagy_*nav` a ## Overview The regular `Pagy::Keyset` uses the fastest technique for pagination, but it cannot work with any Frontend helper because they require numeric -pages. That's why we created `Pagy::Keyset::Numeric`: it uses keyset pagination AND supports `pagy_*navs` and the other Frontend +pages. + +That's why we created `Pagy::Keyset::Numeric`: it uses keyset pagination AND supports `pagy_*navs` and the other Frontend helpers. !!! @@ -27,7 +29,7 @@ You should also familiarize with the [Pagy::Keyset](keyset.md) class. [!button corners="pill" variant="success" text=":icon-play: Try it now!"](/playground.md#5-keyset-apps) -### Glossary +## Glossary Integrates the [Keyset Glossary](keyset_numeric.md#glossary) @@ -35,19 +37,20 @@ Integrates the [Keyset Glossary](keyset_numeric.md#glossary) |-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `keyset numeric pagination` | The pagy exclusive technique to use `keyset pagination` with numeric pages, supporting `pagy_*navs` and the other Frontend helpers.
The best technique for performance AND functionality! | | `page` | The current page **number** | -| `cutoffs` | The known `cutoff` variables used to keep track of the visited pages. | -| `cache_key` | The key used to locate the `cutoffs` in the cache storage | +| `cuts` | The `cut`s of the pagination known so far, used to keep track of the visited pages. | + +## How Pagy::Keyset::Numeric works -### How Pagy::Keyset::Numeric works +### Caching the known cuts -This `Pagy::Keyset` subclass keeps track of the visited numeric pages: as soon as a page is visited, its `cutoff` gets added to the `cutoffs` array (which is cached) hence it can be retrieved in future requests. +`Pagy::Keyset::Numeric` keeps track of the state of the pagination using the `cuts`Array. As soon as a page is visited, its `next_cut` gets added to the `cuts` array (which is cached) hence it can be retrieved by numeric index in future requests. -While the `cutoffs` data must be cached for this class to work, the cache itself is not handled by this class at all: that is a concern of the [keyset_numeric extra](/docs/extras/keyset_numeric). Here is a simplified example of what must happen with the `cutoffs` at each request: +While the `cuts` data must persist between requests for this class to work, this class does not handle the persistency at all: that is a concern of the [keyset_numeric extra](/docs/extras/keyset_numeric). Here is a simplified example of what must happen with the `cuts` at each request: ```ruby -cutoffs = read_from_cache(cache_key) -pagy = Keyset::Numeric.new(set, cutoffs:, **vars) -write_to_cache(cache_key, pagy.cutoffs) +cuts = read_from_cache(cache_key) +pagy = Keyset::Numeric.new(set, cuts:, **vars) +write_to_cache(cache_key, pagy.cuts) ``` !!! Notice @@ -55,6 +58,23 @@ write_to_cache(cache_key, pagy.cutoffs) The cache handling is external to the `Pagy::Keyset::Numeric` class for easy overriding. See [Understanding the cache](/docs/extras/keyset_numeric#understanding-the-cache) for more details. !!! +### Numeric variables for the Frontend Helpers + +Keeping track of the state through the `cuts` allows to set the numeric variables that the Frontend helpers require (see [Attribute Readers](#attribute-readers)). + +However, it's still keyset pagination, which doesn't know any future page after the `next` page, so we add more pages on the go like we do with `Pagy::Countless`... just better for two reasons: + +1. We don't lose the future pages when we jump back because we can count on the cache. +2. Keyset pagination is A LOT faster than offset pagination. + +### Accurate queries + +`Pagy::Keyset::Numeric` knows all the `cuts` of the current pagination. If the `cuts` don't contain the `next_cut`, then it's a new requested page, and it proceeds exactly like the standard `Pagy::Keyset` class. + +However, if the `cuts` contain the `next_cut` then it's already requested page, and from the time of the first request the number of records pulled with LIMIT may have changed for the page. + +Querying with the LIMIT again, might cause records to get skipped or to appear twice! So, to get it right, it pulls the records BETWEEN the `prev_cut` and `next_cut` avoiding the LIMIT and inluding the right records regardless. + ## Setup See the [Keyset Setup](keyset.md#setup). @@ -90,13 +110,9 @@ The `Array` of fetched records for the current page. This section ontegrates the [Pagy::Keyset variables](keyset.md#variables): -==- `:cache_key_param` - -The name of the cache_key variable. `:cache_key` by default. Pass a different symbol to change it. - -==- `:cutoffs` +==- `:cuts` -Mandatory variable that must persist between requests. +Mandatory array that must persist between requests. ==- `:max_pages` @@ -110,7 +126,7 @@ Resets the pagination in case of overflow, instead of raising a `Pagy::OverflowE ## Attribute Readers -`cutoffs`, `in`, `last`, `limit`, `next`, `page`, `prev`, `vars` +`cuts`, `in`, `last`, `limit`, `next`, `page`, `prev`, `vars` ## Troubleshooting diff --git a/docs/extras/keyset_numeric.md b/docs/extras/keyset_numeric.md index bb9a73d0d..1dbbdea0d 100644 --- a/docs/extras/keyset_numeric.md +++ b/docs/extras/keyset_numeric.md @@ -53,10 +53,10 @@ def pagy_cache_new_key = my_custom_cache.generate_key ## Understanding the cache -This extra uses the `session` object as the cache for `cutoffs` by default, because it's simple and works in any app, at least for +This extra uses the `session` object as the cache for the `cuts` by default, because it's simple and works in any app, at least for prototyping. -Notice that the `cutoffs` array can potentially grow big if you don't use `:max_pages`, especially if your `keyset` contains +Notice that the `cuts` array can potentially grow big if you don't use `:max_pages`, especially if your `keyset` contains multiple ordered columns and more if their size is big. You must be aware of it. !!!danger Do not use the cookie-based session as the cache @@ -91,7 +91,17 @@ It might simplify the handling of the cache considerably, but it will require so ## Variables -See the [Pagy::Keyset::Numeric variables](/docs/api/keyset_numeric#variables) +This section integrates the [Pagy::Keyset::Numeric variables](/docs/api/keyset_numeric#variables). + +==- `:cache_key` + +The key used to locate the `cuts` in the cache storage. + +==- `:cache_key_param` + +The name of the cache key param. It is `:cache_key` by default. Pass a different symbol to change it. + +=== ## Methods @@ -111,7 +121,7 @@ This method handles cache writing. It uses the `session` cache by default. Custo ==- `pagy_cache_new_key` This method must generate and return a new cache key. It is called when a new cache entry is needed. It uses a simple algorithm -that allows 1B number shortened to max 5 letters. Customize it adding expiration or other property to the new entry, before +that allows 1B number shortened to max 5 letters. Customize it by adding expiration or other property to the new entry, before returning the key. === diff --git a/gem/lib/pagy/extras/keyset_numeric.rb b/gem/lib/pagy/extras/keyset_numeric.rb index c59c879ee..d16fa10a6 100644 --- a/gem/lib/pagy/extras/keyset_numeric.rb +++ b/gem/lib/pagy/extras/keyset_numeric.rb @@ -14,12 +14,12 @@ module KeysetNumericExtra def pagy_keyset_numeric(set, **vars) vars[:page] ||= pagy_get_page(vars) # numeric page vars[:limit] ||= pagy_get_limit(vars) - vars[:cache_key] ||= key = params[vars[:cache_key_param] || DEFAULT[:cache_key_param]] || - pagy_cache_new_key - vars[:cutoffs] ||= pagy_cache_read(key) + vars[:cache_key] ||= params[vars[:cache_key_param] || DEFAULT[:cache_key_param]] || + pagy_cache_new_key + vars[:cuts] ||= pagy_cache_read(vars[:cache_key]) pagy = Keyset::Numeric.new(set, **vars) - pagy_cache_write(key, pagy.cutoffs) + pagy_cache_write(vars[:cache_key], pagy.cuts) [pagy, pagy.records] end diff --git a/gem/lib/pagy/keyset.rb b/gem/lib/pagy/keyset.rb index aab77d61d..c7d25c660 100644 --- a/gem/lib/pagy/keyset.rb +++ b/gem/lib/pagy/keyset.rb @@ -45,20 +45,20 @@ def initialize(set, **vars) raise InternalError, 'the set must be ordered' if @keyset.empty? assign_page - assign_cutoff - assign_cutoff_args + assign_cuts + assign_cut_args end - # Assign the cutoff from the cache - def assign_cutoff - @cutoff = @vars[:page] + # Assign the prev_cut from the page variable + def assign_cuts + @prev_cut = @vars[:page] end - # Assign the cutoff_args - def assign_cutoff_args - return unless @cutoff + # Assign the cut_args + def assign_cut_args + return unless @prev_cut - @cutoff_args = cutoff_to_args(@cutoff) + @cut_args = cut_to_args(@prev_cut) end # Assign the page @@ -80,7 +80,7 @@ def assign_page # # ("pets"."animal", "pets"."name", "pets"."id") > (:animal, :name, :id) # - def beyond_cutoff_sql(prefix = nil) + def after_cut_sql(prefix = nil) operator = { asc: '>', desc: '<' } directions = @keyset.values table = @set.model.table_name @@ -103,10 +103,10 @@ def beyond_cutoff_sql(prefix = nil) end end - # Decode a cutoff, check its consistency and returns the cutoff args - def cutoff_to_args(cutoff) - args = JSON.parse(B64.urlsafe_decode(cutoff)).transform_keys(&:to_sym) - raise InternalError, 'cutoff and keyset are not consistent' \ + # Decode a cut, check its consistency and returns the cut args + def cut_to_args(cut) + args = JSON.parse(B64.urlsafe_decode(cut)).transform_keys(&:to_sym) + raise InternalError, 'cut and keyset are not consistent' \ unless args.keys == @keyset.keys typecast_args(args) @@ -121,8 +121,8 @@ def default { **default, page: nil } end - # Derive the cutoff of the current page, beyond which the next page starts - def derive_cutoff + # Derive the next_cut + def derive_next_cut hash = keyset_attributes_from(@records.last) json = @vars[:jsonify_keyset_attributes]&.(hash) || hash.to_json B64.urlsafe_encode(json) @@ -140,21 +140,14 @@ def next records return unless @more - @next ||= derive_cutoff + @next ||= derive_next_cut end # Fetch the array of records for the current page def records @records ||= begin @set = apply_select if select? - if @cutoff_args # i.e. page > 1 - # :nocov: - @set = @vars[:after_latest]&.(@set, @cutoff_args) || # deprecated - @vars[:filter_newest]&.(@set, @cutoff_args, @keyset) || # deprecated - @vars[:filter_records]&.(@set, @cutoff_args, @keyset) || - # :nocov: - filter_records - end + @set = filter_records if @cut_args fetch_records end end diff --git a/gem/lib/pagy/keyset/active_record_adapter.rb b/gem/lib/pagy/keyset/active_record_adapter.rb index 418d5dee7..a2268bd96 100644 --- a/gem/lib/pagy/keyset/active_record_adapter.rb +++ b/gem/lib/pagy/keyset/active_record_adapter.rb @@ -18,7 +18,7 @@ def extract_keyset end # Filter the page records - def filter_records = @set.where(beyond_cutoff_sql, **@cutoff_args) + def filter_records = @set.where(after_cut_sql, **@cut_args) # Get the keyset attributes from the record def keyset_attributes_from(record) = record.slice(*@keyset.keys) @@ -26,7 +26,7 @@ def keyset_attributes_from(record) = record.slice(*@keyset.keys) # Set with selected columns? def select? = !@set.select_values.empty? - # Typecast the cutoff args + # Typecast the cut args def typecast_args(args) @set.model.new(args).slice(args.keys) .to_hash.transform_keys(&:to_sym) diff --git a/gem/lib/pagy/keyset/numeric.rb b/gem/lib/pagy/keyset/numeric.rb index 53abdb569..b06f98e06 100644 --- a/gem/lib/pagy/keyset/numeric.rb +++ b/gem/lib/pagy/keyset/numeric.rb @@ -2,7 +2,6 @@ # frozen_string_literal: true require_relative '../keyset' -require 'digest/sha2' class Pagy # :nodoc: class Keyset @@ -17,10 +16,10 @@ class Sequel < Numeric include SequelAdapter end # Avoid args conflicts in composite SQL fragments - LIMIT_PREFIX = 'limit_' # Prefix for cutoff_args + CUTOFF_PREFIX = 'cutoff_' # Prefix for cut_args include SharedNumericMethods - attr_reader :cutoffs + attr_reader :cuts # Finalize the instance variables needed for the UI def initialize(set, **vars) @@ -28,35 +27,32 @@ def initialize(set, **vars) # Ensure next is called, so the last page used by the UI helpers is known self.next @prev = @page - 1 unless @page == 1 - @last = @cutoffs.size - 1 # 1-based array size + @last = @cuts.size - 1 # 1-based array size @in = @records.size end - # Get the cutoff from the cache - def assign_cutoff - @cutoffs = @vars[:cutoffs] || [nil, nil] - pages = @cutoffs.size - 1 + # Get the cut from the cache + def assign_cuts + @cuts = @vars[:cuts] || [nil, nil] + pages = @cuts.size - 1 if @page > pages raise OverflowError.new(self, :page, "in 1..#{pages}", @page) unless @vars[:reset_overflow] - @page = 1 - @cutoffs = [nil, nil] + @page = 1 + @cuts = [nil, nil] end - @cutoff = @cutoffs[@page] + @prev_cut = @cuts[@page] # nil for page 1 (i.e. begins from begin of set) + @next_cut = @cuts[@page + 1] # known page; nil for last page end - # Assign different args to support the BEYOND_LIMIT_CUTOFF SQL if @limit_cutoff - def assign_cutoff_args - # @limit_cutoff is the cached cutoff for the next page: - # the curent page has been visited, hence it's not the last - return super unless @limit_cutoff ||= @cutoffs[@page + 1] # return super only when it's the last page + # Assign different args to support the AFTER_NEXT_CUT SQL if @next_cut + def assign_cut_args + return super unless @next_cut - # The regular cutoff args are missing for page 1 (which doesn't have a cutoff). - @cutoff_args = cutoff_to_args(@cutoff) if @cutoff - - # The limit_cutoff args are preserved by prefixing them before merging - limit_cutoff_args = cutoff_to_args(@limit_cutoff).transform_keys { |key| :"#{LIMIT_PREFIX}#{key}" } - (@cutoff_args ||= {}).merge!(limit_cutoff_args) + @cut_args = cut_to_args(@prev_cut) if @prev_cut + # The next_cut args are preserved by prefixing them before merging + next_args = cut_to_args(@next_cut).transform_keys { |key| :"#{CUTOFF_PREFIX}#{key}" } + (@cut_args ||= {}).merge!(next_args) end # Assign a numeric page @@ -65,43 +61,44 @@ def assign_page end # Prepare the literal SQL string (complete with the placeholders for value interpolation) - # used to filter the page records if @limit_cutoff; super otherwise. + # used to filter the page records if @next_cut; super otherwise. # - # If @limit_cutoff there are two scenarios, depending on the page number: + # If @next_cut there are two scenarios, depending on the page number: # # 1. If page == 1 - # Pull the inital records till the @limit_cutoff - # SQL logic: NOT BEYOND_LIMIT_CUTOFF + # Pull the inital records and filter them out after the @next_cut + # SQL logic: NOT AFTER NEXT_CUT # # 2. If page > 1 - # Pull the records BETWEEN the current @cutoff and the @limit_cutoff - # SQL logic: BEYOND_CUTOFF AND NOT BEYOND_LIMIT_CUTOFF + # Pull the records BETWEEN the @prev_cut and the @next_cut + # SQL logic: AFTER PREV_CUT AND NOT AFTER NEXT_CUT # - # The BEYOND_CUTOFF SQL is like the regular keyset SQL (calling super). For example: + # The AFTER PREV_CUT SQL is like the regular keyset SQL (calling super). For example: # With a set like Pet.order(animal: :asc, name: :desc, id: :asc) it returns the following string: # # ("pets"."animal" = :animal AND "pets"."name" = :name AND "pets"."id" > :id) OR # ("pets"."animal" = :animal AND "pets"."name" < :name) OR # ("pets"."animal" > :animal) # - # The BEYOND_LIMIT_CUTOFF SQL is used as a repacement of the SQL LIMIT. + # Notice that the placeholders are not prefixed + # + # The AFTER NEXT_CUT SQL is used as a repacement of the SQL LIMIT. # That ensures accuracy in case of records added or removed - # Here is how a BEYOND_LIMIT_CUTOFF looks for the same set: + # Here is how an AFTER NEXT_CUT looks for the same set: # # ("pets"."animal" = :limit_animal AND "pets"."name" = :limit_name AND "pets"."id" > :limit_id) OR # ("pets"."animal" = :limit_animal AND "pets"."name" < :limit_name) OR # ("pets"."animal" > :limit_animal) # - # Notice that the :limit_* placeholder will be replaced with the arguments - # of the next cutoff (beyond which the records belong to another page) - def beyond_cutoff_sql - return super unless @limit_cutoff # super for the last page + # Notice that the placeholders are prefixed by ":next_" which matches with the arguments prefixed keys + def after_cut_sql + return super unless @next_cut # super for the last known page sql = +'' - # Generate the BEYOND_CUTOFF SQL unless @page == 1 that doesn't have a cutoff - sql << "(#{super}) AND " if @cutoff - # Add the BEYOND_LIMIT_CUTOFF SQL, passing the prefix for the placeholdars - sql << "NOT (#{super(LIMIT_PREFIX)})" + # Generate the AFTER PREV_CUT SQL unless @page == 1 (which starts from the fist record in the set) + sql << "(#{super}) AND " if @prev_cut + # Add the AFTER NEXT_CUT SQL, passing the prefix for the placeholdars + sql << "NOT (#{super(CUTOFF_PREFIX)})" end # Add the default variables required by the Frontend @@ -109,23 +106,23 @@ def default { **super, **DEFAULT.slice(:ends, :page, :size, :cache_key_param) } end - # Remove the LIMIT if @limit_cutoff + # Remove the LIMIT if @next_cut def fetch_records - return super unless @limit_cutoff # super for the last page + return super unless @next_cut # super for the last known page - # Disable the LIMIT because it is replaced by the BEYOND_LIMIT_CUTOFF SQL. + # Disable the LIMIT because it is replaced by the AFTER NEXT_CUT SQL. # That keeps the fetching accurate also when records are added or removed from a page alredy visited @more = true @set.limit(nil).to_a end - # Return the next page number, and cache the next cutoff if it's missing from the cache (only last page) + # Return the next page number, and cache the next_cut if it's missing from the cache (only last known page) def next records return if !@more || (@vars[:max_pages] && @page >= @vars[:max_pages]) @next ||= (@page + 1).tap do |next_page| - @cutoffs[next_page] = derive_cutoff unless @limit_cutoff + @cuts[next_page] = derive_next_cut unless @next_cut end end end diff --git a/gem/lib/pagy/keyset/sequel_adapter.rb b/gem/lib/pagy/keyset/sequel_adapter.rb index 6de123842..563a984ff 100644 --- a/gem/lib/pagy/keyset/sequel_adapter.rb +++ b/gem/lib/pagy/keyset/sequel_adapter.rb @@ -28,7 +28,7 @@ def extract_keyset end # Filter the page records - def filter_records = @set.where(::Sequel.lit(beyond_cutoff_sql, **@cutoff_args)) + def filter_records = @set.where(::Sequel.lit(after_cut_sql, **@cut_args)) # Get the keyset attributes from the record def keyset_attributes_from(record) = record.to_hash.slice(*@keyset.keys) @@ -36,7 +36,7 @@ def keyset_attributes_from(record) = record.to_hash.slice(*@keyset.keys) # Set with selected columns? def select? = !@set.opts[:select].nil? - # Typecast the cutoff args + # Typecast the cut args def typecast_args(args) model = @set.opts[:model] model.unrestrict_primary_key if (restricted_pk = model.restrict_primary_key?) diff --git a/test/mock_helpers/app.rb b/test/mock_helpers/app.rb index 9643230db..6b947173b 100644 --- a/test/mock_helpers/app.rb +++ b/test/mock_helpers/app.rb @@ -9,16 +9,18 @@ # Backend and Frontend poor man mock app class MockApp attr_reader :params, :request, :response + attr_accessor :session include Pagy::Backend include Pagy::Frontend # App params are merged into the @request.params (and are all strings) # @params are taken from @request.params and merged with app params (which fixes symbols and strings in params) - def initialize(url: 'http://example.com:3000/foo', params: { page: 3 }) + def initialize(url: 'http://example.com:3000/foo', params: { page: 3 }, session: {}) @request = Rack::Request.new(Rack::MockRequest.env_for(url, params: params)) @params = ActiveSupport::HashWithIndifferentAccess.new(@request.params).merge(params) @response = Rack::Response.new + @session = session end def test_i18n_call diff --git a/test/pagy/extras/keyset_numeric_test.rb b/test/pagy/extras/keyset_numeric_test.rb new file mode 100644 index 000000000..d5fa7ad0e --- /dev/null +++ b/test/pagy/extras/keyset_numeric_test.rb @@ -0,0 +1,64 @@ +# frozen_string_literal: true + +require_relative '../../test_helper' +require 'pagy/extras/keyset_numeric' +require 'pagy/extras/limit' + +require_relative '../../files/models' +require_relative '../../mock_helpers/app' + +describe 'pagy/extras/keyset' do + [Pet, PetSequel].each do |model| + describe '#pagy_keyset_numeric' do + it 'returns Pagy::Keyset::Numeric object and records' do + srand(123) # random cache keys "OiYT_", "Sx3RJ", "9g6R_", "JI4l2", "W4K-s"s + # Page 1: cache_key = "OiYT_" + app = MockApp.new(params: {}) + pagy, records = app.send(:pagy_keyset_numeric, + model.order(:id), + tuple_comparison: true, + limit: 10) + _(pagy).must_be_kind_of Pagy::Keyset::Numeric + _(records.size).must_equal 10 + _(pagy.next).must_equal 2 + + # Page 2 + app = MockApp.new(params: { page: 2, limit: 10, cache_key: pagy.vars[:cache_key] }, + session: app.session) + pagy, records = app.send(:pagy_keyset_numeric, + model.order(:id), + tuple_comparison: true) + _(records.first.id).must_equal 11 + _(pagy.next).must_equal 3 + + # Page 3 + app = MockApp.new(params: { page: 3, limit: 10, cache_key: pagy.vars[:cache_key] }, + session: app.session) + pagy, records = app.send(:pagy_keyset_numeric, + model.order(:id), + tuple_comparison: true) + _(records.first.id).must_equal 21 + _(pagy.next).must_equal 4 + + # Manually create the new key in the session, which would be created next autmatically + # To be sure that it will be skipped and a new one created + app.session = app.session.merge("Sx3RJ" => [nil, nil]) + + _(app.send(:pagy_cache_new_key)).must_equal "9g6R_" + + # Add the cache_key query string param + app = MockApp.new(params: { limit: 10 }) + pagy, _records = app.send(:pagy_keyset_numeric, + model.order(:id)) + _(app.send(:pagy_url_for, pagy, pagy.next)).must_equal "/foo?limit=10&page=2&cache_key=JI4l2" + + # Add the cache_key_param query string param + app = MockApp.new(params: { limit: 10 }) + pagy, _records = app.send(:pagy_keyset_numeric, + model.order(:id), + cache_key_param: :ck) + _(app.send(:pagy_url_for, pagy, pagy.next)).must_equal "/foo?limit=10&page=2&ck=W4K-s" + end + end + end +end diff --git a/test/pagy/keyset_numeric_test.rb b/test/pagy/keyset_numeric_test.rb new file mode 100644 index 000000000..ab8ffd2de --- /dev/null +++ b/test/pagy/keyset_numeric_test.rb @@ -0,0 +1,146 @@ +# frozen_string_literal: true + +require_relative '../test_helper' +require_relative '../files/models' +require_relative '../../gem/lib/pagy/b64' + +require 'pagy/keyset/numeric' + +[Pet, PetSequel].each do |model| + describe "Pagy::Keyset with #{model}" do + describe 'uses optional variables' do + it 'use the :tuple_comparison' do + pagy = Pagy::Keyset::Numeric.new(model.order(:animal, :name, :id), + cuts: [nil, nil, "eyJhbmltYWwiOiJjYXQiLCJuYW1lIjoiRWxsYSIsImlkIjoxOH0"], + page: 2, + limit: 10, + tuple_comparison: true) + records = pagy.records + _(records.size).must_equal 10 + _(records.first.id).must_equal 13 + end + it 'uses :jsonify_keyset_attributes' do + pagy = Pagy::Keyset::Numeric.new(model.order(:id), + cuts: [nil, nil, "eyJpZCI6MTB9"], + page: 2, + limit: 10, + jsonify_keyset_attributes: lambda(&:to_json)) + _(pagy.next).must_equal(3) + _(pagy.instance_variable_get(:@cut_args)).must_equal(id: 10) + end + end + describe 'handles the page/cut' do + it 'handles the page/cut for the first page' do + pagy = Pagy::Keyset::Numeric.new(model.order(:id), + cuts: nil, + limit: 10) + _(pagy.instance_variable_get(:@cut)).must_be_nil + _(pagy.next).must_equal 2 + end + it 'handles the page/cut for the second page' do + pagy = Pagy::Keyset::Numeric.new(model.order(:id), + cuts: [nil, nil, "eyJpZCI6MTB9"], + limit: 10, + page: 2) + _(pagy.instance_variable_get(:@cut_args)).must_equal(id: 10) + _(pagy.records.first.id).must_equal 11 + _(pagy.next).must_equal 3 + _(pagy.cuts).must_equal [nil, nil, "eyJpZCI6MTB9", "eyJpZCI6MjB9"] + end + it 'handles the page/cut for the last page' do + pagy = Pagy::Keyset::Numeric.new(model.order(:id), + cuts: [nil, nil, "eyJpZCI6NDB9"], + limit: 10, + page: 2) + _(pagy.instance_variable_get(:@cut_args)).must_equal(id: 40) + _(pagy.records.first.id).must_equal 41 + _(pagy.next).must_be_nil + end + end + describe 'handles overflow' do + it 'raises OverflowError' do + _ do + Pagy::Keyset::Numeric.new(model.order(:id), + cuts: [nil, nil, "eyJpZCI6MTB9"], + limit: 10, + page: 3) + end.must_raise Pagy::OverflowError + end + it 'resets overflow' do + pagy = Pagy::Keyset::Numeric.new(model.order(:id), + cuts: [nil, nil, "eyJpZCI6MTB9", "eyJpZCI6MjB9"], + reset_overflow: true, + limit: 10, + page: 4) + _(pagy.instance_variable_get(:@cut_args)).must_be_nil + _(pagy.records.first.id).must_equal 1 + _(pagy.next).must_equal 2 + _(pagy.cuts).must_equal [nil, nil, "eyJpZCI6MTB9"] + end + end + describe 'handles the jumping back' do + it 'handles the assign_cut_args jump back to the first page' do + pagy = Pagy::Keyset::Numeric.new(model.order(:id), + cuts: [nil, nil, "eyJpZCI6MTB9"], # last visited 2 + page: 1, + limit: 10) + _(pagy.instance_variable_get(:@cut)).must_be_nil + _(pagy.next).must_equal 2 + _(pagy.instance_variable_get(:@cut_args)).must_equal(cutoff_id: 10) + end + it 'handles the assign_cut_args jump back to the second page' do + pagy = Pagy::Keyset::Numeric.new(model.order(:id), + cuts: [nil, nil, "eyJpZCI6MTB9", "eyJpZCI6MjB9"], + page: 2, + limit: 10) + _(pagy.instance_variable_get(:@cut_args)).must_equal({ :id => 10, :cutoff_id => 20 }) + _(pagy.records.first.id).must_equal 11 + _(pagy.next).must_equal 3 + _(pagy.cuts).must_equal [nil, nil, "eyJpZCI6MTB9", "eyJpZCI6MjB9"] + end + end + describe 'other requirements' do + it 'adds the required columns to the selected values' do + set = model.order(:animal, :name, :id).select(:name) + pagy = Pagy::Keyset::Numeric.new(set, + cuts: nil, + limit: 10) + pagy.records + set = pagy.instance_variable_get(:@set) + _((model == Pet ? set.select_values : set.opts[:select]).sort).must_equal %i[animal id name] + end + end + describe 'integrity of results' do + def slurp_by_page(page: nil, records: [], &block) + result = yield(page) + records << result[:records] + result[:page] ? slurp_by_page(page: result[:page], records:, &block) : records + end + + mixed_set = if model == Pet + model.order(animal: :asc, birthdate: :desc, id: :asc) + elsif model == PetSequel + model.order(:animal, Sequel.desc(:birthdate), :id) + end + [model.order(:id), + model.order(:animal, :name, :id), + mixed_set].each_with_index do |set, i| + it "pulls all the records in set#{i} without repetions" do + cuts = nil + pages = slurp_by_page do |page| + pagy = Pagy::Keyset::Numeric.new(set, + cuts:, + page:, + limit: 9) + cuts = pagy.cuts + { records: pagy.records, page: pagy.next } + end + collection = set.to_a + _(collection.size).must_equal 50 + _(pages.flatten).must_equal collection + _(collection.each_slice(9).to_a).must_equal pages + end + end + end + end +end diff --git a/test/pagy/keyset_test.rb b/test/pagy/keyset_test.rb index f2a6733df..5cf01793d 100644 --- a/test/pagy/keyset_test.rb +++ b/test/pagy/keyset_test.rb @@ -28,7 +28,7 @@ err = assert_raises(Pagy::InternalError) do Pagy::Keyset.new(model.order(:id), limit: 10, page: page_animal_id) end - assert_match(/cutoff and keyset are not consistent/, err.message) + assert_match(/cut and keyset are not consistent/, err.message) end end describe 'uses optional variables' do @@ -47,7 +47,7 @@ limit: 10, jsonify_keyset_attributes: lambda(&:to_json)) _(pagy.next).must_equal("eyJpZCI6MjB9") - _(pagy.instance_variable_get(:@cutoff_args)).must_equal({id: 10}) + _(pagy.instance_variable_get(:@cut_args)).must_equal({id: 10}) end it 'uses :filter_records' do filter_records = if model == Pet @@ -98,21 +98,21 @@ end end end - describe 'handles the page/cutoff' do - it 'handles the page/cutoff for the first page' do + describe 'handles the page/cut' do + it 'handles the page/cut for the first page' do pagy = Pagy::Keyset.new(model.order(:id), limit: 10) - _(pagy.instance_variable_get(:@cutoff)).must_be_nil + _(pagy.instance_variable_get(:@cut)).must_be_nil _(pagy.next).must_equal "eyJpZCI6MTB9" end - it 'handles the page/cutoff for the second page' do + it 'handles the page/cut for the second page' do pagy = Pagy::Keyset.new(model.order(:id), limit: 10, page: "eyJpZCI6MTB9") - _(pagy.instance_variable_get(:@cutoff_args)).must_equal(id: 10) + _(pagy.instance_variable_get(:@cut_args)).must_equal(id: 10) _(pagy.records.first.id).must_equal 11 _(pagy.next).must_equal "eyJpZCI6MjB9" end - it 'handles the page/cutoff for the last page' do + it 'handles the page/cut for the last page' do pagy = Pagy::Keyset.new(model.order(:id), limit: 10, page: "eyJpZCI6NDB9") - _(pagy.instance_variable_get(:@cutoff_args)).must_equal(id: 40) + _(pagy.instance_variable_get(:@cut_args)).must_equal(id: 40) _(pagy.records.first.id).must_equal 41 _(pagy.next).must_be_nil end