Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
zolstein authored Mar 22, 2024
1 parent 969f4b4 commit 975bf56
Showing 1 changed file with 31 additions and 8 deletions.
39 changes: 31 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ pgx-collect is a _nearly_ drop-in replacement for the pgx collection functions t
lower overhead.

pgx offers a very convenient set of functions for parsing values from rows, while handling
all of the boilerplate. However, these functions sometimes do non-trivial amounts of
repeated work when processing each row, and cause unnecessary allocations that scale with
O(rows) or even O(rows * columns), which puts more load on the garbage collector. By
rewriting these functions minimize repeated work and allocations, this library provides
the same convenient functionality while improving performance.
all of the boilerplate. However, these functions sometimes repeat significant amounts of
work when processing each row, and cause unnecessary allocations that scale with O(rows)
or even O(rows * columns), which puts more load on the garbage collector. By rewriting
these functions to minimize repeated work and allocations, this library provides the same
convenient functionality while improving performance.

# Usage

Expand Down Expand Up @@ -50,12 +50,35 @@ var customFunc pgx.RowToFunc[Record]
values, err := pgxc.CollectRows(rows, pgxc.Adapt(customFunc))
```

# Who would benefit from this library?

On some level, everyone using the functions CollectRows or AppendRows could benefit.
The biggest wins will be seen by people parsing queries with many rows / columns
using the reflective struct-mapping RowTo functions. I.e. RowToStructBy....

By comparison, people querying small numbers of rows and parsing rows as primatives
or maps, are unlikely to see much benefit (for now). pgx-collect support for these
operations is provided mainly for convenience, rather than because it can provide
a large speedup.

# How much faster is it?

When using AppendRows to query 1000 rows and 4 columns in a benchmark, I've demonstrated
a 3-4x reduction in total runtime, a 10x reduction in memory allocated,
and a 100x reduction in objects allocated. These numbers will vary greatly by workload.
Real-world benefits are unlikely to be this large, but they may still improve query
throughput, especially in pathological cases.

I encourage anyone using these pgx's collect functions to try out pgx-collect, measure
the difference, and share the results, particularly if there are further opportunities
to optimize.

# Why not merge into pgx? Why create a separate library?

I would love for this code to be merged into pgx. However, even though the pgx-collect API
is designed to have the same code produce the same output for all idiomatic use-cases, the
performance improvements require changing the types of some values - thus, if these
changes were integrated, some working code could stop compiling.
is designed as a drop-in replacement for all idiomatic use-cases, the performance improvements
required changing the types of some values in the public API - thus, if these changes were
integrated into pgx, some working code could stop compiling.

If at some time in the future pgx can make backward-incompatible changes, I hope it
integrates these changes. I also hope they can integrate as many of the optimizations as
Expand Down

0 comments on commit 975bf56

Please sign in to comment.