Rewrite Postgres size query and add postgresql.relation.{tuples,pages,all_visible} + toast_size metrics

[bonnefoa]

What does this PR do?

• Change metric: postgresql.table_size now only reports the size of the main table (removing toast size, fsm and vm)
• Report postgresql.toast_size as table_size + index_size + toast_size.
• New metric: postgresql.toast_size reports the size of the associated toast table
• Optimise the query to lower the amount of stat calls
• Add a tag_none type which doesn't report the tag if the value is none
• Add a partition_of tag if the table is a partition

Motivation

Size metrics needs to call admin function pg_table_size/pg_indexes_size/pg_total_relation_size. Those functions will generate multiple stat syscall behind the hood (see dbsize.c) to get the size of the files on disk which can be expensive with a high number of relation.
We also want to keep the precision of how much of the table is stored in toast table or not.
We manually compute the total to avoid the call to pg_total_relation_size, this will effectively divide the number of stat calls by 2.
We rely on pg_relation_size instead of pg_table_size. pg_relation_size don't include the toast so we can generate toast size and table size in different metric. We lose the size of FSM and VM but their sizes are generally dwarfed by the main table and it's a good trade off to skip those to save stat calls.
pg_indexes_size and pg_relation_size calls still need to lock the relation. We know from relhasindex and reltoastrelid if a relation has indexes or a toast table so we can only call the functions only when necessary.

tag_none has been created to only report the tag partition_of when the table is a partition. Normal tables should not report the tag at all.

Benchmarks and timings

Old query

explain analyze SELECT
  N.nspname,
  relname,
  pg_table_size(C.oid) as table_size,
  pg_indexes_size(C.oid) as index_size,
  pg_total_relation_size(C.oid) as total_size
FROM pg_class C
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE nspname NOT IN ('pg_catalog', 'information_schema') AND
  nspname !~ '^pg_toast' AND
  relkind = 'r';
                                                       QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------
 Hash Join  (cost=7.26..1521.93 rows=284 width=152) (actual time=0.214..23.036 rows=325 loops=1)
   Hash Cond: (c.relnamespace = n.oid)
   ->  Seq Scan on pg_class c  (cost=0.00..1511.11 rows=391 width=72) (actual time=0.009..1.365 rows=391 loops=1)
         Filter: (relkind = 'r'::"char")
         Rows Removed by Filter: 1940
   ->  Hash  (cost=7.17..7.17 rows=8 width=68) (actual time=0.025..0.025 rows=8 loops=1)
         Buckets: 1024  Batches: 1  Memory Usage: 9kB
         ->  Seq Scan on pg_namespace n  (cost=0.00..7.17 rows=8 width=68) (actual time=0.008..0.021 rows=8 loops=1)
               Filter: ((nspname <> ALL ('{pg_catalog,information_schema}'::name[])) AND (nspname !~ '^pg_toast'::text))
               Rows Removed by Filter: 3
 Planning Time: 0.151 ms
 Execution Time: 23.081 ms

stat calls

strace -e trace=newfstatat -p $(pgrep -f "postgres.*idle") -c
strace: Process 62163 attached
^Cstrace: Process 62163 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0        86        56 newfstatat
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000           0        86        56 total

New query

 explain analyze SELECT current_database(),
       s.schemaname, s.table, s.parent_table_name,
       s.relpages, s.reltuples, s.relallvisible,
       s.table_size, s.index_size, s.toast_size, s.table_size + s.index_size + s.toast_size
FROM
    (SELECT
      N.nspname as schemaname,
      relname as table,
      I.inhparent::regclass AS parent_table_name,
      C.relpages,
      C.reltuples,
      C.relallvisible,
      pg_relation_size(C.oid) as table_size,
      CASE WHEN C.relhasindex THEN pg_indexes_size(C.oid) ELSE 0 END as index_size,
      CASE WHEN C.reltoastrelid > 0 THEN pg_relation_size(C.reltoastrelid) ELSE 0 END as toast_size
    FROM pg_class C
    LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
    LEFT JOIN pg_inherits I ON (I.inhrelid = C.oid)
    WHERE NOT (nspname = ANY('{pg_catalog,information_schema}')) AND
      relkind = 'r') as s;
                                                             QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------
 Subquery Scan on s  (cost=1523.25..1569.24 rows=320 width=240) (actual time=1.446..10.677 rows=325 loops=1)
   ->  Hash Join  (cost=1523.25..1563.64 rows=320 width=168) (actual time=1.444..10.563 rows=325 loops=1)
         Hash Cond: (c.relnamespace = n.oid)
         ->  Hash Right Join  (cost=1516.00..1551.77 rows=391 width=93) (actual time=1.366..1.454 rows=391 loops=1)
               Hash Cond: (i.inhrelid = c.oid)
               ->  Seq Scan on pg_inherits i  (cost=0.00..30.40 rows=2040 width=8) (actual time=0.002..0.002 rows=0 loops=1)
               ->  Hash  (cost=1511.11..1511.11 rows=391 width=89) (actual time=1.360..1.361 rows=391 loops=1)
                     Buckets: 1024  Batches: 1  Memory Usage: 56kB
                     ->  Seq Scan on pg_class c  (cost=0.00..1511.11 rows=391 width=89) (actual time=0.007..1.284 rows=391 loops=1)
                           Filter: (relkind = 'r'::"char")
                           Rows Removed by Filter: 1940
         ->  Hash  (cost=7.14..7.14 rows=9 width=68) (actual time=0.020..0.021 rows=9 loops=1)
               Buckets: 1024  Batches: 1  Memory Usage: 9kB
               ->  Seq Scan on pg_namespace n  (cost=0.00..7.14 rows=9 width=68) (actual time=0.007..0.017 rows=9 loops=1)
                     Filter: (nspname <> ALL ('{pg_catalog,information_schema}'::name[]))
                     Rows Removed by Filter: 2
 Planning Time: 0.239 ms
 Execution Time: 10.730 ms
(18 rows)

Stat calls

strace -e trace=newfstatat -p $(pgrep -f "postgres.*idle") -c
strace: Process 62163 attached
^Cstrace: Process 62163 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0        23        16 newfstatat
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000           0        23        16 total

We went from 86 stat calls to 23

Additional Notes

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached
• If the PR doesn't need to be tested during QA, please add a qa/skip-qa label.

[codecov]

Codecov Report

Merging #14500 (f780d20) into master (4e078b6) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Flag	Coverage Δ
postgres	91.33% <100.00%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

[github-actions]

Test Results

     12 files       12 suites   11m 45s ⏱️
   210 tests    208 ✔️   2 💤 0 ❌
1 266 runs  1 222 ✔️ 44 💤 0 ❌

Results for commit f780d20.

♻️ This comment has been updated with latest results.

[justiniso]

For my education, the flag -f "postgres.*idle", why is idle in there?

[bonnefoa]

For my education, the flag -f "postgres.*idle", why is idle in there?

I wanted to look at the backend process matching the psql session which will run the query I'm testing.
If you check all processes with a postgres's name, you're gonna match all postgres's processes + psql backend process

pgrep -a -f "postgres:"
1017179 postgres: checkpointer
1017180 postgres: background writer
1017182 postgres: walwriter
1017183 postgres: autovacuum launcher
1017184 postgres: logical replication launcher
1826329 postgres: postgres postgres [local] idle

An idle session will be tagged as idle in the process name so the "postgres.*idle" allowed me to match the backend process of the psql session I have opened.

pgrep -a -f "postgres.*idle"
1826329 postgres: postgres postgres [local] idle

[JordanP]

@bonnefoa thanks for doing this. Is this going to superseed #14156 and #14056 ? (cc @FlipEnergy)

[bonnefoa]

Here are perf before and after the patch


Time spent InterpExpr went from 87% to 56%

[alexandre-normand]

Thanks, @bonnefoa . Nice detailed analysis of the impact of the changes, too.