Update events are missing the "before" in payload

[flexwende]

Update events are missing the "before"-data in the payload, the "after" is present. This makes it hard to further work with the data and perform the correct update in a second database.

example output of an update event using sink.type = 'stdout':

{"payload":{"op":"u","after":{"ts":"2023-09-25T09:55:23.968497Z","details":"asdf"},"source":{"connector":"timescaledb-event-streamer","ts_ms":1695635729265,"snapshot":false,"db":"timescaledb","schema":"public","table":"event_table","lsn":"0/2244CB0","version":"0.10.1","name":"timescaledb","txId":975},"ts_ms":1695635729116}}

I am not sure if this is intended or can be enabled with a setting that I overlooked.

Versions:

• postgresql: 14
• timescaledb: 2.11.2
• timescaledb-event-streamer: 0.10.1

[noctarius]

You have to enable REPLICA IDENTITY FULL. I just don't remember if this is already merged or if you have to wait for 2.12.

[noctarius]

Yes, 2.12 includes all the necessary changes: timescale/timescaledb#6086

Not sure if the packages are already available though. This version also includes a way to prevent decompressions / decompressions to create insert events.

[flexwende]

ok, thanks for the information :)

[noctarius]

Btw, the related commits are:

• Make hypertables support replica identity timescale/timescaledb#5515 Make hypertables support replica identity
• Allow Replica Identity (Alter Table) on CAGGs timescale/timescaledb#5868 Allow ALTER TABLE ... REPLICA IDENTITY (FULL|INDEX) on materialized hypertables (continuous aggregates)
• Make logrepl markers for (partial) decompressions timescale/timescaledb#5805 Make logrepl markers for (partial) decompressions

Specifically the last one is the PR to help with decompression/recompression by providing a begin- and end-marker before larger decompressions which will potentially create insert events. It's basically hiding some internal behavior.

To enable the support you need to set timescaledb.enable_decompression_logrep_markers = on in postgresql.conf.

The support is already available in the tool for a while and should automatically be enabled when it sees a 2.12 TimescaleDB instance (https://github.com/noctarius/timescaledb-event-streamer/blob/main/internal/replication/logicalreplicationresolver/transactiontracker.go#L67). Without the above property the TimescaleDB extension will not generate the necessary markers though.

I'll add a 2.12 test suite when the docker image is available :)