Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PostgresChannelMessageTableSubscriber not properly handling Database Failover #9061

Closed
joshiste opened this issue Apr 1, 2024 · 3 comments · Fixed by #9062
Closed

PostgresChannelMessageTableSubscriber not properly handling Database Failover #9061

joshiste opened this issue Apr 1, 2024 · 3 comments · Fixed by #9062
Labels
(EOL) for: backport-to-6.1.x (EOL) for: backport-to-6.2.x in: jdbc status: backported type: bug
Milestone
6.3.0-RC1

Comments

@joshiste
Copy link
Contributor

joshiste commented Apr 1, 2024 •
edited
Loading

In what version(s) of Spring Integration are you seeing this issue?

6.x

Describe the bug

When the database is doing a failover the PostgresChannelMessageTableSubscriber doesn't notice and will never receive new notifications.
For example on AWS RDS instance with multi-AZ standby, when performing a failover the IP address for the DNS does change. But PostgresChannelMessageTableSubscriber never re-establishes a connection, nor has some other measurement in place to validate that the connection is still valid.

To Reproduce

Perform a DB failover. After this no more messages will be processed.

Expected behavior

Eventually the PostgresChannelMessageTableSubscriber obtains a new connection after failover, and message processing will work again.

Sample

Sorry, not being able to provide a complete sample, as this involves infrastructure 🤷‍♂️
@joshiste joshiste added status: waiting-for-triage The issue need to be evaluated and its future decided type: bug labels Apr 1, 2024
@joshiste
Copy link
Contributor Author

joshiste commented Apr 1, 2024

I'd also be happy to contribute a PR for this.

I would propose to use a configurable timeout for retrieving PGNotifications instead of waiting forever and when the received list is empty to re-new the connection

joshiste added a commit to joshiste/spring-integration that referenced this issue Apr 1, 2024
@joshiste
fix: re-new connection in PostgresChannelMessageTableSubscriber
ef98cb8 
PostgresChannelMessageTableSubscriber currently never renews the connection.
This causes problems on DB failover. With this change the connection is
renewed when notifications are not received for a certain time.

fixes spring-projects#9061
joshiste added a commit to joshiste/spring-integration that referenced this issue Apr 1, 2024
@joshiste
fix: re-new connection in PostgresChannelMessageTableSubscriber
78bc2df 
PostgresChannelMessageTableSubscriber currently never renews the connection.
This causes problems on DB failover. With this change the connection is
renewed when notifications are not received for a certain time.

fixes spring-projects#9061
@joshiste joshiste mentioned this issue Apr 1, 2024
Merged
joshiste added a commit to joshiste/spring-integration that referenced this issue Apr 1, 2024
@joshiste
fix: re-new connection in PostgresChannelMessageTableSubscriber
3221eeb 
PostgresChannelMessageTableSubscriber currently never renews the connection.
This causes problems on DB failover. With this change the connection is
renewed when notifications are not received for a certain time.

fixes spring-projects#9061
@joshiste
Copy link
Contributor Author

joshiste commented Apr 1, 2024

P.S. I also think it's a pity that the PostgresChannelMessageTableSubscriber is a final class. If it either would be an interface or non-final, It would be a lot easier for users to get rid of the faulty behavior...

@artembilan artembilan added this to the 6.3.0-RC1 milestone Apr 1, 2024
@artembilan
Copy link
Member

Thank you for looking into this!
I'm not sure how would an interface for the PostgresChannelMessageTableSubscriber help since this class has a lot of specific logic.

Will look into your PR shortly.

joshiste added a commit to joshiste/spring-integration that referenced this issue Apr 1, 2024
@joshiste
fix: re-new connection in PostgresChannelMessageTableSubscriber
d0e7f7c 
PostgresChannelMessageTableSubscriber never renews the connection.
This causes problems on DB failover. With this change the connection is
renewed when notifications are not received for a certain time.

fixes spring-projects#9061
joshiste added a commit to joshiste/spring-integration that referenced this issue Apr 1, 2024
@joshiste
fix: re-new connection in PostgresChannelMessageTableSubscriber
e6cc266 
PostgresChannelMessageTableSubscriber never renews the connection.
This causes problems on DB failover. With this change the connection is
renewed when notifications are not received for a certain time.

fixes spring-projects#9061
joshiste added a commit to joshiste/spring-integration that referenced this issue Apr 1, 2024
@joshiste
fix: re-new connection in PostgresChannelMessageTableSubscriber
4b3e74c 
PostgresChannelMessageTableSubscriber never renews the connection.
This causes problems on DB failover. With this change the connection is
renewed when notifications are not received for a certain time.

fixes spring-projects#9061
joshiste added a commit to joshiste/spring-integration that referenced this issue Apr 1, 2024
@joshiste
fix: re-new connection in PostgresChannelMessageTableSubscriber
e30bebd 
PostgresChannelMessageTableSubscriber never renews the connection.
This causes problems on DB failover. With this change the connection is
renewed when notifications are not received for a certain time.

fixes spring-projects#9061
artembilan pushed a commit that referenced this issue Apr 1, 2024
@joshiste
GH-9061: renew connection in PostgresChannelMessageTableSubscriber
642278d 
Fixes: #9061

`PostgresChannelMessageTableSubscriber` never renews the connection.
This causes problems on DB failover. 
With this change the connection is renewed when notifications are not received for a certain time.


**Auto-cherry-pick to `6.2.x` & `6.1.x`**
spring-builds pushed a commit that referenced this issue Apr 1, 2024
@joshiste @spring-builds
GH-9061: renew connection in PostgresChannelMessageTableSubscriber
fa6494f 
Fixes: #9061

`PostgresChannelMessageTableSubscriber` never renews the connection.
This causes problems on DB failover.
With this change the connection is renewed when notifications are not received for a certain time.

(cherry picked from commit 642278d)
@spring-builds spring-builds mentioned this issue Apr 1, 2024
Closed
artembilan pushed a commit that referenced this issue Apr 1, 2024
@joshiste @artembilan
GH-9061: renew connection in PostgresChannelMessageTableSubscriber
c42441d 
Fixes: #9061

`PostgresChannelMessageTableSubscriber` never renews the connection.
This causes problems on DB failover.
With this change the connection is renewed when notifications are not received for a certain time.

(cherry picked from commit 642278d)

# Conflicts:
#	spring-integration-jdbc/src/main/java/org/springframework/integration/jdbc/channel/PostgresChannelMessageTableSubscriber.java
#	spring-integration-jdbc/src/test/java/org/springframework/integration/jdbc/channel/PostgresChannelMessageTableSubscriberTests.java
@spring-builds spring-builds mentioned this issue Apr 1, 2024
Closed
@artembilan artembilan mentioned this issue May 3, 2024
Closed
artembilan pushed a commit that referenced this issue May 21, 2024
@joshiste @artembilan
PostgresChannelMessageTableSubscriber: Renew connection only if invalid
da29e2d 
Fixes: #9111

An evolution of the #9061: renew the connection only when we need to.

**Auto-cherry-pick to `6.2.x` & `6.1.x`**
spring-builds pushed a commit that referenced this issue May 21, 2024
@joshiste @spring-builds
PostgresChannelMessageTableSubscriber: Renew connection only if invalid
ace6cfe 
Fixes: #9111

An evolution of the #9061: renew the connection only when we need to.

(cherry picked from commit da29e2d)
artembilan pushed a commit that referenced this issue May 21, 2024
@joshiste @artembilan
PostgresChannelMessageTableSubscriber: Renew connection only if invalid
00c75ac 
Fixes: #9111

An evolution of the #9061: renew the connection only when we need to.

(cherry picked from commit da29e2d)

# Conflicts:
#	spring-integration-jdbc/src/main/java/org/springframework/integration/jdbc/channel/PostgresChannelMessageTableSubscriber.java
EddieChoCho pushed a commit to EddieChoCho/spring-integration that referenced this issue Jun 26, 2024
@joshiste @EddieChoCho
spring-projectsGH-9061: renew connection in PostgresChannelMessageTab…
f1c1942 
…leSubscriber

Fixes: spring-projects#9061

`PostgresChannelMessageTableSubscriber` never renews the connection.
This causes problems on DB failover. 
With this change the connection is renewed when notifications are not received for a certain time.


**Auto-cherry-pick to `6.2.x` & `6.1.x`**
EddieChoCho pushed a commit to EddieChoCho/spring-integration that referenced this issue Jun 26, 2024
@joshiste @EddieChoCho
PostgresChannelMessageTableSubscriber: Renew connection only if invalid
a34d415 
Fixes: spring-projects#9111

An evolution of the spring-projects#9061: renew the connection only when we need to.

**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
(EOL) for: backport-to-6.1.x (EOL) for: backport-to-6.2.x in: jdbc status: backported type: bug
Projects
None yet
Milestone
6.3.0-RC1
Development

Successfully merging a pull request may close this issue.

fix: re-new connection in PostgresChannelMessageTableSubscriber joshiste/spring-integration
3 participants
@artembilan @spring-builds @joshiste