Errors when workers have been idle for one hour

[hyrefeldt-ourstudio]

Hello!

I'm using the Golang SDK v0.11.4 and I get these errors from Riverqueue

time=2024-10-26T15:21:20.147Z level=ERROR msg="Notifier: Error from notification wait" err="unexpected EOF"
time=2024-10-26T15:21:20.147Z level=ERROR msg="Notifier: Error running listener (will attempt reconnect after backoff)" attempt=0 err="unexpected EOF" sleep_duration=529.734059ms

It seems like they happen when the worker has been idle for one hour because the last error message seems to be exactly one hour before these start to occur. And after that, they happen every hour if there are no jobs still.

I'm using this configuration:

	workers := river.NewWorkers()
	if err := river.AddWorkerSafely(workers, myWorker); err != nil {
		return nil, err
	}

	cfg := &river.Config{
		Queues: map[string]river.QueueConfig{
			river.QueueDefault: {
				MaxWorkers: 2,
			},
		},
		Workers:      workers,
		JobTimeout:   5 * time.Hour,
		MaxAttempts:  1,
		ErrorHandler: newErrorHandler(logger),
		TestOnly:     false,
	}

I have tried to find examples of this error and have tried to find hints in the documentation on how the client can be configured but with no success. Would appreciate some help on this.

BR
Johan Hyrefeldt

[bgentry]

Hmm, I don't think we have any timeouts that would align with this. On the notifier's listen connection, we're also pinging every 5 seconds to maintain connection liveness. I'm wondering if this might be a setting at either the pgx.ConnConfig or the database level which could be causing the conns to go away after an hour regardless of activity?

Out of curiosity, have you tried inserting a no-op periodic job on a 55 minute cadence (using periodic jobs) to see if it makes this error go away? Or you could periodically execute a query to SELECT pg_notify('public.river_insert', '{"queue":"default"}); to issue an insert notification.

[bgentry]

@hyrefeldt-ourstudio how does #661 sound to you?

[hyrefeldt-ourstudio]

I changed the value of MaxConnLifetime from the default which is 1 hour to 2 hours but the frequency of the errors I mentioned above still seem to happen after one hour of inactivity.

But maybe you have found something anyway. I will take a look at that PR.

[hyrefeldt-ourstudio]

I took a look at your PR now. Hopefully, that could fix the problem even though my change of MaxConnLifetime didn't have any effect in my particular case.

[hyrefeldt-ourstudio]

But if you were able to reproduce this issue then you should be able to see if it has any effect, I guess.

[hyrefeldt-ourstudio]

@bgentry a bit slow progress here on our side but now I upgraded to v0.14.1 which contains the change in #661 but I still get the error logs:

{"loglevel":"info","@timestamp":"2024-11-12T06:27:02.491Z","message":"started"}
time=2024-11-12T07:27:02.588Z level=ERROR msg="Notifier: Error from notification wait" err="unexpected EOF"
time=2024-11-12T07:27:02.588Z level=ERROR msg="Notifier: Error running listener (will attempt reconnect after backoff)" attempt=0 err="unexpected EOF" sleep_duration=537.776955ms

I will continue to look into this...