Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipelined thread safe but not Pipeline #3248

Closed
jfjm1 opened this issue Jan 28, 2025 · 1 comment
Closed

Pipelined thread safe but not Pipeline #3248

jfjm1 opened this issue Jan 28, 2025 · 1 comment

Comments

@jfjm1
Copy link

jfjm1 commented Jan 28, 2025

I have recently ran into an issue where different Go routines create their own Pipeline and added commands to their respective pipeline. However, when executing, the results appears to be mixed. When moved to Pipelined model, everything appear to be correct. According to the reference, it's safe for concurrent use by multiple goroutines

Expected Behavior

Based on documentation, these 2 should behavior similiarly. Pipelined is merely an alternative to Pipeline with Exec called upon function exit.

Current Behavior

Pipelined appears to be thread safe, but pipeline is not.

Steps to Reproduce

  1. Create multiple Clients, each connected to a unique Replica in a cluster.
  2. Sent ReadOnly successfully.
  3. Each client runs the following:
    1. Scan keys
    2. For batch of 10 keys, run the following:
    3. Pipelined was able to retrieve the data. But Pipeline often returns errors from node indicate MOVED

Pipelined behaviors properly:

go func(keys string) {
	results := make([]*redis.StringStringMapCmd, len(keys))
	_, err := iter.rdb.Pipelined(func(pipe redis.Pipeliner) error {
		for i, key := range keys {
			results[i] = pipe.HGetAll(key)
		}
		return nil
	})
	// processing of err and results...
}()

Pipeline does not:

go func(keys string) {
	results := make([]*redis.StringStringMapCmd, len(keys))
	pipe := iter.replica.Pipeline()
	for i, key := range keys {
		results[i] = pipe.HGetAll(key)
	}
	_, err := pipe.Exec()

	// processing of err and results...
}()

Context (Environment)

Some common use for pipeline is Set along with Expire, or optimization for multiple commands destined for the same node. Each Pipeline will only be created and executed by 1 goroutine, but there will be multiple goroutines operating on their respective pipeline.

@jfjm1
Copy link
Author

jfjm1 commented Jan 30, 2025

I found the cause of the issue. It's when a Client is created, even ReadOnly is issued once immediately after the creation, it does not cover additional connections the Client may make.

@jfjm1 jfjm1 closed this as completed Jan 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant