Handle DCP stream end/reconnection

[tleyden]

Factored out of #922

When a vbucket gets moved/rebalanced, it will send an END STREAM. Currently, our DCP impl does nothing with a stream end. Desired behavior: trigger a reconnection .. might be a different node, but we don't care.

In dcp_feed.go:

func (r *DCPReceiver) OnError(err error) {
    Warn("Feed", "Error processing DCP stream: %v", err)
}

This needs to reconnect rather than just logging an error.

@steveyen do you have any examples of DCP stream reconnection I should be looking at?

[steveyen]

Hi, if you're using cbdatasource, then it should do all the reconnection work for you...

https://github.com/couchbase/go-couchbase/blob/master/cbdatasource/cbdatasource.go#L1136

But, testing back in the day was all manual. Are you seeing cbdatasource not working properly for that case?

[tleyden]

Yep, we are using cbdatasource. Nope, haven't tried to test yet. But thanks, that clears things up regarding expectations.

[tleyden]

But, testing back in the day was all manual.

Do you remember the manual testing you did and how big the cluster was / how many replicas?

[tleyden]

If there was a single node couchbase cluster and the node was abruptly rebooted, would it be expected that cbdatasource would reconnect?

[tleyden]

I caught up with @steveyen in detail regarding the reconnection behavior of cbdatasource and it turns out that #942 is a pre-requisite for this ticket because:

• There was a conscious decision in the cbdatasource design not to use the cluster map to connect to other nodes, due to the usage pattern of some advanced users of couchbase that access it in front of a load balancer.
• Therefore, if you have a case where you have a multi-node cluster and one node (10.1.1.1) is brought down and a rebalance is done, cbdatasource will repeatedly try to connect to the node it knows about (10.1.1.1) and will never be able to reconnect to the other working nodes in the cluster since it doesn't know their URLs.

[tleyden]

Actually, I might be wrong on the dependence on #942 for this, since it looks like we are using the cbdatasource package directly from when initializing the connection to the bucket, and we have an opportunity to pass a list of URLs at that point:

sync_gateway/src/github.com/couchbase/sync_gateway/base/bucket.go

Lines 177 to 187 in d0d5141

	LogTo("Feed+", "Connecting to new bucket datasource. URLs:%s, pool:%s, name:%s, auth:%s", urls, poolName, bucketName, auth)
	bds, err := cbdatasource.NewBucketDataSource(
	urls,
	poolName,
	bucketName,
	"",
	vbucketIdsArr,
	auth,
	dcpReceiver,
	nil,
	)

Doing so there should solve the issue described above.