ES version detection for optype breaks during upgrade to ES 7.5

[igorwwwwwwwwwwwwwwwwwwww]

We (@GitLab) ran into some issues while trying to index into a cluster that is in the process of being upgraded to ES 7.5. Some nodes are running ES v7.4.1, while some are already on v7.5.1.

This occurred with a version of pubsubbeat based on libbeat 7.5.1 -- but since it's an issue with libbeat, it most likely affects all beats.

The error we started getting is:

2020-01-15T15:48:19.956+0100	ERROR	elasticsearch/client.go:344	Failed to perform any bulk index operations: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"optype create not supported for indexing requests without explicit id until all nodes are on version 7.5.0 or higher"}],"type":"illegal_argument_exception","reason":"optype create not supported for indexing requests without explicit id until all nodes are on version 7.5.0 or higher"},"status":400}
2020-01-15T15:48:21.372+0100	ERROR	pipeline/output.go:121	Failed to publish events: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"optype create not supported for indexing requests without explicit id until all nodes are on version 7.5.0 or higher"}],"type":"illegal_argument_exception","reason":"optype create not supported for indexing requests without explicit id until all nodes are on version 7.5.0 or higher"},"status":400}

After digging into the code, it looks like this is related to how libbeat switches the optype from index to create, if it believes it's talking to a ES 7.5 cluster.

The logic for that is in libbeat/outputs/elasticsearch/client.go createEventBulkMeta(). The client uses a Ping() to learn the ES version.

Unfortunately there are several reasons why this version may not be accurate during an ES upgrade. For example, if there is a request-level load balancer in front of ES, then the node receiving the bulk index request may not be the same one as the node the version was learned from. And those nodes may not be running the same ES version.

We believe that this is what is happening to us in this case (on Elastic Cloud ESS).

A possible workaround would be to have a config option to force falling back to the index optype, so that indexing can continue during the cluster upgrade.

[ycombinator]

Looks like this change was introduced in Beats v7.5.0: #13936. So how about upgrading the entire ES cluster to v7.5.x first, then upgrading the Beats to v7.5.x? During the cluster upgrade the Beats will still be running < v7.5.0, which should use the create optype.

[igorwwwwwwwwwwwwwwwwwwww]

Fair point, that would probably work. :) If that's the supported upgrade path, I'm happy to close this.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.