Stack monitoring and version upgrade order

[sebgl]

We ensure a strict ordering of stack product version upgrades: Elasticsearch first, then Enterprise Search, then Kibana, etc.

This is done through our association mechanism: the associated resource can only be updated once all referenced resources are updated:

cloud-on-k8s/pkg/controller/association/conf.go

Lines 111 to 147 in 7c39ff0

	// AllowVersion returns true if the given resourceVersion is lower or equal to the associations' versions.
	// For example: Kibana in version 7.8.0 cannot be deployed if its Elasticsearch association reports version 7.7.0.
	// A difference in the patch version is ignored: Kibana 7.8.1+ can be deployed alongside Elasticsearch 7.8.0.
	// Referenced resources version is parsed from the association conf annotation.
	func AllowVersion(resourceVersion version.Version, associated commonv1.Associated, logger logr.Logger, recorder record.EventRecorder) bool {
	for _, assoc := range associated.GetAssociations() {
	assocRef := assoc.AssociationRef()
	if !assocRef.IsDefined() {
	// no association specified, move on
	continue
	}
	if assoc.AssociationConf() == nil || assoc.AssociationConf().Version == "" {
	// no conf reported yet, this may be the initial resource creation
	logger.Info("Delaying version deployment since the version of an associated resource is not reported yet",
	"version", resourceVersion, "ref_namespace", assocRef.Namespace, "ref_name", assocRef.Name)
	return false
	}
	refVer, err := version.Parse(assoc.AssociationConf().Version)
	if err != nil {
	logger.Error(err, "Invalid version found in association configuration", "association_version", assoc.AssociationConf().Version)
	return false
	}
	compatibleVersions := refVer.GTE(resourceVersion) || ((refVer.Major == resourceVersion.Major) && (refVer.Minor == resourceVersion.Minor))
	if !compatibleVersions {
	// the version of the referenced resource (example: Elasticsearch) is lower than
	// the desired version of the reconciled resource (example: Kibana)
	logger.Info("Delaying version deployment since a referenced resource is not upgraded yet",
	"version", resourceVersion, "ref_version", refVer,
	"ref_type", assoc.AssociationType(), "ref_namespace", assocRef.Namespace, "ref_name", assocRef.Name)
	recorder.Event(associated, corev1.EventTypeWarning, events.EventReasonDelayed,
	fmt.Sprintf("Delaying deployment of version %s since the referenced %s is not upgraded yet", resourceVersion, assoc.AssociationType()))
	return false
	}
	}
	return true
	}

This function gets called as part of individual resource controllers. For example in the Kibana reconciliation:

cloud-on-k8s/pkg/controller/kibana/driver.go

Lines 131 to 133 in 7c39ff0

	if !association.AllowVersion(d.version, kb, logger, d.Recorder()) {
	return results // will eventually retry
	}

We don't do that check in the Elasticsearch controller. Mostly because so far we didn't have any Elasticsearch -> X association. That's not true anymore with stack monitoring.

In the Kibana controller, we do check that any referenced ES cluster for monitoring purposes is running with a higher or equal version.
For consistency, we should do the same in the Elasticsearch controller.

This is where things get a bit tricky :)

• Association to itself: Elasticsearch monitoring data can be configured to be sent to itself. In which case we need to make sure we don't get into a circular race condition where the version cannot be updated at all because the controller waits for its own resource to be updated first (🔁).

• Circular associations: we could configure 2 different clusters to monitor each other (ES A <-> ES B). In which case both would require the other one to be updated first.

(The current situation where we don't do that check in the Elasticsearch controller is maybe an OK solution to that problem, overall).

[thbkrkr]

Currently:

• the circular associations (ES A <-> ES B) works and that with a version upgrade
• the association to itself does not work

[thbkrkr]

Note on version compatibility of monitored and monitoring clusters: Ideally the monitoring cluster and the production cluster run on the same Elastic Stack version. However, a monitoring cluster on the latest release of 7.x also works with production clusters that use the same major version. Monitoring clusters that use 7.x also work with production clusters that use the latest release of 6.x (source: https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-production.html#monitoring-production).

[thbkrkr]

Closed by:

• Only configure Stack Monitoring if association reconciled #5339
• Make sure to read association configuration again from annotations if it was cleared #5489