Wait for ES deployment to be ready and then deploy jaeger

[pavolloffay]

Now operator creates ES CR and jaeger objects (deployments) at the same time. This is causing jaeger pods to fail multiple times until Elasticsearch is up.

[pavolloffay]

I think this one of the most important features for ES deployment.

Thinking about how to tackle it. Maybe the best would be to use k8s client and wait until ES deployment is up - readiness probe responds positively.

[jpkrohling]

I'd say that the cleanest would be to create a Job that blocks until it can make a connection to ES. This Job can be added to the list of Dependencies, similar to how the CassandraCreateSchema works.

jaeger-operator/pkg/storage/dependency.go

Lines 12 to 18 in 401ef74

	func Dependencies(jaeger *v1alpha1.Jaeger) []batchv1.Job {
	if strings.EqualFold(jaeger.Spec.Storage.Type, "cassandra") {
	return cassandraDeps(jaeger)
	}
	return nil
	}

jaeger-operator/pkg/storage/cassandra_dependencies.go

Line 15 in 401ef74

func cassandraDeps(jaeger *v1alpha1.Jaeger) []batchv1.Job {

[pavolloffay]

I was thinking about this but - it seems more complicated and harder to maintain.

With job we need a separate docker image with code which calls ES - this will require mounting secrets etc.

I would rather use operator to do this using the similar way how it looks for dependencies jobs to be finished.

[pavolloffay]

Dependencies for C* makes sense - it initializes schema. Whereas in this case we just wait until deployment is ready.

For rollover support using dependencies makes sense - as we need to initialize mappings and create indices. The rollover image would fail if the ES is not up.

[jpkrohling]

Then probably something like:

jaeger-operator/pkg/controller/jaeger/deployment.go

Lines 53 to 62 in 401ef74

	for _, d := range depInventory.Create {
	if err := r.waitForStability(d); err != nil {
	return err
	}
	}
	for _, d := range depInventory.Update {
	if err := r.waitForStability(d); err != nil {
	return err
	}
	}

and

jaeger-operator/pkg/controller/jaeger/deployment.go

Lines 74 to 95 in 401ef74

	func (r *ReconcileJaeger) waitForStability(dep appsv1.Deployment) error {
	// TODO: decide what's a good timeout... the first cold run might take a while to download
	// the images, subsequent runs should take only a few seconds
	return wait.PollImmediate(time.Second, 5*time.Minute, func() (done bool, err error) {
	d := &appsv1.Deployment{}
	if err := r.client.Get(context.Background(), types.NamespacedName{Name: dep.Name, Namespace: dep.Namespace}, d); err != nil {
	return false, err
	}
	if d.Status.ReadyReplicas != d.Status.Replicas {
	log.WithFields(log.Fields{
	"namespace": dep.Namespace,
	"name": dep.Name,
	"ready": d.Status.ReadyReplicas,
	"desired": d.Status.Replicas,
	}).Debug("Waiting for deployment to estabilize")
	return false, nil
	}
	return true, nil
	})
	}