7.17.0 fleet-server looping error #5323
Comments
|
This is a fairly recent bug (#5250) that also occurs with version A workaround is to override the entrypoint of the agent containers to fix how the CA store is updated for the entire container ( apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
...
spec:
...
deployment:
...
podTemplate:
spec:
...
containers:
- name: agent
command:
- bash
- -c
- |
#!/usr/bin/env bash
set -e
if [[ -f /mnt/elastic-internal/elasticsearch-association/<agent-ns>/<es-name>/certs/ca.crt ]]; then
cp /mnt/elastic-internal/elasticsearch-association/<agent-ns>/<es-name>/certs/ca.crt /usr/local/share/ca-certificates
update-ca-certificates
fi
/usr/bin/tini -- /usr/local/bin/docker-entrypoint -eFull YAMLapiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: fleet-server-quickstart
namespace: default
spec:
version: 7.17.0
kibanaRef:
name: kibana-quickstart
elasticsearchRefs:
- name: elasticsearch-quickstart
mode: fleet
fleetServerEnabled: true
deployment:
replicas: 1
podTemplate:
spec:
serviceAccountName: elastic-agent
automountServiceAccountToken: true
securityContext:
runAsUser: 0
containers:
- name: agent
command:
- bash
- -c
- |
#!/usr/bin/env bash
set -e
if [[ -f /mnt/elastic-internal/elasticsearch-association/default/elasticsearch-quickstart/certs/ca.crt ]]; then
cp /mnt/elastic-internal/elasticsearch-association/default/elasticsearch-quickstart/certs/ca.crt /usr/local/share/ca-certificates
update-ca-certificates
fi
/usr/bin/tini -- /usr/local/bin/docker-entrypoint -e
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: elastic-agent-quickstart
namespace: default
spec:
version: 7.17.0
kibanaRef:
name: kibana-quickstart
fleetServerRef:
name: fleet-server-quickstart
mode: fleet
daemonSet:
podTemplate:
spec:
serviceAccountName: elastic-agent
automountServiceAccountToken: true
securityContext:
runAsUser: 0
containers:
- name: agent
command:
- bash
- -c
- |
#!/usr/bin/env bash
set -e
if [[ -f /mnt/elastic-internal/elasticsearch-association/agent-ns/elasticsearch/certs/ca.crt ]]; then
cp /mnt/elastic-internal/elasticsearch-association/agent-ns/elasticsearch/certs/ca.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust
fi
/usr/bin/tini -- /usr/local/bin/docker-entrypoint -e
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana-quickstart
namespace: default
spec:
version: 7.17.0
count: 1
elasticsearchRef:
name: elasticsearch-quickstart
config:
xpack.fleet.agents.elasticsearch.host: "https://elasticsearch-quickstart-es-http.default.svc:9200"
xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-quickstart-agent-http.default.svc:8220"]
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch-quickstart
namespace: default
spec:
version: 7.17.0
nodeSets:
- name: default
count: 3
config:
node.store.allow_mmap: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- pods
- nodes
verbs:
- get
- watch
- list
- apiGroups: ["coordination.k8s.io"]
resources:
- leases
verbs:
- get
- create
- update
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: elastic-agent
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: elastic-agent
subjects:
- kind: ServiceAccount
name: elastic-agent
namespace: default
roleRef:
kind: ClusterRole
name: elastic-agent
apiGroup: rbac.authorization.k8s.ioI'm sorry for the inconvenience. |
Does this conditional not only check if the CA is there? Seem in my image, the directory that holds the actual CA is not present. It is easy enough to take what you provided and do a Also, thank you for the response :) |
No, if a ca.crt is present (it is optional, e.g. the ES cert is signed by a well-known certificate authority), it is copied to Starting |
|
Hi, we upgraded the operator to 2.0.0 but we have exactly the same error. Seems bug is not fixed in 2.0.0? |
Yes you are right. This is unfortunately a known issue in ECK 2.0, https://www.elastic.co/guide/en/cloud-on-k8s/master/release-highlights-2.0.0.html#k8s-200-known-issues. The fix did not make it into the release by mistake. You still need to use the workaround. I've updated the comment above to say it is fixed for the upcoming |
Bug Report
What did you do?
Starting a new fleet-managed instance as described in the documentation
https://www.elastic.co/guide/en/cloud-on-k8s/1.9/k8s-elastic-agent-fleet-quickstart.html
This points to the YAML
What did you expect to see?
Running instances of Elasticsearch, Kibana, fleet-agent-server and fleet-agents.
What did you see instead? Under which circumstances?
Running instances of Elasticsearch and Kibana.
Fleet-agent-server crashes at boot and fleet-agent does not start due to server not started.
Fleet-agent-server logs seems to point no non-existent directories for CA
$ kubectl logs -n eck fleet-server-agent-6bfb5cc49d-6gfft cp: cannot create regular file '/etc/pki/ca-trust/source/anchors/': No such file or directoryEnvironment
Test environment of k3s nodes
ECK version:
1.9.1
The text was updated successfully, but these errors were encountered: