Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Stack Monitoring] Elasticsearch monitoring with Metricbeat and Filebeat as sidecars #4528

Merged
merged 86 commits into from
Jul 6, 2021
Merged

[Stack Monitoring] Elasticsearch monitoring with Metricbeat and Filebeat as sidecars #4528

Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
2c1619a
Stack monitoring for Elasticsearch
thbkrkr May 27, 2021
e621963
Regenerate
thbkrkr May 27, 2021
1839614
Model with only one associationi es->es
thbkrkr May 31, 2021
cd1e21e
Regenerate
thbkrkr May 31, 2021
cc931cd
Restore hash-based annotation name
thbkrkr Jun 1, 2021
e018707
Simple stack monitoring E2E test
thbkrkr Jun 3, 2021
a27f910
Move beats config in embedded files
thbkrkr Jun 3, 2021
b298efc
Renaming
thbkrkr Jun 3, 2021
29e3cd9
Typo
thbkrkr Jun 3, 2021
964bedd
Revert bad replace all
thbkrkr Jun 3, 2021
9a2a5e3
Avoid to use configmap subpath
thbkrkr Jun 3, 2021
3f5d8ee
containerImage func godoc
thbkrkr Jun 3, 2021
064fd50
Introduce WithContainers to avoid weirdness with container defaulter
thbkrkr Jun 7, 2021
76f4f27
Wording
thbkrkr Jun 7, 2021
9f16362
Remove unused code
thbkrkr Jun 7, 2021
27ab36e
Sort imports
thbkrkr Jun 7, 2021
02400fd
Reject unsupported version
thbkrkr Jun 8, 2021
fc4c782
Naming
thbkrkr Jun 8, 2021
d3d97ab
Add stack mon doc link in es spec doc
thbkrkr Jun 8, 2021
0a7528d
Typo
thbkrkr Jun 8, 2021
069eace
Merge remote-tracking branch 'upstream/master' into stack-monitoring
thbkrkr Jun 8, 2021
75fcde3
go fmt
thbkrkr Jun 8, 2021
dfe83f5
doc
thbkrkr Jun 8, 2021
ac6d4f9
Dedicated role
thbkrkr Jun 8, 2021
ff98c86
Update Test_aggregateRoles
thbkrkr Jun 10, 2021
d8b8fc0
Merge remote-tracking branch 'upstream/master' into stack-monitoring
thbkrkr Jun 16, 2021
450492c
Renaming
thbkrkr Jun 16, 2021
e01836d
End of revert about configmap subpath
thbkrkr Jun 16, 2021
46ff83a
Move ES monitoring config in stackmon package
thbkrkr Jun 16, 2021
b3fcdc6
Review's input
thbkrkr Jun 16, 2021
337e96e
Generate
thbkrkr Jun 16, 2021
4abcc0c
Revert
thbkrkr Jun 16, 2021
cc48fd1
Fix monitoring config
thbkrkr Jun 17, 2021
074d760
Add unit tests
thbkrkr Jun 17, 2021
9346b4a
Use GetCASecretName()
thbkrkr Jun 17, 2021
6c96979
lint
thbkrkr Jun 17, 2021
4d62d07
Move warning comment
thbkrkr Jun 18, 2021
b2cfbce
Unit tests for fullContainerImage
thbkrkr Jun 18, 2021
ce855c0
Future proof elasticsearchRefs for Elastic Agent
thbkrkr Jun 21, 2021
86e9fbd
Add kubebuilder annotations
thbkrkr Jun 22, 2021
9a81c1c
Adjust association name
thbkrkr Jun 22, 2021
3fb22c4
Refactoring validation
thbkrkr Jun 22, 2021
6b9c2d3
Cleaning beat yaml config
thbkrkr Jun 22, 2021
482784a
Refactoring beat config in secrets
thbkrkr Jun 22, 2021
2d1c67e
godoc
thbkrkr Jun 23, 2021
a2fd5d2
Put monitoringXxxEnvVars funcs side by side
thbkrkr Jun 23, 2021
048f8e4
Update descriptions for API doc
thbkrkr Jun 23, 2021
7b249bd
Apply review's input for API doc
thbkrkr Jun 23, 2021
eb9de17
Run generate
thbkrkr Jun 23, 2021
541ab64
Fix receiver name
thbkrkr Jun 24, 2021
175bcf3
Factorize ElasticsearchConfigAnnotationName shared with agent
thbkrkr Jun 24, 2021
ac8cbb2
Adjust association type and labels
thbkrkr Jun 24, 2021
a52d1f1
Typo and new line
thbkrkr Jun 24, 2021
e52d785
Decouple beat_config.go and es_config.go
thbkrkr Jun 24, 2021
a9b4e49
Adjust TargetEsCaCertMountPathFormat
thbkrkr Jun 24, 2021
ffa0aa0
Reuse IsStackMonitoringDefined
thbkrkr Jun 24, 2021
6218a47
Move and rename fileLogStyleEnvVar
thbkrkr Jun 24, 2021
170f106
Adjust comments in filebeat.yml
thbkrkr Jun 24, 2021
00042a6
Document TestEsStackMonitoring
thbkrkr Jun 24, 2021
f6975f3
Add CheckBeatSidecars to TestESStackMonitoring
thbkrkr Jun 24, 2021
cf5c596
Revert fullContainerImage
thbkrkr Jun 24, 2021
742fe9f
Unit tests for Validate
thbkrkr Jun 24, 2021
d45f5a2
Unit tests for WithMonitoring
thbkrkr Jun 24, 2021
05dcd97
Remove useless health=green query param
thbkrkr Jun 24, 2021
12f3bd4
Unit test TestPodTemplateBuilder_WithContainers
thbkrkr Jun 24, 2021
8e72f7c
Support container override with pod template
thbkrkr Jun 24, 2021
2f5b796
common/stackmon package
thbkrkr Jul 1, 2021
a9584a2
ES impl for HasMonitoring
thbkrkr Jul 1, 2021
e60c0d9
Refactoring of elasticsearch/stackmon
thbkrkr Jul 1, 2021
7737cef
Remove redundant with_monitoring unit tests with container_test.go
thbkrkr Jul 1, 2021
5b92f6d
Dedicated es user to collect metrics
thbkrkr Jul 1, 2021
f1139cc
Adjust dedicated monitoring role
thbkrkr Jul 1, 2021
6e788c8
Adjust unit tests following the addition of user elastic-internal-mon…
thbkrkr Jul 1, 2021
f531e93
Use 7.14.0-SNAPSHOT as MinStackVersion
thbkrkr Jul 5, 2021
1b035c9
BeatSidecar constructor and public fields
thbkrkr Jul 5, 2021
f1a45c9
Simplify HasMonitoring interface with metav1.Object
thbkrkr Jul 5, 2021
324bb05
Files renaming
thbkrkr Jul 5, 2021
d7b7b13
indentation
thbkrkr Jul 5, 2021
07c1bdd
Accumulate validation errors
thbkrkr Jul 5, 2021
718eed6
Remove redundant code
thbkrkr Jul 5, 2021
b49530f
Update ES secrets check in E2E tests
thbkrkr Jul 5, 2021
f4759fd
Fix FilebeatImage
thbkrkr Jul 5, 2021
c26faa5
Better control of volume names
thbkrkr Jul 5, 2021
fc508c8
Merge remote-tracking branch 'upstream/master' into stack-monitoring
thbkrkr Jul 5, 2021
31c2465
Run generate
thbkrkr Jul 5, 2021
ccd7e12
Update E2E test Kibana CheckStatus
thbkrkr Jul 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cmd/manager/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -677,6 +677,7 @@ func registerControllers(mgr manager.Manager, params operator.Parameters, access
{name: "BEAT-KB", registerFunc: associationctl.AddBeatKibana},
{name: "AGENT-ES", registerFunc: associationctl.AddAgentES},
{name: "EMS-ES", registerFunc: associationctl.AddMapsES},
{name: "ES-MONITORING", registerFunc: associationctl.AddEsMonitoring},
}

for _, c := range assocControllers {
Expand Down
4,778 changes: 4,778 additions & 0 deletions config/all-in-one.yaml
View file
Open in desktop

Large diffs are not rendered by default.

165 changes: 124 additions & 41 deletions config/crds/v1/all-crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,10 +202,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced
resource is used.
type: string
required:
- name
Expand Down Expand Up @@ -276,8 +276,8 @@ spec:
type: string
description: AssociationStatusMap is the map of association's namespaced
name string to its AssociationStatus. For resources that have a
single Association of a given type (eg. single ES reference), this
map will contain a single entry.
single Association of a given type (for ex. single ES reference),
this map contains a single entry.
type: object
expectedNodes:
format: int32
Expand Down Expand Up @@ -378,10 +378,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand Down Expand Up @@ -808,10 +808,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand Down Expand Up @@ -1657,10 +1657,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand All @@ -1683,10 +1683,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand Down Expand Up @@ -1874,10 +1874,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand Down Expand Up @@ -2833,6 +2833,80 @@ spec:
image:
description: Image is the Elasticsearch Docker image to deploy.
type: string
monitoring:
description: Monitoring enables you to collect and ship log and monitoring
data of this Elasticsearch cluster. See https://www.elastic.co/guide/en/elasticsearch/reference/current/monitor-elasticsearch-cluster.html.
Metricbeat and Filebeat are deployed in the same Pod as sidecars
and each one sends data to one or two different Elasticsearch monitoring
clusters running in the same Kubernetes cluster.
properties:
logs:
description: Logs holds references to Elasticsearch clusters which
thbkrkr marked this conversation as resolved.
Show resolved Hide resolved
receive log data from this Elasticsearch cluster.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
monitoring Elasticsearch clusters running in the same Kubernetes
cluster. Due to existing limitations, only a single Elasticsearch
cluster is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object.
properties:
name:
description: Name of the Kubernetes object.
type: string
namespace:
description: Namespace of the Kubernetes object. If
empty, defaults to the current namespace.
type: string
serviceName:
description: ServiceName is the name of an existing
Kubernetes service which is used to make requests
to the referenced object. It has to be in the same
namespace as the referenced resource. If left empty,
the default HTTP service of the referenced resource
is used.
type: string
required:
- name
type: object
type: array
type: object
metrics:
description: Metrics holds references to Elasticsearch clusters
thbkrkr marked this conversation as resolved.
Show resolved Hide resolved
which receive monitoring data from this Elasticsearch cluster.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
monitoring Elasticsearch clusters running in the same Kubernetes
cluster. Due to existing limitations, only a single Elasticsearch
cluster is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object.
properties:
name:
description: Name of the Kubernetes object.
type: string
namespace:
description: Namespace of the Kubernetes object. If
empty, defaults to the current namespace.
type: string
serviceName:
description: ServiceName is the name of an existing
Kubernetes service which is used to make requests
to the referenced object. It has to be in the same
namespace as the referenced resource. If left empty,
the default HTTP service of the referenced resource
is used.
type: string
required:
- name
type: object
type: array
type: object
type: object
nodeSets:
description: NodeSets allow specifying groups of Elasticsearch nodes
sharing the same configuration and Pod templates.
Expand Down Expand Up @@ -3230,10 +3304,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
service which is used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the
referenced resource will be used.
resource. If left empty, the default HTTP service of the
referenced resource is used.
type: string
required:
- name
Expand Down Expand Up @@ -3724,6 +3798,15 @@ spec:
description: ElasticsearchHealth is the health of the cluster as returned
by the health API.
type: string
monitoringAssociationStatus:
additionalProperties:
description: AssociationStatus is the status of an association resource.
type: string
description: AssociationStatusMap is the map of association's namespaced
name string to its AssociationStatus. For resources that have a
single Association of a given type (for ex. single ES reference),
this map contains a single entry.
type: object
phase:
description: ElasticsearchOrchestrationPhase is the phase Elasticsearch
is in from the controller point of view.
Expand Down Expand Up @@ -4743,10 +4826,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand Down Expand Up @@ -5272,10 +5355,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand Down Expand Up @@ -5816,10 +5899,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand Down
12 changes: 6 additions & 6 deletions config/crds/v1/bases/agent.k8s.elastic.co_agents.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13380,10 +13380,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced
resource is used.
type: string
required:
- name
Expand Down Expand Up @@ -13454,8 +13454,8 @@ spec:
type: string
description: AssociationStatusMap is the map of association's namespaced
name string to its AssociationStatus. For resources that have a
single Association of a given type (eg. single ES reference), this
map will contain a single entry.
single Association of a given type (for ex. single ES reference),
this map contains a single entry.
type: object
expectedNodes:
format: int32
Expand Down
16 changes: 8 additions & 8 deletions config/crds/v1/bases/apm.k8s.elastic.co_apmservers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,10 +76,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand Down Expand Up @@ -506,10 +506,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand Down
16 changes: 8 additions & 8 deletions config/crds/v1/bases/beat.k8s.elastic.co_beats.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13380,10 +13380,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand All @@ -13406,10 +13406,10 @@ spec:
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which will be used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty the default HTTP service of the referenced
resource will be used.
service which is used to make requests to the referenced object.
It has to be in the same namespace as the referenced resource.
If left empty, the default HTTP service of the referenced resource
is used.
type: string
required:
- name
Expand Down
Loading