Deploy Fluent Bit #2723

alexandre-allard · 2020-08-06T07:51:07Z

Component: salt, build, log

Context:

Summary:

Acceptance criteria:

Closes: #2701

bert-e · 2020-08-06T07:51:09Z

Hello alexandre-allard-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2020-08-06T07:51:13Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

bert-e · 2020-08-12T14:26:55Z

Conflict

There is a conflict between your branch improvement/2701-deploy-fluent-bit-from-charts and the
destination branch development/2.6.

Please resolve the conflict on the feature branch (improvement/2701-deploy-fluent-bit-from-charts).

 $ git fetch
 $ git checkout origin/improvement/2701-deploy-fluent-bit-from-charts
 $ git merge origin/development/2.6
 $ # <intense conflict resolution>
 $ git commit
 $ git push origin HEAD:improvement/2701-deploy-fluent-bit-from-charts

bert-e · 2020-08-12T14:38:28Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

charts/fluent-bit.yaml

salt/metalk8s/addons/logging/fluent-bit/deployed/configmap.sls

TeddyAndrieux · 2020-08-19T08:36:51Z

salt/metalk8s/addons/logging/fluent-bit/deployed/configmap.sls

+            [FILTER]
+                Name           modify
+                Match          host.*
+                Rename         HOSTNAME node


Not sure we should rename it to node because hostname may be different from node name :/
I would instead retrieve the node name of the current host here if it's doable (or add a hostname label on k8s logs but not sure it's easy to do as well)
If it's not possible or to much work, then ignore this comment and let's keep hostname as node for the moment

PS look like hostname is not really good, on my test env I got some logs with localhost and localhost.localdomain as node (so likely HOSTNAME)
Looking on log it's seems these logs are not really relevant (maybe we should filter out logs with hostname localhost and localhost.localdomain )

Edit: Ok log with localhost and localhost.localdomain are only logs at the beginning before systemd-hostnamed update the hostname

It seems to be a little bit tricky to achieve that.
I can't find an easy way to retrieve the real hostname from the kubernetes logs or to retrieve the nodename from journal logs.
I've tried to use $HOSTNAME env var, but it's set to the container hostname... not really useful.
Plus, the helm chart does not allow to set up env vars based on pod specs... otherwise we could have done:

[...] spec: containers: - env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName [...]

And use the env var in fluent bit configuration.
Maybe we should just have 2 different labels host and node... or we should not use this chart.

Ok let's use hostname and pod for the moment, I updated the dashboard accordingly

tests/post/steps/test_logging.py

tests/post/steps/files/logger-pod.yaml.tpl

salt/metalk8s/addons/logging/fluent-bit/deployed/configmap.sls

helm repo add loki https://grafana.github.io/loki/charts helm repo update helm fetch -d charts --untar loki/fluent-bit Refs: #2701

this file is used to overwrite default values. Refs: #2701

Refs: #2701

there is a bug introduced by the latest change on how we handle CSC in this script, the script fails if no --service-config or --remove-manifest options are provided.

./charts/render.py fluent-bit --namespace metalk8s-logging \ --remove-manifest ConfigMap fluent-bit \ charts/fluent-bit.yaml charts/fluent-bit/ \ > salt/metalk8s/addons/logging/fluent-bit/deployed/chart.sls Refs: #2701

TeddyAndrieux

Just a comment about tests

tests/post/steps/test_logging.py

salt/metalk8s/addons/logging/loki/deployed/services.sls

salt/metalk8s/addons/logging/fluent-bit/deployed/configmap.sls

Refs: #2701

spawn a container that logs a single line then try to retrieve it from Loki API Refs: #2701

This is needed for Fluent Bit to be able to send every logs to each Loki instance. Refs: #2682

We write all logs to each Loki replicas, so we need to define one Output block, in Fluent Bit configuration, per loki instance. Refs: #2701

TeddyAndrieux

LGTM

alexandre-allard · 2020-08-24T10:06:30Z

/approve

bert-e · 2020-08-24T10:06:39Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.6

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2020-08-24T11:03:38Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.6

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5

Please check the status of the associated issue None.

Goodbye alexandre-allard-scality.

gdemonet changed the base branch from development/2.6 to improvement/2682-deploy-loki August 6, 2020 08:15

TeddyAndrieux force-pushed the improvement/2682-deploy-loki branch from 2209681 to a7e07bb Compare August 6, 2020 08:32

TeddyAndrieux force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from 31e4e2a to 693fe20 Compare August 6, 2020 08:34

TeddyAndrieux force-pushed the improvement/2682-deploy-loki branch from a7e07bb to 34cd3e2 Compare August 6, 2020 16:24

TeddyAndrieux force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from 693fe20 to bf86469 Compare August 6, 2020 16:25

TeddyAndrieux force-pushed the improvement/2682-deploy-loki branch from 34cd3e2 to 8ffd6e1 Compare August 7, 2020 15:13

TeddyAndrieux force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from bf86469 to a59134f Compare August 7, 2020 15:14

TeddyAndrieux force-pushed the improvement/2682-deploy-loki branch from 8ffd6e1 to 7d03763 Compare August 7, 2020 15:42

TeddyAndrieux force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from a59134f to 4612ca4 Compare August 7, 2020 15:42

alexandre-allard force-pushed the improvement/2682-deploy-loki branch from 7d03763 to e5fce98 Compare August 10, 2020 11:57

TeddyAndrieux mentioned this pull request Aug 11, 2020

Deploy Promtail #2054

Closed

TeddyAndrieux force-pushed the improvement/2682-deploy-loki branch from e5fce98 to 784ae86 Compare August 11, 2020 08:40

alexandre-allard force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch 2 times, most recently from 25bb1e4 to 6527baa Compare August 11, 2020 15:17

Base automatically changed from improvement/2682-deploy-loki to development/2.6 August 12, 2020 14:26

alexandre-allard force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from 6527baa to 6a13bbe Compare August 12, 2020 14:38

alexandre-allard force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch 2 times, most recently from 5433397 to 96c2445 Compare August 17, 2020 14:32

alexandre-allard marked this pull request as ready for review August 17, 2020 14:32

alexandre-allard requested a review from a team as a code owner August 17, 2020 14:32

TeddyAndrieux mentioned this pull request Aug 18, 2020

Add Logging dashboards (Loki, Logs, Fluent-bit) #2745

Merged

alexandre-allard force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from 96c2445 to a0acb6b Compare August 19, 2020 07:44

TeddyAndrieux reviewed Aug 19, 2020

View reviewed changes

alexandre-allard added 3 commits August 19, 2020 17:07

charts/log: initial import of fluent-bit helm charts

e1c4687

helm repo add loki https://grafana.github.io/loki/charts helm repo update helm fetch -d charts --untar loki/fluent-bit Refs: #2701

charts/log: add fluent-bit helm configuration values

f1378e0

this file is used to overwrite default values. Refs: #2701

build: fetch fluent-bit image during build

c059dc9

Refs: #2701

alexandre-allard added 2 commits August 19, 2020 17:21

charts: fix render.py script

d52813b

there is a bug introduced by the latest change on how we handle CSC in this script, the script fails if no --service-config or --remove-manifest options are provided.

salt: render fluent-bit chart into Salt formula

4d18d5d

./charts/render.py fluent-bit --namespace metalk8s-logging \ --remove-manifest ConfigMap fluent-bit \ charts/fluent-bit.yaml charts/fluent-bit/ \ > salt/metalk8s/addons/logging/fluent-bit/deployed/chart.sls Refs: #2701

alexandre-allard force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from a0acb6b to 0765628 Compare August 20, 2020 07:38

TeddyAndrieux reviewed Aug 20, 2020

View reviewed changes

tests/post/steps/test_logging.py Outdated Show resolved Hide resolved

alexandre-allard force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from 16ea9b1 to be626c7 Compare August 20, 2020 16:53

TeddyAndrieux reviewed Aug 21, 2020

View reviewed changes

salt/metalk8s/addons/logging/loki/deployed/services.sls Outdated Show resolved Hide resolved

TeddyAndrieux reviewed Aug 21, 2020

View reviewed changes

salt/metalk8s/addons/logging/fluent-bit/deployed/configmap.sls Outdated Show resolved Hide resolved

alexandre-allard added 6 commits August 24, 2020 10:32

salt: add fluent-bit configmap and init

c22539c

Refs: #2701

build: add fluent-bit formulas to salt tree

a1b8c1f

Refs: #2701

salt: add fluent-bit deployed to logging addon

384c08a

Refs: #2701

tests: add scenario for logging pipeline

e5d3901

spawn a container that logs a single line then try to retrieve it from Loki API Refs: #2701

salt: add a service for each loki replica

51a6291

This is needed for Fluent Bit to be able to send every logs to each Loki instance. Refs: #2682

salt: add 1 output block per loki replica

c0a0a46

We write all logs to each Loki replicas, so we need to define one Output block, in Fluent Bit configuration, per loki instance. Refs: #2701

alexandre-allard force-pushed the improvement/2701-deploy-fluent-bit-from-charts branch from be626c7 to c0a0a46 Compare August 24, 2020 08:33

TeddyAndrieux approved these changes Aug 24, 2020

View reviewed changes

bert-e merged commit 6f4c524 into development/2.6 Aug 24, 2020

bert-e deleted the improvement/2701-deploy-fluent-bit-from-charts branch August 24, 2020 11:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploy Fluent Bit #2723

Deploy Fluent Bit #2723

alexandre-allard commented Aug 6, 2020 •

edited

Loading

bert-e commented Aug 6, 2020

bert-e commented Aug 6, 2020

bert-e commented Aug 12, 2020

bert-e commented Aug 12, 2020

TeddyAndrieux Aug 19, 2020 •

edited

Loading

TeddyAndrieux Aug 19, 2020 •

edited

Loading

alexandre-allard Aug 20, 2020 •

edited

Loading

TeddyAndrieux Aug 24, 2020

TeddyAndrieux left a comment

TeddyAndrieux left a comment

alexandre-allard commented Aug 24, 2020

bert-e commented Aug 24, 2020

bert-e commented Aug 24, 2020

Deploy Fluent Bit #2723

Deploy Fluent Bit #2723

Conversation

alexandre-allard commented Aug 6, 2020 • edited Loading

bert-e commented Aug 6, 2020

Hello alexandre-allard-scality,

bert-e commented Aug 6, 2020

Waiting for approval

bert-e commented Aug 12, 2020

Conflict

bert-e commented Aug 12, 2020

Waiting for approval

TeddyAndrieux Aug 19, 2020 • edited Loading

Choose a reason for hiding this comment

TeddyAndrieux Aug 19, 2020 • edited Loading

Choose a reason for hiding this comment

alexandre-allard Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

TeddyAndrieux Aug 24, 2020

Choose a reason for hiding this comment

TeddyAndrieux left a comment

Choose a reason for hiding this comment

TeddyAndrieux left a comment

Choose a reason for hiding this comment

alexandre-allard commented Aug 24, 2020

bert-e commented Aug 24, 2020

In the queue

bert-e commented Aug 24, 2020

alexandre-allard commented Aug 6, 2020 •

edited

Loading

TeddyAndrieux Aug 19, 2020 •

edited

Loading

TeddyAndrieux Aug 19, 2020 •

edited

Loading

alexandre-allard Aug 20, 2020 •

edited

Loading