Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deployment of Fluent-Bit from Helm charts #2701

Closed
alexandre-allard opened this issue Aug 3, 2020 · 0 comments · Fixed by #2723
Closed

Deployment of Fluent-Bit from Helm charts #2701

alexandre-allard opened this issue Aug 3, 2020 · 0 comments · Fixed by #2723
Assignees
@alexandre-allard @TeddyAndrieux
Labels
complexity:easy Something that requires less than a day to fix kind:enhancement New feature or request topic:log Anything related to log centralization system

Comments

@alexandre-allard
Copy link
Contributor

alexandre-allard commented Aug 3, 2020 •
edited
Loading

Component: salt, charts, log, build

Why this is needed:
See #2668
This implementation may not be the final one, as we will probably change it afterwards to use the Fluent-Bit Operator, but we need it to have a functional logging pipeline for now.

What should be done:

Implementation proposal (strongly recommended):
As we already did for other add-ons (Prometheus, Dex, ...), we can use the charts provided by Fluent-Bit to ease the generation of Salt formulas, using the charts/renderer.py script:

  helm repo add loki https://grafana.github.io/loki/charts
  helm repo update
  helm fetch -d charts --untar loki/fluent-bit
  ./charts/render.py fluent-bit --namespace metalk8s-logging \
    charts/fluent-bit.yaml charts/fluent-bit/ \
    > salt/metalk8s/addons/fluent-bit/deployed/charts.sls

Test plan:
Ensure the fluent-bit pods are running (1 per node).
@alexandre-allard alexandre-allard added kind:enhancement New feature or request topic:log Anything related to log centralization system labels Aug 3, 2020
@alexandre-allard alexandre-allard self-assigned this Aug 3, 2020
@alexandre-allard alexandre-allard added complexity:medium Something that requires one or few days to fix complexity:easy Something that requires less than a day to fix and removed complexity:medium Something that requires one or few days to fix labels Aug 5, 2020
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
charts/log: initial import of fluent-bit helm charts
33713e7 
  helm repo add loki https://grafana.github.io/loki/charts
  helm repo update
  helm fetch -d charts --untar loki/fluent-bit

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
charts/log: add fluent-bit helm configuration values
28899db 
this file is used to overwrite default values.

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
build: fetch fluent-bit image during build
9829b4d 
Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
salt: render fluent-bit chart into Salt formula
0f6d728 
  ./charts/render.py fluent-bit --namespace metalk8s-logging \
    charts/fluent-bit.yaml charts/fluent-bit/ \
    > salt/metalk8s/addons/logging/fluent-bit/deployed/chart.sls

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
build: add fluent-bit formulas to salt tree
c19a251 
Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
salt: add fluent-bit deployed to logging addon
31e4e2a 
Refs: #2701
@alexandre-allard alexandre-allard mentioned this issue Aug 6, 2020
Merged
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
charts/log: initial import of fluent-bit helm charts
6be0cce 
  helm repo add loki https://grafana.github.io/loki/charts
  helm repo update
  helm fetch -d charts --untar loki/fluent-bit

Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
charts/log: add fluent-bit helm configuration values
a99a96c 
this file is used to overwrite default values.

Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
build: fetch fluent-bit image during build
5131ead 
Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
salt: render fluent-bit chart into Salt formula
cd7c004 
  ./charts/render.py fluent-bit --namespace metalk8s-logging \
    charts/fluent-bit.yaml charts/fluent-bit/ \
    > salt/metalk8s/addons/logging/fluent-bit/deployed/chart.sls

Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
build: add fluent-bit formulas to salt tree
ac27719 
Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
salt: add fluent-bit deployed to logging addon
693fe20 
Refs: #2701
@TeddyAndrieux TeddyAndrieux self-assigned this Aug 6, 2020
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
charts/log: initial import of fluent-bit helm charts
91883c5 
  helm repo add loki https://grafana.github.io/loki/charts
  helm repo update
  helm fetch -d charts --untar loki/fluent-bit

Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
charts/log: add fluent-bit helm configuration values
b148ff1 
this file is used to overwrite default values.

Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
build: fetch fluent-bit image during build
28c9e5b 
Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
salt: render fluent-bit chart into Salt formula
641a406 
  ./charts/render.py fluent-bit --namespace metalk8s-logging \
    charts/fluent-bit.yaml charts/fluent-bit/ \
    > salt/metalk8s/addons/logging/fluent-bit/deployed/chart.sls

Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
build: add fluent-bit formulas to salt tree
cc97b90 
Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
salt: add fluent-bit deployed to logging addon
bf86469 
Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 7, 2020
@alexandre-allard @TeddyAndrieux
charts/log: initial import of fluent-bit helm charts
2e1d5df 
  helm repo add loki https://grafana.github.io/loki/charts
  helm repo update
  helm fetch -d charts --untar loki/fluent-bit

Refs: #2701
TeddyAndrieux pushed a commit that referenced this issue Aug 7, 2020
@alexandre-allard @TeddyAndrieux
charts/log: add fluent-bit helm configuration values
a4fc115 
this file is used to overwrite default values.

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 19, 2020
@alexandre-allard
build: add fluent-bit formulas to salt tree
ea27caa 
Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 19, 2020
@alexandre-allard
salt: add fluent-bit deployed to logging addon
d476699 
Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 19, 2020
@alexandre-allard
tests: add scenario for logging pipeline
a0acb6b 
spawn a container that logs a single line
then try to retrieve it from Loki API

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
charts/log: initial import of fluent-bit helm charts
e1c4687 
  helm repo add loki https://grafana.github.io/loki/charts
  helm repo update
  helm fetch -d charts --untar loki/fluent-bit

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
charts/log: add fluent-bit helm configuration values
f1378e0 
this file is used to overwrite default values.

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
build: fetch fluent-bit image during build
c059dc9 
Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
salt: render fluent-bit chart into Salt formula
4d18d5d 
  ./charts/render.py fluent-bit --namespace metalk8s-logging \
    --remove-manifest ConfigMap fluent-bit \
    charts/fluent-bit.yaml charts/fluent-bit/ \
    > salt/metalk8s/addons/logging/fluent-bit/deployed/chart.sls

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
salt: add fluent-bit configmap and init
9aa20dd 
Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
build: add fluent-bit formulas to salt tree
da56eb0 
Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
salt: add fluent-bit deployed to logging addon
432bfdb 
Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
tests: add scenario for logging pipeline
0765628 
spawn a container that logs a single line
then try to retrieve it from Loki API

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
salt: add 1 output block per loki replica
16ea9b1 
We write all logs to each Loki replicas,
so we need to define one Output block,
in Fluent Bit configuration, per loki instance.

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
tests: add scenario for logging pipeline
596bb9c 
spawn a container that logs a single line
then try to retrieve it from Loki API

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
salt: add 1 output block per loki replica
be626c7 
We write all logs to each Loki replicas,
so we need to define one Output block,
in Fluent Bit configuration, per loki instance.

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 24, 2020
@alexandre-allard
salt: add fluent-bit configmap and init
c22539c 
Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 24, 2020
@alexandre-allard
build: add fluent-bit formulas to salt tree
a1b8c1f 
Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 24, 2020
@alexandre-allard
salt: add fluent-bit deployed to logging addon
384c08a 
Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 24, 2020
@alexandre-allard
tests: add scenario for logging pipeline
e5d3901 
spawn a container that logs a single line
then try to retrieve it from Loki API

Refs: #2701
alexandre-allard added a commit that referenced this issue Aug 24, 2020
@alexandre-allard
salt: add 1 output block per loki replica
c0a0a46 
We write all logs to each Loki replicas,
so we need to define one Output block,
in Fluent Bit configuration, per loki instance.

Refs: #2701
ChengYanJin pushed a commit that referenced this issue Sep 3, 2020
@alexandre-allard @ChengYanJin
charts/log: initial import of fluent-bit helm charts
c422522 
  helm repo add loki https://grafana.github.io/loki/charts
  helm repo update
  helm fetch -d charts --untar loki/fluent-bit

Refs: #2701
ChengYanJin pushed a commit that referenced this issue Sep 3, 2020
@alexandre-allard @ChengYanJin
charts/log: add fluent-bit helm configuration values
e561e1e 
this file is used to overwrite default values.

Refs: #2701
ChengYanJin pushed a commit that referenced this issue Sep 3, 2020
@alexandre-allard @ChengYanJin
build: fetch fluent-bit image during build
d11addc 
Refs: #2701
ChengYanJin pushed a commit that referenced this issue Sep 3, 2020
@alexandre-allard @ChengYanJin
salt: render fluent-bit chart into Salt formula
eb94a3d 
  ./charts/render.py fluent-bit --namespace metalk8s-logging \
    --remove-manifest ConfigMap fluent-bit \
    charts/fluent-bit.yaml charts/fluent-bit/ \
    > salt/metalk8s/addons/logging/fluent-bit/deployed/chart.sls

Refs: #2701
ChengYanJin pushed a commit that referenced this issue Sep 3, 2020
@alexandre-allard @ChengYanJin
salt: add fluent-bit configmap and init
6ec5e65 
Refs: #2701
ChengYanJin pushed a commit that referenced this issue Sep 3, 2020
@alexandre-allard @ChengYanJin
build: add fluent-bit formulas to salt tree
6d42700 
Refs: #2701
ChengYanJin pushed a commit that referenced this issue Sep 3, 2020
@alexandre-allard @ChengYanJin
salt: add fluent-bit deployed to logging addon
59d0215 
Refs: #2701
ChengYanJin pushed a commit that referenced this issue Sep 3, 2020
@alexandre-allard @ChengYanJin
tests: add scenario for logging pipeline
7ff9d97 
spawn a container that logs a single line
then try to retrieve it from Loki API

Refs: #2701
ChengYanJin pushed a commit that referenced this issue Sep 3, 2020
@alexandre-allard @ChengYanJin
salt: add 1 output block per loki replica
d037ab6 
We write all logs to each Loki replicas,
so we need to define one Output block,
in Fluent Bit configuration, per loki instance.

Refs: #2701
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexandre-allard alexandre-allard

@TeddyAndrieux TeddyAndrieux

Labels
complexity:easy Something that requires less than a day to fix kind:enhancement New feature or request topic:log Anything related to log centralization system
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Deploy Fluent Bit scality/metalk8s
2 participants
@alexandre-allard @TeddyAndrieux