Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deployment of Loki #2682

Closed
alexandre-allard opened this issue Jul 29, 2020 · 0 comments · Fixed by #2722
Closed

Deployment of Loki #2682

alexandre-allard opened this issue Jul 29, 2020 · 0 comments · Fixed by #2722
Assignees
@alexandre-allard @TeddyAndrieux
Labels
complexity:medium Something that requires one or few days to fix kind:enhancement New feature or request topic:log Anything related to log centralization system

Comments

@alexandre-allard
Copy link
Contributor

alexandre-allard commented Jul 29, 2020 •
edited
Loading

Component: salt, charts, log, build

Why this is needed: See #2668

What should be done:

Implementation proposal (strongly recommended):
As we already did for other add-ons (Prometheus, Dex, ...), we can use the charts provided by Loki to ease the generation of Salt formulas, using the charts/renderer.py script:

  helm repo add loki https://grafana.github.io/loki/charts
  helm repo update
  helm fetch -d charts --untar loki/loki
  ./charts/render.py loki --namespace metalk8s-logging \
    charts/loki.yaml charts/loki/ \
    > salt/metalk8s/addons/loki/deployed/charts.sls

Loki helm chart: https://github.com/grafana/loki/tree/master/production/helm

Test plan:
Add simple test to ensure loki is OK during pre-merge by sending a log on Loki API and reading it afterwards.
@alexandre-allard alexandre-allard added kind:enhancement New feature or request topic:log Anything related to log centralization system labels Jul 29, 2020
@alexandre-allard alexandre-allard changed the title Deployment of Loki & Fluent Bit Deployment of Loki Aug 3, 2020
@alexandre-allard alexandre-allard self-assigned this Aug 5, 2020
@alexandre-allard alexandre-allard added the complexity:medium Something that requires one or few days to fix label Aug 5, 2020
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
charts/log: initial import of loki helm charts
20aaf2b 
  helm repo add loki https://grafana.github.io/loki/charts
  helm repo update
  helm fetch -d charts --untar loki/loki

Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
charts: add option to remove manifests
403a7b6 
Add `--remove-manifest KIND NAME` option allowing
to remove some manifests from the resulting chart.
Useful when we want to deploy and manage ourself
some object of an addon (e.g. Loki secret).

Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
charts/log: add loki helm configuration values
e8bdd91 
this file is used to overwrite default values.

Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
build: fetch loki image during build
d6e7663 
Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
salt: render loki chart into Salt formula
42a975d 
  ./charts/render.py loki --namespace metalk8s-logging \
    --remove-manifest Secret loki \
    charts/loki.yaml charts/loki/ \
    > salt/metalk8s/addons/logging/loki/deployed/chart.sls

Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
salt: deploy metalk8s-logging namespace
263d5cc 
Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
build: add loki formulas to salt tree
3127c9e 
Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
salt: deploy logging stack during MetalK8s install
62f64d4 
For now it only deploys loki, but it'll
also include Fluent Bit installation.

Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
salt: add CSC mechanism to Loki
edde110 
Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 6, 2020
@alexandre-allard
salt: add Loki storageclass
2209681 
Refs: #2682
@alexandre-allard alexandre-allard mentioned this issue Aug 6, 2020
Merged
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
salt: add CSC mechanism to Loki
213d546 
Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
salt: add Loki storageclass
a7e07bb 
Refs: #2682
@TeddyAndrieux TeddyAndrieux self-assigned this Aug 6, 2020
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
charts/log: add loki helm configuration values
824f9c0 
this file is used to overwrite default values.

Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
build: fetch loki image during build
01dbca8 
Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
salt: render loki chart into Salt formula
69afe9a 
  ./charts/render.py loki --namespace metalk8s-logging \
    --remove-manifest Secret loki \
    charts/loki.yaml charts/loki/ \
    > salt/metalk8s/addons/logging/loki/deployed/chart.sls

Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
salt: deploy metalk8s-logging namespace
542e145 
Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
build: add loki formulas to salt tree
f956fc8 
Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
salt: deploy logging stack during MetalK8s install
c26e51b 
For now it only deploys loki, but it'll
also include Fluent Bit installation.

Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
salt: add CSC mechanism to Loki
f5706cd 
Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 6, 2020
@alexandre-allard @TeddyAndrieux
salt: add Loki storageclass
34cd3e2 
Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 7, 2020
@alexandre-allard @TeddyAndrieux
charts/log: initial import of loki helm charts
c36920b 
  helm repo add loki https://grafana.github.io/loki/charts
  helm repo update
  helm fetch -d charts --untar loki/loki

Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 10, 2020
@alexandre-allard
salt: deploy logging stack during MetalK8s install
1c8218c 
For now it only deploys loki, but it'll
also include Fluent Bit installation.

Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 10, 2020
@alexandre-allard
charts: add option to change path of CSC defaults
d12fc75 
This is needed as we put Loki inside logging directory,
so the path is not metalk8s/addons/loki/config/loki.yaml
but metalk8s/addons/logging/loki/config/loki.yaml, so
we need to be able to set the path.

Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 10, 2020
@alexandre-allard
salt: add CSC mechanism to Loki
3feae7b 
Re-render the chart with CSC:

  ./charts/render.py loki \
    --remove-manifest Secret loki \
    --namespace metalk8s-logging \
    --service-config loki metalk8s-loki-config \
    metalk8s/addons/logging/loki/config/loki.yaml \
    charts/loki.yaml charts/loki/ \
    > salt/metalk8s/addons/logging/loki/deployed/chart.sls

Refs: #2682
alexandre-allard pushed a commit that referenced this issue Aug 10, 2020
@TeddyAndrieux @alexandre-allard
build,example,eve: Add example YAML for Loki volume
e5fce98 
Add an example file to create a sparseLoopDevice volume for Loki, add
this file in the ISO and also use it for CI and vagrant deployment

Refs: #2682
@TeddyAndrieux TeddyAndrieux mentioned this issue Aug 11, 2020
Closed
TeddyAndrieux pushed a commit that referenced this issue Aug 11, 2020
@alexandre-allard @TeddyAndrieux
charts/log: initial import of loki helm charts
41acd19 
  helm repo add loki https://grafana.github.io/loki/charts
  helm repo update
  helm fetch -d charts --untar loki/loki

Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 11, 2020
@alexandre-allard @TeddyAndrieux
charts: add option to remove manifests
65cf7bf 
Add `--remove-manifest KIND NAME` option allowing
to remove some manifests from the resulting chart.
Useful when we want to deploy and manage ourself
some object of an addon (e.g. Loki secret).

Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 11, 2020
@alexandre-allard @TeddyAndrieux
charts/log: add loki helm configuration values
2da2ddd 
this file is used to overwrite default values.

Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 11, 2020
@alexandre-allard @TeddyAndrieux
build: fetch loki image during build
79e58f8 
Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 11, 2020
@alexandre-allard @TeddyAndrieux
salt: add Loki storageclass
6628336 
Refs: #2682
TeddyAndrieux added a commit that referenced this issue Aug 11, 2020
@TeddyAndrieux
charts: add persistence storage to Loki
97cd2ff 
Enable persistence from Loki chart
NOTE: That Loki chart does not allow to provide selector to claim
Volume, so any volume using `metalk8s-loki` storageClass can be bound to
the Loki container

Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 11, 2020
@alexandre-allard @TeddyAndrieux
salt: render loki chart into Salt formula
4e5d4d7 
  ./charts/render.py loki --namespace metalk8s-logging \
    --remove-manifest Secret loki \
    charts/loki.yaml charts/loki/ \
    > salt/metalk8s/addons/logging/loki/deployed/chart.sls

Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 11, 2020
@alexandre-allard @TeddyAndrieux
salt: deploy metalk8s-logging namespace
fce1062 
Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 11, 2020
@alexandre-allard @TeddyAndrieux
build: add loki formulas to salt tree
e98d6ad 
Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 11, 2020
@alexandre-allard @TeddyAndrieux
salt: deploy logging stack during MetalK8s install
a66ef4d 
For now it only deploys loki, but it'll
also include Fluent Bit installation.

Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 11, 2020
@alexandre-allard @TeddyAndrieux
charts: add option to change path of CSC defaults
daea4b2 
This is needed as we put Loki inside logging directory,
so the path is not metalk8s/addons/loki/config/loki.yaml
but metalk8s/addons/logging/loki/config/loki.yaml, so
we need to be able to set the path.

Refs: #2682
TeddyAndrieux pushed a commit that referenced this issue Aug 11, 2020
@alexandre-allard @TeddyAndrieux
salt: add CSC mechanism to Loki
97b15c9 
Re-render the chart with CSC:

  ./charts/render.py loki \
    --remove-manifest Secret loki \
    --namespace metalk8s-logging \
    --service-config loki metalk8s-loki-config \
    metalk8s/addons/logging/loki/config/loki.yaml \
    charts/loki.yaml charts/loki/ \
    > salt/metalk8s/addons/logging/loki/deployed/chart.sls

Refs: #2682
TeddyAndrieux added a commit that referenced this issue Aug 11, 2020
@TeddyAndrieux
build,example,eve: Add example YAML for Loki volume
784ae86 
Add an example file to create a sparseLoopDevice volume for Loki, add
this file in the ISO and also use it for CI and vagrant deployment

Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 13, 2020
@alexandre-allard
salt: enable Loki table manager purge
df165e6 
set to 2 weeks which means we can have 3 weeks
of retention at most.

Refs: #2682
@alexandre-allard alexandre-allard mentioned this issue Aug 13, 2020
Merged
Ebaneck pushed a commit that referenced this issue Aug 17, 2020
@alexandre-allard @Ebaneck
charts: add option to change path of CSC defaults
df0452b 
This is needed as we put Loki inside logging directory,
so the path is not metalk8s/addons/loki/config/loki.yaml
but metalk8s/addons/logging/loki/config/loki.yaml, so
we need to be able to set the path.

Refs: #2682
Ebaneck pushed a commit that referenced this issue Aug 17, 2020
@alexandre-allard @Ebaneck
charts: add option to change path of CSC defaults
e22be54 
This is needed as we put Loki inside logging directory,
so the path is not metalk8s/addons/loki/config/loki.yaml
but metalk8s/addons/logging/loki/config/loki.yaml, so
we need to be able to set the path.

Refs: #2682
(cherry picked from commit daea4b2)
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
salt: add a service for each loki replica
a7550b3 
This is needed for Fluent Bit to be able to
send every logs to each Loki instance.

Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 20, 2020
@alexandre-allard
salt: add a service for each loki replica
2b96a7a 
This is needed for Fluent Bit to be able to
send every logs to each Loki instance.

Refs: #2682
alexandre-allard added a commit that referenced this issue Aug 24, 2020
@alexandre-allard
salt: add a service for each loki replica
51a6291 
This is needed for Fluent Bit to be able to
send every logs to each Loki instance.

Refs: #2682
ChengYanJin pushed a commit that referenced this issue Sep 3, 2020
@alexandre-allard @ChengYanJin
salt: add a service for each loki replica
b710d71 
This is needed for Fluent Bit to be able to
send every logs to each Loki instance.

Refs: #2682
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexandre-allard alexandre-allard

@TeddyAndrieux TeddyAndrieux

Labels
complexity:medium Something that requires one or few days to fix kind:enhancement New feature or request topic:log Anything related to log centralization system
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Deploy Loki scality/metalk8s
2 participants
@alexandre-allard @TeddyAndrieux