Enrich SOS report content #3222

alexandre-allard · 2021-03-23T10:35:43Z

Component: sosreport

Context:
In our CI, we do not have enough data collected when a build fails, so it's hard to investigate flakies or other issue that might happen.

Summary:

Backport of Fix nested sosreport plugin options #3013
Add Salt minion service logs
Add Salt configuration directory (/etc/salt)
Add a Prometheus snapshot
Add kubectl top nodes & pods
Add bootstrap and solutions configuration files
Fix an issue in both containerd and metalk8s plugins that were using an undefined method.

Still need to enable it in the CI (especially enabling Prometheus admin API to be able to generate a snapshot).

Acceptance criteria:
We can retrieve an sosreport with all elements listed above.

Closes: #3218

bert-e · 2021-03-23T10:35:44Z

Hello alexandre-allard-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2021-03-23T10:35:49Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

bert-e · 2021-03-25T13:23:26Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

(cherry picked from commit ea6fe1f)

Replace self.get_command_output() by self.exec_cmd() method as the former does not exist (anymore?).

Retrieve /etc/salt (excluding /etc/salt/pki) and the journal logs from salt-minion service Refs: #3128

This is mostly needed to be able to generate snapshots

Add the prometheus-snapshot option to sosreport metalk8s plugin. This option generate a snapshot using the Prometheus API and then put it in the archive. The Prometheus admin API must be enabled for this option to work.

Use right `kubectl` flags to remove unneeded output to avoid having to parse it.

Refs: #3218

Add bootstrap and solutions configuration files to the sosreport archive. Refs: #3218

bert-e · 2021-03-25T16:22:13Z

Branches have diverged

This pull request's source branch improvement/3218-enrich-sos-report-content has diverged from
development/2.9 by more than 50 commits.

To avoid any integration risks, please re-synchronize them using one of the
following solutions:

Merge origin/development/2.9 into improvement/3218-enrich-sos-report-content
Rebase improvement/3218-enrich-sos-report-content onto origin/development/2.9

Note: If you choose to rebase, you may have to ask me to rebuild
integration branches using the reset command.

bert-e · 2021-03-25T16:22:25Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.6/improvement/3218-enrich-sos-report-content with contents from improvement/3218-enrich-sos-report-content
and development/2.6.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.6/improvement/3218-enrich-sos-report-content origin/development/2.6
 $ git merge origin/improvement/3218-enrich-sos-report-content
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.6/improvement/3218-enrich-sos-report-content

bert-e · 2021-03-26T08:49:04Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.7/improvement/3218-enrich-sos-report-content with contents from w/2.6/improvement/3218-enrich-sos-report-content
and development/2.7.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.7/improvement/3218-enrich-sos-report-content origin/development/2.7
 $ git merge origin/w/2.6/improvement/3218-enrich-sos-report-content
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.7/improvement/3218-enrich-sos-report-content

bert-e · 2021-03-26T10:08:21Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.8/improvement/3218-enrich-sos-report-content with contents from w/2.7/improvement/3218-enrich-sos-report-content
and development/2.8.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.8/improvement/3218-enrich-sos-report-content origin/development/2.8
 $ git merge origin/w/2.7/improvement/3218-enrich-sos-report-content
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.8/improvement/3218-enrich-sos-report-content

bert-e · 2021-03-26T11:10:24Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/3218-enrich-sos-report-content into
development/2.5
w/2.6/improvement/3218-enrich-sos-report-content will be merged into development/2.6
w/2.7/improvement/3218-enrich-sos-report-content will be merged into development/2.7
w/2.8/improvement/3218-enrich-sos-report-content will be merged into development/2.8
w/2.9/improvement/3218-enrich-sos-report-content will be merged into development/2.9

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2021-03-26T11:10:25Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

Refs: #3218

TeddyAndrieux

Just a question and comments (already said offline):

changelog entry
Add the command to render the prometheus chart in the commit message (just to be consistent)
A bit of documentation, at least about prometheus snapshot generation

Otherwise it looks good to me

packages/common/metalk8s-sosreport/metalk8s.py

TeddyAndrieux

LGTM

alexandre-allard · 2021-04-07T06:45:26Z

/approve

bert-e · 2021-04-07T06:45:46Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.5
✔️ development/2.6
✔️ development/2.7
✔️ development/2.8
✔️ development/2.9

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2021-04-07T15:36:38Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.5
✔️ development/2.6
✔️ development/2.7
✔️ development/2.8
✔️ development/2.9

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3
development/2.4

Please check the status of the associated issue None.

Goodbye alexandre-allard-scality.

alexandre-allard requested a review from a team as a code owner March 23, 2021 10:35

NicolasT changed the title ~~Rnrich SOS report content~~ Enrich SOS report content Mar 23, 2021

alexandre-allard force-pushed the improvement/3218-enrich-sos-report-content branch from 2ab4f7f to 049454e Compare March 25, 2021 13:23

TrevorBenson and others added 8 commits March 25, 2021 15:44

Dependency on option all removed from option describe.

d66cbd9

(cherry picked from commit ea6fe1f)

packages: Fix sosreport plugins

b6afc17

Replace self.get_command_output() by self.exec_cmd() method as the former does not exist (anymore?).

packages: Add salt-minion in metalk8s sos plugin

addd121

Retrieve /etc/salt (excluding /etc/salt/pki) and the journal logs from salt-minion service Refs: #3128

salt,charts: Allow to enable Prometheus admin API with CSC

32c52bc

This is mostly needed to be able to generate snapshots

packages: Add prometheus-snapshot option to sosreport

b1e0fff

Add the prometheus-snapshot option to sosreport metalk8s plugin. This option generate a snapshot using the Prometheus API and then put it in the archive. The Prometheus admin API must be enabled for this option to work.

packages: remove unnecessary output from exec cmd

2ae0300

Use right `kubectl` flags to remove unneeded output to avoid having to parse it.

packages: Add kubectl top to sosreport

3754c81

Refs: #3218

packages: Add MetalK8s configuration files to sosreport

f282f78

Add bootstrap and solutions configuration files to the sosreport archive. Refs: #3218

alexandre-allard force-pushed the improvement/3218-enrich-sos-report-content branch from 049454e to f282f78 Compare March 25, 2021 16:22

alexandre-allard changed the base branch from development/2.9 to development/2.5 March 25, 2021 16:22

Add CHANGELOG entry for sosreport plugins enrichment

2e2bb8c

Refs: #3218

TeddyAndrieux reviewed Mar 31, 2021

View reviewed changes

packages/common/metalk8s-sosreport/metalk8s.py Show resolved Hide resolved

TeddyAndrieux approved these changes Apr 6, 2021

View reviewed changes

bert-e merged commit 2e2bb8c into development/2.5 Apr 7, 2021

bert-e deleted the improvement/3218-enrich-sos-report-content branch April 7, 2021 15:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enrich SOS report content #3222

Enrich SOS report content #3222

alexandre-allard commented Mar 23, 2021 •

edited

Loading

bert-e commented Mar 23, 2021

bert-e commented Mar 23, 2021

bert-e commented Mar 25, 2021

bert-e commented Mar 25, 2021

bert-e commented Mar 25, 2021

bert-e commented Mar 26, 2021

bert-e commented Mar 26, 2021

bert-e commented Mar 26, 2021

bert-e commented Mar 26, 2021

TeddyAndrieux left a comment

TeddyAndrieux left a comment

alexandre-allard commented Apr 7, 2021

bert-e commented Apr 7, 2021

bert-e commented Apr 7, 2021

Enrich SOS report content #3222

Enrich SOS report content #3222

Conversation

alexandre-allard commented Mar 23, 2021 • edited Loading

bert-e commented Mar 23, 2021

Hello alexandre-allard-scality,

bert-e commented Mar 23, 2021

Waiting for approval

bert-e commented Mar 25, 2021

Waiting for approval

bert-e commented Mar 25, 2021

Branches have diverged

bert-e commented Mar 25, 2021

Conflict

bert-e commented Mar 26, 2021

Conflict

bert-e commented Mar 26, 2021

Conflict

bert-e commented Mar 26, 2021

Integration data created

bert-e commented Mar 26, 2021

Waiting for approval

TeddyAndrieux left a comment

Choose a reason for hiding this comment

TeddyAndrieux left a comment

Choose a reason for hiding this comment

alexandre-allard commented Apr 7, 2021

bert-e commented Apr 7, 2021

In the queue

bert-e commented Apr 7, 2021

alexandre-allard commented Mar 23, 2021 •

edited

Loading