-
Notifications
You must be signed in to change notification settings - Fork 546
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[4.0] Initial tree reorganization and new binary implementation #1993
Conversation
This pull request introduces 1 alert when merging 6baace7 into a240961 - view on LGTM.com new alerts:
|
/var/tmp vs /tmp I was not expecting the commands to move to bin/. I'm not strongly against it, just wondering why? I somewhat feel we should have just reviewed and committed #1508/#1873 separately first. Why does this make it easier? |
There are a few utilities that will collect an sosreport today, e.g. abrt. For one example, say abrt generates an sosreport and then a system experiences a kernel panic - you have now lost the pre-reboot sosreport. There's also a not-uncommon pattern of asking for an sosreport before and after some upgrade operation that includes a reboot for a kernel upgrade. Keeping them in place through reboots means less hassle for the end user in terms of number of trips to retrieve the archives.
That's just the location in the tree. Packaging would still place the binaries wherever they need to go. I.E. if you're running from the git checkout, yes you'd need to invoke using They need to be in different directory in the tree since we obviously can't have an
These are both relatively big changes to |
That's a good reason :).
We did have a discussion internally about some customers wanting to have sosreports generated on a regular basis - we would have that tool manage what ones need to be cleaned and when. Do you have anything that help keep /var/tmp clean? |
We don't have anything specific for it to my knowledge. |
Maybe |
6baace7
to
726d728
Compare
This pull request introduces 1 alert when merging 726d728 into a240961 - view on LGTM.com new alerts:
|
Realized there was a missing fixup when compared to my local branch, so I've repushed that after squashing it into the set. I'll fixup the LGTM alert when I inevitably make another push after getting feedback on this. |
We use |
726d728
to
7160256
Compare
I've updated the actual |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 with the sosreport arg fix
Re-pushed with this fixup change to have the
|
Few comments (that I might add more during this weekend):
So I have to call
|
|
||
for container in containers: | ||
self.add_cmd_output("podman inspect %s" % container, | ||
subdir='containers') | ||
|
||
for img in images: | ||
name, img_id = img.strip().split() | ||
name, img_id = img | ||
insp = name if 'none' not in name else img_id |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is not related to the change itself, but either way: what if a container name is e.g. danone
, i.e. containing none
string? I think it does not matter much, as img_id
is always set (am I right here?). So the only impact would be less human-readable file podman_inspect_<id>
will be stored instead of podman_inspect_<name>
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are situations where podman
(and docker
for that matter) will report an image name as a string of just 'none' or "<none>", so we could potentially make this a more exact match if needed. However, yes the image ID is always present, which is why we fallback like we do here.
"nvme smart-log /dev/%s" % dev, | ||
"nvme error-log /dev/%s" % dev, | ||
"nvme show-regs /dev/%s" % dev, | ||
"nvme get-ns-id /dev/%s" % dev |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This command is missing in cmds
. Is that intentional?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep - we talked about that in #1508 too. It's duplicate data.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think docs/plugins.rst
and setup.py
shall replace sos.plugins
to sos.report.plugins
as well, but definitely not required now (likewise Makefile
changes). Just a note for further reference/check.
Hmm, I would expect argparse to catch that and print the help message. Will dig into that to see if there's a way to force that.
Fair point, I'll drop that commit.
That is strange - it appears that python 3.6.8 on RHEL 8 is not attempting the last location in I'll raise a RHBZ for this to ask what the maintainers team for the python interpreter suggest.
Shouldn't be an issue, I'll swing back around to that one. |
The more I think about the EDIT: Oh, derp. This is actually obvious what's going on. You have a locally installed (older) version of sos, that is superseding the checkout path for the module imports. In other words, yes we need to change the append to an insert for all purposes. |
Parsing from
then most or all plugins raise an uncaught exception:
The traceback:
Similar problem happens for The cause is on line https://github.com/sosreport/sos/pull/1993/files#diff-7e7bbf33e24f9243b70a0371399d2fe4R166 where |
First, thanks for such prompt answers and fixes (and for the PR in general, of course! a tremendous amount of work).
(no further review points from me, yay!) |
@pmoravec nice catch! I thought I had caught the type casting, but apparently was overlooking it for the config file. I'm pushing a branch update that does explicit checks against the default type as defined by the component loading options into
Interesting. The way I understand aliases to work is that
My current view is that sos-collector integration will use policies as well. |
Ok, I've done a bit more digging. And this exact topic is a matter of (recurring) debate in upstream python - the prevailing recommendation right now though is to use
|
As an aside, what are the thoughts on the term |
This is the first patch to add a new `sos` binary to act as the new commandline interface for users. It adds the basic `SoS()` and `SoSComponent()` classes that will be built out and used going forward for new subcommand development. Actual functionality will be added in coming patches, as this is mainly serving as a checkpoint to avoid over-large commits affecting wide swathes of changes at once. Related: sosreport#1986 Signed-off-by: Jake Hunsaker <[email protected]>
Moves `SoSOptions()` and the related bits to a new sos/options.py location. This should allow for easier maintenance and ongoing development as our option handling approach will be shared across components. Signed-off-by: Jake Hunsaker <[email protected]>
Signed-off-by: Jake Hunsaker <[email protected]>
Signed-off-by: Jake Hunsaker <[email protected]>
Now dynamically build the usage string reported by --help for the top-level parser (I.E. `sos --help`) to print available components. Additionally, rework the subparser usage strings to accurately report the syntax for subcommands. Signed-off-by: Jake Hunsaker <[email protected]>
This is the first of probably a number of commits aimed at making SoSOptions() a generic option handling class for all components in sos, rather than being tightly bound to `report`. Signed-off-by: Jake Hunsaker <[email protected]>
Moves logging setup into `SoSComponent`, and logging is setup by default at component initialization, unless the `configure_logging` class attr is set to `False`. Moves the TempFileUtil() class out of report and into utilities for easier usage by all sos components. Signed-off-by: Jake Hunsaker <[email protected]>
As we are moving to decouple SoSOptions() from report, we need to adjust how some of the option loading is handled, such as loading values from a conf file. The previous approach was to instantiate a new SoSOptions() object, read into it from the config file, and then merge that with the "main" SoSOptions() object. This worked well when we only had report to worry about since the arguments were more or less static. However, in allowing new subcommands to define their own options this presented a problem due to the behavior of merging. This was compounded by the fact that merging still functioned how we expected/needed for actions like merging commandline options against the defaults. As such, the new approach is to use the "main" SoSOptions() instance to read the config file specifications and then directly modify the needed values rather than relying on the merging behavior. Signed-off-by: Jake Hunsaker <[email protected]>
Adds the exit handling from SoSReport() directly into SoSComponent() Signed-off-by: Jake Hunsaker <[email protected]>
Adds Policy loading into `SoSComponent()` so that any subcommand can, if it needs it, have policies loaded consistenly. This is controlled via the `load_policy` class attr, which defaults to True. Future subcommands that do not need to have a policy loaded (E.G. the planned `info` command), can simply set this to False in order to skip this step. Moves `SoSComponent()` out of sos/__init__.py and into sos/component.py to ease import conflicts arising from policy loading. Signed-off-by: Jake Hunsaker <[email protected]>
Moving the `_sos_ definition wrapper for gettext into utilities proved to be the wrong direction, so this moves it back into sos/__init__.py in order to preserve the ability to properly import it elsewhere. Signed-off-by: Jake Hunsaker <[email protected]>
For the 3.x line, many classes subclassed `object` directly, in order to use the 'new-style' objects introduced with python3, while still maintaining python2 functionality. As sos-4.0 is python3 only, we no longer need to do this. Signed-off-by: Jake Hunsaker <[email protected]>
Signed-off-by: Jake Hunsaker <[email protected]>
Moves SoSReport() from a standalone class to a subclass of SoSComponent() in order to take advantage of the new 4.0 redesign. This commit also pulls out methods that were previously transitioned to other places, e.g. logging setup moving from SoSReport() to the base SoSComponent() class. Signed-off-by: Jake Hunsaker <[email protected]>
For python3, calling `.keys()` on a dict no longer returns a list, so we need to explicitly make it a list of values to behave as expected. Signed-off-by: Jake Hunsaker <[email protected]>
Removes an extraneous setting of self.sys_tmp, which is now handled via `SoSComponent`. This was getting overridden to `None` which was causing the final archive creation to fail. Signed-off-by: Jake Hunsaker <[email protected]>
Signed-off-by: Jake Hunsaker <[email protected]>
When loading from a config file, ensure that verbosity is set to and integer rather than a string. Signed-off-by: Jake Hunsaker <[email protected]>
Adds the `desc` class attr to SoSReport() for use by the help text when `sos --help` is run. Signed-off-by: Jake Hunsaker <[email protected]>
Since sos-4.0 is py3 only, set the interpreset of the binary as such Signed-off-by: Jake Hunsaker <[email protected]>
We need to carry a `sosreport` binary for some time to allow for end users and downstreams to adjust to the new binary. It was hoped originally that the old `sosreport` binary could be maintained in place and simply provide the older set of functionality. This has proven to not be possible givent he overhaul of the options handling that allows us to have multiple subcommands. So while we will still ship an `sosreport` binary, and it will be locked to `report` functionality, it is now a simple redirection script that also makes the user aware of the new `sos` binary. Closes: sosreport#1986 Signed-off-by: Jake Hunsaker <[email protected]>
Moves the HTML/JSON reporting bits under sos/report. Signed-off-by: Jake Hunsaker <[email protected]>
Adds support to the parser to allow aliases for subcommands in the parser. At the moment, this only applies to `report`, with the 'rep' shorthand. This is however intended for further 4.0 use via the integration of sos-collector and soscleaner. For example, invoking 'sos clean' and 'sos cleaner' would both be possible. Signed-off-by: Jake Hunsaker <[email protected]>
Consolidated PEP8 fixes following reorganization and redeisgn work. Signed-off-by: Jake Hunsaker <[email protected]>
Updates the test script `simple.sh` used by Travis to use the new `sos` binary with the `report` subcommand. Signed-off-by: Jake Hunsaker <[email protected]>
Removes an extraneous import identified by LGTM Signed-off-by: Jake Hunsaker <[email protected]>
Ok, trivial fixups have been merged and this PR has been rebased on current master. We have acks from both the Red Hat and Canonical sides, so I'm viewing this as good-to-go in its current state. However, I will leave this PR open for further comments until tomorrow morning (-ish) for any last minute critiques. Should there be none, I'll merge at that point. |
This is the initial proposal for the reorganization of the project tree, and the introduction of the new
sos
binary as the command of choice for running sos.In no particular order, here are some highlights that deserve attention:
This includes [Plugin] Add iterator for block devices, update relevant plugins, and remove add_udev_info #1508 and [Policy|Plugin] Provide a Container Runtime Abstraction #1873. This is done now to make the merging process easier, and to close out two of the goals highlighted in the 4.0 RFC.
Policy initialization is now done after logging setup. I did this because in previous
Policy()
work I have found it frustrating that logging was not available. This plays into the request for [policies] Detect the RedHat OS variant using 'ID' field #1934.tmp_dir
, which is needed to setup logging within the temp directory created for sos. I have tentatively set this to be/var/tmp
if one is not specified by--tmp-dir
by the user. This appears to be a change for Ubuntu, away from/tmp
, and so I'd like @battlemidget and @BryanQuigley to weigh in on if that is acceptable or if that poses some major issue for that distro line.Subcommands can be added by subclassing
SoSComponent
from sos/component.py. Report has been updated to make use of this mechanism. Subcommands must also be explicitly added insos/__init__.py/SoS()
handling to be supported. I wanted to make the tree as clean as possible, and dynamic discovery and loading of subcommands proved to be at odds with that.SoSOptions
required a lot of rework. Most of the functionality remains the same however, with two major exceptions.SoSComponent
in thearg_defaults
class attribute. This is a dict of key/value pairings for any options that subcommand implements. This replaces the previous design wherearg_defaults
was statically set forSoSOptions
.SoSOptions
instance that gets used and morphed intoself.opts
for anySoSComponent
.The interpreter has been set to
python3
, in order to kickstart the python3-only development mindset for 4.0. I am not sure if it is still functional if you manually shove it through a python2 interpreter, and I have not tested doing so.Do not expect
make
orsetuputils
to work at this time. Packaging will be addressed further into the 4.0 development cycle.There are a lot of commits in this PR. I tried to condense where I could and where it made sense, however with how involved this work is I think it's reasonable to say that any developer would assume that individual commits can be breaking until the end of this patchset.
Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines
Closes: #ISSUENUMBER
included in an independent line?Resolves: #PRNUMBER
included in an independent line?