[plugins] Implement foreman-debug in native sos plugins #1525

TurboTurtle · 2018-12-28T21:41:56Z

According to senior resources within the RH Satellite support team, it would be desirable to have sos natively collect what the foreman-debug script collects. This patch set is an effort to port that script to sos.

This creates 4 new plugins - ruby, katello, candlepin, and pulp and overhauls the foreman plugin to gather the bulk of what foreman-debug collects.

Where advantageous, particularly database output, the sos plugins retain the filenames used by foreman-debug, for example foreman_settings_table rather than the long postgres command use to access the database, which includes switching to the postgres user as that is the only way to attach to the database by configuration design. This also is to ensure we don't inadvertently break any automated tooling that may be looking for those filenames.

Additionally, instead of duplicating add_copy_specs across the foreman and apache plugins, the Satellite preset now enables the apache.log option to gather everything under /var/log/httpd.

As noted in the foreman plugin we are adding the catalina add_copy_spec there since, according to support resources, it is very common for support cases to need to go back several days in catalina logs and we don't want to needlessly clutter other sosreports that enable tomcat.

Many thanks to Paul Dudley from Red Hat for assistance with verification and initial identification of the need/desire for this work, and for providing an environment to test these changes in.

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

Is the commit message split over multiple lines and hard-wrapped at 72 characters?
Is the subject and message clear and concise?
Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
Does the commit contain a Signed-off-by: First Lastname [email protected]?

TurboTurtle · 2018-12-28T21:43:51Z

Tagging @pmoravec for review as I'm sure you'll be particularly keen on making sure this has full coverage.

pmoravec · 2018-12-29T21:39:31Z

Cc @lzap as the main foreman-debug contributor for his awareness / possible feedback.

My very first reaction:

On one side, importing data collected by foreman-debug into sosreport simplifies some stuff. On the other side, maintaining sosreport will be harder - any change in foreman-debug will have to be newly/additionally manually added into sosreport as well. I dont have strong opinion on weights of these pros/negs.

If I am not mistaken, the PR does not carry forward https://github.com/theforeman/foreman-tasks/blob/master/extra/dynflow-debug.sh or https://github.com/theforeman/foreman_hooks/blob/master/extra/foreman-debug.sh that typical deployment of foreman-debug calls (there are symlinks in /usr/share/foreman/script/foreman-debug.d to these files, and foreman-debug calls all scripts within this directory). (btw. adding such extra scripts by foreman-debug community must be monitored by sosreport maintainers as well, to keep parity/inclusion of collected data).

There is duplicated method build_query_cmd (that's name shall refer to postgres, imho). But I dont see an easy way of merging that code to one place - until we have some "layered plugins structure" (one plugin calls commands from another one), or putting the code into parent Plugin class (does not sound much appropriate to me).

TurboTurtle · 2018-12-30T15:09:46Z

If I am not mistaken, the PR does not carry forward https://github.com/theforeman/foreman-tasks/blob/master/extra/dynflow-debug.sh or https://github.com/theforeman/foreman_hooks/blob/master/extra/foreman-debug.sh that typical deployment of foreman-debug calls (there are symlinks in /usr/share/foreman/script/foreman-debug.d to these files, and foreman-debug calls all scripts within this directory). (btw. adding such extra scripts by foreman-debug community must be monitored by sosreport maintainers as well, to keep parity/inclusion of collected data).

Yep, looks like I completely glossed over that. I'll get something worked up soon-ish and added here.

There is duplicated method build_query_cmd (that's name shall refer to postgres, imho). But I dont see an easy way of merging that code to one place - until we have some "layered plugins structure" (one plugin calls commands from another one), or putting the code into parent Plugin class (does not sound much appropriate to me).

Yeah, that's what I came up with to - I didn't have a good approach besides putting it in both plugins. It's small enough that I don't think it's worth worrying about, but if we do have a growing need for something similar we should talk about that. I flirted with the idea of putting build_query_cmd into sos.utilities, but the fact that it's specific to exactly 2 plugins (and postgres like you mention) dissuaded me.

lzap · 2019-01-07T14:49:09Z

According to senior resources within the RH Satellite support team, it would be desirable to have sos natively collect what the foreman-debug script collects.

Simple question - why? What are the reasons? I am against rewriting this because it's "cleaner" or "nicer".

Our foreman-debug works fine for upstream project, it was designed with sosreport in mind (option -g), it works on non-redhat systems and provides us our own releasing pace which is very important. Sosreport is released with RHEL cadence which is causing huge troubles in Satellite and Foreman release processes, similarly to SELinux base policy. If we want to go this path we need to make sure that:

a) sosreport is on par with features and all the logs and commands collected by foreman-debug
b) we have a plan on cadence and release engineering
c) it works for debian and ubuntu users at least
d) we must be sure that we can't evolve what we have rather than starting from scratch

TurboTurtle · 2019-01-07T15:17:53Z

According to senior resources within the RH Satellite support team, it would be desirable to have sos natively collect what the foreman-debug script collects.

Simple question - why? What are the reasons? I am against rewriting this because it's "cleaner" or "nicer".

From a purely RH perspective, consolidating support tooling is a big gain for customer experience - both with the product and with support engagement in general.

From a sos perspective, this allows us to take advantage of the sos infrastructure - time limiting, size limiting, ability to avoid certain collection entirely (for whatever reason - most notably hanging collection on seriously broken systems), etc... A bash script circumvents all of this.

As for the Satellite team's specifics, I'll let @pdudley speak to that.

Our foreman-debug works fine for upstream project, it was designed with sosreport in mind (option -g), it works on non-redhat systems and provides us our own releasing pace which is very important.

Even with -g, foreman-debug duplicates some information already collected by sos.

Sosreport is released with RHEL cadence which is causing huge troubles in Satellite and Foreman release processes, similarly to SELinux base policy. If we want to go this path we need to make sure that:

sosreport on RHEL is released with the RHEL cadence. The upstream sos project does 2 releases a year (or at least we aim to), one of which does traditionally line up with a RHEL release with the other not doing so.

a) sosreport is on par with features and all the logs and commands collected by foreman-debug

I'm curious if you have anything in mind that sosreport is lacking in comparison to foreman-debug today?

b) we have a plan on cadence and release engineering
c) it works for debian and ubuntu users at least

sos is available on the Debian and Ubuntu distros, all we'd need to do is verify package names and log locations. If either of those are different, then adjusting the plugin for different distros is trivial.

d) we must be sure that we can't evolve what we have rather than starting from scratch

I'd argue this isn't starting from scratch - we're extending a well maintained and supported collection utility and thus evolving the current collection strategy for a specific product/subset of components.

TurboTurtle · 2019-01-07T23:02:05Z

I've updated the PR to include dynflow query output from the foreman-debug extension in the foreman plugin. I elected to not do this as a new dynflow plugin as the logs collected by the extension were already being grabbed by foreman, and the db queries reference the same foreman db.

Note that I made a slight change to the psql syntax for the timestamp as the existing syntax was causing quoting issues for the commands being run. Rather than setting a DYNFLOW_EXPORT_MONTHS env var, this is now controlled by a new foreman.months plugin option that defaults to 6 (like the existing behavior).

lzap · 2019-01-08T10:23:58Z

From a purely RH perspective

No discussion that this is a great benefit for RH support, it's pretty clear.

foreman-debug duplicates some information already collected by sos.

Let's fix that.

The upstream sos project does 2 releases

That sounds good, although I still have some concerns about RHEL cadence being too slow for us. I guess in case there is some important bug, errata is possible.

I'm curious if you have anything in mind that sosreport is lacking in comparison to foreman-debug today?

I was not comparing features of a generic support collector tool itself, sosreport wins here that's clear but rather amount of files and commends collected by foreman-debug. I am curious if you implemented everything. Can you at least give me git commit sha of the foreman-debug you used as the starting point? I'd like to do side-by-side comparison.

Having that said, we have a nice feature - ability to upload debug tarball to our rsync write-only site. I spent one minute digging in the sos codebase but was unable to find relevant code. If there is a way to create some "profile" called foreman with ability to upload tarball that would be great. We can switch from rsync to FTP if this is the only protocol sosreport supports (I think I've heard sosreport can do FTP upload not sure).

sos is available on the Debian and Ubuntu distros, all we'd need to do is verify package names and log locations.

Foreman-debug has all the log files locations written in a way that it should also work on Debian/Ubuntu, note /etc/apache2 as an example. I would hate having two tools for debugging purposes, if we decide to go with sosreport path I would rather discourage use of foreman-debug in a warning message asking to install and run sosreport.

I'd argue this isn't starting from scratch

Again, I did not mean sosreport as something from scratch, I thought you are coming up with new list of files we should be collecting. But after a closer look it looks like you took existing foreman-debug and implemented the same. After a side-by-side comparison this should be good.

As original author of foreman-debug, I have no problems moving to sosreport honestly. I do not like idea of having two tools however and I would like to see a smooth transition period. I suggest the following:

Do side-by-side comparison of foreman-debug and this PR to make sure we are not dropping any important file or command.
Present this as a proposal to RH support folks for feedback.
Present this as a proposal to Foreman community for feedback.
Implement deprecated warning in foreman-debug.
Ship this code in sosreport upstream release.
After transition period, remove all code from foreman-debug in late 2019 leaving it as a simple stub: use sosreport.

pmoravec · 2019-01-08T11:47:38Z

* Do side-by-side comparison of foreman-debug and this PR to make sure we are not dropping any important file or command.

As foreman-debug calls few "hooks", I am comparing now if current upstream sos + this PR is on par with:

bmr-cymru · 2019-01-08T12:16:12Z

That sounds good, although I still have some concerns about RHEL cadence being too slow for us. I guess in case there is some important bug, errata is possible.

Just to squash this myth once and for all: sos does not (and has not for several years), have a "RHEL cadence". We started moving away from that with the release of 3.0, in 2014. We now aim, as @TurboTurtle explained, to release twice a year, roughly every six months - this is the goal, but as we are a small team with no full-time resources, and as we are often flooded with downstream requests from certain distributors around their release times, it is not something we are always able to keep to perfectly.

At the same time, we normally maintain master in a stable state, and new work is carried out on branches before merging, so we are usually able to issue point releases for downstreams that have particular requirements or schedules - we have done this for Fedora, Ubuntu, Debian and others in the past (including layered Red Hat products), so if there is a problem with our schedules for the pieces you work on, the best solution is to talk to us and give us enough time to find an answer.

pmoravec · 2019-01-09T08:07:12Z

What is missing to collect from files (add_files command in f-d) - other missing stuff will be identified later on.

@TurboTurtle if you wish I can incorporate the changes (as I shall know where/what/why to collect more, knowing both sos/f-d/Satellite/foreman/..)

@lzap: please reply to questions with "LZAP:" if sosreport collects sufficient data (less than what f-d does but still enough).

@TurboTurtle and @bmr-cymru : two questions:

does it make sense to collect /etc/{sysconfig,default}/* files explicitly in individual plugins, when system plugin collects them automatically? It makes sense when --onlyplugin option is used; worth to explicitly collect several such files for foreman-debugas well?
does it make sense to collect whole /etc/profile.d? IMHO worth to do to know various customizations but unsure what all secret info might be there..?

Here is the (bigger) list of files that f-d collects but this PR does not:

/var/log/foreman/dynflow_executor*.log*
/var/log/foreman/dynflow_executor*.output*
/var/log/foreman/apipie_cache*.log*
/var/log/foreman/cron*.log*
/var/log/foreman/db_migrate*.log*
/var/log/foreman/db_seed*.log*
/var/log/foreman/production.log-*
/var/log/foreman/production.log
/var/log/foreman/dynflow_executor.output*
  (foreman needs all individual file patterns to collect, not only /var/log/foreman that would skip most of those types and collect e.g. just 2 newest (and big) production.log*)
/var/log/audit/audit.log*
  (logrotated, sos collects audit.log only)
/var/log/syslog*
  (logrotated, sos collects syslog and syslog.1 only)
/usr/share/foreman/Gemfile*
  (other than Gemfile.in; LZAP: can be there a different file?)
/etc/libvirt*
  (LZAP: something other than sos/plugins/libvirt.py collects?)
/etc/libvirt/storage/*
  (LZAP: something other than /etc/libvirt/storage/*.xml ?)
/etc/libvirt/qemu/
  (LZAP: something other than /etc/libvirt/qemu/*.xml ?)
/etc/libvirt/qemu/networks
  (LZAP: something other than /etc/libvirt/qemu/networks/*.xml ?)
/var/lib/puppet/ssl/certs/$(hostname -f).pem
/var/lib/puppet/ssl/certs/ca.pem
/etc/httpd/conf/*
  (LZAP: something else than httpd.conf ?)
/etc/httpd/conf.d/*
  (LZAP: something else than *.conf ?)
/etc/httpd/conf.d/*/*
/etc/apache2/*
  (LZAP: collecting that just on debian/ubuntu is ok, I expect?)
/var/log/{httpd,apache2}/*error_log*
  (other than just error_log)
/var/log/{httpd,apache2}/foreman-ssl_access_ssl.log*
/var/lib/pgsql/data/pg_log/*
  (sosreport collects "just" *log files, so this should be covered, right LZAP: ?)
several /etc/{sysconfig,default}/* files
  (collected by system plugin, but many other plugins collect equivalent stuff directly)
/var/log/foreman-proxy/cron*.log*
/var/log/foreman-proxy/migrate_settings*.log*
/var/log/foreman-proxy/proxy*.log*
/var/log/foreman-proxy/smart_proxy_dynflow_core*.log*
  (as for foreman logs, we need individual types of logs to collect separately, smart_proxy_dynflow_core*.log* properly collected)
/var/lib/dhcp/*.leases
/var/log/named*.log
/var/named
  (LZAP: (when) is this really required? can't it be big? or can't it contain some secure data?)
/var/log/puppet/*.log*
  (other than *.log)
/etc/puppet/ssl/ca/inventory.txt
/var/lib/puppet/ssl/ca/inventory.txt
/etc/puppetlabs/puppet/ssl/certs/$(hostname-f).pem
/etc/puppetlabs/puppet/ssl/certs/ca.pem
/etc/puppetlabs/puppet/*.conf
/etc/puppetlabs/puppetserver/conf.d/*.conf
/etc/puppetlabs/puppet/rack/*
/etc/puppetlabs/puppet/manifests/*
/var/log/puppetlabs/puppetserver/*.log*
/etc/puppetlabs/puppet/ssl/ca/inventory.txt
/var/lib/puppetlabs/puppet/ssl/ca/inventory.txt
/var/log/httpd/katello-reverse-proxy_access_ssl.log*
/var/log/httpd/katello-reverse-proxy_error_ssl.log*
/var/log/candlepin/audit*.log*
/var/log/candlepin/candlepin*.log*
/var/log/candlepin/cpdb*.log*
/var/log/candlepin/cpinit*.log*
/var/log/candlepin/error*.log*
  (again, we need individual types of logs collected separately)
/var/log/tomcat*/catalina*.log*
/var/log/tomcat*/host-manager*.log*
/var/log/tomcat*/localhost*.log*
/var/log/tomcat*/manager*.log*
  (also worth to split, catalina*.log* can be too big to not collecting others)
/var/log/rhsm/*
  (other than rhsm.log and rhsmcertd.log ? worth to take at least logrotated files but imho no other file patterns can appear there, am I right LZAP: ?)
/var/log/httpd/pulp-http{s,}_access_ssl.log*
/var/log/httpd/pulp-http{s,}_error_ssl.log*
  (also required to split current /var/log/httpd/pulp-http*.log* )
/var/log/mongodb/*
  (other than mongodb.log - imho mongodb.log* is sufficient (appears on older mongo versions upgraded to Sat6.4 equivalent one)
/var/lib/mongodb/mongodb.log*
  (appears since Sat6.4)
/var/log/qdrouterd/qdrouterd.log
  (LZAP: is this required? that was relevant in past only, no? qdrouterd logs to syslog now (again)?)
/var/log/ipa*-install.log
  (LZAP: something else than /var/log/ipaclient-install.log ?)
/var/log/squid/access*.log*
/var/log/squid/cache*.log*
/var/log/squid/store*.log*
/var/log/squid/squid*.out*
  (other than access.log, cache.log, squid.out)
/etc/profile.d/*
  (LZAP: why is that required? all files or just some foreman/katello/puppet/pulp/.. relevant? but IMHO this is worth to be collected by sos every time..)

lzap · 2019-01-09T09:00:33Z

Before going one by one, I want to point out one important thing - size vs how many files (and logrotated files) should we collect. This has been big issue for foreman-debug, we ended up in neverending discussions. Some people had strong opinion of not collecting too much of historical data because this can easily lead to too big support tarballs (> 500 MB), others argued that then the report is useless etc. I think sosreport guys probably have both opinion on that and also solutions (via some parameters etc). So we should stick what's default for sosreport. But customers should be definitely able to specify more than the default (I see there is --all-logs option).

something other than sos/plugins/libvirt.py collects?