Call ResetFailedUnit when cleaning up failed services

[agrare]

If a systemd service has failed it stays around until you call systemd reset-failed unit-name otherwise it stays around as:

# systemctl status
● vmware_infra_manager_event_catcher@8f2f33a3-d1b1-4603-a8aa-44a7156347d9.service   loaded failed failed    vmware_infra_manager_event_catcher@8f2f33a3-d1b1-4603-a8aa-44a7156347d9.service

# systemctl status vmware_infra_manager_event_catcher@8f2f33a3-d1b1-4603-a8aa-44a7156347d9.service
Warning: The unit file, source configuration file or drop-ins of vmware_infra_manager_event_catcher@8f2f33a3-d1b1-4603-a8aa-44a7156347d9.service changed on disk. Run 'systemctl daemon-reload' to reload units.
● vmware_infra_manager_event_catcher@8f2f33a3-d1b1-4603-a8aa-44a7156347d9.service
   Loaded: loaded (/etc/systemd/system/[email protected]; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/vmware_infra_manager_event_catcher@8f2f33a3-d1b1-4603-a8aa-44a7156347d9.service.d
           └─override.conf
   Active: inactive (dead) since Thu 2020-11-12 11:26:12 EST; 1h 25min ago
  Process: 178314 ExecStart=/bin/bash -lc exec ruby lib/workers/bin/run_single_worker.rb ManageIQ::Providers::Vmware::InfraManager::EventCatcher --heartbeat --guid=8f2f33a3-d1b1-4603-a8aa-44a7156347d9 (code=exi>
 Main PID: 178314 (code=exited, status=1/FAILURE)

[agrare]

WIP because I want to confirm that this order is correct, (stop, reset, disable)

[agrare]

Okay taking out of WIP, tested this live on an appliance by kill -9'ing a worker and the failed systemd service is properly cleaned up

[jrafanie]

Does it stay around in the miq_workers table as a running worker while it's failed in systemd?

[agrare]

No it is not in the workers table (checked that on the appliance that I pulled that systemd status from)

[jrafanie]

I know this is still wip, but the travis failure looks relevant for this change:

  1) MiqServer::WorkerManagement::Monitor::Systemd#cleanup_failed_systemd_services with failed services calls DisableUnitFiles with the service name
     Failure/Error: systemd_manager.ResetFailedUnit(service_name)
       #<Double "DBus::Systemd::Manager"> received unexpected message :ResetFailedUnit with ("[email protected]")
     # ./app/models/miq_server/worker_management/monitor/systemd.rb:32:in `block in systemd_stop_services'
     # ./app/models/miq_server/worker_management/monitor/systemd.rb:30:in `each'
     # ./app/models/miq_server/worker_management/monitor/systemd.rb:30:in `systemd_stop_services'
     # ./app/models/miq_server/worker_management/monitor/systemd.rb:9:in `cleanup_failed_systemd_services'
     # ./spec/models/miq_server/worker_management/monitor/systemd_spec.rb:28:in `block (4 levels) in <top (required)>'

[miq-bot]

Checked commit agrare@1678e35 with ruby 2.6.3, rubocop 0.82.0, haml-lint 0.35.0, and yamllint
2 files checked, 0 offenses detected
Everything looks fine. 🍪

[agrare]

@jrafanie this is ready to go, please take a look

[simaishi]

Kasparov backport details:

$ git log -1
commit 2be0d2302b7873fbb0e6f67a6d5a708a3d73f073
Author: Joe Rafaniello <[email protected]>
Date:   Thu Nov 19 10:23:12 2020 -0500

    Merge pull request #20810 from agrare/reset_failed_systemd_unit_files

    Call ResetFailedUnit when cleaning up failed services

    (cherry picked from commit 0a5655118d8b2817400406b44d697292c47b0893)