fix: Restart=always docker systemd service

[jackfrancis]

Reason for Change:

This PR ensures that Restart=always is configured for the docker systemd service, and that the included monitor script for both docker and kubelet has a fail-safe "start the systemd service" if the systemd job spec doesn't do so itself.

Issue Fixed:

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version

Notes:

[jackfrancis]

@xuto2 @yangl900 FYI, this pathology may exist for AKS node systemd definitions as well

[codecov]

Codecov Report

Merging #3758 into master will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3758   +/-   ##
=======================================
  Coverage   73.19%   73.20%           
=======================================
  Files         148      148           
  Lines       25367    25372    +5     
=======================================
+ Hits        18568    18573    +5     
  Misses       5663     5663           
  Partials     1136     1136           

Impacted Files	Coverage Δ
pkg/engine/templates_generated.go	53.42% <ø> (ø)
pkg/api/addons.go	97.80% <0.00%> (+0.01%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b05d7a8...f7ce667. Read the comment docs.

[devigned]

lgtm

[xuto2]

/lgtm.

Thanks @jackfrancis for sharing!

[mboersma]

/lgtm

[acs-bot]

New changes are detected. LGTM label has been removed.

[acs-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jackfrancis, mboersma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis,mboersma]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jackfrancis]

After more investigation, it appears that Restart=always is already present for the docker systemd service:

$ grep Restart /lib/systemd/system/docker.service
RestartSec=2
Restart=always

(Though not via aks-engine-delivered config).

So, we should still not just rely upon that being there, and explicitly declare that config (like we're doing in this PR). However, the fact that we're still seeing terminally stopped docker systemd services in the wild suggests that we aren't as resilient as we want to in terms of guaranteeing the availability of the docker systemd service. These docs suggest that Restart=always in fact has limits:

https://www.freedesktop.org/software/systemd/man/systemd.service.html#Restart=

The key statement is:

As exceptions to the setting above, the service will not be restarted if the exit code or signal is specified in RestartPreventExitStatus= (see below) or the service is stopped with systemctl stop or an equivalent operation.

We aren't specifying any non-qualifying exit codes for docker:

$ sudo grep -R "RestartPreventExitStatus" /etc/systemd/
/etc/systemd/system/sshd.service:RestartPreventExitStatus=255
/etc/systemd/system/multi-user.target.wants/snapd.service:RestartPreventExitStatus=42
/etc/systemd/system/multi-user.target.wants/ssh.service:RestartPreventExitStatus=255

So there must be some systemctl stop-equivalent events that occasionally happen. In order to restart docker after that, I've added systemctl start logic to the script that the pre-existing docker-monitor service uses. I think with this combination, we'll be in better shape to deal with docker systemd service events.