feat: configurable disk caching

[jackfrancis]

Reason for Change:

This PR introduces configurable disk caching:

For OS disk configuration on master VMs:

• either "ReadOnly", "ReadWrite", or "None"

For OS disk configuration on agent pool VMs:

• either "ReadOnly", "ReadWrite", or "None"
  For data disk configuration on agent pool VMs:
• either "ReadOnly", "ReadWrite", or "None"
  • supports a single configuration option across all data disks

An example api model w/ this new configuration:

{
...
        "masterProfile": {
            ...
            "osDiskCachingType": "ReadOnly",
            ...
        },
        "agentPoolProfiles": [
            {
            ...
            "osDiskCachingType": "ReadWrite",
            "dataDiskCachingType": "None",
            ...
            }
        ]
...
}

Issue Fixed:

Fixes #2249

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version

Notes:

[jackfrancis]

@PacoGuy How does the api model spec in the PR description look in terms of what you'd want in a configurable disk caching configuration?

And @mboersma FYI on intent, and @marosset to make sure we ship this w/ Windows 1st class support as well.

[codecov]

Codecov Report

Merging #2863 into master will increase coverage by 0.09%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2863      +/-   ##
==========================================
+ Coverage   71.13%   71.22%   +0.09%     
==========================================
  Files         147      147              
  Lines       25670    25738      +68     
==========================================
+ Hits        18261    18333      +72     
+ Misses       6272     6268       -4     
  Partials     1137     1137              

Impacted Files	Coverage Δ
pkg/api/types.go	94.37% <ø> (ø)
pkg/api/vlabs/types.go	73.30% <ø> (ø)
pkg/api/converterfromapi.go	93.64% <100.00%> (+0.03%)	⬆️
pkg/api/convertertoapi.go	93.15% <100.00%> (+0.03%)	⬆️
pkg/api/defaults.go	92.58% <100.00%> (+0.10%)	⬆️
pkg/api/vlabs/validate.go	80.15% <100.00%> (+0.88%)	⬆️
pkg/engine/virtualmachines.go	80.85% <100.00%> (+0.19%)	⬆️
pkg/engine/virtualmachinescalesets.go	80.93% <100.00%> (+0.14%)	⬆️
pkg/engine/engine.go	85.85% <0.00%> (+0.05%)	⬆️
pkg/engine/armvariables.go	86.17% <0.00%> (+0.08%)	⬆️
... and 4 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 414828f...89f1739. Read the comment docs.

[PacoGuy]

Looks like that PR model spec would work for us, Thanks!

[marosset]

@jackfrancis This should only add changes to the ARM correct?
If so I don't think anything specific needs to be done from windows (other than adding a test config or something to verify/validate)
LMK if you do need help tho.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jackfrancis]

@marosset FYI, adding this new configuration to a Windows pool in our E2E cluster configuration to confirm that this is not a Linux/Windows thing.

Btw, should we add a data disk to this pool to (1) test that functionality generally, and to also validate the data disk caching type override?

[marosset]

Sure.

[mboersma]

/lgtm

[mboersma]

I thought "qux" followed "baz." 😜

[jackfrancis]

There is no spoon.

[acs-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jackfrancis, mboersma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis,mboersma]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment