-
Notifications
You must be signed in to change notification settings - Fork 193
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: Update device settings to drive better performance (#843)
* feat: Update device settings to drive better performance In this PR: 1. Add ability for customer to enable perf optimization in storageclass. 2. Ability to select perf profile in storageclass. 3. Added scripts to run perf tests against the driver. 4. Added scripts to get sku\latency for azure disk. 5. Added unit tests. * Address review comments * Address review comments and fix unit tests * fix unit tests * Increase test coverage * Add e2e test for pv perf optimization * Fix bugs in perf optimization * fix tests * fix e2e tests * fiz e2e test * fix e2e test * address review comments * regenerate chart index * fix chart index * delete v1.2 index * fix v2 unit test
- Loading branch information
1 parent
a5907be
commit 1af2e64
Showing
41 changed files
with
2,953 additions
and
73 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
89 changes: 89 additions & 0 deletions
89
docs/enhancements/feat-add-ability-to-tune-azuredisk-performance-parameters.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| # Add ability to tune azuredisk performance parameters | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| <!-- toc --> | ||
| - [Summary](#summary) | ||
| - [Motivation](#motivation) | ||
| - [Proposal](#proposal) | ||
| - [Perf Profile](#perf-profile) | ||
| - [Scope Of the Change](#scope-of-the-change) | ||
| - [Limitations](#limitations) | ||
| - [Future Considerations](#future-considerations) | ||
| <!-- /toc --> | ||
|
|
||
| ## Summary | ||
|
|
||
| Persistent volumes in kubernetes are used for wide variety of stateful workloads. | ||
| These workloads have different runtime/IO characterstics, certain device level config settings | ||
| on the data disk can make a huge difference in performance of the application. | ||
|
|
||
| Azure storage publishes [guidelines](https://docs.microsoft.com/en-us/azure/virtual-machines/premium-storage-performance) | ||
| for the applications to configure the disks' guest OS settings to drive maximum IOPS and Bandwidth. | ||
|
|
||
| Azure Disk CSI driver users currently do not have an easy way to tune disk device configuration to | ||
| get better performance out of them for their workloads. | ||
|
|
||
| This proposal proposes to provide users with ability to enable automatic perf optimization of data | ||
| disks by tweaking guest OS level device/disk/driver parameters. | ||
|
|
||
| ## Motivation | ||
|
|
||
| As the adoption of the Azure Disk CSI driver increases, we will encounter different type | ||
| of workloads which have different disk IO chracterstics. Some may require to drive the data | ||
| disks to maximum IOPS, while others may need to do larger size writes and drive maximum throughput. | ||
|
|
||
| Azure Disk CSI driver should evolve to let users tune the data disks configurations, to get optimal | ||
| performance for their workloads. | ||
|
|
||
| With this feature, we try to provide a configurable and handsfree way for the users to enable | ||
| perf optimizations of data disk by enabling a feature in storageclass and be able to select from | ||
| multiple optimization profiles based on their requirement. | ||
|
|
||
| We intend to provide a way for the users to opt-in for the new behavior and expect not to break | ||
| any existing applications/configurations. | ||
|
|
||
| ## Proposal | ||
|
|
||
| Azure Disk CSI driver will read one extra parameter from the storageclass, `perfProfile`. | ||
|
|
||
| - *perfprofile*: Users will be able to select a certain performance profile for their device to match their | ||
| workload needs. In the beginning we will expose `Basic` profile which sets the disks for balanaced IO and throughput | ||
| workloads. If users want to keep this feature disabled they can skip adding `perfProfile` parameter in the storage class | ||
| or set `perfProfile` to `None`. If `perfProfile` is set to an unknown value, it will result in CreateVolume failure. | ||
| Please read the limitations to fully understand the options available today. | ||
|
|
||
| ### Perf Profile | ||
|
|
||
| ```yaml | ||
| apiVersion: storage.k8s.io/v1 | ||
| kind: StorageClass | ||
| metadata: | ||
| name: sc-kubestone-perf-optimized-premium-ssd-csi | ||
| provisioner: disk.csi.azure.com | ||
| parameters: | ||
| skuName: Premium_LRS | ||
| perfProfile: Basic # available values: None(by default), Basic. These are case insensitive. | ||
| reclaimPolicy: Delete | ||
| volumeBindingMode: Immediate | ||
| allowVolumeExpansion: true | ||
| ``` | ||
| ### Scope Of the Change | ||
| In this iteration we will only enable the feature for StandardSSD and Premium disks. | ||
| ## Limitations | ||
| - This feature is not supported for HDD or UltraDisk right now. | ||
| - The current implementation only optimizes the disks which use the storVsc linux disk driver. | ||
| - Only `Basic` `perfProfile` is available today, which would provide balanced IO and throughput performance. | ||
|
|
||
| ## Future Considerations | ||
|
|
||
| - We will consider exposing more perf profiles, tailor made for different IO characterstics. | ||
| - We will consider allowing users to create their own perf profile and express the workloads characterstics for | ||
| the storage class using CRD or some other configuration. | ||
| - We will consider expanding perf optimization for HDD and UltraDisks. | ||
| - We will consider expanding perf optimization for other disk drivers such as NVME etc. | ||
| - We will consider expanding perf optimization for other OS' such as Windows. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,108 @@ | ||
| #!/usr/bin/env python | ||
|
|
||
| # Copyright 2015 The Kubernetes Authors. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| import argparse | ||
| import json | ||
| import math | ||
| import os.path | ||
| import shlex | ||
| import subprocess | ||
| import requests | ||
| import sys | ||
| import logging | ||
|
|
||
| logger = logging.getLogger() | ||
|
|
||
| def measure_latency(directory, iops, request_size, rw, name): | ||
| # Through experimentation, 1000 was the fewest number of IOs with a consistent average | ||
| number_ios = 1000 | ||
| # Rate limit the fio command to non-burst IOPs to avoid inaccurate latency measurement due to throttling | ||
| fio_command = [ | ||
| '/usr/bin/fio', '--direct=1', '--ioengine=libaio', '--iodepth=1', | ||
| '--time_based=0', f'--name={name}', f'--size=10MB', | ||
| '--output-format=json', '--overwrite=1', f'--rw={rw}', | ||
| f'--bs={request_size}k', f'--number_ios={number_ios}', | ||
| f'--rate_iops={iops}', f'--directory={directory}' | ||
| ] | ||
| # For usability, print an approximate expected runtime | ||
| expected_runtime = 2 | ||
|
|
||
| logger.info('Running: %s.\nThis will take approximately %d seconds.' % (' '.join(shlex.quote(i) for i in fio_command), expected_runtime)) | ||
| return json.loads(subprocess.check_output(fio_command)) | ||
|
|
||
| def get_latency(directory, iops, request_size, random): | ||
| write = ('rand' if random else '') + 'write' | ||
| read = ('rand' if random else '') + 'read' | ||
|
|
||
| jobname = 'test' | ||
| # The fio filename_format is, by default, $jobname.$jobnum.$filenum (when | ||
| # no other format specifier is given). This script only runs a single job | ||
| # with a single file, so the suffix will always be 0.0 | ||
| job_filename = jobname + '.0.0' | ||
| try: | ||
| os.remove(job_filename) | ||
| except FileNotFoundError: | ||
| pass | ||
|
|
||
| parsed = measure_latency(directory, iops, request_size, write, jobname) | ||
| write_clat_mean = parsed['jobs'][0]['write']['clat_ns']['mean'] | ||
|
|
||
| parsed = measure_latency(directory, iops, request_size, read, jobname) | ||
| read_clat_mean = parsed['jobs'][0]['read']['clat_ns']['mean'] | ||
|
|
||
| winner = max(write_clat_mean, read_clat_mean) | ||
|
|
||
| # convert to seconds | ||
| return winner / 1_000_000_000 | ||
|
|
||
| parser = argparse.ArgumentParser(description=""" | ||
| Calculate the latencies azure disk is seeing for read\writes. | ||
| """.strip()) | ||
|
|
||
| parser.add_argument('--maxIops', | ||
| type=int, | ||
| required=True, | ||
| default=1, | ||
| help='Max IOs possible') | ||
|
|
||
| parser.add_argument('--ioSize', | ||
| required=True, | ||
| help='Size of the IOs for calculating latency.') | ||
|
|
||
| parser.add_argument('--directory', | ||
| required=True, | ||
| help='Directory at which read/writes need to be made.') | ||
|
|
||
| args = parser.parse_args() | ||
|
|
||
| try: | ||
| handler = logging.StreamHandler() | ||
| logger.addHandler(handler) | ||
| if not sys.stderr.isatty(): | ||
| handler.setFormatter(logging.Formatter('%(asctime)s %(message)s')) | ||
| logger.setLevel(logging.DEBUG) | ||
|
|
||
|
|
||
| # completion latency of a single direct IO request of RS_min_seq_IO | ||
| latency_max_BW = get_latency(args.directory, args.maxIops, args.ioSize, False) | ||
| logger.info(f'Measured {latency_max_BW}s latency for a sequential request of size {args.ioSize}kB for disk at LUN {args.directory}.') | ||
|
|
||
| # completion latency of a single direct IO request of the smallest expected size | ||
| latency_base_rand = get_latency(args.directory, args.maxIops, 8, True) | ||
| logger.info(f'Measured {latency_base_rand}s latency for a random request of size 8kB for disk at LUN {args.directory}.') | ||
|
|
||
| except Exception as e: | ||
| logger.critical('An exception was encountered while calculating the desired block device settings. As such, no settings can be recommended', exc_info=e) |
Oops, something went wrong.