-
Notifications
You must be signed in to change notification settings - Fork 129
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
sync main branch with feature/dmlib (#4075)
* Update VCPKG Baseline due to zlib (#4027) * override zlib version * shwitch baseline * macro update * override zlib * Fixed CURL build breaks discovered by newer openssl baselines (#4034) * Vcpkg baseline to latest (#4036) * vcpkg baseline * f4b262b259145adb2ab0116a390b08642489d32b * 79fa9d85cb4510b05c2657bd7ec8a6a2cb266dc5 * 6ca56aeb457f033d344a7106cb3f9f1abf8f4e98 * override * 1.4.1` * Removed version override for opentelemetry (#4037) * Removed version override for opentelemetry * Disable deprecation warnings for open telemety tests to guard against future deprecations * Added explanation for 4996 error; removed InMemoryExporter from documentation * Sync eng/common directory with azure-sdk-tools for PR 3790 (#4041) * Added yaml support for job matrix creation * autogen scenario matrix for stress test * Temporary Working State * update to default sparse * pr comments and some error handling * custom matrixfilename and ordering of generatedValues.yaml * common module import * JobMatrix write host Co-authored-by: Albert Cheng <[email protected]> * Storageblob (#4043) * first take * grrrrr * qwqwq * ewqw * eewew * put back * pipeline * Set read-only VCPKG_BINARY_SOURCES_SECRET * Pass InstallLanguageSteps through the perf template * blop pipeline * path * ';l'l; * add support for sync param * --sync flag support * typo * sync * sync to y * for PR * clang Co-authored-by: Daniel Jurek <[email protected]> * Upgrade cspell packages to version `^6.12.0` (#4044) to address dependabot issue https://github.com/Azure/azure-sdk-for-js/security/dependabot/13 Co-authored-by: Jeremy Meng <[email protected]> * Sync eng/common directory with azure-sdk-tools for PR 4480 (#4046) * Add extra parameters for .net readme path * Use another way for default value * Remove custom source Co-authored-by: sima-zhu <[email protected]> * Converted WinHTTP to Async. (#4015) * Async WinHTTP * Added cancellation test; added support for request cancellation. Co-authored-by: Anton Kolesnyk <[email protected]> Co-authored-by: Ahson Khan <[email protected]> Co-authored-by: Rick Winter <[email protected]> * Updated OpenTelemetry tests to use their own `SpanExporter` rather than depend on the `InMemorySpanExporter`. (#4042) * Removed version override for opentelemetry * Disable deprecation warnings for open telemety tests to guard against future deprecations * Added explanation for 4996 error; removed InMemoryExporter from documentation * Checkpoint adding manual telemetry exporter * Switch from using OpenTelemetry's InMemoryExporter (which is an internal-to-opentelemetry test hook) to our own SpanExporter * removed unused variables because gcc doesn't like them * Fixed doxygen comments * Removed a bunch of unnecessary headers * Update sdk/core/azure-core-tracing-opentelemetry/test/ut/test_exporter.hpp Co-authored-by: Rick Winter <[email protected]> Co-authored-by: Rick Winter <[email protected]> * [Perf] Include GA versions of core package (#4053) * Fix to for a SAL annotation (#4054) * In-memory Identity token cache (#4024) * identicalize the comments type (#4063) * Improve scenario matrix edge case handling (#4065) Co-authored-by: Ben Broderick Phillips <[email protected]> * docV1 (#4051) * docV1 * Update doc/PerformanceTesting.md Co-authored-by: Larry Osterman <[email protected]> * Update doc/PerformanceTesting.md Co-authored-by: Larry Osterman <[email protected]> * Update doc/PerformanceTesting.md Co-authored-by: Larry Osterman <[email protected]> * Update doc/PerformanceTesting.md Co-authored-by: Larry Osterman <[email protected]> * comment * some more updates * Update doc/PerformanceTesting.md Co-authored-by: Larry Osterman <[email protected]> * Update doc/PerformanceTesting.md Co-authored-by: Larry Osterman <[email protected]> * Update doc/PerformanceTesting.md Co-authored-by: Larry Osterman <[email protected]> * PR * Update doc/PerformanceTesting.md Co-authored-by: Larry Osterman <[email protected]> Co-authored-by: Larry Osterman <[email protected]> * bump version to newest (#4066) Co-authored-by: scbedd <[email protected]> * Sync eng/common directory with azure-sdk-tools for PR 4543 (#4071) * stress test addons version check * cleanup Co-authored-by: Albert Cheng <[email protected]> * Added the ability to select OpenSSL 1.1.1n if desired. (#4045) * Added ability to switch to OpenSSL 1.1.1n * Fixed OpenSSL compilation challenges - the CRL stuff works now * Added instructions on using OpenSSL 1.1.1 to README * Added vcpkg args to handle openssl variants * Dump out correlation id without verbose logging for resource deployment (#4069) Co-authored-by: Ben Broderick Phillips <[email protected]> * fix typo in PerformanceTesting.md (#4076) Co-authored-by: George Arama <[email protected]> Co-authored-by: Larry Osterman <[email protected]> Co-authored-by: Azure SDK Bot <[email protected]> Co-authored-by: Albert Cheng <[email protected]> Co-authored-by: Daniel Jurek <[email protected]> Co-authored-by: Jeremy Meng <[email protected]> Co-authored-by: sima-zhu <[email protected]> Co-authored-by: Anton Kolesnyk <[email protected]> Co-authored-by: Ahson Khan <[email protected]> Co-authored-by: Rick Winter <[email protected]> Co-authored-by: Mike Harder <[email protected]> Co-authored-by: Ben Broderick Phillips <[email protected]> Co-authored-by: scbedd <[email protected]>
- Loading branch information
1 parent
585869a
commit dcf62d2
Showing
97 changed files
with
5,147 additions
and
1,306 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
macro(SetPerfDeps PACKAGE VAR_RESULT) | ||
string(TOUPPER ${PACKAGE} SUFFIX) | ||
string(CONCAT VAR_TRIGGER "VCPKG-" ${SUFFIX}) | ||
message(STATUS "trigger name ${VAR_TRIGGER}") | ||
if(DEFINED ENV{${VAR_TRIGGER}}) | ||
find_package(${PACKAGE} $ENV{${VAR_TRIGGER}} EXACT) | ||
add_compile_definitions(${VAR_RESULT}="$ENV{${VAR_TRIGGER}}") | ||
else() | ||
add_compile_definitions(${VAR_RESULT}="source") | ||
endif() | ||
endmacro() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,171 @@ | ||
# Performance Testing | ||
|
||
## Writing a test | ||
|
||
Adding tests , at the moment relies in making modifications in a couple of repositories. | ||
|
||
## Azure-SDK-For-Cpp repo | ||
|
||
E.G. https://github.com/Azure/azure-sdk-for-cpp/tree/main/sdk/storage/azure-storage-blobs | ||
|
||
### Location | ||
|
||
Performance tests are located under **"test/perf"** folder under the service folder(e.g. alongside ut folder). | ||
|
||
The perf test folder should be added in the parent CMakeLists.txt guarded by the **BUILD_PERFORMANCE_TESTS** flag. | ||
|
||
E.G. | ||
|
||
```markdown | ||
|
||
if(BUILD_PERFORMANCE_TESTS) | ||
add_subdirectory(test/perf) | ||
endif() | ||
|
||
``` | ||
|
||
### Structure | ||
|
||
As any other CPP project you will need a **CMakeLists.txt** file , along side **src** and **inc** folders | ||
|
||
#### Contents of the **inc** directory | ||
|
||
Under inc folder create a subfolder structure following your namespace hierarchy ending with another test folder since these are in the ...Test namespace. | ||
|
||
E.G. test/perf/inc/azure/storage/blobs/test | ||
|
||
The tests are defined in .hpp files under this folder. | ||
|
||
##### Test definition | ||
|
||
In a .hpp file we define a class that will be contain the tests methods. The class is defined in the ::Test namespace for your service. | ||
|
||
The class will inherit from the **PerfTest** base class and will override several methods as follows: | ||
- Constructor(Azure::Perf::TestOptions options) : PerfTest(options) | ||
Options field are passed from the perf test framework and constain the various options defined for running the test. | ||
- void Run(Azure::Core::Context const&) override {...}. | ||
Runs the actual test code. It is strongly recommended that the test code does as little extra work here as possible, it should consist solely of the actual test invocation. The test code should remove all assert, conditional statements ("if"/"else") or any other unnecessary work as any extra work will skew the results. | ||
- std::vector<Azure::Perf::TestOption> GetTestOptions() override | ||
Defines the various parameters for the test run that can be passed to the test from the performance framework. The perf framework uses these params to run various combinations(e.g. blob size) | ||
- static Azure::Perf::TestMetadata GetTestMetadata() | ||
Returns TestMetadate object used to identify the test. | ||
|
||
#### Contents of the **src** directory | ||
|
||
Contains one cpp file that contains the main method defintion | ||
|
||
```cpp | ||
int main(int argc, char** argv) | ||
{ | ||
std::cout << "SERVICE VERSION " << VCPKG_SERVICE_VERSION << std::endl; | ||
// Create the test list | ||
std::vector<Azure::Perf::TestMetadata> tests{ | ||
Service::Namespace::Test::TestName::GetTestMetadata(), | ||
}; | ||
|
||
Azure::Perf::Program::Run(Azure::Core::Context::ApplicationContext, tests, argc, argv); | ||
|
||
return 0; | ||
} | ||
``` | ||
#### CMakeLists.txt | ||
Beyond the regular cmake defintion in your cmake file make sure to add | ||
```makefile | ||
include(AzureVcpkg) | ||
az_vcpkg_integrate() | ||
... | ||
include(PerfTest) | ||
SETPERFDEPS(azure-storage-blobs-cpp VCPKG_SERVICE_VERSION) | ||
``` | ||
|
||
The crucial part here is the SETPERFDEPS cmake macro. | ||
The perf framework will set an environment variable based on the service name with the value representing a version, thus allowing to run the tests against diffrent VCPKG published versions. If the env is not defined then the test will build against the current source code of the service. | ||
There can be multiple set perf for each dependency of the service ( e.g. identity, storage). | ||
|
||
## Pipeline definition | ||
The file should be named `perf.yml`, and should be located in the service directory that is to be tested, for consistency and to keep service related resources in the service folder. | ||
|
||
```yml | ||
parameters: | ||
- name: PackageVersions | ||
displayName: PackageVersions (regex of package versions to run) | ||
type: string | ||
default: '12|source' | ||
- name: Tests | ||
displayName: Tests (regex of tests to run) | ||
type: string | ||
default: '^(download|upload|list-blobs)$' | ||
- name: Arguments | ||
displayName: Arguments (regex of arguments to run) | ||
type: string | ||
default: '(10240)|(10485760)|(1073741824)|(5 )|(500 )|(50000 )' | ||
- name: Iterations | ||
displayName: Iterations (times to run each test) | ||
type: number | ||
default: '5' | ||
- name: AdditionalArguments | ||
displayName: AdditionalArguments (passed to PerfAutomation) | ||
type: string | ||
default: ' ' | ||
|
||
extends: | ||
template: /eng/pipelines/templates/jobs/perf.yml | ||
parameters: | ||
ServiceDirectory: service folder | ||
Services: "^service name$" | ||
PackageVersions: ${{ parameters.PackageVersions }} | ||
Tests: ${{ parameters.Tests }} | ||
Arguments: ${{ parameters.Arguments }} | ||
Iterations: ${{ parameters.Iterations }} | ||
AdditionalArguments: ${{ parameters.AdditionalArguments }} | ||
InstallLanguageSteps: | ||
- pwsh: | | ||
Write-Host "##vso[task.setvariable variable=VCPKG_BINARY_SOURCES_SECRET;issecret=true;]clear;x-azblob,https://cppvcpkgcache.blob.core.windows.net/public-vcpkg-container,,read" | ||
displayName: Set Vcpkg Variables | ||
EnvVars: | ||
# This is set in the InstallLanguageSteps | ||
VCPKG_BINARY_SOURCES_SECRET: $(VCPKG_BINARY_SOURCES_SECRET) | ||
|
||
``` | ||
|
||
The fields of interest here are the parameters: | ||
- Tests which define which test s run in this pipeline ( which tests are available) | ||
- Arguments which allow to filter which subset of parameters are acceptable for the tests from the plurality in the framework | ||
- ServiceDirectory which represents the root folder of the service | ||
- Services which represents the services which these tests target. | ||
|
||
## Resources | ||
|
||
If the tests require certain resources to exist beforehand the pipeline can deploy them based on the presence of the **perf-resources.bicep** file which will be used to deploy the required resources defined within. | ||
|
||
## Azure-SDK-Tools repo | ||
|
||
## Location | ||
|
||
The performance automation is located under azure-sdk-tools\tools\perf-automation\Azure.Sdk.Tools.PerfAutomation folder. | ||
(https://github.com/Azure/azure-sdk-tools/tree/main/tools/perf-automation/Azure.Sdk.Tools.PerfAutomation) | ||
|
||
## Test definition | ||
|
||
in the above mentioned folder resided the test defintion file "tests.yml". | ||
|
||
### The tests exists in other languages | ||
|
||
If the test exists in other languages then making the CPP version visible to the framework requires adding an entry with the name **CPP** under **Service**/**Languages** followed by the packages and versions available for the testing(this ties into the CMakeLists SetPerfDeps macro). | ||
|
||
Next under the **Tests**/**Test** node with the desired name add the CPP test name. In this section mind the aArguments list , this ies with the regex in the cpp sdk pipeline.yml definition. | ||
|
||
### The test does not exist in other languages | ||
|
||
In this case you are the first to add the required nodes. The defintion is fairly simple and straightforward. | ||
|
||
## Pipeline | ||
|
||
Once you have everything in place create a pipeline using the definition in your in the cpp repo by going to https://dev.azure.com/azure-sdk/internal/_build?definitionScope=%5Cperf and create a new one under the cpp node. | ||
|
||
To test intermediate definitions of your pipeline you can run the https://dev.azure.com/azure-sdk/internal/_build?definitionId=5121 pipline and set the proper values for the cpp node( make sure to deselect all other languages except cpp unless you want to run them). | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.