diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 66dbd8da66b..965467fb175 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -50,3 +50,4 @@ # .NET Client Tools ########### /src/dotnet/Azure.ClientSdk.Analyzers @jsquire @pallavit @JoshLove-msft @christothes @annelo-msft @KrzysztofCwalina @tg-msft @heaths @m-nash +/src/dotnet/APIView @chidozieononiwu diff --git a/eng/common/testproxy/dotnet-devcert.crt b/eng/common/testproxy/dotnet-devcert.crt index 07976e52ef0..254432686db 100644 --- a/eng/common/testproxy/dotnet-devcert.crt +++ b/eng/common/testproxy/dotnet-devcert.crt @@ -1,20 +1,21 @@ -----BEGIN CERTIFICATE----- -MIIDSDCCAjCgAwIBAgIUIoKu8Oao7j10TLNxaUG2Bs0FrRwwDQYJKoZIhvcNAQEL -BQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTIyMDgwNTIxMTcyM1oXDTIzMDgw -NTIxMTcyM1owFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEF -AAOCAQ8AMIIBCgKCAQEA0UPG7ER++5/9D/qa4SCtt7QvdHwcpidbwktPNU8iRW7V -pIDPWS4goLp/+7+maT0Z/mqwSO3JDtm/dtdlr3F/5EMgyUExnYcvUixZAiyFyEwj -j6wnAtNvqsg4rDqBlD17fuqTVsZm9Yo7QYub6p5PeznWYucOxRrczqFCiW4uj0Yk -GgUHPPmCvhSDKowV8CYRHfkD6R8R4SFkoP3/uejXHxeXoYJNMWq5K0GqGaOZtNFB -F7QWZHoLrRpZcY4h+DxwP3c+/FdlVcs9nstkF+EnTnwx5IRyKsaWb/pUEmYKvNDz -wi6qnRUdu+DghZuvyZZDgwoYrSZokcbKumk0MsLC3QIDAQABo4GRMIGOMA8GA1Ud +MIIDZzCCAk+gAwIBAgIUXjY6UxqL53TvxH8dtPNZm6/getIwDQYJKoZIhvcNAQEL +BQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTIzMDcyODIxMDM1MloXDTI0MDcy +NzIxMDM1MlowFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEF +AAOCAQ8AMIIBCgKCAQEAsTPe57bim6NTmBHS1ldLAr7dnMT7AMtfttfjqPzggDgR +kNKj/mi5Xd86AOR6QfLMMipdL2NpPTZP557t4V1oVgODv+M3SiKRriY01TNbL9K5 +zjYlPDik1BzKJgHiLmHPmuKsWslTTMO86nau5YNzKfaOIFbiV5uTUCLTZh3NspDo +OaIeJ4Efud/6bHQkbIXggAt2TFjVum+jMXLYeyA8ZjwFgW1ENAlOOV5Gm8eFjkIt +OhQSZLlLc9BnJkOAhT0v6Xq0oRwCm0YW42+JFzmIvjK0cU/sFmjDqzKAxhtWexz7 +WT7KDiJU+GNsZmm8KjeU0EaQpzpK8q/MEbNRX1OqrQIDAQABo4GwMIGtMA8GA1Ud EwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGmMBYGA1UdJQEB/wQMMAoGCCsGAQUF BwMBMBcGA1UdEQEB/wQNMAuCCWxvY2FsaG9zdDA6BgorBgEEAYI3VAEBBCwMKkFT -UC5ORVQgQ29yZSBIVFRQUyBkZXZlbG9wbWVudCBjZXJ0aWZpY2F0ZTANBgkqhkiG -9w0BAQsFAAOCAQEARX4NxGbycdPVuqvu/CO+/LpWrEm1OcOl7N57/mD5npTIJT78 -TYtXk1J61akumKdf5CaBgCDRcl35LhioFZIMEsiOidffAp6t493xocncFBhIYYrZ -HS6aKsZKPu8h3wOLpYu+zh7f0Hx6pkHPAfw4+knmQjDYomz/hTwuo/MuT8k6Ee7B -NGWqxUamLI8bucuf2ZfT1XOq83uWaFF5KwAuVLhpzo39/TmPyYGnaoKRYf9QjabS -LUjecMNLJFWHUSD4cKHvXJjDYZEiCiy+MdUDytWIsfw0fzAUjz9Qaz8YpZ+fXufM -MNMNfyJHSMEMFIT2D1UaQiwryXWQWJ93OiSdjA== +UC5ORVQgQ29yZSBIVFRQUyBkZXZlbG9wbWVudCBjZXJ0aWZpY2F0ZTAdBgNVHQ4E +FgQU+9BDdfBwXH38ti7IGpL4Kn7IDVgwDQYJKoZIhvcNAQELBQADggEBAKoy2dp8 +gsKk1U51164IIhOYisaJ1egOI0++STd8j94SX9XlCAgwLaPr/mU/U438xKI1inSA +Miaboqtt2tqvfP5nceSYL3FPJ3K0ADw8UFgwApKiRYpRevAIspG+OaqHwHUFRhyG +bxkUZ4w96IEpVtDOGoy12sCmChZgdVk44+y8uurSza18Vj1LfkrN6ppZLt4FII5e +p8BFKtqCRToFRJIIjMePOdTUbeRwUCjBPyYv/h5jcJUfFXQJpPXvJs4LFcUivqA3 +sAut3tut6CDzToTMtAD5ebxSPh3DDM6JOsWhrGoT0if5qoio75tG6yV40gi3Tocr +KCwDu8B2O2HlL5Q= -----END CERTIFICATE----- diff --git a/eng/common/testproxy/dotnet-devcert.pfx b/eng/common/testproxy/dotnet-devcert.pfx index a971cd950a1..d8f717ce3cb 100644 Binary files a/eng/common/testproxy/dotnet-devcert.pfx and b/eng/common/testproxy/dotnet-devcert.pfx differ diff --git a/eng/common/testproxy/onboarding/README.me b/eng/common/testproxy/onboarding/README.me index 09c2b364f0f..aeaa1377656 100644 --- a/eng/common/testproxy/onboarding/README.me +++ b/eng/common/testproxy/onboarding/README.me @@ -6,16 +6,16 @@ The azure-sdk monorepos are growing quickly due to the presence of recordings. D The script `generate-assets-json.ps1` will execute the initial migration of your recordings from within a language repo to the [assets repo](https://github.com/Azure/azure-sdk-assets) as well as creating the assets.json file for those assets. -The script is [generate-assets-json.ps1](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/testproxy/transition-scripts/generate-assets-json.ps1) +The script is [generate-assets-json.ps1](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/testproxy/onboarding/generate-assets-json.ps1) ### Download the transition script locally ```powershell -Invoke-WebRequest -OutFile "generate-assets-json.ps1" https://raw.githubusercontent.com/Azure/azure-sdk-tools/main/eng/common/testproxy/transition-scripts/generate-assets-json.ps1 +Invoke-WebRequest -OutFile "generate-assets-json.ps1" https://raw.githubusercontent.com/Azure/azure-sdk-tools/main/eng/common/testproxy/onboarding/generate-assets-json.ps1 ``` ```bash -wget https://raw.githubusercontent.com/Azure/azure-sdk-tools/main/eng/common/testproxy/transition-scripts/generate-assets-json.ps1 -o generate-assets-json.ps1 +wget https://raw.githubusercontent.com/Azure/azure-sdk-tools/main/eng/common/testproxy/onboarding/generate-assets-json.ps1 -o generate-assets-json.ps1 ``` ## Setup @@ -73,7 +73,7 @@ The location of the automatically restored assets is colloquially referred to as ## Running the script -[generate-assets-json.ps1](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/testproxy/transition-scripts/generate-assets-json.ps1) is a standalone powershell script with no supporting script requirements. The easiest way to run the script would be to use a one-liner [defined above](#download-the-transition-script-locally) to grab the file directly. **Please ensure you have the newest version of this script before continuing!** +[generate-assets-json.ps1](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/testproxy/onboarding/generate-assets-json.ps1) is a standalone powershell script with no supporting script requirements. The easiest way to run the script would be to use a one-liner [defined above](#download-the-transition-script-locally) to grab the file directly. **Please ensure you have the newest version of this script before continuing!** ```powershell # if downloading the file singly, cd to the directory containing generate-assets-json.ps1 @@ -124,7 +124,7 @@ This is necessary because the language is used in several places. 1. The AssetsRepoPrefixPath in assets.json is set to the language. 2. The TagPrefix is set to the `/` or `//` etc. -3. The language also used to determine what the [recording directories within a repository are named](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/testproxy/transition-scripts/generate-assets-json.ps1#L47). +3. The language also used to determine what the [recording directories within a repository are named](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/testproxy/onboarding/generate-assets-json.ps1#L47). ## A final note about the initial push diff --git a/eng/common/testproxy/target_version.txt b/eng/common/testproxy/target_version.txt index faaab72a41d..3ebae04c347 100644 --- a/eng/common/testproxy/target_version.txt +++ b/eng/common/testproxy/target_version.txt @@ -1 +1 @@ -1.0.0-dev.20230728.1 +1.0.0-dev.20230802.1 diff --git a/src/dotnet/APIView/APIViewUITests/APIViewUITests.csproj b/src/dotnet/APIView/APIViewUITests/APIViewUITests.csproj index 905505e1274..3c7fd939050 100644 --- a/src/dotnet/APIView/APIViewUITests/APIViewUITests.csproj +++ b/src/dotnet/APIView/APIViewUITests/APIViewUITests.csproj @@ -18,7 +18,7 @@ - + diff --git a/src/dotnet/APIView/APIViewWeb/Repositories/DevopsArtifactRepository.cs b/src/dotnet/APIView/APIViewWeb/Repositories/DevopsArtifactRepository.cs index d4397dc6e60..dacf2c41006 100644 --- a/src/dotnet/APIView/APIViewWeb/Repositories/DevopsArtifactRepository.cs +++ b/src/dotnet/APIView/APIViewWeb/Repositories/DevopsArtifactRepository.cs @@ -30,7 +30,7 @@ public class DevopsArtifactRepository : IDevopsArtifactRepository private readonly string _hostUrl; private readonly TelemetryClient _telemetryClient; - public DevopsArtifactRepository(IConfiguration configuration, IHttpContextAccessor httpContextAccessor) + public DevopsArtifactRepository(IConfiguration configuration) { _configuration = configuration; _devopsAccessToken = Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(string.Format("{0}:{1}", "", _configuration["Azure-Devops-PAT"]))); @@ -39,7 +39,7 @@ public DevopsArtifactRepository(IConfiguration configuration, IHttpContextAccess _devopsClient = new HttpClient(); _devopsClient.DefaultRequestHeaders.Accept.Clear(); - _devopsClient.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json")); + _devopsClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json")); _devopsClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", _devopsAccessToken); } @@ -68,11 +68,11 @@ private async Task GetFromDevopsAsync(string request) { var downloadResp = await _devopsClient.GetAsync(request); int count = 0; - while ((downloadResp.StatusCode == HttpStatusCode.TooManyRequests || downloadResp.StatusCode == HttpStatusCode.BadRequest) && count < 5) + int[] waitTimes = new int[] { 0, 1, 2, 4, 8, 16, 32, 64, 128, 256 }; + while ((downloadResp.StatusCode == HttpStatusCode.TooManyRequests || downloadResp.StatusCode == HttpStatusCode.BadRequest) && count < waitTimes.Length) { - var retryAfter = (downloadResp.Headers.RetryAfter is null) ? "10" : downloadResp.Headers.RetryAfter.ToString(); - _telemetryClient.TrackTrace($"Download request from devops artifact is throttled. Retry After: {retryAfter}, Retry count: {count}"); - await Task.Delay(int.Parse(retryAfter) * 1000); + _telemetryClient.TrackTrace($"Download request from devops artifact is either throttled or flaky, waiting {waitTimes[count]} seconds before retrying, Retry count: {count}"); + await Task.Delay(TimeSpan.FromSeconds(waitTimes[count])); downloadResp = await _devopsClient.GetAsync(request); count++; } diff --git a/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstDumper.cpp b/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstDumper.cpp index 04bbcd30a35..b43f4b617e0 100644 --- a/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstDumper.cpp +++ b/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstDumper.cpp @@ -48,7 +48,7 @@ void AstDumper::OpenNamespace(std::string_view const& namespaceName) LeftAlign(); InsertKeyword("namespace"); InsertWhitespace(); - InsertMemberName(namespaceName); + InsertIdentifier(namespaceName); InsertWhitespace(); InsertPunctuation('{'); AdjustIndent(namespaceIndent); diff --git a/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstDumper.hpp b/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstDumper.hpp index 9e30bc46efd..0e1df127f08 100644 --- a/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstDumper.hpp +++ b/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstDumper.hpp @@ -38,11 +38,12 @@ class AstDumper { virtual void InsertText(std::string_view const& text) = 0; virtual void InsertPunctuation(char punctuation) = 0; virtual void InsertLineIdMarker() = 0; + virtual void InsertIdentifier(std::string_view const& identifier) = 0; virtual void InsertTypeName( std::string_view const& type, std::string_view const& typeNavigationId) = 0; - virtual void InsertMemberName(std::string_view const& member) = 0; + virtual void InsertMemberName(std::string_view const& member, std::string_view const& memberFullName) = 0; virtual void InsertStringLiteral(std::string_view const& str) = 0; virtual void InsertLiteral(std::string_view const& str) = 0; virtual void InsertComment(std::string_view const& comment) = 0; diff --git a/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstNode.cpp b/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstNode.cpp index 5e0694d76e3..f8efa485969 100644 --- a/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstNode.cpp +++ b/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/AstNode.cpp @@ -585,8 +585,7 @@ class AstDeclRefExpr : public AstExpr { } else { - dumper->InsertWhitespace(); - dumper->InsertMemberName(m_referencedName); + dumper->InsertIdentifier(m_referencedName); } } }; @@ -604,7 +603,7 @@ class AstDependentDeclRefExpr : public AstExpr { m_type.Dump(dumper, dumpOptions); dumper->InsertPunctuation(':'); dumper->InsertPunctuation(':'); - dumper->InsertMemberName(m_referencedName); + dumper->InsertIdentifier(m_referencedName); } }; @@ -700,7 +699,7 @@ class AstCallExpr : public AstExpr { { dumper->InsertPunctuation('{'); } - dumper->InsertMemberName(m_methodToCall); + dumper->InsertIdentifier(m_methodToCall); dumper->InsertPunctuation('('); DumpList( m_arguments.begin(), @@ -1169,8 +1168,61 @@ void AstBaseClass::DumpNode(AstDumper* dumper, DumpNodeOptions const& dumpOption m_baseClass.Dump(dumper, dumpOptions); } +// For functions, we want the navigation ID to be the full signature, including the return type +// to handle overloads. +static std::string GetNavigationId(clang::FunctionDecl const* func) +{ + std::string navigationId; + clang::PrintingPolicy pp{LangOptions{}}; + pp.adjustForCPlusPlus(); + pp.FullyQualifiedName = true; + pp.TerseOutput = true; + llvm::raw_string_ostream os(navigationId); + + func->print(os, pp); + return navigationId; +} + +static std::string GetNavigationId(clang::ParmVarDecl const* param) +{ + + std::string navigationId; + auto dc = param->getParentFunctionOrMethod(); + if (dc) + { + auto fd = cast(dc); + navigationId = GetNavigationId(fd) + " param " + param->getNameAsString(); + } + else + { + param->dump(llvm::outs()); + } + return navigationId; +} + +static std::string GetNavigationId( + clang::ClassTemplateSpecializationDecl const* templateSpecialization) +{ + std::string navigationId = templateSpecialization->getQualifiedNameAsString(); + navigationId += "<"; + for (const auto& arg : templateSpecialization->getTemplateArgs().asArray()) + { + navigationId += " " + arg.getAsType().getAsString(); + } + navigationId += ">"; + return navigationId; +} + +static std::string GetNavigationId(UsingDecl const* usingDeclaration) +{ + std::string navigationId = "using "; + navigationId += usingDeclaration->getQualifiedNameAsString(); + return navigationId; +} + class AstParamVariable : public AstNamedNode { std::string m_typeAsString; + std::string m_fullTypeAsString; AstType m_type; bool m_isStatic{}; bool m_isArray{}; @@ -1183,9 +1235,11 @@ class AstParamVariable : public AstNamedNode { AzureClassesDatabase* const database, std::shared_ptr parentNode) : AstNamedNode(var, database, parentNode), m_type{var->getType(), var->getASTContext()}, - m_isStatic(var->isStaticDataMember()) + m_fullTypeAsString{var->getQualifiedNameAsString()}, m_isStatic(var->isStaticDataMember()) { + m_navigationId = GetNavigationId(var); + clang::PrintingPolicy pp{LangOptions{}}; pp.adjustForCPlusPlus(); @@ -1222,7 +1276,12 @@ class AstParamVariable : public AstNamedNode { } dumper->InsertLiteral(m_typeAsString); dumper->InsertWhitespace(); - dumper->InsertMemberName(Name()); + + // If the parameter name isn't provided, there's nothing to insert into the output. + if (!Name().empty()) + { + dumper->InsertMemberName(Name(), m_navigationId); + } if (m_isArray) { dumper->InsertPunctuation('['); @@ -1313,7 +1372,7 @@ class AstVariable : public AstNamedNode { } dumper->InsertLiteral(m_typeAsString); dumper->InsertWhitespace(); - dumper->InsertMemberName(Name()); + dumper->InsertMemberName(Name(), m_navigationId); if (m_isArray) { dumper->InsertPunctuation('['); @@ -1382,7 +1441,7 @@ class AstTemplateParameter : public AstNamedNode { if (!m_paramName.empty()) { dumper->InsertWhitespace(); - dumper->InsertMemberName(m_paramName); + dumper->InsertIdentifier(m_paramName); if (m_defaultValue) { dumper->InsertWhitespace(); @@ -1478,13 +1537,13 @@ class AstTemplateTemplateParameter : public AstNamedNode { dumper->InsertWhitespace(); dumper->InsertKeyword("class"); dumper->InsertWhitespace(); - dumper->InsertMemberName(m_paramName); + dumper->InsertIdentifier(m_paramName); if (!m_defaultTypeName.empty()) { dumper->InsertWhitespace(); dumper->InsertPunctuation('='); dumper->InsertWhitespace(); - dumper->InsertMemberName(m_defaultTypeName); + dumper->InsertIdentifier(m_defaultTypeName); } } }; @@ -1550,7 +1609,7 @@ class AstTypeAlias : public AstNamedNode { } dumper->InsertKeyword("using"); dumper->InsertWhitespace(); - dumper->InsertMemberName(Name()); + dumper->InsertTypeName(Name(), m_navigationId); dumper->InsertWhitespace(); dumper->InsertPunctuation('='); dumper->InsertWhitespace(); @@ -1643,6 +1702,7 @@ class AstFunction : public AstNamedNode { func->getKind() == Decl::CXXConstructor || func->getKind() == Decl::CXXDestructor}, m_exceptionSpecification{func->getExceptionSpecType()} { + m_navigationId = GetNavigationId(func); if (m_exceptionSpecification == EST_DependentNoexcept) { auto typePtr = func->getType().getTypePtr(); @@ -1706,13 +1766,13 @@ class AstFunction : public AstNamedNode { } if (dumpOptions.IncludeNamespace) { - dumper->InsertMemberName(Namespace()); + dumper->InsertIdentifier(Namespace()); dumper->InsertPunctuation(':'); dumper->InsertPunctuation(':'); } if (dumpOptions.IncludeContainingClass && !m_parentClass.empty()) { - dumper->InsertMemberName(m_parentClass); + dumper->InsertIdentifier(m_parentClass); dumper->InsertPunctuation(':'); dumper->InsertPunctuation(':'); } @@ -1772,7 +1832,6 @@ class AstMethod : public AstFunction { } void DumpNode(AstDumper* dumper, DumpNodeOptions const& dumpOptions) const override { - DumpAttributes(dumper, dumpOptions); if (dumpOptions.NeedsLeftAlign) { dumper->LeftAlign(); @@ -1858,7 +1917,6 @@ class AstConstructor : public AstMethod { } void DumpNode(AstDumper* dumper, DumpNodeOptions const& dumpOptions) const override { - DumpAttributes(dumper, dumpOptions); if (dumpOptions.NeedsLeftAlign) { dumper->LeftAlign(); @@ -1964,23 +2022,21 @@ class AstAccessSpec : public AstNode { { } AstAccessSpec(AccessSpecifier specifier) : AstNode(nullptr), m_accessSpecifier{specifier} {} - void DumpNode(AstDumper* dumper, DumpNodeOptions const& dumpOptions) const override; -}; - -void AstAccessSpec::DumpNode(AstDumper* dumper, DumpNodeOptions const& dumpOptions) const -{ - // We want to left-indent the "public:", "private:" and "protected" items so they stick - // out from the fields in the class. - dumper->AdjustIndent(-2); - if (dumpOptions.NeedsLeftAlign) + void DumpNode(AstDumper* dumper, DumpNodeOptions const& dumpOptions) const override { - dumper->LeftAlign(); + // We want to left-indent the "public:", "private:" and "protected" items so they stick + // out from the fields in the class. + dumper->AdjustIndent(-2); + if (dumpOptions.NeedsLeftAlign) + { + dumper->LeftAlign(); + } + dumper->InsertKeyword(AccessSpecifierToString(m_accessSpecifier)); + dumper->InsertPunctuation(':'); + dumper->AdjustIndent(2); + dumper->Newline(); } - dumper->InsertKeyword(AccessSpecifierToString(m_accessSpecifier)); - dumper->InsertPunctuation(':'); - dumper->AdjustIndent(2); - dumper->Newline(); -} +}; /** * Represents an AST class or structure. @@ -2234,6 +2290,7 @@ class AstClassTemplateSpecialization : public AstClassLike { : AstClassLike(templateDecl, azureClassesDatabase, parentNode) { + m_navigationId = GetNavigationId(templateDecl); for (const auto& arg : templateDecl->getTemplateArgs().asArray()) { m_arguments.push_back(std::make_unique(arg.getAsType())); @@ -2375,7 +2432,7 @@ class AstFriend : public AstNode { } else { - dumper->InsertTypeName(m_friendType, m_friendType); + dumper->InsertIdentifier(m_friendType); if (dumpOptions.NeedsTrailingSemi) { dumper->InsertPunctuation(';'); @@ -2431,7 +2488,7 @@ class AstUsingDecl : public AstNamedNode { std::shared_ptr parentNode) : AstNamedNode(usingDecl, azureClassesDatabase, parentNode) { - usingDecl->dump(llvm::outs()); + m_navigationId = GetNavigationId(usingDecl); if (usingDecl->getQualifier()) { std::string qualifier; @@ -2459,7 +2516,7 @@ class AstUsingDecl : public AstNamedNode { { fullName.erase(0, Namespace().size()); } - dumper->InsertTypeName(fullName, Namespace()); + dumper->InsertTypeName(fullName, m_navigationId); dumper->InsertPunctuation(';'); if (dumpOptions.NeedsTrailingNewline) { @@ -2487,7 +2544,7 @@ class AstEnumerator : public AstNamedNode { { DumpAttributes(dumper, dumpOptions); dumper->LeftAlign(); - dumper->InsertMemberName(Name()); + dumper->InsertMemberName(Name(), m_navigationId); if (m_initializer) { dumper->InsertWhitespace(); @@ -2505,6 +2562,7 @@ class AstEnum : public AstNamedNode { bool m_isScoped; bool m_isScopedWithClass; bool m_isFixed; + bool m_isForwardDeclaration; public: AstEnum( @@ -2513,8 +2571,9 @@ class AstEnum : public AstNamedNode { std::shared_ptr parentNode) : AstNamedNode(enumDecl, azureClassesDatabase, parentNode), m_underlyingType{enumDecl->getIntegerType().getAsString()}, - m_isScoped{enumDecl->isScoped()}, - m_isScopedWithClass{enumDecl->isScopedUsingClassTag()}, m_isFixed{enumDecl->isFixed()} + m_isScoped{enumDecl->isScoped()}, m_isScopedWithClass{enumDecl->isScopedUsingClassTag()}, + m_isFixed{enumDecl->isFixed()}, m_isForwardDeclaration{ + enumDecl != enumDecl->getDefinition()} { if (!m_isScoped) { @@ -2666,9 +2725,9 @@ AstClassLike::AstClassLike( // virtual functions are always included in the API View. We need to adjust the access // specifier for these functions so that they are emitted as private. // - // Friend nodes ignore access specifiers, so there is no need to adjust access specifiers - // for them. - if (child->getKind() != Decl::Kind::Friend) + // Friend nodes and static_assert nodes ignore access specifiers, so there is no need to + // adjust access specifiers for them. + if (child->getKind() != Decl::Kind::Friend && child->getKind() != Decl::Kind::StaticAssert) { if (child->getKind() == Decl::Kind::AccessSpec) @@ -2800,7 +2859,14 @@ void AstClassLike::DumpNode(AstDumper* dumper, DumpNodeOptions const& dumpOption } DumpTag(dumper, dumpOptions); dumper->InsertWhitespace(); - dumper->InsertTypeName(Name(), m_navigationId); + if (m_isForwardDeclaration) + { + dumper->InsertIdentifier(Name()); + } + else + { + dumper->InsertTypeName(Name(), m_navigationId); + } DumpTemplateSpecializationArguments(dumper, dumpOptions); if (!m_isForwardDeclaration) { @@ -2846,7 +2912,8 @@ void AstClassLike::DumpNode(AstDumper* dumper, DumpNodeOptions const& dumpOption if (m_isAnonymousNamedStruct && !m_anonymousNamedStructName.empty()) { dumper->InsertWhitespace(); - dumper->InsertTypeName(m_anonymousNamedStructName, m_navigationId); + dumper->InsertTypeName( + m_anonymousNamedStructName, m_navigationId + m_anonymousNamedStructName); } } if (dumpOptions.NeedsTrailingSemi) @@ -2887,36 +2954,44 @@ void AstEnum::DumpNode(AstDumper* dumper, DumpNodeOptions const& dumpOptions) co } } dumper->InsertWhitespace(); - dumper->InsertTypeName(Name(), m_navigationId); - if (m_isFixed) + if (m_isForwardDeclaration) { - dumper->InsertWhitespace(); - dumper->InsertPunctuation(':'); - dumper->InsertWhitespace(); - dumper->InsertMemberName(m_underlyingType); + dumper->InsertIdentifier(Name()); } - dumper->Newline(); - dumper->LeftAlign(); - dumper->InsertPunctuation('{'); - dumper->AdjustIndent(2); - dumper->Newline(); - + else { - DumpNodeOptions innerOptions{dumpOptions}; - innerOptions.NeedsLeadingNewline = true; - DumpList( - m_enumerators.begin(), - m_enumerators.end(), - dumper, - innerOptions, - [&](AstDumper* dumper, std::unique_ptr const& enumerator) { - enumerator->DumpNode(dumper, dumpOptions); - }); + dumper->InsertTypeName(Name(), m_navigationId); + + if (m_isFixed) + { + dumper->InsertWhitespace(); + dumper->InsertPunctuation(':'); + dumper->InsertWhitespace(); + dumper->InsertIdentifier(m_underlyingType); + } + dumper->Newline(); + dumper->LeftAlign(); + dumper->InsertPunctuation('{'); + dumper->AdjustIndent(2); + dumper->Newline(); + + { + DumpNodeOptions innerOptions{dumpOptions}; + innerOptions.NeedsLeadingNewline = true; + DumpList( + m_enumerators.begin(), + m_enumerators.end(), + dumper, + innerOptions, + [&](AstDumper* dumper, std::unique_ptr const& enumerator) { + enumerator->DumpNode(dumper, dumpOptions); + }); + } + dumper->Newline(); + dumper->AdjustIndent(-2); + dumper->LeftAlign(); + dumper->InsertPunctuation('}'); } - dumper->Newline(); - dumper->AdjustIndent(-2); - dumper->LeftAlign(); - dumper->InsertPunctuation('}'); if (dumpOptions.NeedsTrailingSemi) { dumper->InsertPunctuation(';'); @@ -2936,7 +3011,7 @@ void AstField::DumpNode(AstDumper* dumper, DumpNodeOptions const& dumpOptions) c } m_fieldType.Dump(dumper, dumpOptions); dumper->InsertWhitespace(); - dumper->InsertMemberName(Name()); + dumper->InsertMemberName(Name(), m_navigationId); // if (m_initializer) //{ // DumpNodeOptions innerOptions{dumpOptions}; @@ -2964,7 +3039,7 @@ void AstField::DumpNode(AstDumper* dumper, DumpNodeOptions const& dumpOptions) c void AstType::Dump(AstDumper* dumper, DumpNodeOptions const& dumpOptions) const { - dumper->InsertMemberName(m_internalTypeName); + dumper->InsertIdentifier(m_internalTypeName); } void AstExpr::Dump(AstDumper* dumper, DumpNodeOptions const& dumpOptions) const @@ -2988,7 +3063,7 @@ void AstMemberExpr::Dump(AstDumper* dumper, DumpNodeOptions const& dumpOptions) } m_member->Dump(dumper, dumpOptions); dumper->InsertPunctuation('.'); - dumper->InsertMemberName(m_memberMethod); + dumper->InsertIdentifier(m_memberMethod); dumper->InsertPunctuation('('); dumper->InsertPunctuation(')'); if (dumpOptions.DumpListInitializer) diff --git a/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/JsonDumper.hpp b/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/JsonDumper.hpp index 512e33bafb8..71e1a49ff10 100644 --- a/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/JsonDumper.hpp +++ b/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/JsonDumper.hpp @@ -7,6 +7,7 @@ #include #include #include +#include using namespace nlohmann::literals; @@ -37,6 +38,32 @@ class JsonDumper : public AstDumper { InheritanceInfoEnd = 18 }; + // Validate that the json we've created won't cause problems for ApiView. + void ValidateJson() + { + std::unordered_set definitions; + for (const auto& token : m_json["Tokens"]) + { + if (token.contains("DefinitionId") && token["DefinitionId"].is_string()) + { + auto definitionId = token["DefinitionId"].get(); + if (definitions.find(definitionId) != definitions.end()) + { + throw std::runtime_error("Duplicate DefinitionId: " + definitionId); + } + definitions.emplace(definitionId); + } + if (!token.contains("Value")) + { + throw std::runtime_error("Missing Value in token"); + } + if (!token.contains("Kind")) + { + throw std::runtime_error("Missing Kind in token"); + } + } + } + public: JsonDumper( std::string_view reviewName, @@ -56,13 +83,17 @@ class JsonDumper : public AstDumper { m_json["Tokens"] = nlohmann::json::array(); } - void DumpToFile(std::ostream& outfile) { outfile << m_json; } + void DumpToFile(std::ostream& outfile) + { + ValidateJson(); + outfile << m_json; + } nlohmann::json const& GetJson() { return m_json; } // Each ApiView node has 4 mandatory members: // - // DefinitionId: ID used in the Navigation pane for type navigation. - // NavigateToId: ??? + // DefinitionId: A unique value used to represent an entity where comments can be left. This MUST be unique. + // NavigateToId: ID used in the Navigation pane for type navigation. // Value: Value to display in ApiView (mandatory) // Kind: Type of node, used for color coding output. @@ -116,6 +147,14 @@ class JsonDumper : public AstDumper { {"Value", nullptr}, {"Kind", TokenKinds::LineIdMarker}}); // Not clear if this is used at all. } + virtual void InsertIdentifier(std::string_view const& id) override + { + m_json["Tokens"].push_back( + {{"DefinitionId", nullptr}, + {"NavigateToId", nullptr}, + {"Value", id}, + {"Kind", TokenKinds::TypeName}}); + } virtual void InsertTypeName(std::string_view const& type, std::string_view const& navigationId) override { @@ -125,10 +164,12 @@ class JsonDumper : public AstDumper { {"Value", type}, {"Kind", TokenKinds::TypeName}}); } - virtual void InsertMemberName(std::string_view const& member) override + virtual void InsertMemberName( + std::string_view const& member, + std::string_view const& memberFullName) override { m_json["Tokens"].push_back( - {{"DefinitionId", member}, + {{"DefinitionId", memberFullName}, {"NavigateToId", nullptr}, {"Value", member}, {"Kind", TokenKinds::MemberName}}); diff --git a/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/TextDumper.hpp b/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/TextDumper.hpp index 8a1668fd6dc..7f5170c38c2 100644 --- a/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/TextDumper.hpp +++ b/tools/apiview/parsers/cpp-api-parser/ApiViewProcessor/TextDumper.hpp @@ -24,9 +24,13 @@ class TextDumper : public AstDumper { { m_stream << type; } - virtual void InsertMemberName(std::string_view const& member) override { m_stream << member; } + virtual void InsertMemberName(std::string_view const& member, std::string_view const&) override + { + m_stream << member; + } virtual void InsertStringLiteral(std::string_view const& str) override { m_stream << str; } virtual void InsertLiteral(std::string_view const& str) override { m_stream << str; } + virtual void InsertIdentifier(std::string_view const& str) override { m_stream << str; } virtual void InsertComment(std::string_view const& comment) override { m_stream << comment; } virtual void AddDocumentRangeStart() override { m_stream << "/*"; } virtual void AddDocumentRangeEnd() override { m_stream << "*/"; } diff --git a/tools/apiview/parsers/cpp-api-parser/ParseTests/tests.cpp b/tools/apiview/parsers/cpp-api-parser/ParseTests/tests.cpp index 019eab608c2..085f56567ef 100644 --- a/tools/apiview/parsers/cpp-api-parser/ParseTests/tests.cpp +++ b/tools/apiview/parsers/cpp-api-parser/ParseTests/tests.cpp @@ -252,7 +252,8 @@ struct NsDumper : AstDumper std::string_view const& typeNavigationId) override { } - virtual void InsertMemberName(std::string_view const& member) override {} + virtual void InsertMemberName(std::string_view const& member, std::string_view const&) override {} + virtual void InsertIdentifier(std::string_view const& identifier) override {} virtual void InsertStringLiteral(std::string_view const& str) override {} virtual void InsertLiteral(std::string_view const& str) override {} virtual void InsertComment(std::string_view const& comment) override {} @@ -304,7 +305,7 @@ TEST_F(TestParser, NamespaceFilter1) EXPECT_EQ("CPA0003", dumper.Messages[0].DiagnosticId); EXPECT_EQ("GlobalFunction4", dumper.Messages[0].FailingId); EXPECT_EQ("CPA0002", dumper.Messages[1].DiagnosticId); - EXPECT_EQ("GlobalFunction4", dumper.Messages[1].FailingId); + EXPECT_EQ("char *GlobalFunction4(int character)", dumper.Messages[1].FailingId); EXPECT_EQ("CPA0003", dumper.Messages[2].DiagnosticId); EXPECT_EQ("A::AB::ABC::FunctionABC", dumper.Messages[2].FailingId); EXPECT_EQ("CPA0003", dumper.Messages[3].DiagnosticId); @@ -344,7 +345,7 @@ TEST_F(TestParser, NamespaceFilter2) EXPECT_EQ("CPA0003", dumper.Messages[2].DiagnosticId); EXPECT_EQ("GlobalFunction4", dumper.Messages[2].FailingId); EXPECT_EQ("CPA0002", dumper.Messages[3].DiagnosticId); - EXPECT_EQ("GlobalFunction4", dumper.Messages[3].FailingId); + EXPECT_EQ("char *GlobalFunction4(int character)", dumper.Messages[3].FailingId); EXPECT_EQ("CPA0003", dumper.Messages[4].DiagnosticId); EXPECT_EQ("A::AB::ABC::FunctionABC", dumper.Messages[4].FailingId); EXPECT_EQ("CPA0003", dumper.Messages[5].DiagnosticId); @@ -386,7 +387,7 @@ TEST_F(TestParser, NamespaceFilter3) EXPECT_EQ("CPA0003", dumper.Messages[3].DiagnosticId); EXPECT_EQ("GlobalFunction4", dumper.Messages[3].FailingId); EXPECT_EQ("CPA0002", dumper.Messages[4].DiagnosticId); - EXPECT_EQ("GlobalFunction4", dumper.Messages[4].FailingId); + EXPECT_EQ("char *GlobalFunction4(int character)", dumper.Messages[4].FailingId); EXPECT_EQ("CPA0003", dumper.Messages[5].DiagnosticId); EXPECT_EQ("A::AB::ABC::FunctionABC", dumper.Messages[5].FailingId); EXPECT_EQ("CPA0003", dumper.Messages[6].DiagnosticId); @@ -427,7 +428,7 @@ TEST_F(TestParser, NamespaceFilter4) EXPECT_EQ("CPA0003", dumper.Messages[2].DiagnosticId); EXPECT_EQ("GlobalFunction4", dumper.Messages[2].FailingId); EXPECT_EQ("CPA0002", dumper.Messages[3].DiagnosticId); - EXPECT_EQ("GlobalFunction4", dumper.Messages[3].FailingId); + EXPECT_EQ("char *GlobalFunction4(int character)", dumper.Messages[3].FailingId); EXPECT_TRUE(SyntaxCheckClassDb(db, "SimpleTestGenerated4.cpp")); } diff --git a/tools/perf-automation/Azure.Sdk.Tools.PerfAutomation/Python.cs b/tools/perf-automation/Azure.Sdk.Tools.PerfAutomation/Python.cs index 5dd754a89c3..06b05610e28 100644 --- a/tools/perf-automation/Azure.Sdk.Tools.PerfAutomation/Python.cs +++ b/tools/perf-automation/Azure.Sdk.Tools.PerfAutomation/Python.cs @@ -57,7 +57,7 @@ public class Python : LanguageBase { if (packageName == primaryPackage) { - await Util.RunAsync(pip, "install -e .", projectDirectory, outputBuilder: outputBuilder, errorBuilder: errorBuilder); + await Util.RunAsync(pip, "install .", projectDirectory, outputBuilder: outputBuilder, errorBuilder: errorBuilder); } // TODO: Consider installing source versions of non-primary packages. Would require finding package in source tree. // So far, this seems unnecessary, since dev-requirements.txt usually includes core. diff --git a/tools/pylint-extensions/azure-pylint-guidelines-checker/CHANGELOG.md b/tools/pylint-extensions/azure-pylint-guidelines-checker/CHANGELOG.md index 7d9eab18e7e..893571419a9 100644 --- a/tools/pylint-extensions/azure-pylint-guidelines-checker/CHANGELOG.md +++ b/tools/pylint-extensions/azure-pylint-guidelines-checker/CHANGELOG.md @@ -1,5 +1,11 @@ # Release History +## 0.1.0 (2023-08-02) + +Add two new checkers: +- Checker to warn against importing the package `six` +- Checker to warn against importing `HttpResponse` from `azure.core.pipeline.transport` + ## 0.0.9 (2023-06-26) Fix bug with varargs in CheckDocstringParameters checker. Updated client-paging-methods-use-list checker to include private methods. \ No newline at end of file diff --git a/tools/pylint-extensions/azure-pylint-guidelines-checker/pylint_guidelines_checker.py b/tools/pylint-extensions/azure-pylint-guidelines-checker/pylint_guidelines_checker.py index a43d68c633f..53f093a532e 100644 --- a/tools/pylint-extensions/azure-pylint-guidelines-checker/pylint_guidelines_checker.py +++ b/tools/pylint-extensions/azure-pylint-guidelines-checker/pylint_guidelines_checker.py @@ -1383,11 +1383,9 @@ def check_return(self, node): # Get decorators on the function function_decorators = node.decoratornames() try: - returns = next(node.infer_call_result()) - # If returns None, or raises an error ignore - if returns == astroid.Uninferable: - return - if returns.as_string() == "None": + returns = next(node.infer_call_result()).as_string() + # If returns None ignore + if returns == "None": return except (astroid.exceptions.InferenceError, AttributeError): # this function doesn't return anything, just return diff --git a/tools/pylint-extensions/azure-pylint-guidelines-checker/setup.py b/tools/pylint-extensions/azure-pylint-guidelines-checker/setup.py index 51b46f851a4..435487173f9 100644 --- a/tools/pylint-extensions/azure-pylint-guidelines-checker/setup.py +++ b/tools/pylint-extensions/azure-pylint-guidelines-checker/setup.py @@ -6,7 +6,7 @@ setup( name="azure-pylint-guidelines-checker", - version="0.0.9", + version="0.1.0", url='http://github.com/Azure/azure-sdk-for-python', license='MIT License', description="A pylint plugin which enforces azure sdk guidelines.", diff --git a/tools/pylint-extensions/azure-pylint-guidelines-checker/tests/test_pylint_custom_plugins.py b/tools/pylint-extensions/azure-pylint-guidelines-checker/tests/test_pylint_custom_plugins.py index e00214ccaad..9e26a9567d7 100644 --- a/tools/pylint-extensions/azure-pylint-guidelines-checker/tests/test_pylint_custom_plugins.py +++ b/tools/pylint-extensions/azure-pylint-guidelines-checker/tests/test_pylint_custom_plugins.py @@ -3523,19 +3523,6 @@ def function_foo(): with self.assertNoMessages(): self.checker.visit_functiondef(node) - def test_docstring_raises_uninferable(self): - node = astroid.extract_node( - """ - def function_foo(): - ''' - :raises: ValueError - ''' - raise ValueError("hello") - """ - ) - with self.assertNoMessages(): - self.checker.visit_functiondef(node) - class TestDoNotImportLegacySix(pylint.testutils.CheckerTestCase): """Test that we are blocking disallowed imports and allowing allowed imports.""" CHECKER_CLASS = checker.DoNotImportLegacySix diff --git a/tools/sdk-testgen/packages/autorest.testmodeler/CHANGELOG.json b/tools/sdk-testgen/packages/autorest.testmodeler/CHANGELOG.json index b5b6cf95175..2ef04b3f916 100644 --- a/tools/sdk-testgen/packages/autorest.testmodeler/CHANGELOG.json +++ b/tools/sdk-testgen/packages/autorest.testmodeler/CHANGELOG.json @@ -1,6 +1,18 @@ { "name": "@autorest/testmodeler", "entries": [ + { + "version": "2.6.1", + "tag": "@autorest/testmodeler_v2.6.1", + "date": "Fri, 21 Jul 2023 08:36:29 GMT", + "comments": { + "patch": [ + { + "comment": "Support DataFactoryKeyVaultSecretReference of DataFactoryElement" + } + ] + } + }, { "version": "2.6.0", "tag": "@autorest/testmodeler_v2.6.0", diff --git a/tools/sdk-testgen/packages/autorest.testmodeler/CHANGELOG.md b/tools/sdk-testgen/packages/autorest.testmodeler/CHANGELOG.md index ef2635f44ff..c188b218489 100644 --- a/tools/sdk-testgen/packages/autorest.testmodeler/CHANGELOG.md +++ b/tools/sdk-testgen/packages/autorest.testmodeler/CHANGELOG.md @@ -1,6 +1,13 @@ # Change Log - @autorest/testmodeler -This log was last generated on Mon, 19 Jun 2023 08:50:04 GMT and should not be manually modified. +This log was last generated on Fri, 21 Jul 2023 08:36:29 GMT and should not be manually modified. + +## 2.6.1 +Fri, 21 Jul 2023 08:36:29 GMT + +### Patches + +- Support DataFactoryKeyVaultSecretReference of DataFactoryElement ## 2.6.0 Mon, 19 Jun 2023 08:50:04 GMT diff --git a/tools/sdk-testgen/packages/autorest.testmodeler/package.json b/tools/sdk-testgen/packages/autorest.testmodeler/package.json index 1e052a69044..c719c66ba0a 100644 --- a/tools/sdk-testgen/packages/autorest.testmodeler/package.json +++ b/tools/sdk-testgen/packages/autorest.testmodeler/package.json @@ -1,6 +1,6 @@ { "name": "@autorest/testmodeler", - "version": "2.6.0", + "version": "2.6.1", "description": "Autorest extension for testmodeler", "main": "dist/index.js", "scripts": { diff --git a/tools/sdk-testgen/packages/autorest.testmodeler/src/core/model.ts b/tools/sdk-testgen/packages/autorest.testmodeler/src/core/model.ts index cf9b37de312..eb1373dd99c 100644 --- a/tools/sdk-testgen/packages/autorest.testmodeler/src/core/model.ts +++ b/tools/sdk-testgen/packages/autorest.testmodeler/src/core/model.ts @@ -218,7 +218,7 @@ export class ExampleValue { const format = extensions[xMsFormat]; const elementFormat = extensions[xMsFormatElementType]; - const dfeObjSchema = ExampleValue.createSchemaForDfeObject(rawValue, format); + const dfeObjSchema = ExampleValue.createSchemaForDfeObject(session, rawValue, format); if (dfeObjSchema) { return this.createInstance(session, rawValue, usedProperties, dfeObjSchema, language, undefined, searchDescents); } else { @@ -231,17 +231,30 @@ export class ExampleValue { return instance; } - private static createSchemaForDfeObject(raw: any, dfeFormat: string): ObjectSchema | undefined { + private static createSchemaForDfeObject(session: Session, raw: any, dfeFormat: string): ObjectSchema | undefined { const dfeObjectType = 'type'; const dfeObjectValue = 'value'; - const dfeObjectTypeValues = ['Expression', 'SecureString', 'AzureKeyVaultSecretReference']; const dfeObjectSchemaPrefix = 'DataFactoryElement-'; - if (Object(raw) && raw[dfeObjectType] && raw[dfeObjectValue] && dfeObjectTypeValues.includes(raw[dfeObjectType])) { + if (Object(raw) && raw[dfeObjectType] && raw[dfeObjectValue]) { const r = new ObjectSchema(dfeObjectSchemaPrefix + raw[dfeObjectType], ''); r.addProperty(new Property(dfeObjectType, '', new StringSchema(`${dfeFormat}-${dfeObjectType}`, ''))); - r.addProperty(new Property(dfeObjectValue, '', new StringSchema(`${dfeFormat}-${dfeObjectValue}`, ''))); - return r; + switch (raw[dfeObjectType]) { + case 'Expression': + case 'SecureString': + r.addProperty(new Property(dfeObjectValue, '', new StringSchema(`${dfeFormat}-${dfeObjectValue}`, ''))); + return r; + case 'AzureKeyVaultSecretReference': { + const valueSchema = session.model.schemas.objects.find((s) => s.language.default.name === `AzureKeyVaultSecretReference`); + if (!valueSchema) { + throw new Error('Cant find schema for the value of DataFactoryElement KeyVaultSecret Reference'); + } + r.addProperty(new Property(dfeObjectValue, '', valueSchema)); + return r; + } + default: + return undefined; + } } return undefined; } diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy.Tests/Directory.Build.targets b/tools/test-proxy/Azure.Sdk.Tools.TestProxy.Tests/Directory.Build.targets new file mode 100644 index 00000000000..e76b95e4b90 --- /dev/null +++ b/tools/test-proxy/Azure.Sdk.Tools.TestProxy.Tests/Directory.Build.targets @@ -0,0 +1,9 @@ + + + + + true + false + $(RepoEngPath)\AzureSDKToolsKey.snk + + \ No newline at end of file diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy.Tests/LoggingTests.cs b/tools/test-proxy/Azure.Sdk.Tools.TestProxy.Tests/LoggingTests.cs new file mode 100644 index 00000000000..b06106317d8 --- /dev/null +++ b/tools/test-proxy/Azure.Sdk.Tools.TestProxy.Tests/LoggingTests.cs @@ -0,0 +1,115 @@ +using System.IO; +using System.Net.Http; +using System.Text; +using System.Threading.Tasks; +using Azure.Sdk.Tools.TestProxy.Common; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Mvc; +using Microsoft.Extensions.Logging.Abstractions; +using Xunit; + +namespace Azure.Sdk.Tools.TestProxy.Tests +{ + /// + /// Logging tests cannot be run in parallel with other tests because they share a static logger. + /// + [Collection(nameof(LoggingCollection))] + public class LoggingTests + { + [Fact] + public async Task PlaybackLogsSanitizedRequest() + { + var logger = new TestLogger(); + DebugLogger.Logger = logger; + + try + { + RecordingHandler testRecordingHandler = new RecordingHandler(Directory.GetCurrentDirectory()); + var httpContext = new DefaultHttpContext(); + var body = "{\"x-recording-file\":\"Test.RecordEntries/request_with_binary_content.json\"}"; + httpContext.Request.Body = TestHelpers.GenerateStreamRequestBody(body); + httpContext.Request.ContentLength = body.Length; + + var controller = new Playback(testRecordingHandler, new NullLoggerFactory()) + { + ControllerContext = new ControllerContext() + { + HttpContext = httpContext + } + }; + await controller.Start(); + + var recordingId = httpContext.Response.Headers["x-recording-id"].ToString(); + Assert.NotNull(recordingId); + Assert.True(testRecordingHandler.PlaybackSessions.ContainsKey(recordingId)); + var entry = testRecordingHandler.PlaybackSessions[recordingId].Session.Entries[0]; + HttpRequest request = TestHelpers.CreateRequestFromEntry(entry); + request.Headers["Authorization"] = "fake-auth-header"; + + HttpResponse response = new DefaultHttpContext().Response; + await testRecordingHandler.HandlePlaybackRequest(recordingId, request, response); + + Assert.Single(logger.Logs); + var logEntry = logger.Logs[0].ToString(); + Assert.DoesNotContain(@"""Authorization"":[""fake-auth-header""]", logEntry); + Assert.Contains(@"""Authorization"":[""Sanitized""]", logEntry); + } + finally + { + DebugLogger.Logger = null; + } + } + + [Fact] + public async Task RecordingHandlerLogsSanitizedRequests() + { + var logger = new TestLogger(); + DebugLogger.Logger = logger; + var httpContext = new DefaultHttpContext(); + var bodyBytes = Encoding.UTF8.GetBytes("{\"hello\":\"world\"}"); + var mockClient = new HttpClient(new MockHttpHandler(bodyBytes, "application/json")); + var path = Directory.GetCurrentDirectory(); + var recordingHandler = new RecordingHandler(path) + { + RedirectableClient = mockClient, + RedirectlessClient = mockClient + }; + + var relativePath = "recordings/logs"; + var fullPathToRecording = Path.Combine(path, relativePath) + ".json"; + + await recordingHandler.StartRecordingAsync(relativePath, httpContext.Response); + + var recordingId = httpContext.Response.Headers["x-recording-id"].ToString(); + + httpContext.Request.ContentType = "application/json"; + httpContext.Request.Headers["Authorization"] = "fake-auth-header"; + httpContext.Request.ContentLength = 0; + httpContext.Request.Headers["x-recording-id"] = recordingId; + httpContext.Request.Headers["x-recording-upstream-base-uri"] = "http://example.org"; + httpContext.Request.Method = "GET"; + httpContext.Request.Body = new MemoryStream(CompressionUtilities.CompressBody(bodyBytes, httpContext.Request.Headers)); + + await recordingHandler.HandleRecordRequestAsync(recordingId, httpContext.Request, httpContext.Response); + recordingHandler.StopRecording(recordingId); + + try + { + Assert.Single(logger.Logs); + var logEntry = logger.Logs[0].ToString(); + Assert.DoesNotContain(@"""Authorization"":[""fake-auth-header""]", logEntry); + Assert.Contains(@"""Authorization"":[""Sanitized""]", logEntry); + } + finally + { + File.Delete(fullPathToRecording); + DebugLogger.Logger = null; + } + } + } + + [CollectionDefinition(nameof(LoggingCollection), DisableParallelization = true)] + public class LoggingCollection + { + } +} \ No newline at end of file diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy.Tests/TestLogger.cs b/tools/test-proxy/Azure.Sdk.Tools.TestProxy.Tests/TestLogger.cs new file mode 100644 index 00000000000..60116223ca6 --- /dev/null +++ b/tools/test-proxy/Azure.Sdk.Tools.TestProxy.Tests/TestLogger.cs @@ -0,0 +1,27 @@ +using System; +using System.Collections.Generic; +using Microsoft.Extensions.Logging; + +namespace Azure.Sdk.Tools.TestProxy.Tests +{ + public class TestLogger : ILogger + { + internal List Logs { get; }= new List(); + + public IDisposable BeginScope(TState state) + { + throw new NotImplementedException(); + } + + public bool IsEnabled(LogLevel logLevel) + { + return true; + } + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception exception, + Func formatter) + { + Logs.Add(state); + } + } +} \ No newline at end of file diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Admin.cs b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Admin.cs index 0b00144138b..3c26851eaa8 100644 --- a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Admin.cs +++ b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Admin.cs @@ -28,25 +28,25 @@ public Admin(RecordingHandler recordingHandler, ILoggerFactory loggingFactory) } [HttpPost] - public async Task Reset() + public void Reset() { - await DebugLogger.LogRequestDetailsAsync(_logger, Request); + DebugLogger.LogRequestDetails(_logger, Request); var recordingId = RecordingHandler.GetHeader(Request, "x-recording-id", allowNulls: true); _recordingHandler.SetDefaultExtensions(recordingId); } [HttpGet] - public async Task IsAlive() + public void IsAlive() { - await DebugLogger.LogRequestDetailsAsync(_logger, Request); + DebugLogger.LogRequestDetails(_logger, Request); Response.StatusCode = 200; } [HttpPost] public async Task AddTransform() { - await DebugLogger.LogRequestDetailsAsync(_logger, Request); + DebugLogger.LogRequestDetails(_logger, Request); var tName = RecordingHandler.GetHeader(Request, "x-abstraction-identifier"); var recordingId = RecordingHandler.GetHeader(Request, "x-recording-id", allowNulls: true); @@ -65,7 +65,7 @@ public async Task AddTransform() [HttpPost] public async Task AddSanitizer() { - await DebugLogger.LogRequestDetailsAsync(_logger, Request); + DebugLogger.LogRequestDetails(_logger, Request); var sName = RecordingHandler.GetHeader(Request, "x-abstraction-identifier"); var recordingId = RecordingHandler.GetHeader(Request, "x-recording-id", allowNulls: true); @@ -84,7 +84,7 @@ public async Task AddSanitizer() [HttpPost] public async Task SetMatcher() { - await DebugLogger.LogRequestDetailsAsync(_logger, Request); + DebugLogger.LogRequestDetails(_logger, Request); var mName = RecordingHandler.GetHeader(Request, "x-abstraction-identifier"); var recordingId = RecordingHandler.GetHeader(Request, "x-recording-id", allowNulls: true); @@ -102,9 +102,9 @@ public async Task SetMatcher() [HttpPost] [AllowEmptyBody] - public async Task SetRecordingOptions([FromBody()] IDictionary options = null) + public void SetRecordingOptions([FromBody()] IDictionary options = null) { - await DebugLogger.LogRequestDetailsAsync(_logger, Request); + DebugLogger.LogRequestDetails(_logger, Request); var recordingId = RecordingHandler.GetHeader(Request, "x-recording-id", allowNulls: true); diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Common/DebugLogger.cs b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Common/DebugLogger.cs index 3ff9760b04e..d95564e1ddd 100644 --- a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Common/DebugLogger.cs +++ b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Common/DebugLogger.cs @@ -1,14 +1,9 @@ using Microsoft.Extensions.Logging; using Microsoft.AspNetCore.Http; -using System.Threading.Tasks; using System.Text; -using System.IO; using System; -using System.Diagnostics; +using System.Collections.Generic; using System.Text.Json; -using Microsoft.AspNetCore.Http.Extensions; -using Microsoft.Extensions.Hosting; -using Microsoft.Extensions.Configuration; namespace Azure.Sdk.Tools.TestProxy.Common { @@ -32,13 +27,14 @@ namespace Azure.Sdk.Tools.TestProxy.Common /// public static class DebugLogger { - private static ILogger logger = null; + // internal for testing + internal static ILogger Logger { get; set; } public static void ConfigureLogger(ILoggerFactory factory) { - if (logger == null && factory != null) + if (Logger == null && factory != null) { - logger = factory.CreateLogger("DebugLogging"); + Logger = factory.CreateLogger("Azure.Sdk.Tools.TestProxy"); } } @@ -50,7 +46,7 @@ public static void ConfigureLogger(ILoggerFactory factory) /// public static bool CheckLogLevel(LogLevel level) { - var result = logger?.IsEnabled(LogLevel.Debug); + var result = Logger?.IsEnabled(LogLevel.Debug); if (result.HasValue) { @@ -68,9 +64,9 @@ public static bool CheckLogLevel(LogLevel level) /// The content which should be logged. public static void LogInformation(string details) { - if (null != logger) + if (null != Logger) { - logger.LogInformation(details); + Logger.LogInformation(details); } else { @@ -80,9 +76,9 @@ public static void LogInformation(string details) public static void LogError(string details) { - if (null != logger) + if (null != Logger) { - logger.LogError(details); + Logger.LogError(details); } else { @@ -93,9 +89,9 @@ public static void LogError(string details) public static void LogError(int statusCode, Exception e) { var details = statusCode.ToString() + Environment.NewLine + e.Message + Environment.NewLine + e.StackTrace; - if (null != logger) + if (null != Logger) { - logger.LogError(details); + Logger.LogError(details); } else { @@ -109,9 +105,9 @@ public static void LogError(int statusCode, Exception e) /// The content which should be logged. public static void LogDebug(string details) { - if (logger != null) + if (Logger != null) { - logger.LogDebug(details); + Logger.LogDebug(details); } else { @@ -151,11 +147,11 @@ public static void LogDebug(string details) /// Usually will be the DI-ed individual ILogger instance from a controller. However any valid ILogger instance is fine here. /// The http request which needs to be detailed. /// - public static async Task LogRequestDetailsAsync(ILogger loggerInstance, HttpRequest req) + public static void LogRequestDetails(ILogger loggerInstance, HttpRequest req) { if(CheckLogLevel(LogLevel.Debug)) { - loggerInstance.LogDebug(await _generateLogLine(req)); + loggerInstance.LogDebug(_generateLogLine(req, null)); } } @@ -164,12 +160,13 @@ public static async Task LogRequestDetailsAsync(ILogger loggerInstance, HttpRequ /// actually logging anything, this function is entirely passthrough. /// /// The http request which needs to be detailed. - /// - public static async Task LogRequestDetailsAsync(HttpRequest req) + /// The set of sanitizers to apply before logging. + /// The log line. + public static void LogRequestDetails(HttpRequest req, IEnumerable sanitizers) { if (CheckLogLevel(LogLevel.Debug)) { - logger.LogDebug(await _generateLogLine(req)); + Logger.LogDebug(_generateLogLine(req, sanitizers)); } } @@ -177,20 +174,26 @@ public static async Task LogRequestDetailsAsync(HttpRequest req) /// Generate a line of data from an http request. This is non-destructive, which means it does not mess /// with the request Body stream at all. /// - /// - /// - private static async Task _generateLogLine(HttpRequest req) + /// The request + /// The set of sanitizers to apply before logging. + /// The log line. + private static string _generateLogLine(HttpRequest req, IEnumerable sanitizers) { - StringBuilder sb = new StringBuilder(); - string headers = string.Empty; + RecordEntry entry = RecordingHandler.CreateNoBodyRecordEntry(req); - using (MemoryStream ms = new MemoryStream()) + if (sanitizers != null) { - await JsonSerializer.SerializeAsync(ms, req.Headers); - headers = Encoding.UTF8.GetString(ms.ToArray()); + foreach (var sanitizer in sanitizers) + { + sanitizer.Sanitize(entry); + } } - sb.AppendLine("URI: [ " + req.GetDisplayUrl() + "]"); + var headers = Encoding.UTF8.GetString(JsonSerializer.SerializeToUtf8Bytes(entry.Request.Headers)); + + StringBuilder sb = new StringBuilder(); + + sb.AppendLine("URI: [ " + entry.RequestUri + "]"); sb.AppendLine("Headers: [" + headers + "]"); return sb.ToString(); diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Playback.cs b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Playback.cs index c7d82c44d8b..44b62c38f31 100644 --- a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Playback.cs +++ b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Playback.cs @@ -63,7 +63,7 @@ public void Stop() [HttpPost] public async Task Reset([FromBody()] IDictionary options = null) { - await DebugLogger.LogRequestDetailsAsync(_logger, Request); + DebugLogger.LogRequestDetails(_logger, Request); var pathToAssets = RecordingHandler.GetAssetsJsonLocation(StoreResolver.ParseAssetsJsonBody(options), _recordingHandler.ContextDirectory); @@ -73,7 +73,7 @@ public async Task Reset([FromBody()] IDictionary options = null) [HttpPost] public async Task Restore([FromBody()] IDictionary options = null) { - await DebugLogger.LogRequestDetailsAsync(_logger, Request); + DebugLogger.LogRequestDetails(_logger, Request); var pathToAssets = RecordingHandler.GetAssetsJsonLocation(StoreResolver.ParseAssetsJsonBody(options), _recordingHandler.ContextDirectory); diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Properties/AssemblyInfo.cs b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Properties/AssemblyInfo.cs new file mode 100644 index 00000000000..7ad94ee8711 --- /dev/null +++ b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Properties/AssemblyInfo.cs @@ -0,0 +1,7 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +using System.Runtime.CompilerServices; + +[assembly: InternalsVisibleTo("Azure.Sdk.Tools.TestProxy.Tests, PublicKey=0024000004800000940000000602000000240000525341310004000001000100259ae92701e6c1d912e6126950be871a0aa0bc76c69b573a8f549708e4f5b9658246d97f239964447af47052f09df117f955af39c1bfc43c369ada5460750e7dd0b0f178a70bb970a8fb74f9d892636a4ac38234157de5482482d3debd80f082d6b55a5761cc97c261e5ad3ba3025c06990011f1f86cc021de48381c8174049a")] + diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md index e4fb13d28f1..3848772e414 100644 --- a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md +++ b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md @@ -269,10 +269,11 @@ test-proxy -- --urls "http://localhost:9000;https://localhost:9001" The test-proxy is integrated with the following environment variables. -| Variable | Usage | -|---|---| -| `TEST_PROXY_FOLDER` | if command-line argument `storage-location` is not provided when invoking the proxy, this environment variable is also checked for a valid directory to use as test-proxy context. | -| `Logging__LogLevel__Default` | Defaults to `Information`. Possible valid values are `Information`, `Warning`, `Error`, `Critical`. | +| Variable | Usage | +|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `TEST_PROXY_FOLDER` | if command-line argument `storage-location` is not provided when invoking the proxy, this environment variable is also checked for a valid directory to use as test-proxy context. | +| `Logging__LogLevel__Default` | Defaults to `Information`. Possible valid values are

`Debug`, `Information`, `Warning`, `Error`, `Critical`.

Do not set for .NET test runs as it would cause the tests *themselves* to start emitting logs.| +| `Logging__LogLevel__Azure.Sdk.Tools.TestProxy`| Set to `Debug` to see request level logs emitted by the Test Proxy.| Both of the above variables can be set in the `docker` runtime by providing additional arguments EG: `docker run -e Logging__LogLevel__Default=Warning azsdkengsys.azurecr.io/engsys/test-proxy:latest`. For multiple environment variables, just use multiple `-e` provisions. diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Record.cs b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Record.cs index fb12f08d5c1..d63be791388 100644 --- a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Record.cs +++ b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Record.cs @@ -56,7 +56,7 @@ public async Task Start() [HttpPost] public async Task Push([FromBody()] IDictionary options = null) { - await DebugLogger.LogRequestDetailsAsync(_logger, Request); + DebugLogger.LogRequestDetails(_logger, Request); var pathToAssets = RecordingHandler.GetAssetsJsonLocation(StoreResolver.ParseAssetsJsonBody(options), _recordingHandler.ContextDirectory); diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/RecordingHandler.cs b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/RecordingHandler.cs index 0094e47ea1f..f6272b4d7d3 100644 --- a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/RecordingHandler.cs +++ b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/RecordingHandler.cs @@ -187,13 +187,15 @@ public async Task StartRecordingAsync(string sessionId, HttpResponse outgoingRes public async Task HandleRecordRequestAsync(string recordingId, HttpRequest incomingRequest, HttpResponse outgoingResponse) { - await DebugLogger.LogRequestDetailsAsync(incomingRequest); - if (!RecordingSessions.TryGetValue(recordingId, out var session)) { throw new HttpException(HttpStatusCode.BadRequest, $"There is no active recording session under id {recordingId}."); } + var sanitizers = session.AdditionalSanitizers.Count > 0 ? Sanitizers.Concat(session.AdditionalSanitizers) : Sanitizers; + + DebugLogger.LogRequestDetails(incomingRequest, sanitizers); + (RecordEntry entry, byte[] requestBody) = await CreateEntryAsync(incomingRequest).ConfigureAwait(false); var upstreamRequest = CreateUpstreamRequest(incomingRequest, requestBody); @@ -429,13 +431,15 @@ public void StopPlayback(string recordingId, bool purgeMemoryStore = false) public async Task HandlePlaybackRequest(string recordingId, HttpRequest incomingRequest, HttpResponse outgoingResponse) { - await DebugLogger.LogRequestDetailsAsync(incomingRequest); - if (!PlaybackSessions.TryGetValue(recordingId, out var session)) { throw new HttpException(HttpStatusCode.BadRequest, $"There is no active playback session under recording id {recordingId}."); } + var sanitizers = session.AdditionalSanitizers.Count > 0 ? Sanitizers.Concat(session.AdditionalSanitizers) : Sanitizers; + + DebugLogger.LogRequestDetails(incomingRequest, sanitizers); + var entry = (await CreateEntryAsync(incomingRequest).ConfigureAwait(false)).Item1; // If request contains "x-recording-remove: false", then request is not removed from session after playback. @@ -532,6 +536,16 @@ public async Task WriteBodyBytes(byte[] bodyData, int playbackResponseTime, Http } public static async Task<(RecordEntry, byte[])> CreateEntryAsync(HttpRequest request) + { + var entry = CreateNoBodyRecordEntry(request); + + byte[] bytes = await ReadAllBytes(request.Body).ConfigureAwait(false); + entry.Request.Body = CompressionUtilities.DecompressBody(bytes, request.Headers); + + return (entry, bytes); + } + + public static RecordEntry CreateNoBodyRecordEntry(HttpRequest request) { var entry = new RecordEntry(); entry.RequestUri = GetRequestUri(request).AbsoluteUri; @@ -545,10 +559,7 @@ public async Task WriteBodyBytes(byte[] bodyData, int playbackResponseTime, Http } } - byte[] bytes = await ReadAllBytes(request.Body).ConfigureAwait(false); - - entry.Request.Body = CompressionUtilities.DecompressBody(bytes, request.Headers); - return (entry, bytes); + return entry; } #endregion diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Startup.cs b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Startup.cs index 26ac09a4f4b..5508a54b2b9 100644 --- a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Startup.cs +++ b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Startup.cs @@ -21,9 +21,9 @@ using System.Linq; using System.CommandLine; using Azure.Sdk.Tools.TestProxy.CommandOptions; -using Microsoft.AspNetCore.Identity; -using Microsoft.CodeAnalysis.VisualBasic.Syntax; using System.Text.Json; +using Microsoft.Extensions.Logging.Console; +using Microsoft.Extensions.Options; namespace Azure.Sdk.Tools.TestProxy { @@ -147,9 +147,12 @@ private static void StartServer(StartOptions startOptions) { loggingBuilder.ClearProviders(); loggingBuilder.AddConfiguration(hostBuilder.Configuration.GetSection("Logging")); - loggingBuilder.AddSimpleConsole(formatterOptions => + loggingBuilder.AddConsole(options => { - formatterOptions.TimestampFormat = "[HH:mm:ss] "; + options.LogToStandardErrorThreshold = LogLevel.Error; + }).AddSimpleConsole(options => + { + options.TimestampFormat = "[HH:mm:ss] "; }); loggingBuilder.AddDebug(); loggingBuilder.AddEventSourceLogger(); diff --git a/tools/test-proxy/documentation/_images/assets_on_disk.png b/tools/test-proxy/documentation/_images/assets_on_disk.png new file mode 100644 index 00000000000..45fcbb77d1a Binary files /dev/null and b/tools/test-proxy/documentation/_images/assets_on_disk.png differ diff --git a/tools/test-proxy/documentation/_images/example_assets_store.png b/tools/test-proxy/documentation/_images/example_assets_store.png deleted file mode 100644 index 8535e01af06..00000000000 Binary files a/tools/test-proxy/documentation/_images/example_assets_store.png and /dev/null differ diff --git a/tools/test-proxy/documentation/_images/organization_of_assets.png b/tools/test-proxy/documentation/_images/organization_of_assets.png index 8ebb183e648..4d2348e5a21 100644 Binary files a/tools/test-proxy/documentation/_images/organization_of_assets.png and b/tools/test-proxy/documentation/_images/organization_of_assets.png differ diff --git a/tools/test-proxy/documentation/asset-sync/README.md b/tools/test-proxy/documentation/asset-sync/README.md index a92c52f5615..89039a6ea2d 100644 --- a/tools/test-proxy/documentation/asset-sync/README.md +++ b/tools/test-proxy/documentation/asset-sync/README.md @@ -185,7 +185,7 @@ First, ensure that your language-specific "shim" supports the automatic addition - [PR Enabling in JS](https://github.com/Azure/azure-sdk-for-js/pull/23405) - [PR Enabling in Go](https://github.com/Azure/azure-sdk-for-go/pull/19322) -Use [the transition script](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/testproxy/transition-scripts/generate-assets-json.ps1) and follow the [readme](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/testproxy/transition-scripts/README.md)! +Use [the transition script](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/testproxy/onboarding/generate-assets-json.ps1) and follow the [readme](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/testproxy/transition-scripts/README.md)! In summary, once an assets.json is present, the shim _must_ be updated to **actually send** a reference to that assets.json inside the `record/start` or `playback/start` requests! @@ -197,19 +197,17 @@ In summary, once an assets.json is present, the shim _must_ be updated to **actu Once a package or service has an assets.json and a targeted tag, each language-repo test framework will automatically provide this assets.json alongside the recording file path. This will allow the test-proxy to automatically restore recordings as necessary. -Here is the result of running tests in `playback`-mode for a couple packages within the python repo: +In practice, this is what a code repo will look like after running `playback`-mode for a couple packages. Specifically, this is for the python repo. -![assets store](../_images/example_assets_store.png) +![assets store](../_images/assets_on_disk.png) One can see the automatically restored assets repos within the `.assets` folder. Each of the top-level folders within the `.assets` folder contains a single slice of the assets repo. -The below diagram illustrates how an individual assets.json, language repo, and assets repo relate to each other. Text that appears as a specific color can be traced to its source in the assets.json. +The below diagram illustrates how an individual assets.json, language repo, and assets repo relate to each other. -![assets diagram](../_images/organization_of_assets.png) +![assets diagram](../_images/organization_of_assets1.png) -> Side note: the `.breadcrumb` file is created/updated as an artifact of the test-proxy restore/push/reset operations. Don't look for one if you haven't restored at least one assets.json first! - -One can use visual inspection of the `.breadcrumb` file to _find_ which folder contains the files for your assets.json. Or, more conveniently, a user can use the `config` verb to access this data! Using assets diagram directly above. we can work an example: +A user can use the `config` verb to access this the location of their assets on disk! Using assets diagram directly as a reference. we can work an example: ```powershell # from the root of the azure-sdk-for-net repo, run: diff --git a/tools/test-proxy/tests.yml b/tools/test-proxy/tests.yml index dc9a4b3410d..c4f3c0a763c 100644 --- a/tools/test-proxy/tests.yml +++ b/tools/test-proxy/tests.yml @@ -169,6 +169,50 @@ stages: env: PROXY_MANUAL_START: "true" + - job: Test_Net_Core + + displayName: Invoke .NET Azure.Core.Testframework CI Tests + + strategy: + matrix: + Windows: + Pool: 'azsdk-pool-mms-win-2022-general' + OS: 'Windows' + Linux: + Pool: azsdk-pool-mms-ubuntu-2204-general + OS: 'Linux' + Mac: + Pool: 'Azure Pipelines' + OS: 'Mac' + + pool: + name: $(Pool) + + variables: + REPO: "Azure/azure-sdk-for-net" + CLONE_LOCATION: "$(Agent.BuildDirectory)/net_repo" + + steps: + - template: /eng/pipelines/templates/steps/install-dotnet.yml + + - pwsh: | + git clone https://github.com/$(REPO) $(CLONE_LOCATION) --depth 1 + displayName: Clone Repo + + - template: /eng/pipelines/templates/steps/test-proxy-local-tool.yml + parameters: + runProxy: true + rootFolder: $(CLONE_LOCATION) + + - pwsh: | + dotnet test + displayName: 'Invoke Tests' + workingDirectory: $(CLONE_LOCATION)/sdk/core/Azure.Core.TestFramework + env: + PROXY_MANUAL_START: "true" + PROXY_DEBUG_MODE: "true" + + - job: Test_Python_Tables displayName: Run Python Tables CI Tests