Go recorded test framework

[christothes]

Merging the testFramework module to master.

[mohsha-msft]

So far the test has been able to work perfectly. Appreciate your work @christothes. There are few questions I am curious about

I see we're using DefaultMatchers here. While the default matchers work perfectly in most of the cases, there are few cases where I believe users may need to use custom matchers. So they should be provided options to supply their custom matchers.

Here are the couple of scenarios which I found while working on integrating testframework in azblob SDK

1. For upload/download 1MB/2MB/4MB/8MB file to Azure Service (which is a common scenario), the recording file is storing body in request/response which is increasing the size of file and make it infeasible to keep in the repo. And depending on the use-case, I may not be interesting in comparing body content at all.
2. While dealing with Shared Access Signature (SAS, I want to omit the request URL and ignore it from comparisons because those URLs hold secret. I want to provide my custom headers to match apart from

var ignoredHeaders = map[string]*string{
	"Date":                   nil,
	"X-Ms-Date":              nil,
	"x-ms-date":              nil,
	"x-ms-client-request-id": nil,
	"User-Agent":             nil,
	"Request-Id":             nil,
	"traceparent":            nil,
	"Authorization":          nil,
}

supplied in DefaultMatchers

matcher := &RequestMatcher{
		context:        testContext,
		IgnoredHeaders: ignoredHeaders,
	}

[mohsha-msft]

Also curious on your opinion on whether we can omit body from the recording as well

[christothes]

Some of the most recent commits on this PR added the ability to configure custom matchers. For example, see the SetBodyMatcher func here.

For setting custom headers to ignore for matching, you'd add items to the IgnoredHeaders map

[christothes]

As far as eliminating the need to record the body content at all, I think this should be acheivable by setting a custom BodySanitizer where the "sanitization" is to simply return and empty object (or perhaps one that contains only a single "sanitized" property to make it clear that it was due to sanitization and not a truly empty body)

[jhendrixMSFT]

Here's an observation I had while testing this out. I don't know what the fix looks like, perhaps it's just documentation/examples.

I wrote a test for azidentity that uses recordings, retrieving an access token for client secret credentials. Using a Secret_String for the tenant ID is problematic as this is part of the request URL so there's no match during play-back.

Changing the tenant ID to Default works around the above (not sure this is how we'd want to check it in though). Now, playback fails due to a mismatch of Content-Length headers. I believe this is because the sanitized client ID and secrets are used in the request body which is different than the recording.

The recording contained the credentials as well as the access token which we'd obviously not want to commit, or even expose in a PR.

I suppose these problems might be unique to azidentity since it's directly testing endpoints with sensitive data, and will require heavy use of custom matchers etc.

[christothes]

I wrote a test for azidentity that uses recordings, retrieving an access token for client secret credentials. Using a Secret_String for the tenant ID is problematic as this is part of the request URL so there's no match during play-back.

For this scenario I think you'd want to set a custom URLSanitizer.

azure-sdk-for-go/sdk/internal/testframework/recording_sanitizer.go

Line 52 in 438d7e1

func (s *RecordingSanitizer) AddUrlSanitizer(sanitizer StringSanitizer) {

The recording contained the credentials as well as the access token which we'd obviously not want to commit, or even expose in a PR.

This should just require setting a custom body sanitizer that knows how to parse the token in the response and replace it with a sanitized value.

azure-sdk-for-go/sdk/internal/testframework/recording_sanitizer.go

Line 47 in 438d7e1

func (s *RecordingSanitizer) AddBodysanitizer(sanitizer StringSanitizer) {

I suppose these problems might be unique to azidentity since it's directly testing endpoints with sensitive data, and will require heavy use of custom matchers etc.

Actually it should be custom sanitizers you need here. Although I don't think the sanitization is all that complex in these cases.

[catalinaperalta]

Actually it should be custom sanitizers you need here. Although I don't think the sanitization is all that complex in these cases.

I just wanted to chime in here since I also worked on getting some recordings set up for azidentity. Aside from the custom sanitizers, there needs to be a custom request matcher due to the problem Joel pointed out about the mismatched headers.

[christothes]

Actually it should be custom sanitizers you need here. Although I don't think the sanitization is all that complex in these cases.

I just wanted to chime in here since I also worked on getting some recordings set up for azidentity. Aside from the custom sanitizers, there needs to be a custom request matcher due to the problem Joel pointed out about the mismatched headers.

Could you give an example of where sanitization wouldn't do the right thing in terms of header matching?

[catalinaperalta]

Could you give an example of where sanitization wouldn't do the right thing in terms of header matching?

I suppose you could just sanitize all the headers. I just recall tracking this down to the matcher and for a header that azidentity doesnt particularly change...if I recall correctly.

[christothes]

Could you give an example of where sanitization wouldn't do the right thing in terms of header matching?

I suppose you could just sanitize all the headers. I just recall tracking this down to the matcher and for a header that azidentity doesnt particularly change...if I recall correctly.

Hmm - no big deal, but if you happen to find the real example of this, I'd be curious in which scenario headers wouldn't match regardless if they were sanitized.

[check-enforcer]

This repository is protected by Check Enforcer. The check-enforcer check-run will not pass until there is at least one more check-run successfully passing. Check Enforcer supports the following comment commands:

• /check-enforcer evaluate; tells Check Enforcer to evaluate this pull request.
• /check-enforcer override; by-pass Check Enforcer (approvals still required).