Added PlaybackOnlyAttribute #12663
Added PlaybackOnlyAttribute #12663
Conversation
|
Why do you want to ever avoid running these tests live? |
|
Read through the test. Would we ever reenable it? If we ever add this attribute I think it needs to take a reason for playback only. |
In the case of ORS, we would have to take a dependency on the Storage Management SDK to set up and tear down the ORS policy with every test run. This turned out to be a non-viable solution, so instead, we are going to run the tests in playback only. We've had a few other similar cases, where it's not worth the effort to set up the infrastructure to live test a minor feature. @tg-msft recommended we create this annotation at one point. |
|
Another example is #12623. BlockBlobStorage storage accounts are legency, you can't even create one in the Azure Portal. I'm pretty sure its not possible to create one via ARM templates. To sidestep this, we gain access to a BlockBlobStorage account, create the test, record it, and we're good. |
| /// Attribute on test assemblies, classes, or methods that run only against playback resources. | ||
| /// </summary> | ||
| [AttributeUsage(AttributeTargets.Method | AttributeTargets.Class | AttributeTargets.Assembly, AllowMultiple = true, Inherited = true)] | ||
| public class PlaybackOnlyAttribute : NUnitAttribute, IApplyToTest |
There was a problem hiding this comment.
Would this prevent someone from being able to re-record the test just by changing the ctor parameter?
There was a problem hiding this comment.
Which constructor parameter are you referring to?
There was a problem hiding this comment.
There was a problem hiding this comment.
I assume so, yes, but this is by design. The workflow to re-record would be to comment out the attribute.
| /// </summary> | ||
| [AttributeUsage(AttributeTargets.Method | AttributeTargets.Class | AttributeTargets.Assembly, AllowMultiple = true, Inherited = true)] | ||
| public class PlaybackOnlyAttribute : NUnitAttribute, IApplyToTest | ||
| { |
There was a problem hiding this comment.
Let's add a "reason" required ctor parameter.
|
I was thinking of something a little more like a |
|
We can also skip the test dynamically based on the existence of particular environment setting. |
|
As long as the env setting was present in our DevOps pipeline because I'm pretty sure those machines are refurbished raspberry pi zeros and struggle getting some of these longer tests to finish before our timeout kills the whole run. |
|
Another example of why we need this functionality - https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/storage/Azure.Storage.Blobs/tests/PageBlobClientTests.cs#L1057. It's not programmatically possible to create a managed disk storage account. That test will never run in live mode. |
|
@seanmcc-msft what are your thoughts about #12663 (comment)? |
I don't think I fully understand that comment. My objective is to have the ability to have tests that only run in playback mode, both locally and in the pipeline. These tests would rarely be re-recorded, and never run live. I never want to skip these tests in playback mode, and always skip them in recorded and live modes. To re-record, we'd comment out the annotation. |
We would like the ability to mark some tests as playback-only.
For example: https://github.com/Azure/azure-sdk-for-net/pull/12609/files#diff-e152777102db76590298fbda43777754R1769
Related #12609