-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] Smoke Test Pipeline Should Report Metrics on Application Sizes #3600
Comments
Alright so the SmokeTests drop to Each |
I think the strategy here I'm going to follow is to index all the files/sizes in the UWP Baseline project. Then for each other project, find the outliers to that list and report what those are and their sizes above the baseline. This should give us not only the overall footprint each package adds to the toolkit, but it should also help us understand other dependencies each of those packages bring and how they contribute to that total. I'll then try and output that to the console. Luckily this is something I can easily test locally. I'm going to make it a separate PowerShell script in the SmokeTests project, and then we can just call it from the DevOps YAML file after the Smoke Tests run. @azchohfi do you know if there's a known location where we can store a file/metadata associated the last main build in DevOps or would we have to check it in somewhere to some repo? Curious if in the future we could have the script fetch that from somewhere and compare against it to know how things are changing for a given PR? |
Not sure what the best way to do this would be. We could upload an xml or json file with these results somewhere and use that, but I don't know if we could just use the same repo for that. Not sure if we could use the ADO artifacts for that either, since there is a retention policy for builds. Maybe a release pipeline could push this to an Azure storage, and then the PRs pipelines could compare it with a known URL. I'm not sure if there is a more standard way of doing this. Lets brainstorm. |
@azchohfi Hmm, was just messing with the |
Not sure, but it seems like it might work. |
Right now it just outputs to the host, need to add more to actually spit out a file that we could compare against. When we do that, we can also output this cleaner as a table and have an extra column for the comparison. And we could colorize it fancier too! https://stackoverflow.com/a/49038815/8798708 |
…and package sizes Provides a more human readible way to understand our package sizes and dependencies against a baseline blank application See more info in CommunityToolkit#3600 for potential improvements
…and package sizes Provides a more human readible way to understand our package sizes and dependencies against a baseline blank application See more info in CommunityToolkit#3600 for potential improvements
## Fixes #3560 This PR does the following: - [x] Split our tests out of the single Job into two separates Jobs, these can hopefully run in parallel? We should also get separate status checks for each job? (Will see when we run this here.) - [x] Adds a UWP Baseline Blank App, this will be useful for #3600 to compare what the minimum app size is currently with no extra dependencies. - [x] Split out UI Tests as a separate step from the Unit Tests - [x] Add Application Metrics Report to Smoke Tests #3600 (could be a separate PR I suppose) ## PR Type - Build or CI related changes ## What is the current behavior? Everything is done in one big job. ## What is the new behavior? We have more granularity on our pipeline ## PR Checklist Please check if your PR fulfills the following requirements: - [ ] Tested code with current [supported SDKs](../readme.md#supported) - [ ] Pull Request has been submitted to the documentation repository [instructions](..\contributing.md#docs). Link: <!-- docs PR link --> - [ ] Sample in sample app has been added / updated (for bug fixes / features) - [ ] Icon has been created (if new sample) following the [Thumbnail Style Guide and templates](https://github.com/windows-toolkit/WindowsCommunityToolkit-design-assets) - [ ] Tests for the changes have been added (for bug fixes / features) (if applicable) - [ ] Header has been added to all new source files (run *build/UpdateHeaders.bat*) - [ ] Contains **NO** breaking changes <!-- If this PR contains a breaking change, please describe the impact and migration path for existing applications below. Please note that breaking changes are likely to be rejected within minor release cycles or held until major versions. --> ## Other information
Describe the problem this feature would solve
Besides just checking the final output of the NuGet package, one reason for the Smoke Tests are to help us understand the impact the toolkit packages have on the final size of an application.
While the Smoke Tests currently run in the CI and generate these independent apps. The size isn't currently reported back in a nicely consumable way either at the end of that job or with some sort of bot back to the issue itself.
Describe the solution
MVP: When smoke test is completed, it lists a table at the end of each package and the resulting app size to the console. This way we can scroll to the end of the job and get that data when needed.
Nice to haves:
Additional context & Screenshots
Somewhat related to #3560 as well. And to address being able to check and help with #3145 more explicitly.
The text was updated successfully, but these errors were encountered: