Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

choco outdated.outdated ignore-pinned uses correct enhanced exit codes Pester tests fails in some Test Kitchen scenarios #2801

Closed
corbob opened this issue Aug 22, 2022 · 1 comment · Fixed by #2802
Closed

choco outdated.outdated ignore-pinned uses correct enhanced exit codes Pester tests fails in some Test Kitchen scenarios #2801

corbob opened this issue Aug 22, 2022 · 1 comment · Fixed by #2802
Assignees
@corbob
Labels
5 - Released NO RELEASE NOTES Should not be included in the release notes - not enhancing or fixing end product. Tests Issues related to the tests (NUnit or Pester)
Milestone
1.2.0

Comments

@corbob
Copy link
Member

corbob commented Aug 22, 2022

What You Are Seeing?

When Team City kicks off a Test Kitchen run for a freshly completed Chocolatey component version, and while the Test Kitchen is running a new version of the component is published to internal feeds, this particular test reports a failure:

Expected 0, because Chocolatey v1.2.0-alpha-20220813-1f34adf2 Business
Outdated Packages
 Output is package name | current version | available version | pinned?
chocolatey-agent|1.1.0-alpha-20220818-67|1.1.0-alpha-20220818-68|false
Chocolatey has determined 1 package(s) are outdated., but got 2.
at $Output.ExitCode | Should -Be $ExitCode -Because $Output.String, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26
at <ScriptBlock>, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26

What is Expected?

This test should not fail, because in this case the package is expected to be outdated.

How Did You Get This To Happen? (Steps to Reproduce)

In this specific instance:

  1. Chocolatey Agent PR was merged to develop branch
  2. Team City kicks off the chained Test Kitchen build
  3. Another Chocolatey Agent PR was merged to develop branch while the Test Kitchen build is running
  4. Test Kitchen fails this specific test
  5. Team City kicks off a chained Test Kitchen build for step 3's version
  6. Test Kitchen succeeds this test (assuming no further merges to develop

System Details

Test Kitchen related, occurs on both 2016 and 2019 kitchens

Output Log

Output from the Team City email:

 Failed tests summary: 2 (2 new)
(new) Pester - <chocolatey-service-tests-windows-2016>: choco outdated.outdated ignore-pinned uses correct enhanced exit codes.Exits with correct exit code (0) 		details »
(new) Pester - <chocolatey-service-tests-windows-2019>: choco outdated.outdated ignore-pinned uses correct enhanced exit codes.Exits with correct exit code (0) 		details »

Stacktraces:
(new) Pester - <chocolatey-service-tests-windows-2016>: choco outdated.outdated ignore-pinned uses correct enhanced exit codes.Exits with correct exit code (0)

Expected 0, because Chocolatey v1.2.0-alpha-20220813-1f34adf2 Business
Outdated Packages
 Output is package name | current version | available version | pinned?
chocolatey-agent|1.1.0-alpha-20220818-67|1.1.0-alpha-20220818-68|false
Chocolatey has determined 1 package(s) are outdated., but got 2.
at $Output.ExitCode | Should -Be $ExitCode -Because $Output.String, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26
at <ScriptBlock>, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26


(new) Pester - <chocolatey-service-tests-windows-2019>: choco outdated.outdated ignore-pinned uses correct enhanced exit codes.Exits with correct exit code (0)

Expected 0, because Chocolatey v1.2.0-alpha-20220813-1f34adf2 Business
Outdated Packages
 Output is package name | current version | available version | pinned?
chocolatey-agent|1.1.0-alpha-20220818-67|1.1.0-alpha-20220818-68|false
Chocolatey has determined 1 package(s) are outdated., but got 2.
at $Output.ExitCode | Should -Be $ExitCode -Because $Output.String, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26
at <ScriptBlock>, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26

Proposed solution: Maintain a list of internal Chocolatey components that are pinned during the setup of these tests. As they sometimes encounter the described scenario, they should rightly be pinned for this particular test.
@corbob corbob added 0 - _Triaging Tests Issues related to the tests (NUnit or Pester) labels Aug 22, 2022
@corbob corbob self-assigned this Aug 22, 2022
corbob added a commit to corbob/choco that referenced this issue Aug 22, 2022
@corbob
(chocolatey#2801) Pin Chocolatey packages for outdated tests
833b5bf 
Pin the chocolatey packages so that they don't inadvertently interfere
with the outdated command tests.
@corbob corbob mentioned this issue Aug 22, 2022
Merged
9 tasks
corbob added a commit to corbob/choco that referenced this issue Sep 26, 2022
@corbob
(chocolatey#2801) Pin Chocolatey packages for outdated tests
3df174a 
Pin the chocolatey packages so that they don't inadvertently interfere
with the outdated command tests.
@AdmiringWorm AdmiringWorm added 4 - Done NO RELEASE NOTES Should not be included in the release notes - not enhancing or fixing end product. and removed 2 - Working labels Sep 27, 2022
@AdmiringWorm AdmiringWorm added this to the 1.2.0 milestone Sep 27, 2022
AdmiringWorm added a commit to AdmiringWorm/choco that referenced this issue Oct 19, 2022
@AdmiringWorm
Merge branch 'release/1.2.0'
edde979 
* release/1.2.0: (374 commits)
  (maint) Add incoming arguments available in Licensed Extension
  (chocolatey#798) Add missing pin option in tab completion
  (chocolatey#2860) Add e2e tests for new PS cmdlet
  (maint) Fix Chocolatey path in docs generation
  (chocolatey#2860) Add new PS helper to get maintainer specific paths
  (chocolatey#2816) Update to latest Chocolatey NuGet.Core
  (chocolatey#2693) Re-add old resolve or load method
  (chocolatey#886) Update packages.config install logging
  (chocolatey#1185) Always pass in hook scripts event if empty
  Update tests/chocolatey-tests/commands/choco-outdated.Tests.ps1
  (chocolatey#2801) Pin Chocolatey packages for outdated tests
  (chocolatey#886) Use specific strings for configurations
  (chocolatey#886) Add Pester tests for packages.config
  (chocolatey#886) Debug log Package configuration
  (chocolatey#1185) Be explicit with expected exit codes
  (maint) Pester test directory structure shuffle
  (chocolatey#1185) Add tests for hook scripts
  (maint) Minor formatting change
  (chocolatey#798) Add tests for pinning package on install
  (chocolatey#2787) Restore Chocolatey snapshot before sxs
  ...
@AdmiringWorm
Copy link
Member

🎉 This issue has been resolved in version 1.2.0 🎉

The release is available on:

Your GitReleaseManager bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@corbob corbob

Labels
5 - Released NO RELEASE NOTES Should not be included in the release notes - not enhancing or fixing end product. Tests Issues related to the tests (NUnit or Pester)
Projects
None yet
Milestone
1.2.0
Development

Successfully merging a pull request may close this issue.

(#2801) Pin Chocolatey packages for outdated tests corbob/choco
2 participants
@AdmiringWorm @corbob