Fix flakey tentacle command line tests

[sburmanoctopus]

[sc-65398]

Background

The TentacleCommandLineTests.ShouldLogStartupDiagnosticsToInstanceLogFileOnly test was flaky. This needed to be fixed.

Results

Before

The test would create an instance of Tentacle by running Tentacle.exe (setting everything up). Then, while this instance was running, it would run the show-thumbprint command against the same executable, creating a second instance.

What we believe was causing the failure was that the first instance was writing to the log file while the second instance was starting up. This would, lock the log file and prevent the second instance from writing logs.

This test does its assertion by looking at the log file. Since the second instance logs were not being logged, it would therefore fail.

One would expect the second instance to throw an error when file locking occurs. However, we have configured NLog with throwExceptions="false", so if any errors are thrown while logging, it will simply not log and continue as if nothing has happened. This is great for customers, but not for finding test issues.

After

To solve this, we stop the first instance of the running tentacle before we attempt to run the command. This will prevent the file lock, and will therefore allows the logs to be populated as expected.

To make the other tests in this file safer, we also stopped the initial Tentacle instance for each test that also ran a second instance over the first.

To help diagnose future issues, we have also set throwExceptions="true" for tests only. We proved this does indeed work with this test run (where we purposely locked the file)

How to review this PR

Quality ✔️

Pre-requisites

• I have read How we use GitHub Issues for help deciding when and where it's appropriate to make an issue.
• I have considered informing or consulting the right people, according to the ownership map.
• I have considered appropriate testing for my change.

[shortcut-integration]

This pull request has been linked to Shortcut Story #65398: Fix flakey tentacle command line tests.

[sburmanoctopus]

This was the test that was flaky

[sburmanoctopus]

FYI, turns out this was not the primary reason for the flaky test.

The main reason was that when Tentacle starts up, it checks to see if it CanLoadCurrentInstance before configuring logging.

If that fails, it will fall back to logging into the default log file (that all tentacles log into). This is what happened when the logs were not showing up.

CanLoadCurrentInstance was claiming to not be able to load an instance because it loads all instances before it tries to find the instance we are looking for. So when another test is manipulating that directory (e.g. deleting instances), we may encounter an exception from that one unrelated instance.

This PR will solve that issue for tests.

[nathanwoctopusdeploy]

LGTM

[sburmanoctopus]

We decided to remove throwExceptions="true" adjustment for testing, as it was causing a test to fail because NLog was trying to create the "Octopus Tentacle" event source, and it already existed. This is not an issue normally, as throwExceptions="false" normally.

It was considered not worth investing time to fix this. So we deleted this enhancement.