ci: do not test a config that will never succeed #388
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this test, we are asserting that the instrumentation is not
installed when the
Rack
constant is not present (see thebefore
block for this test case). We then go on to test that the configuration
is the "default" configuration that you'd get with a fresh installation.
The problem is that if the
Rack
constant is not present, then theinstrumentation-base
code that sets all the defaults for our optionsis never run (and logically, why would it?). So these test lines can
never actually succeed.
Unless, of course, the
instrumentation
object we're referring to isnot a pristine, new object. And in fact, depending on what order the
tests run in, our object is not a pristine, new object. If you run
basically any other test in this suite before, then we actually end up
recycling an object that is partially initialzied from a previous test,
and has an internal
@config
object that has some default options.And so, the test is actually passing some times because of this
non-deterministic behavior. For example, if you run with
SEED=1
, thenthe test suite passes (because other tests run first that initialize the
object completely). If you run with
SEED=6372
, it fails (because it isthe very first test run).
The real bug here is that we do not have proper test isolation. I
think that's actually a problem all over the code base, if I'm being
honest about it.
However, I elect not to fix that problem today. We'd need some way to
"reset" instrumentation and the registry in between tests (maybe not
that hard, honestly). That's more than I want to do on a Saturday
afternoon. So to fix the current issue, we just don't bother testing
things that logically could never pass anyways. What we actually care
about is that the instrumentation reports it was not installed, which
does work correctly as it is.
Fixes #387