feat: mutator tracing

[alnr]

#1001

This slightly refactors #1049 and keeps more of the timeout behavior unchanged.

[codecov]

Codecov Report

Merging #1050 (ddda29c) into master (8f42940) will decrease coverage by 0.35%.
The diff coverage is 69.23%.

@@            Coverage Diff             @@
##           master    #1050      +/-   ##
==========================================
- Coverage   78.09%   77.74%   -0.36%     
==========================================
  Files          83       83              
  Lines        3977     3841     -136     
==========================================
- Hits         3106     2986     -120     
+ Misses        593      578      -15     
+ Partials      278      277       -1     

Impacted Files	Coverage Δ
driver/registry_memory.go	93.75% <ø> (+3.92%)	⬆️
x/registry.go	0.00% <0.00%> (ø)
pipeline/mutate/mutator_hydrator.go	67.82% <81.81%> (+1.73%)	⬆️
internal/driver.go	77.77% <0.00%> (-22.23%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[aeneasr]

Just one minor thing for tests otherwise looks good!

[alnr]

• updated config.schema.json
• bumped ory/x
• reverted test change (that snuck in during 7c7f21f)

[zepatrik]

IMO this needs to be moved one scope up, like it was before, so that the default is also set when the config is not set.

[zepatrik]

And we need a test for this, so e.g. only setting cfg.Api.Retry.MaxDelay, and expecting the giveUpAfter default to be set.

[alnr]

After looking at this in more detail, the retry config was completely broken in 2a97e05.

I'll revisit this...

[alnr]

Alrighty... the retry configuration is broken all over Oathkeeper. Here's my understanding of what is currently happening:

• The give_up_after property specifies the per-retry timeout. That's surprising to me, since it sounds like it means the overall timeout, including retries.
• The max_delay specifies the maximum backoff wait time in between retries.
• The maximum number of retries is not configurable, and is always 4 (the default in hashicorp/retryablehttp).
• This config migration is incorrect. It thinks give_up_after is the overall timeout (including retries).

Not sure now is the time to fix this fully, since that could also break users.

With the goal of moving this PR over the line, I propose to keep as much backward-compatibility as possible. This particular mutator previously had no overall timeout and no retries if the retry config was not specified at all. If the retry config was specified (even if {}), it peformed up to 4 retries. If the retry config is empty ({}), the timeouts also changed.

This patch keeps that behavior, while fixing bugs:

• the fallback timeouts (retry config present but empty) were wrong: 50ms timeout and 1s delay, even though the config schema promises a more reasonable 1s timeout and 100ms delay. Those defaults now at least work as advertised.
• faulty error messages

We should consider revamping the timeout handling all over Oathkeeper in a follow-up ticket.

Thoughts? @zepatrik @aeneasr @daviddelucca ?

[daviddelucca]

This PR is related to activate tracing on mutator. I think we could merge it and open another ticket/branch to work on revamping on timeouts.

Does make sense?

[zepatrik]

Nice, I agree to merge this as is and fix the other issues you found in a separate PR 👍

[alnr]

This needs more work to synchronize the tracing config schemata with ory/x.

[alnr]

I've synced the tracing config schema with the one from ory/x. Initially, I tried to reference it by URL, but that (logically) requires Oathkeeper to load the remote schema on startup, which is often not possible due to network segmentation.

@daviddelucca can you try this branch, please?