From c17d841cc558a9b85f732272bdfac8f0bcb97e97 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 18 Oct 2021 21:45:00 -0700 Subject: [PATCH 01/45] draft tracestate probability sampling spec --- .../trace/tracestate-probability-sampling.md | 266 ++++++++++++++++++ 1 file changed, 266 insertions(+) create mode 100644 specification/trace/tracestate-probability-sampling.md diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md new file mode 100644 index 00000000000..784de142c75 --- /dev/null +++ b/specification/trace/tracestate-probability-sampling.md @@ -0,0 +1,266 @@ +# TraceState: Probability Sampling + + + +- [Definitions Used in this Document](#definitions-used-in-this-document) + * [Sampling](#sampling) + * [Sampler](#sampler) + * [Parent-based sampler](#parent-based-sampler) + * [Probability sampler](#probability-sampler) + * [Consistent probability sampler](#consistent-probability-sampler) + * [Always-on sampler](#always-on-sampler) + * [Always-off sampler](#always-off-sampler) + * [Non-probability sampler](#non-probability-sampler) + * [Adjusted count](#adjusted-count) + * [Unbiased probability sampling](#unbiased-probability-sampling) + * [Power-of-two random sampling](#power-of-two-random-sampling) + + + +**Status**: [Experimental](../document-status.md) + +Probability sampling allows OpenTelemetry tracing users to lower their +collection costs with the use of randomized sampling techniques. +OpenTelemetry specifies how to convey and record the results of +probability sampling using the W3C `tracestate` in a way that allows +Span-to-Metrics pipelines to be built that accurately count sampled +spans. + +The specification in this document is semantic in nature. Two +`tracestate` fields, known as "r-value" and "p-value", are defined to +enable the development of interoperable probability Sampler +implementations. OpenTelemetry is gathering experience with Samplers +based on this specification while the group considers how to add +probability sampling support in the default SDKs, via the specified +built-in Samplers. + +## Definitions Used in this Document + +### Sampling + +Sampling is a family of techniques for collecting and analyzing only a +fraction of a complete data set. Individual items that are "sampled" +are taken to represent one or more spans when collected and counted. +The representivity of each span is used in a Span-to-Metrics pipeline +to accurately count spans. + +Sampling terminology uses "population" to refer to the complete set of +data being sampled from. In OpenTelemetry tracing, "population" +refers to all spans. + +In probability sampling, the representivity of individual sample items +is generally known, whereas OpenTelemetry also recognizes +"non-probability" sampling approaches, in which representivity is not +explicitly quantified. + +### Sampler + +A Sampler provides configurable logic, used by the SDK, for selecting +which Spans are "recorded" and/or "sampled" in a tracing client +library. To "record" a span means to build a representation of it in +the client's memory, which makes it eligible for being exported. To +"sample" a span implies setting the W3C `sampled` flag, recording the +span, and exporting the span when it is finished. + +OpenTelemetry supports spans that are "recorded" and not "sampled" +for in-process observability of live spans (e.g., z-pages). + +The Sampler interface and the built-in Samplers defined by +OpenTelemetry must be capable of deciding immediately whether to +sample a span, since the child context immediately propagates the +decision. + +### Parent-based sampler + +A Sampler that makes its decision to sample based on the W3C `sampled` +flag from the context is said to use parent-based sampling. + +### Probability sampler + +A probability Sampler is a Sampler that knows immediately, for each +of its decisions, the probability that the span had of being selected. + +Sampling probability is defined as a number less than or equal to 1 +and greater than 0 (i.e., `0 < probability <= 1`). The case of 0 +probability is treated as a special, non-probabilistic case. + +### Consistent probability sampler + +A consistent probability sampler is a Sampler that supports independent +sampling decisions at each span in a trace while maintaining that +traces will be complete with probability equal to the minimum sampling +probability across the trace. Consistent probability sampling requires that +for any span in a given trace, if a Sampler with lesser sampling probability +selects the span for sampling, then the span would also be selected by a +Sampler configured with greater sampling probability. + +In OpenTelemetry, consistent probability samplers are limited to +power-of-two probabilities. OpenTelemetry consistent probability +sampling is defined in terms of "p-value" and "r-value", both +integers, which are propagated via the context to assist in making +consistent sampling decisions. + +### Always-on sampler + +An always-on sampler is another name for a consistent probability +sampler with probability equal to one. + +### Always-off sampler + +An always-off Sampler has the effect of disabling a span completely, +effectively excluding it from the population. This is not defined as +a probability sampler with zero probability, because these spans are +effectively unrepresented. + +### Non-probability sampler + +A non-probability sampler is a Sampler that makes its decisions not +based on chance, but instead uses arbitrary logic and internal state. + +### Adjusted count + +Adjusted count is defined as a measure of representivity, the number +of spans in the population that are represented by the individually +sampled span. Span-to-metrics pipelines can be built by adding the +adjusted count of each sample span to a counter of matching spans. +Likewise, span-to-metrics pipelines can be built by observing the +duration of each sample span in a histogram, the span's adjusted count +number of times each. + +The adjusted count 1 means one-to-one sampling. Adjusted counts +greater than 1 indicate the use of a probability sampler. Adjusted +counts are unknown when using a non-probability sampler. + +Zero adjusted count is defined in a way that supports composition of +probability and non-probability samplers. In effect, spans that are +"recorded" but not "sampled" have adjusted count of zero. + +### Unbiased probability sampling + +The statistical term "unbiased" is a requirement applied to the +adjusted count of a span, which states that the expected value of the +sum of adjusted counts across all exported spans MUST equal the true +number of spans in the population. Statistical bias, a measure of the +difference between an estimate and its true value, of the estimated +span count in the population should equal zero. Moreover, this +requirement must be true for all subsets of the span population for a +sampler to be considered an unbiased probability sampler. + +Sampling schemes that are not explicitly unbiased should be +categorized as non-probability samplers because they cannot record +unbiased adjusted counts. Here are example Samplers that do not +qualify as unbiased: + +- A traditional form of "leaky-bucket" sampler applies a rate limit to + the starting of new sampled traces. When the configured limit is + not exceeded, all spans pass through with adjusted count 1. When + the configured rate limit is exceeded, it is impossible to set + adjusted count without introducing bias because future arrivals are + not known. +- A "every-N" sampler records spans on a regular interval, but instead + of making a probabilistic decision it makes an exact decision + (e.g., every 10,000 spans). This sampler knows the representivity + of the spans it samples, but the selection process is biased. +- A "at least once per time period" sampler remembers the last time + each distinct span name exported a span. When a span occurs after + more than the specified interval, it samples one (e.g., to ensure + that receivers know about these spans). This sampler introduces + bias because spans that happen between the intervals do not receive + consideration. +- The "always off" sampler is biased by definition. Since it exports + no spans, the sum of adjusted count is always zero. + +As an example, Simple Random Sampling is an unbiased sampling +algorithm. The algorithm is given a sampling probability `p` in the +interval `(0, 1]` and a source of randomness. Then for each item: + +1. Generate a uniform floating point value `r` in the range `[0, 1)` +2. If `r < p`, select the item with adjusted count `1 / p`. + +This algorithm is unbiased because every item in the population +receives equal consideration. + +### Power-of-two random sampling + +An unbiased sampling scheme can be implemented using a random bit +string as the input. This scheme is limited to power-of-two sampling +probabilities, as follows. + +1. Express the sampling probability as `2**-s`. For example, 25% + equals `2**-2` +2. Count `r`, the number of consecutive zero bits in the input string +3. If `s <= r`, select the item with adjusted coubnt `2**s`. + +This algorithm is the basis of the consistent probability sampling +approach used in OpenTelemetry, defined in greater detail below. + +## Probability sampling tracestate fields + +The consistent sampling scheme adopted by OpenTelemetry propagates two +values, p-value and r-value, via the context. + +1. r-value: this "randomness" value is determined and propagated from the root of the trace to all spans and serves to make sampling decisions consistent +2. p-value: the "parent probability" value can be modified by any span in the trace and informs child parent-based Samplers their adjusted count + +Both fields are propagated via the OpenTelemetry `tracestate` under +the `ot` vendor tag using [syntax defined +here](tracestate-handling.md). Both fields are represented as +unsigned six-bit integers (i.e., in the inclusive interval [0, 63]). + +### p-value + +The p-value SHOULD be set in the `tracestate` when an unbiased +probability sampler configured with power-of-two probability selected +the parent span for sampling. p-value SHOULD be set in the +`tracestate` when the W3C `sampled` flag is set in the corresponding +`traceparent`. Non-probability samplers, having unknown adjusted +count, SHOULD unset p-value when making sampling decisions. + +Zero adjusted count is represented by the special p-value 63. +Otherwise, the p-value is set to the negative base-2 logarithm of +sampling probability: + +| p-value | Parent Probability | +| ----- | ----------- | +| 0 | 1 | +| 1 | 1/2 | +| 2 | 1/4 | +| ... | ... | +| N | 2**-N | +| ... | ... | +| 61 | 2**-61 | +| 62 | 2**-62 | +| 63 | 0 | + +### r-value + +The r-value SHOULD be set in the `tracestate` by the Sampler at the +root of the trace in order to support consistent probability sampling. +When the value is omitted or not present, child spans in the trace are +not able to participate in consistent probability sampling. + +R-value determines which sampling probabilities and will not sample +for spans of a given trace, as follows: + +| r-value | Probability of r-value | Implied sampling probabilities | +| ---------------- | ------------------------ | ---------------------- | +| 0 | 1/2 | 1 | +| 1 | 1/4 | 1/2 and above | +| 2 | 1/8 | 1/4 and above | +| 3 | 1/16 | 1/8 and above | +| ... | ... | ... | +| 0 <= r <= 61 | 1/(2**(-r-1)) | 2**(-r) and above | +| ... | ... | ... | +| 59 | 2**-60 | 2**-59 and above | +| 60 | 2**-61 | 2**-60 and above | +| 61 | 2**-62 | 2**-61 and above | +| 62 | 2**-62 | 2**-62 and above | + +### Probability Sampler behavior + + + +#### Parent-based + +#### Consistent probability-based + From 37616be0125b267f48bc3916f53679625ed173aa Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Wed, 20 Oct 2021 10:01:29 -0700 Subject: [PATCH 02/45] 14 res --- specification/trace/tracestate-handling.md | 2 +- .../trace/tracestate-probability-sampling.md | 350 +++++++++++------- 2 files changed, 219 insertions(+), 133 deletions(-) diff --git a/specification/trace/tracestate-handling.md b/specification/trace/tracestate-handling.md index 8dd24d45b46..47240553a52 100644 --- a/specification/trace/tracestate-handling.md +++ b/specification/trace/tracestate-handling.md @@ -10,7 +10,7 @@ When setting [TraceState](api.md#tracestate) values that are part of the OTel ec they MUST all be contained in a single entry using the `ot` key, with the value being a semicolon separated list of key-value pairs such as: -* `ot=p:8;r:64` +* `ot=p:8;r:62` * `ot=foo:bar;k1:13` The [TraceContext](https://www.w3.org/TR/trace-context/) specification defines support for multiple "tenants" each to use their own `tracestate` entry by prefixing `tenant@` to tenant-specific values in a mixed tracing environment. OpenTelemetry recognizes this syntax but does not specify an interpretation for multi-tenant `tracestate`. diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 784de142c75..fb77305551b 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -2,18 +2,38 @@ -- [Definitions Used in this Document](#definitions-used-in-this-document) +- [Definitions used in this document](#definitions-used-in-this-document) * [Sampling](#sampling) + + [Adjusted count](#adjusted-count) + + [Power-of-two random sampling](#power-of-two-random-sampling) * [Sampler](#sampler) - * [Parent-based sampler](#parent-based-sampler) - * [Probability sampler](#probability-sampler) - * [Consistent probability sampler](#consistent-probability-sampler) - * [Always-on sampler](#always-on-sampler) - * [Always-off sampler](#always-off-sampler) - * [Non-probability sampler](#non-probability-sampler) - * [Adjusted count](#adjusted-count) - * [Unbiased probability sampling](#unbiased-probability-sampling) - * [Power-of-two random sampling](#power-of-two-random-sampling) + + [Parent-based sampler](#parent-based-sampler) + + [Probability sampler](#probability-sampler) + + [Consistent probability sampler](#consistent-probability-sampler) + + [Always-on sampler](#always-on-sampler) + + [Always-off sampler](#always-off-sampler) + + [Non-probability sampler](#non-probability-sampler) +- [Probability sampling](#probability-sampling) + * [Context: traceparent](#context-traceparent) + + [Sampled flag](#sampled-flag) + - [Requirement 1](#requirement-1) + - [Requirement 2](#requirement-2) + * [Context: tracestate](#context-tracestate) + + [P-value](#p-value) + - [Requirement 1](#requirement-1-1) + - [Requirement 2](#requirement-2-1) + - [Requirement 3](#requirement-3) + - [Requirement 4](#requirement-4) + - [Requirement 5](#requirement-5) + - [Requirement 6](#requirement-6) + + [R-value](#r-value) + - [Requirement 1](#requirement-1-2) + - [Requirement 2](#requirement-2-2) + + [Composition rules](#composition-rules) + - [Requirement 1](#requirement-1-3) + - [Requirement 2](#requirement-2-3) + - [Requirement 3](#requirement-3-1) + - [Requirement 4](#requirement-4-1) @@ -31,10 +51,9 @@ The specification in this document is semantic in nature. Two enable the development of interoperable probability Sampler implementations. OpenTelemetry is gathering experience with Samplers based on this specification while the group considers how to add -probability sampling support in the default SDKs, via the specified -built-in Samplers. +probability sampling support to the default SDK specification. -## Definitions Used in this Document +## Definitions used in this document ### Sampling @@ -53,6 +72,36 @@ is generally known, whereas OpenTelemetry also recognizes "non-probability" sampling approaches, in which representivity is not explicitly quantified. +#### Adjusted count + +Adjusted count is a measure of representivity, the number of spans in +the population that are represented by the individually sampled span. +Span-to-metrics pipelines can be built by adding the adjusted count of +each sample span to a counter of matching spans. + +For probability sampling, adjusted count is defined as the reciprocal +(i.e., mathematical inverse) of sampling probability. + +For non-probability sampling, adjusted count is unknown. + +Zero adjusted count is defined in a way that supports composition of +probability and non-probability sampling. Zero is assigned as the +adjusted count when a probability sampler does not select a span. + +#### Power-of-two random sampling + +A simple sampling scheme can be implemented using a random bit string +as the input. This scheme is limited to power-of-two sampling +probabilities, as follows. + +1. Express the sampling probability as `2**-s`. For example, 25% + equals `2**-2` with `s=2` +2. Count `r`, the number of consecutive zero bits in the input string +3. If `s <= r`, select the item with adjusted count `2**s`. + +This algorithm is the basis of the consistent probability sampling +approach used in OpenTelemetry, defined in greater detail below. + ### Sampler A Sampler provides configurable logic, used by the SDK, for selecting @@ -66,16 +115,15 @@ OpenTelemetry supports spans that are "recorded" and not "sampled" for in-process observability of live spans (e.g., z-pages). The Sampler interface and the built-in Samplers defined by -OpenTelemetry must be capable of deciding immediately whether to -sample a span, since the child context immediately propagates the -decision. +OpenTelemetry decide immediately whether to sample a span, and the +child context immediately propagates the decision. -### Parent-based sampler +#### Parent-based sampler A Sampler that makes its decision to sample based on the W3C `sampled` flag from the context is said to use parent-based sampling. -### Probability sampler +#### Probability sampler A probability Sampler is a Sampler that knows immediately, for each of its decisions, the probability that the span had of being selected. @@ -84,7 +132,7 @@ Sampling probability is defined as a number less than or equal to 1 and greater than 0 (i.e., `0 < probability <= 1`). The case of 0 probability is treated as a special, non-probabilistic case. -### Consistent probability sampler +#### Consistent probability sampler A consistent probability sampler is a Sampler that supports independent sampling decisions at each span in a trace while maintaining that @@ -94,145 +142,138 @@ for any span in a given trace, if a Sampler with lesser sampling probability selects the span for sampling, then the span would also be selected by a Sampler configured with greater sampling probability. -In OpenTelemetry, consistent probability samplers are limited to -power-of-two probabilities. OpenTelemetry consistent probability -sampling is defined in terms of "p-value" and "r-value", both -integers, which are propagated via the context to assist in making -consistent sampling decisions. - -### Always-on sampler +#### Always-on sampler An always-on sampler is another name for a consistent probability sampler with probability equal to one. -### Always-off sampler +#### Always-off sampler An always-off Sampler has the effect of disabling a span completely, effectively excluding it from the population. This is not defined as a probability sampler with zero probability, because these spans are effectively unrepresented. -### Non-probability sampler +#### Non-probability sampler A non-probability sampler is a Sampler that makes its decisions not based on chance, but instead uses arbitrary logic and internal state. -### Adjusted count +## Probability sampling -Adjusted count is defined as a measure of representivity, the number -of spans in the population that are represented by the individually -sampled span. Span-to-metrics pipelines can be built by adding the -adjusted count of each sample span to a counter of matching spans. -Likewise, span-to-metrics pipelines can be built by observing the -duration of each sample span in a histogram, the span's adjusted count -number of times each. +The consistent sampling scheme adopted by OpenTelemetry propagates two +values via the context, termed "p-value" and "r-value": -The adjusted count 1 means one-to-one sampling. Adjusted counts -greater than 1 indicate the use of a probability sampler. Adjusted -counts are unknown when using a non-probability sampler. +1. p-value: the "parent probability" value can be set independently by any span in the trace, for its children, and informs child parent-based Samplers of their adjusted count +2. r-value: the "randomness" value is determined and propagated from the root to all spans in the trace and serves to make sampling decisions consistent -Zero adjusted count is defined in a way that supports composition of -probability and non-probability samplers. In effect, spans that are -"recorded" but not "sampled" have adjusted count of zero. - -### Unbiased probability sampling - -The statistical term "unbiased" is a requirement applied to the -adjusted count of a span, which states that the expected value of the -sum of adjusted counts across all exported spans MUST equal the true -number of spans in the population. Statistical bias, a measure of the -difference between an estimate and its true value, of the estimated -span count in the population should equal zero. Moreover, this -requirement must be true for all subsets of the span population for a -sampler to be considered an unbiased probability sampler. - -Sampling schemes that are not explicitly unbiased should be -categorized as non-probability samplers because they cannot record -unbiased adjusted counts. Here are example Samplers that do not -qualify as unbiased: - -- A traditional form of "leaky-bucket" sampler applies a rate limit to - the starting of new sampled traces. When the configured limit is - not exceeded, all spans pass through with adjusted count 1. When - the configured rate limit is exceeded, it is impossible to set - adjusted count without introducing bias because future arrivals are - not known. -- A "every-N" sampler records spans on a regular interval, but instead - of making a probabilistic decision it makes an exact decision - (e.g., every 10,000 spans). This sampler knows the representivity - of the spans it samples, but the selection process is biased. -- A "at least once per time period" sampler remembers the last time - each distinct span name exported a span. When a span occurs after - more than the specified interval, it samples one (e.g., to ensure - that receivers know about these spans). This sampler introduces - bias because spans that happen between the intervals do not receive - consideration. -- The "always off" sampler is biased by definition. Since it exports - no spans, the sum of adjusted count is always zero. - -As an example, Simple Random Sampling is an unbiased sampling -algorithm. The algorithm is given a sampling probability `p` in the -interval `(0, 1]` and a source of randomness. Then for each item: - -1. Generate a uniform floating point value `r` in the range `[0, 1)` -2. If `r < p`, select the item with adjusted count `1 / p`. - -This algorithm is unbiased because every item in the population -receives equal consideration. - -### Power-of-two random sampling - -An unbiased sampling scheme can be implemented using a random bit -string as the input. This scheme is limited to power-of-two sampling -probabilities, as follows. +Both fields are propagated via the OpenTelemetry `tracestate` under +the `ot` vendor tag using the rules for [tracestate +handling](tracestate-handling.md). Both fields are represented as +unsigned integers requiring at most 6 bits of information. An +invariant will be stated that connects the `sampled` trace flag found +in `traceparent` context to the r-value and p-value found in +`tracestate` context. -1. Express the sampling probability as `2**-s`. For example, 25% - equals `2**-2` -2. Count `r`, the number of consecutive zero bits in the input string -3. If `s <= r`, select the item with adjusted coubnt `2**s`. +### Context: traceparent -This algorithm is the basis of the consistent probability sampling -approach used in OpenTelemetry, defined in greater detail below. +The W3C `traceparent` (version 0) contains three fields of +information: the TraceId, the SpanId, and the trace flags. The +`sampled` trace flag has been defined by W3C to signal an intent to +sample the context. -## Probability sampling tracestate fields +The [Sampler API](sdk.md#sampler) is responsible for setting the +`sampled` flag. -The consistent sampling scheme adopted by OpenTelemetry propagates two -values, p-value and r-value, via the context. +#### Sampled flag -1. r-value: this "randomness" value is determined and propagated from the root of the trace to all spans and serves to make sampling decisions consistent -2. p-value: the "parent probability" value can be modified by any span in the trace and informs child parent-based Samplers their adjusted count +Probability sampling uses additional information to enable consistent +decision making and to record the adjusted count of sampled spans. +When both values are defined and in the specified range, the invariant +between r-value and p-value and the `sampled` trace flag states that +`sampled` is equivalent to the expression `p <= r || p == 63`. -Both fields are propagated via the OpenTelemetry `tracestate` under -the `ot` vendor tag using [syntax defined -here](tracestate-handling.md). Both fields are represented as -unsigned six-bit integers (i.e., in the inclusive interval [0, 63]). +When the invariant is violated, the `sampled` flag takes precedence +and `p` is unset from `tracestate` in order to signal unknown adjusted count. + +##### Requirement 1 + +If `sampled` is set, the `r` and `p` values are valid, `p < 63`, and +`p > r`, then the invariant is violated. In this case, Samplers +SHOULD honor the `sampled` flag and unset `p` from the OpenTelemetry +`tracestate`. -### p-value +##### Requirement 2 -The p-value SHOULD be set in the `tracestate` when an unbiased -probability sampler configured with power-of-two probability selected -the parent span for sampling. p-value SHOULD be set in the -`tracestate` when the W3C `sampled` flag is set in the corresponding -`traceparent`. Non-probability samplers, having unknown adjusted -count, SHOULD unset p-value when making sampling decisions. +If `sampled` is not set, the `r` and `p` values are valid, and `p <= +r` or `p == 63`, the invariant is violated. In this case, +implementations SHOULD honor the `sampled` flag and unset `p` from the +OpenTelemetry `tracestate`. -Zero adjusted count is represented by the special p-value 63. -Otherwise, the p-value is set to the negative base-2 logarithm of +### Context: tracestate + +P-value and r-value are set in the OpenTelemetry `tracestate` using +the identifiers `p` and `r`, each an unsigned base-16 integer. + +P-value is valid in the inclusive range `[0, 63]` (i.e., there are 64 +valid values). + +R-value is valid in the inclusive range `[0, 62]` (i.e., there are 63 +valid values). + +P-value and r-value are independent settings, each can be meaningfully +set without the other present. The invariant between `sampled`, `p`, +and `r` only applies when both `p` and `r` are present. + +#### P-value + +Zero adjusted count is represented by the special p-value 63, +otherwise the p-value is set to the negative base-2 logarithm of sampling probability: -| p-value | Parent Probability | -| ----- | ----------- | -| 0 | 1 | -| 1 | 1/2 | -| 2 | 1/4 | -| ... | ... | -| N | 2**-N | -| ... | ... | -| 61 | 2**-61 | -| 62 | 2**-62 | -| 63 | 0 | +| p-value | Parent Probability | Adjusted count | +| ----- | ----------- | -- | +| 0 | 1 | 1 | +| 1 | 1/2 | 2 | +| 2 | 1/4 | 4 | +| ... | ... | ... | +| N | 2**-N | 2**N | +| ... | ... | ... | +| 61 | 2**-61 | 2**61 | +| 62 | 2**-62 | 2**62 | +| 63 | 0 | 0 | + +##### Requirement 1 + +Samplers SHOULD unset `p` from the tracestate if the unsigned value is +greater than 63. + +##### Requirement 2 + +Parent-based Samplers SHOULD NOT modify a valid `tracestate`. + +##### Requirement 3 + +Non-probability samplers, having unknown adjusted count, SHOULD unset +`p` from the `tracestate`. + +##### Requirement 4 + +If p-value is set without r-value, the consumer SHOULD interpret the +adjusted count from the context, which is provided without the ability +to make new consistent sampling decisions. -### r-value +##### Requirement 5 + +Consistent probability samplers, when they decide not sample a span, +MUST unset `p`. + +##### Requirement 6 + +Consistent probability samplers, when they decide to sample a span, +MUST set `p` to the base-2 logarithm of the adjusted count. + +#### R-value The r-value SHOULD be set in the `tracestate` by the Sampler at the root of the trace in order to support consistent probability sampling. @@ -256,11 +297,56 @@ for spans of a given trace, as follows: | 61 | 2**-62 | 2**-61 and above | | 62 | 2**-62 | 2**-62 and above | -### Probability Sampler behavior +##### Requirement 1 + +Samplers SHOULD unset both `r` and `p` if the unsigned value is +greater than 62. + +##### Requirement 2 + +Samplers SHOULD NOT modify `r` when it is already set in the `tracestate`. + +#### Composition rules + +When more than one Sampler participates in the decision to sample a +context, their decisions can be combined using composition rules. In +all cases, the combined decision to sample is the logical-OR of the +Samplers' decisions (i.e., sample if at least one of the composite +Samplers decides to sample). + +To combine p-values from two consistent probability Sampler decisions, +the Sampler with the greater probability takes effect. The output +p-value becomes the minimum of the two values for `p`. + +To combine a consistent probability Sampler decision with a +non-probability Sampler decision, p-value 63 is used to signify zero +adjusted count. If the probability Sampler decides to sample, its +p-value takes effect. If the probability Sampler decides not to +sample when the non-probability sample does sample, p-value 63 takes +effect signifying zero adjusted count. + +##### Requirement 1 + +When combining Sampler decisions for multiple consistent probability +Samplers and at least one decides to sample, the minimum of the "yes" +decision `p` values MUST be set in the `tracestate`. + +##### Requirement 2 +When combining Sampler decisions for multiple consistent probability +Samplers and none decides to sample, p-value MUST be unset in the +`tracestate`. +##### Requirement 3 -#### Parent-based +When combining Sampler decisions for a consistent probability Sampler +and a non-probability Sampler, and the probabilty Sampler decides to +sample, its p-value MUST be set in the `tracestate` regardless of the +non-probability Sampler decision. -#### Consistent probability-based +##### Requirement 4 +When combining Sampler decisions for a consistent probability Sampler +and a non-probability Sampler, and the probabilty Sampler decides not +to sample but the non-probability does sample, p-value 63 MUST be set +in the `tracestate`. From 780a7a206bdb853e84d4b3aad38cc309fca9285b Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 25 Oct 2021 14:17:38 -0700 Subject: [PATCH 03/45] require samplers --- .../trace/tracestate-probability-sampling.md | 94 +++++++++++-------- 1 file changed, 56 insertions(+), 38 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index fb77305551b..3f6924fc1c7 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -46,16 +46,23 @@ probability sampling using the W3C `tracestate` in a way that allows Span-to-Metrics pipelines to be built that accurately count sampled spans. -The specification in this document is semantic in nature. Two -`tracestate` fields, known as "r-value" and "p-value", are defined to -enable the development of interoperable probability Sampler -implementations. OpenTelemetry is gathering experience with Samplers -based on this specification while the group considers how to add -probability sampling support to the default SDK specification. +## Approach used in this document -## Definitions used in this document +### Objective -### Sampling +This document specifies two `tracestate` fields, known as "r-value" +and "p-value" meant to support interoperable Sampler implementations. +Rules are given for creating, validating, interpreting, and mutating +these fields in an OpenTelemetry [context](../context/context.md). + +Two Samplers are specified that for optional use by OpenTelemetry +tracing SDKs named `ConsistentProbabilitityParent` and +`ConsistentProbability`, meant as optional replacements for the +built-in `Parent` and `TraceIdRatio` Samplers, respectively. + +### Definitions + +#### Sampling Sampling is a family of techniques for collecting and analyzing only a fraction of a complete data set. Individual items that are "sampled" @@ -102,7 +109,7 @@ probabilities, as follows. This algorithm is the basis of the consistent probability sampling approach used in OpenTelemetry, defined in greater detail below. -### Sampler +#### Sampler A Sampler provides configurable logic, used by the SDK, for selecting which Spans are "recorded" and/or "sampled" in a tracing client @@ -175,7 +182,20 @@ invariant will be stated that connects the `sampled` trace flag found in `traceparent` context to the r-value and p-value found in `tracestate` context. -### Context: traceparent +### Conformance + +Samplers that conform to this specification have both behavioral and +statistical requirements. Consumers of OpenTelemetry `tracestate` +data are expected to validate the probability sampling fields before +interpreting the data. + +Producers of OpenTelemetry `tracestate` are required to meet the +behavioral requirements and ensure statistically valid outcomes using +tests that are included in this specification, so that users and +consumers of OpenTelemetry `tracestate` can be assured of the accuracy +of their data. + +### Context invariants The W3C `traceparent` (version 0) contains three fields of information: the TraceId, the SpanId, and the trace flags. The @@ -183,7 +203,15 @@ information: the TraceId, the SpanId, and the trace flags. The sample the context. The [Sampler API](sdk.md#sampler) is responsible for setting the -`sampled` flag. +`sampled` flag and the `tracestate`. + +P-value and r-value are set in the OpenTelemetry `tracestate`, under +the vendor tag `ot`, using the identifiers `p` and `r`. P-value is an +unsigned integer valid in the inclusive range `[0, 63]` (i.e., there +are 64 valid values). R-value is an unsigned integer valid in the +inclusive range `[0, 62]` (i.e., there are 63 valid values). P-value +and r-value are independent settings, each can be meaningfully set +without the other present. #### Sampled flag @@ -193,8 +221,10 @@ When both values are defined and in the specified range, the invariant between r-value and p-value and the `sampled` trace flag states that `sampled` is equivalent to the expression `p <= r || p == 63`. -When the invariant is violated, the `sampled` flag takes precedence -and `p` is unset from `tracestate` in order to signal unknown adjusted count. +The invariant between `sampled`, `p`, and `r` only applies when both +`p` and `r` are present. When the invariant is violated, the +`sampled` flag takes precedence and `p` is unset from `tracestate` in +order to signal unknown adjusted count. ##### Requirement 1 @@ -210,21 +240,6 @@ r` or `p == 63`, the invariant is violated. In this case, implementations SHOULD honor the `sampled` flag and unset `p` from the OpenTelemetry `tracestate`. -### Context: tracestate - -P-value and r-value are set in the OpenTelemetry `tracestate` using -the identifiers `p` and `r`, each an unsigned base-16 integer. - -P-value is valid in the inclusive range `[0, 63]` (i.e., there are 64 -valid values). - -R-value is valid in the inclusive range `[0, 62]` (i.e., there are 63 -valid values). - -P-value and r-value are independent settings, each can be meaningfully -set without the other present. The invariant between `sampled`, `p`, -and `r` only applies when both `p` and `r` are present. - #### P-value Zero adjusted count is represented by the special p-value 63, @@ -260,13 +275,12 @@ Non-probability samplers, having unknown adjusted count, SHOULD unset ##### Requirement 4 If p-value is set without r-value, the consumer SHOULD interpret the -adjusted count from the context, which is provided without the ability -to make new consistent sampling decisions. +adjusted count from the context. ##### Requirement 5 -Consistent probability samplers, when they decide not sample a span, -MUST unset `p`. +Consistent probability samplers, when they decide not to sample a +span, MUST unset `p`. ##### Requirement 6 @@ -275,10 +289,10 @@ MUST set `p` to the base-2 logarithm of the adjusted count. #### R-value -The r-value SHOULD be set in the `tracestate` by the Sampler at the -root of the trace in order to support consistent probability sampling. -When the value is omitted or not present, child spans in the trace are -not able to participate in consistent probability sampling. +R-value is set in the `tracestate` by the Sampler at the root of the +trace, in order to support consistent probability sampling. When the +value is omitted or not present, child spans in the trace are not able +to participate in consistent probability sampling. R-value determines which sampling probabilities and will not sample for spans of a given trace, as follows: @@ -299,13 +313,17 @@ for spans of a given trace, as follows: ##### Requirement 1 -Samplers SHOULD unset both `r` and `p` if the unsigned value is -greater than 62. +Samplers SHOULD unset both `r` and `p` if the unsigned value is of `r` +is greater than 62. ##### Requirement 2 Samplers SHOULD NOT modify `r` when it is already set in the `tracestate`. +### Sampler interface + +TODO + #### Composition rules When more than one Sampler participates in the decision to sample a From 64146a6cd98520f2cb37fcda02a3387765f9fafd Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 26 Oct 2021 01:03:18 -0700 Subject: [PATCH 04/45] move requirements --- .../trace/tracestate-probability-sampling.md | 196 ++++++++++++------ 1 file changed, 135 insertions(+), 61 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 3f6924fc1c7..3042de1ae19 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -2,11 +2,13 @@ -- [Definitions used in this document](#definitions-used-in-this-document) - * [Sampling](#sampling) +- [Approach used in this document](#approach-used-in-this-document) + * [Objective](#objective) + * [Definitions](#definitions) + + [Sampling](#sampling) + [Adjusted count](#adjusted-count) + [Power-of-two random sampling](#power-of-two-random-sampling) - * [Sampler](#sampler) + + [Sampler](#sampler) + [Parent-based sampler](#parent-based-sampler) + [Probability sampler](#probability-sampler) + [Consistent probability sampler](#consistent-probability-sampler) @@ -14,26 +16,33 @@ + [Always-off sampler](#always-off-sampler) + [Non-probability sampler](#non-probability-sampler) - [Probability sampling](#probability-sampling) - * [Context: traceparent](#context-traceparent) + * [Conformance](#conformance) + * [Context invariants](#context-invariants) + [Sampled flag](#sampled-flag) - [Requirement 1](#requirement-1) - - [Requirement 2](#requirement-2) - * [Context: tracestate](#context-tracestate) + [P-value](#p-value) - [Requirement 1](#requirement-1-1) - - [Requirement 2](#requirement-2-1) - - [Requirement 3](#requirement-3) - - [Requirement 4](#requirement-4) - - [Requirement 5](#requirement-5) - - [Requirement 6](#requirement-6) + [R-value](#r-value) - [Requirement 1](#requirement-1-2) - - [Requirement 2](#requirement-2-2) - + [Composition rules](#composition-rules) + * [Samplers](#samplers) + + [ParentConsistentProbabilityBased sampler](#parentconsistentprobabilitybased-sampler) - [Requirement 1](#requirement-1-3) - - [Requirement 2](#requirement-2-3) + - [Requirement 2](#requirement-2) + - [Requirement 3](#requirement-3) + + [ConsistentProbabilityBased sampler](#consistentprobabilitybased-sampler) + - [Requirement 1](#requirement-1-4) + - [Requirement 2](#requirement-2-1) - [Requirement 3](#requirement-3-1) + - [Requirement 4](#requirement-4) + + [Composition rules](#composition-rules) + - [Requirement 1](#requirement-1-5) + - [Requirement 2](#requirement-2-2) + - [Requirement 3](#requirement-3-2) - [Requirement 4](#requirement-4-1) + * [Testing requirements](#testing-requirements) + + [Power of two sampling probability](#power-of-two-sampling-probability) + + [Arbitrary sampling probability](#arbitrary-sampling-probability) + + [Uniform attribute probability](#uniform-attribute-probability) @@ -56,9 +65,9 @@ Rules are given for creating, validating, interpreting, and mutating these fields in an OpenTelemetry [context](../context/context.md). Two Samplers are specified that for optional use by OpenTelemetry -tracing SDKs named `ConsistentProbabilitityParent` and -`ConsistentProbability`, meant as optional replacements for the -built-in `Parent` and `TraceIdRatio` Samplers, respectively. +tracing SDKs named `ParentConsistentProbabilitityBased` and +`ConsistentProbabilityBased`, meant as optional replacements for the +built-in `ParentBased` and `TraceIdRatioBased` Samplers, respectively. ### Definitions @@ -228,17 +237,9 @@ order to signal unknown adjusted count. ##### Requirement 1 -If `sampled` is set, the `r` and `p` values are valid, `p < 63`, and -`p > r`, then the invariant is violated. In this case, Samplers -SHOULD honor the `sampled` flag and unset `p` from the OpenTelemetry -`tracestate`. - -##### Requirement 2 - -If `sampled` is not set, the `r` and `p` values are valid, and `p <= -r` or `p == 63`, the invariant is violated. In this case, -implementations SHOULD honor the `sampled` flag and unset `p` from the -OpenTelemetry `tracestate`. +Samplers SHOULD unset `p` when the invariant between the `sampled`, +`p`, and `r` values is violated before using the `tracestate` to make +a sampling decision. #### P-value @@ -260,32 +261,8 @@ sampling probability: ##### Requirement 1 -Samplers SHOULD unset `p` from the tracestate if the unsigned value is -greater than 63. - -##### Requirement 2 - -Parent-based Samplers SHOULD NOT modify a valid `tracestate`. - -##### Requirement 3 - -Non-probability samplers, having unknown adjusted count, SHOULD unset -`p` from the `tracestate`. - -##### Requirement 4 - -If p-value is set without r-value, the consumer SHOULD interpret the -adjusted count from the context. - -##### Requirement 5 - -Consistent probability samplers, when they decide not to sample a -span, MUST unset `p`. - -##### Requirement 6 - -Consistent probability samplers, when they decide to sample a span, -MUST set `p` to the base-2 logarithm of the adjusted count. +Samplers SHOULD unset `p` from the `tracestate` if the unsigned value is +greater than 63 before using the `tracestate` to make a sampling decision. #### R-value @@ -313,16 +290,74 @@ for spans of a given trace, as follows: ##### Requirement 1 -Samplers SHOULD unset both `r` and `p` if the unsigned value is of `r` -is greater than 62. +Samplers SHOULD unset both `r` and `p` from the `tracestate` if the +unsigned value is of `r` is greater than 62 before using the +`tracestate` to make a sampling decision. + +### Samplers + +#### ParentConsistentProbabilityBased sampler + +The `ParentConsistentProbabilityBased` sampler is meant as an optional +replacement for the [`ParentBased` Sampler](sdk.md#parentbased). It is +required to first validate the `tracestate` and then behave as the +`ParentBased` sampler would. + +##### Requirement 1 + +The `ParentConsistentProbabilityBased` Sampler MUST have the same +constructor signature as the built-in `ParentBased` sampler in each +OpenTelemetry SDK. + +##### Requirement 2 + +The `ParentConsistentProbabilityBased` Sampler MUST NOT modify a +valid `tracestate`. + +##### Requirement 3 + +The `ParentConsistentProbabilityBased` Sampler MUST make the same +decision specified for the `ParentBased` sampler for valid contexts. + +#### ConsistentProbabilityBased sampler + +The `ConsistentProbabilityBased` sampler is meant as an optional +replacement for the [`TraceIdRatioBased` +Sampler](sdk.md#traceidratiobased). In the case where it is used as a +root sampler, it is required to produce a valid `tracestate`. In the +case where it is used in a non-root context, it is required to +validate the incoming `tracestate` and to produce a valid `tracestate` +for the outgoing context. + +The `ConsistentProbabilityBased` sampler is required to support +probabilities that are not exact powers of two. To do so, +implementations are required to select between the nearest powers of +two probabilistically. For example, 5% sampling can be achieved by +selecting 1/16 sampling 60% of the time and 1/32 sampling 40% of the +time. + +##### Requirement 1 + +The `ConsistentProbabilityBased` Sampler MUST have the same +constructor signature as the built-in `TraceIdRatioBased` sampler in +each OpenTelemetry SDK. ##### Requirement 2 -Samplers SHOULD NOT modify `r` when it is already set in the `tracestate`. +The `ConsistentProbabilityBased` Sampler MUST set `r` when it makes a +root sampling decision. + +##### Requirement 3 + +The `ConsistentProbabilityBased` Sampler MUST unset `p` from the +`tracestate` when it decides not to sample. + +##### Requirement 4 -### Sampler interface +The `ConsistentProbabilityBased` Sampler MUST set `p` when it decides +to sample to the base-2 logarithm of the unbiased adjusted count. -TODO +A test specification for this requirement is given in the appendix. #### Composition rules @@ -358,13 +393,52 @@ Samplers and none decides to sample, p-value MUST be unset in the ##### Requirement 3 When combining Sampler decisions for a consistent probability Sampler -and a non-probability Sampler, and the probabilty Sampler decides to +and a non-probability Sampler, and the probability Sampler decides to sample, its p-value MUST be set in the `tracestate` regardless of the non-probability Sampler decision. ##### Requirement 4 When combining Sampler decisions for a consistent probability Sampler -and a non-probability Sampler, and the probabilty Sampler decides not +and a non-probability Sampler, and the probability Sampler decides not to sample but the non-probability does sample, p-value 63 MUST be set in the `tracestate`. + +### Testing requirements + +TODO describe these tests: Overview for hypothesis testing, use of +large N, use of significance levels. Expectation of a strong general +purpose non-cryptographic random number generator. Tests are meant to +validate end-to-end sampling logic and span-to-metrics accounting, not +validate the RNG. + +Tests are expected to use fixed seeds. Tests are expected to +demonstrate and document that the statistical tests fail at +approximately at the expected level of statistical significance. +E.g., a 1% significance level test should be repeated and demonstrate +occasional failure, then be saved with the seed that produced the +passing result. This should be documented. + +#### Power of two sampling probability + +This MAY use an exact binomial test or it may use a Chi-squared test +with 1 degree of freedom. + +Repeat for sampling probabilities: TBD. + +#### Arbitrary sampling probability + +Chi-squared test with three categories: + +1. Unsampled +2. Sampled with lesser probability +3. Sampled with greater/equal probability + +Repeat for sampling probabilities: TBD. + +#### Uniform attribute probability + +Chi-squared test with arbitrary number of categories. In either of +the above tests, use a secondary attribute with K categorical values. + +Repeat for number of categories: TBD. From 68fb444a7a59f99381f68b951dd752a2ec5dc95a Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 26 Oct 2021 01:07:18 -0700 Subject: [PATCH 05/45] note --- specification/trace/tracestate-probability-sampling.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 3042de1ae19..ffa13eb9f1d 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -419,6 +419,8 @@ E.g., a 1% significance level test should be repeated and demonstrate occasional failure, then be saved with the seed that produced the passing result. This should be documented. +Tests can be implemented using a another SDK's test, for example. + #### Power of two sampling probability This MAY use an exact binomial test or it may use a Chi-squared test From f02863e541e0cf4f897ea713292877c55315c59c Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 2 Nov 2021 01:54:40 -0700 Subject: [PATCH 06/45] add test spec --- .../trace/tracestate-probability-sampling.md | 199 +++++++++++++----- 1 file changed, 149 insertions(+), 50 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index ffa13eb9f1d..7a5493db5db 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -64,10 +64,11 @@ and "p-value" meant to support interoperable Sampler implementations. Rules are given for creating, validating, interpreting, and mutating these fields in an OpenTelemetry [context](../context/context.md). -Two Samplers are specified that for optional use by OpenTelemetry -tracing SDKs named `ParentConsistentProbabilitityBased` and -`ConsistentProbabilityBased`, meant as optional replacements for the -built-in `ParentBased` and `TraceIdRatioBased` Samplers, respectively. +Two Samplers are specified for optional inclusion in OpenTelemetry +tracing SDKs, named `ParentConsistentProbabilitityBased` and +`ConsistentProbabilityBased`. These are meant as optional +replacements for the built-in `ParentBased` and `TraceIdRatioBased` +Samplers. ### Definitions @@ -193,16 +194,18 @@ in `traceparent` context to the r-value and p-value found in ### Conformance -Samplers that conform to this specification have both behavioral and -statistical requirements. Consumers of OpenTelemetry `tracestate` -data are expected to validate the probability sampling fields before -interpreting the data. +Consumers of OpenTelemetry `tracestate` data are expected to validate +the probability sampling fields before interpreting the data. This +applies to the two samplers specified here as well as consumers of +span data, who are expected to validate `tracestate` before +interpreting span adjusted counts. -Producers of OpenTelemetry `tracestate` are required to meet the -behavioral requirements and ensure statistically valid outcomes using -tests that are included in this specification, so that users and -consumers of OpenTelemetry `tracestate` can be assured of the accuracy -of their data. +Producers of OpenTelemetry `tracestate` containing p-value and r-value +fields are required to meet the behavioral requirements stated for the +ConsistentProbabilityBased sampler, to ensure statistically valid +outcomes, using a test suite included in this specification, so that +users and consumers of OpenTelemetry `tracestate` can be assured of +accurate span counts in a span-to-metrics pipeline. ### Context invariants @@ -239,7 +242,7 @@ order to signal unknown adjusted count. Samplers SHOULD unset `p` when the invariant between the `sampled`, `p`, and `r` values is violated before using the `tracestate` to make -a sampling decision. +a sampling decision or interpret adjusted count. #### P-value @@ -261,8 +264,9 @@ sampling probability: ##### Requirement 1 -Samplers SHOULD unset `p` from the `tracestate` if the unsigned value is -greater than 63 before using the `tracestate` to make a sampling decision. +Consumers SHOULD unset `p` from the `tracestate` if the unsigned value +is greater than 63 before using the `tracestate` to make a sampling +decision or interpret adjusted count. #### R-value @@ -317,17 +321,18 @@ valid `tracestate`. ##### Requirement 3 The `ParentConsistentProbabilityBased` Sampler MUST make the same -decision specified for the `ParentBased` sampler for valid contexts. +decision specified for the `ParentBased` sampler when the context is +valid. #### ConsistentProbabilityBased sampler The `ConsistentProbabilityBased` sampler is meant as an optional replacement for the [`TraceIdRatioBased` Sampler](sdk.md#traceidratiobased). In the case where it is used as a -root sampler, it is required to produce a valid `tracestate`. In the -case where it is used in a non-root context, it is required to -validate the incoming `tracestate` and to produce a valid `tracestate` -for the outgoing context. +root sampler, the `ConsistentProbabilityBased` sampler is required to +produce a valid `tracestate`. In the case where it is used in a +non-root context, it is required to validate the incoming `tracestate` +and to produce a valid `tracestate` for the outgoing context. The `ConsistentProbabilityBased` sampler is required to support probabilities that are not exact powers of two. To do so, @@ -355,11 +360,35 @@ The `ConsistentProbabilityBased` Sampler MUST unset `p` from the ##### Requirement 4 The `ConsistentProbabilityBased` Sampler MUST set `p` when it decides -to sample to the base-2 logarithm of the unbiased adjusted count. +to sample. + +##### Requirement 5 + +When it decides to sample, the `ConsistentProbabilityBased` Sampler +MUST set `p` to the negative base-2 logarithm of a power-of-two +sampling probability. + +##### Requirement 6 + +The `ConsistentProbabilityBased` Sampler MUST set `p` so that the +adjusted count interpreted from the `tracestate` is an unbiased +estimate of the number of representative spans in the population. A test specification for this requirement is given in the appendix. -#### Composition rules +##### Requirement 7 + +If `r` is not set on the input `tracecontext` and the Span is not a +root span, `ConsistentProbabilityBased` SHOULD set `r` as if it were a +root span. + +##### Requirement 8 + +If the configured sampling probability is less than `2**-62`, the +Sampler should round down to zero probability and make the same +sampling decision as the builtin `AlwaysOff` sampler would. + +### Composition rules When more than one Sampler participates in the decision to sample a context, their decisions can be combined using composition rules. In @@ -378,6 +407,8 @@ p-value takes effect. If the probability Sampler decides not to sample when the non-probability sample does sample, p-value 63 takes effect signifying zero adjusted count. +#### List of requirements + ##### Requirement 1 When combining Sampler decisions for multiple consistent probability @@ -404,43 +435,111 @@ and a non-probability Sampler, and the probability Sampler decides not to sample but the non-probability does sample, p-value 63 MUST be set in the `tracestate`. -### Testing requirements +### Consumer-only requirements + +#### Trace consumers + +Due to the `ConsistentProbabilityBased` Sampler requirement about +setting `r` when it is unset for a non-root span, trace consumers are +advised to check traces for r-value consistency. + +When a single trace contains more than a single distinct `r` value, it +means the trace was not correctly sampled at the root for probability +sampling. While the adjusted count of each span is correct in this +scenario, it may be impossible to detect complete traces. + +##### Requirement 1 -TODO describe these tests: Overview for hypothesis testing, use of -large N, use of significance levels. Expectation of a strong general -purpose non-cryptographic random number generator. Tests are meant to -validate end-to-end sampling logic and span-to-metrics accounting, not -validate the RNG. +When a single trace contains spans with `tracestate` values containing +more than one distinct value for `r`, the consumer SHOULD recognize +the trace as inconsistently sampled. -Tests are expected to use fixed seeds. Tests are expected to -demonstrate and document that the statistical tests fail at -approximately at the expected level of statistical significance. -E.g., a 1% significance level test should be repeated and demonstrate -occasional failure, then be saved with the seed that produced the -passing result. This should be documented. +### Appendix: Statistical test requirements -Tests can be implemented using a another SDK's test, for example. +This section specifies a test that can be implemented to ensures +basic conformance to the requirement that sampling decisions be unbiased. -#### Power of two sampling probability +The goal of this test specification is to be simple to implement and +not require advanced statistical skills or libraries to be successful. -This MAY use an exact binomial test or it may use a Chi-squared test -with 1 degree of freedom. +This test is not meant to evaluate the performance of a random number +generator. This test assumes the underlying RNG is good quality and +checks that the sampler produces the expected proportion of sampling +decisions. -Repeat for sampling probabilities: TBD. +One of the challenges of this kind of test is that probabilistic tests +are expected to occasionally produce exceptional results. To make +this a strict test for randomness, we take the following approach to +locate a "golden" seed: -#### Arbitrary sampling probability +- Use fixed values for significance level (5%) and trials (20) +- Use a population size of one million spans +- Locate a seed value such that the Chi-Squared test fails exactly once. -Chi-squared test with three categories: +For a correct implementation, this is a fairly easy test to pass. The +author of the test is expected to document how the test was produced. +As specified, the Chi-Squared test has either one or two degrees of +freedom, depending on whether the sampling probability is an exact +power of two or not. -1. Unsampled -2. Sampled with lesser probability -3. Sampled with greater/equal probability +#### Test procedure: exact powers of two -Repeat for sampling probabilities: TBD. +In this case there is one degree of freedom for the Chi-Squared test. +The following table summarizes the test parameters. -#### Uniform attribute probability +| Sampling probability | p-value when sampled | Expectsampled | Expectunsampled | +| --- | --- | --- | --- | +| 0.5 (2**-1) | 1 | 500000 | 500000 | +| .0625 (2**-4) | 4 | 62500 | 937500 | +| .0078125 (2**-7) | 7 | 7812.5 | 992187.5 | +| 976.5625 (2**-10) | 10 | 976.5625 | 999023.4375 | +| .0001220703125 (2**-13) | 13 | 122.0703125 | 999877.9296875 | -Chi-squared test with arbitrary number of categories. In either of -the above tests, use a secondary attribute with K categorical values. +The formula for computing Chi-Squared in this case is: + +``` +ChiSquared = math.Pow(sampled - expectsampled, 2) / expectsampled + + math.Pow(1000000 - sampled - expectunsampled, 2) / expectunsampled +``` + +This should be compared with 0.003932, the value of the Chi-Squared +distribution for one degree of freedom with significance level 5%. +For each probability in the table above, the test is required to +demonstrate a seed that produces exactly one ChiSquared value less +than 0.003932. + +##### Requirement 1 + +For the teset with 20 trials and 1 million spans each, the test MUST +demonstrate a random number generator seed such that the ChiSquared +test statistic is below 0.003932 exactly 1 out of 20 times. + +#### Test procedure: non-powers of two + +In this case there are two degrees of freedom for the Chi-Squared test. +The following table summarizes the test parameters. + +| Sampling probability | Lower p-value | Upper p-value | Expectlower | Expectupper | Expectunsampled | +| --- | --- | --- | --- | | | +| 0.6 | 0 | 1 | 400000 | 200000 | 400000 | +| 0.1 | 3 | 4 | ... | ... | ... | +| ... | ... | ... | ... | ... | ... | +| TODO | | | | | | + +The formula for computing Chi-Squared in this case is: + +``` +TODO +``` + +This should be compared with 0.102587, the value of the Chi-Squared +distribution for two degrees of freedom with significance level 5%. +For each probability in the table above, the test is required to +demonstrate a seed that produces exactly one ChiSquared value less +than 0.102587. + +##### Requirement 1 -Repeat for number of categories: TBD. +For the teset with 20 trials and 1 million spans each, the test MUST +demonstrate a random number generator seed such that the ChiSquared +test statistic is below 0.102587 exactly 1 out of 20 times. From 93fdd0c5ef8e1f718e1d2d1678245ef53c6f6e07 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 2 Nov 2021 12:38:22 -0700 Subject: [PATCH 07/45] update test spec with more clarity --- .../trace/tracestate-probability-sampling.md | 103 +++++++++++------- 1 file changed, 64 insertions(+), 39 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 7a5493db5db..3fff157aea3 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -456,90 +456,115 @@ the trace as inconsistently sampled. ### Appendix: Statistical test requirements -This section specifies a test that can be implemented to ensures -basic conformance to the requirement that sampling decisions be unbiased. +This section specifies a test that can be implemented to ensure basic +conformance with the requirement that sampling decisions are unbiased. The goal of this test specification is to be simple to implement and not require advanced statistical skills or libraries to be successful. This test is not meant to evaluate the performance of a random number -generator. This test assumes the underlying RNG is good quality and -checks that the sampler produces the expected proportion of sampling -decisions. +generator. This test assumes the underlying RNG is of good quality +and checks that the sampler produces the expected proportionality with +a high degree of statistical confidence. One of the challenges of this kind of test is that probabilistic tests are expected to occasionally produce exceptional results. To make -this a strict test for randomness, we take the following approach to -locate a "golden" seed: +this a strict test for random behavior, we take the following approach: +- Generate a pre-determined list of 20 random seeds - Use fixed values for significance level (5%) and trials (20) - Use a population size of one million spans -- Locate a seed value such that the Chi-Squared test fails exactly once. +- For each trial, simulate the population and compute ChiSquared + test statistic +- Locate the first seed value in the ordered list such that the + Chi-Squared significance test fails exactly once out of 20 trials + +To create this test, perform the above sequence using the seed values +from the predetermined list, in order, until a seed value is found +with exactly one failure. This is expected to happen fairly often and +is required to happen once among the 20 available seeds. After +calculating the index of the first seed with exactly one ChiSquared +failure, record it in the test. For continuous integration testing, +it is only necessary to re-run the test using the predetermined seed +index. -For a correct implementation, this is a fairly easy test to pass. The -author of the test is expected to document how the test was produced. As specified, the Chi-Squared test has either one or two degrees of freedom, depending on whether the sampling probability is an exact power of two or not. -#### Test procedure: exact powers of two +#### Test procedure: non-powers of two -In this case there is one degree of freedom for the Chi-Squared test. +In this case there are two degrees of freedom for the Chi-Squared test. The following table summarizes the test parameters. -| Sampling probability | p-value when sampled | Expectsampled | Expectunsampled | -| --- | --- | --- | --- | -| 0.5 (2**-1) | 1 | 500000 | 500000 | -| .0625 (2**-4) | 4 | 62500 | 937500 | -| .0078125 (2**-7) | 7 | 7812.5 | 992187.5 | -| 976.5625 (2**-10) | 10 | 976.5625 | 999023.4375 | -| .0001220703125 (2**-13) | 13 | 122.0703125 | 999877.9296875 | +| Test case | Sampling probability | Lower, Upper p-value when sampled | Expectlower | Expectupper | Expectunsampled | +| --- | --- | --- | | | | +| 0 | 0.900000 | 0, 1 | 100000 | 800000 | 100000 | +| 1 | 0.600000 | 0, 1 | 400000 | 200000 | 400000 | +| 2 | 0.330000 | 1, 2 | 170000 | 160000 | 670000 | +| 3 | 0.130000 | 2, 3 | 120000 | 10000 | 870000 | +| 4 | 0.100000 | 3, 4 | 25000 | 75000 | 900000 | +| 5 | 0.050000 | 4, 5 | 12500 | 37500 | 950000 | +| 6 | 0.017000 | 5, 6 | 14250 | 2750 | 983000 | +| 7 | 0.010000 | 6, 7 | 5625 | 4375 | 990000 | +| 8 | 0.005000 | 7, 8 | 2812.5 | 2187.5 | 995000 | +| 9 | 0.002900 | 8, 9 | 1006.25 | 1893.75 | 997100 | +| 10 | 0.001000 | 9, 10 | 953.125 | 46.875 | 999000 | +| 11 | 0.000500 | 10, 11 | 476.5625 | 23.4375 | 999500 | +| 12 | 0.000260 | 11, 12 | 228.28125 | 31.71875 | 999740 | +| 13 | 0.000230 | 12, 13 | 14.140625 | 215.859375 | 999770 | +| 14 | 0.000100 | 13, 14 | 22.0703125 | 77.9296875 | 999900 | The formula for computing Chi-Squared in this case is: ``` -ChiSquared = math.Pow(sampled - expectsampled, 2) / expectsampled + - math.Pow(1000000 - sampled - expectunsampled, 2) / expectunsampled +ChiSquared = math.Pow(sampledlower - expectlower, 2) / expectlower + + math.Pow(sampledupper - expectupper, 2) / expectupper + + math.Pow(1000000 - sampledlower - sampledupper - expectunsampled, 2) / expectunsampled + ``` -This should be compared with 0.003932, the value of the Chi-Squared -distribution for one degree of freedom with significance level 5%. +This should be compared with 0.102587, the value of the Chi-Squared +distribution for two degrees of freedom with significance level 5%. For each probability in the table above, the test is required to demonstrate a seed that produces exactly one ChiSquared value less -than 0.003932. +than 0.102587. ##### Requirement 1 -For the teset with 20 trials and 1 million spans each, the test MUST +For the test with 20 trials and 1 million spans each, the test MUST demonstrate a random number generator seed such that the ChiSquared -test statistic is below 0.003932 exactly 1 out of 20 times. +test statistic is below 0.102587 exactly 1 out of 20 times. -#### Test procedure: non-powers of two +#### Test procedure: exact powers of two -In this case there are two degrees of freedom for the Chi-Squared test. +In this case there is one degree of freedom for the Chi-Squared test. The following table summarizes the test parameters. -| Sampling probability | Lower p-value | Upper p-value | Expectlower | Expectupper | Expectunsampled | -| --- | --- | --- | --- | | | -| 0.6 | 0 | 1 | 400000 | 200000 | 400000 | -| 0.1 | 3 | 4 | ... | ... | ... | -| ... | ... | ... | ... | ... | ... | -| TODO | | | | | | +| Test case | Sampling probability | P-value when sampled | Expectsampled | Expectunsampled | +| --- | --- | --- | --- | --- | +| 15 | 0x1p-01 (0.500000) | 1 | 500000 | 500000 | +| 16 | 0x1p-04 (0.062500) | 4 | 62500 | 937500 | +| 17 | 0x1p-07 (0.007812) | 7 | 7812.5 | 992187.5 | +| 18 | 0x1p-10 (0.000977) | 10 | 976.5625 | 999023.4375 | +| 19 | 0x1p-13 (0.000122) | 13 | 122.0703125 | 999877.9297 | The formula for computing Chi-Squared in this case is: ``` -TODO +ChiSquared = math.Pow(sampled - expectsampled, 2) / expectsampled + + math.Pow(1000000 - sampled - expectunsampled, 2) / expectunsampled ``` -This should be compared with 0.102587, the value of the Chi-Squared -distribution for two degrees of freedom with significance level 5%. +This should be compared with 0.003932, the value of the Chi-Squared +distribution for one degree of freedom with significance level 5%. For each probability in the table above, the test is required to demonstrate a seed that produces exactly one ChiSquared value less -than 0.102587. +than 0.003932. ##### Requirement 1 For the teset with 20 trials and 1 million spans each, the test MUST demonstrate a random number generator seed such that the ChiSquared -test statistic is below 0.102587 exactly 1 out of 20 times. +test statistic is below 0.003932 exactly 1 out of 20 times. + From 18c29040e24f6b031f1c464bd05401756e999121 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 2 Nov 2021 12:41:18 -0700 Subject: [PATCH 08/45] use r:62 in examples (which is in-range) --- specification/trace/tracestate-handling.md | 4 ++-- specification/trace/tracestate-probability-sampling.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/trace/tracestate-handling.md b/specification/trace/tracestate-handling.md index 47240553a52..cd310fb9216 100644 --- a/specification/trace/tracestate-handling.md +++ b/specification/trace/tracestate-handling.md @@ -65,8 +65,8 @@ Set values MUST be either updated or added to the `ot` entry in `TraceState`, in order to preserve existing values belonging to other OTel concerns. For example, if a given concern K wants to set `k1:13`: -* `ot=p:8;r:64` will become `ot=p:8;r:64;k1:13`. -* `ot=p:8;k1:7;r:64` will become `ot=p8;r:64;k1:13`. Preserving the order is not required. +* `ot=p:8;r:62` will become `ot=p:8;r:62;k1:13`. +* `ot=p:8;k1:7;r:62` will become `ot=p8;r:62;k1:13`. Preserving the order is not required. If setting a value ends up making the entire `ot` entry exceed the 256 characters limit, SDKs are advised to abort the operation and signal the user about the error, e.g. diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 3fff157aea3..c70f65a060e 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -498,7 +498,7 @@ In this case there are two degrees of freedom for the Chi-Squared test. The following table summarizes the test parameters. | Test case | Sampling probability | Lower, Upper p-value when sampled | Expectlower | Expectupper | Expectunsampled | -| --- | --- | --- | | | | +| --- | --- | --- | --- | --- | --- | | 0 | 0.900000 | 0, 1 | 100000 | 800000 | 100000 | | 1 | 0.600000 | 0, 1 | 400000 | 200000 | 400000 | | 2 | 0.330000 | 1, 2 | 170000 | 160000 | 670000 | From 29e343010bdefd3c0fb36d861edf568998aef3b8 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 2 Nov 2021 12:49:04 -0700 Subject: [PATCH 09/45] reformat test spec table --- .../trace/tracestate-probability-sampling.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index c70f65a060e..608cda359f6 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -518,10 +518,9 @@ The following table summarizes the test parameters. The formula for computing Chi-Squared in this case is: ``` -ChiSquared = math.Pow(sampledlower - expectlower, 2) / expectlower + - math.Pow(sampledupper - expectupper, 2) / expectupper + - math.Pow(1000000 - sampledlower - sampledupper - expectunsampled, 2) / expectunsampled - +ChiSquared = math.Pow(sampled_lowerP - expect_lowerP, 2) / expect_lowerP + + math.Pow(sampled_upperP - expect_upperP, 2) / expect_upperP + + math.Pow(1000000 - sampled_lowerP - sampled_upperP - expect_unsampled, 2) / expect_unsampled ``` This should be compared with 0.102587, the value of the Chi-Squared @@ -552,8 +551,8 @@ The following table summarizes the test parameters. The formula for computing Chi-Squared in this case is: ``` -ChiSquared = math.Pow(sampled - expectsampled, 2) / expectsampled + - math.Pow(1000000 - sampled - expectunsampled, 2) / expectunsampled +ChiSquared = math.Pow(sampled - expect_sampled, 2) / expect_sampled + + math.Pow(1000000 - sampled - expect_unsampled, 2) / expect_unsampled ``` This should be compared with 0.003932, the value of the Chi-Squared From 269bdc5939787541efb2e74c10a20e504c30b8c1 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 2 Nov 2021 12:51:27 -0700 Subject: [PATCH 10/45] toc --- .../trace/tracestate-probability-sampling.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 608cda359f6..0390531953b 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -34,15 +34,24 @@ - [Requirement 2](#requirement-2-1) - [Requirement 3](#requirement-3-1) - [Requirement 4](#requirement-4) - + [Composition rules](#composition-rules) + - [Requirement 5](#requirement-5) + - [Requirement 6](#requirement-6) + - [Requirement 7](#requirement-7) + - [Requirement 8](#requirement-8) + * [Composition rules](#composition-rules) + + [List of requirements](#list-of-requirements) - [Requirement 1](#requirement-1-5) - [Requirement 2](#requirement-2-2) - [Requirement 3](#requirement-3-2) - [Requirement 4](#requirement-4-1) - * [Testing requirements](#testing-requirements) - + [Power of two sampling probability](#power-of-two-sampling-probability) - + [Arbitrary sampling probability](#arbitrary-sampling-probability) - + [Uniform attribute probability](#uniform-attribute-probability) + * [Consumer-only requirements](#consumer-only-requirements) + + [Trace consumers](#trace-consumers) + - [Requirement 1](#requirement-1-6) + * [Appendix: Statistical test requirements](#appendix-statistical-test-requirements) + + [Test procedure: non-powers of two](#test-procedure-non-powers-of-two) + - [Requirement 1](#requirement-1-7) + + [Test procedure: exact powers of two](#test-procedure-exact-powers-of-two) + - [Requirement 1](#requirement-1-8) From ed02c3737525dda8c24dac6e3f1eba32493b3b46 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 2 Nov 2021 13:36:56 -0700 Subject: [PATCH 11/45] give each requirement a name --- .../trace/tracestate-probability-sampling.md | 104 +++++++++--------- 1 file changed, 52 insertions(+), 52 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 0390531953b..57bb6f4d30e 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -19,39 +19,39 @@ * [Conformance](#conformance) * [Context invariants](#context-invariants) + [Sampled flag](#sampled-flag) - - [Requirement 1](#requirement-1) + - [Requirement: Inconsistent p-values are unset](#requirement-inconsistent-p-values-are-unset) + [P-value](#p-value) - - [Requirement 1](#requirement-1-1) + - [Requirement: Out-of-range p-values are unset](#requirement-out-of-range-p-values-are-unset) + [R-value](#r-value) - - [Requirement 1](#requirement-1-2) + - [Requirement: Out-of-range r-values unset both p and r](#requirement-out-of-range-r-values-unset-both-p-and-r) * [Samplers](#samplers) + [ParentConsistentProbabilityBased sampler](#parentconsistentprobabilitybased-sampler) - - [Requirement 1](#requirement-1-3) - - [Requirement 2](#requirement-2) - - [Requirement 3](#requirement-3) + - [Requirement: ParentBased API compatibility](#requirement-parentbased-api-compatibility) + - [Requirement: ParentConsistentProbabilityBased does not modify valid tracestate](#requirement-parentconsistentprobabilitybased-does-not-modify-valid-tracestate) + - [Requirement: ParentConsistentProbabilityBased and ParentBased make identical decisions](#requirement-parentconsistentprobabilitybased-and-parentbased-make-identical-decisions) + [ConsistentProbabilityBased sampler](#consistentprobabilitybased-sampler) - - [Requirement 1](#requirement-1-4) - - [Requirement 2](#requirement-2-1) - - [Requirement 3](#requirement-3-1) - - [Requirement 4](#requirement-4) - - [Requirement 5](#requirement-5) - - [Requirement 6](#requirement-6) - - [Requirement 7](#requirement-7) - - [Requirement 8](#requirement-8) + - [Requirement: TraceIdRatioBased API compatibility](#requirement-traceidratiobased-api-compatibility) + - [Requirement: ConsistentProbabilityBased sampler sets r for root span](#requirement-consistentprobabilitybased-sampler-sets-r-for-root-span) + - [Requirement: ConsistentProbabilityBased sampler unsets p when not sampling](#requirement-consistentprobabilitybased-sampler-unsets-p-when-not-sampling) + - [Requirement: ConsistentProbabilityBased sampler sets p when sampling](#requirement-consistentprobabilitybased-sampler-sets-p-when-sampling) + - [Requirement: ConsistentProbabilityBased sampler sets p using an unbiased algorithm](#requirement-consistentprobabilitybased-sampler-sets-p-using-an-unbiased-algorithm) + - [Requirement: ConsistentProbabilityBased sampler sets r for non-root span](#requirement-consistentprobabilitybased-sampler-sets-r-for-non-root-span) + - [Requirement: ConsistentProbabilityBased sampler acts like AlwaysOff sampler for probabilities less than 2**-62](#requirement-consistentprobabilitybased-sampler-acts-like-alwaysoff-sampler-for-probabilities-less-than-2-62) * [Composition rules](#composition-rules) + [List of requirements](#list-of-requirements) - - [Requirement 1](#requirement-1-5) - - [Requirement 2](#requirement-2-2) - - [Requirement 3](#requirement-3-2) - - [Requirement 4](#requirement-4-1) - * [Consumer-only requirements](#consumer-only-requirements) + - [Requirement: Combining multiple sampling decisions using logical `or`](#requirement-combining-multiple-sampling-decisions-using-logical-or) + - [Requirement: Combine multiple consistent probability samplers using the minimum p-value](#requirement-combine-multiple-consistent-probability-samplers-using-the-minimum-p-value) + - [Requirement: Unset p when multiple consistent probability samplers decide not to sample](#requirement-unset-p-when-multiple-consistent-probability-samplers-decide-not-to-sample) + - [Requirement: Use probability sampler p-value when its decision to sample is combined with non-probability samplers](#requirement-use-probability-sampler-p-value-when-its-decision-to-sample-is-combined-with-non-probability-samplers) + - [Requirement: Use p-value 63 when a probability sampler decision not to sample when combined with non-probability sampler decision to sample](#requirement-use-p-value-63-when-a-probability-sampler-decision-not-to-sample-when-combined-with-non-probability-sampler-decision-to-sample) + * [Consumer recommendations](#consumer-recommendations) + [Trace consumers](#trace-consumers) - - [Requirement 1](#requirement-1-6) + - [Recommendation: Recognize inconsistent r-values](#recommendation-recognize-inconsistent-r-values) * [Appendix: Statistical test requirements](#appendix-statistical-test-requirements) + [Test procedure: non-powers of two](#test-procedure-non-powers-of-two) - - [Requirement 1](#requirement-1-7) + - [Requirement: Pass 15 non-power-of-two statistical tests](#requirement-pass-15-non-power-of-two-statistical-tests) + [Test procedure: exact powers of two](#test-procedure-exact-powers-of-two) - - [Requirement 1](#requirement-1-8) + - [Requirement: Pass 5 power-of-two statistical tests](#requirement-pass-5-power-of-two-statistical-tests) @@ -247,7 +247,7 @@ The invariant between `sampled`, `p`, and `r` only applies when both `sampled` flag takes precedence and `p` is unset from `tracestate` in order to signal unknown adjusted count. -##### Requirement 1 +##### Requirement: Inconsistent p-values are unset Samplers SHOULD unset `p` when the invariant between the `sampled`, `p`, and `r` values is violated before using the `tracestate` to make @@ -271,7 +271,7 @@ sampling probability: | 62 | 2**-62 | 2**62 | | 63 | 0 | 0 | -##### Requirement 1 +##### Requirement: Out-of-range p-values are unset Consumers SHOULD unset `p` from the `tracestate` if the unsigned value is greater than 63 before using the `tracestate` to make a sampling @@ -301,7 +301,7 @@ for spans of a given trace, as follows: | 61 | 2**-62 | 2**-61 and above | | 62 | 2**-62 | 2**-62 and above | -##### Requirement 1 +##### Requirement: Out-of-range r-values unset both p and r Samplers SHOULD unset both `r` and `p` from the `tracestate` if the unsigned value is of `r` is greater than 62 before using the @@ -316,22 +316,21 @@ replacement for the [`ParentBased` Sampler](sdk.md#parentbased). It is required to first validate the `tracestate` and then behave as the `ParentBased` sampler would. -##### Requirement 1 +##### Requirement: ParentBased API compatibility The `ParentConsistentProbabilityBased` Sampler MUST have the same constructor signature as the built-in `ParentBased` sampler in each OpenTelemetry SDK. -##### Requirement 2 +##### Requirement: ParentConsistentProbabilityBased does not modify valid tracestate The `ParentConsistentProbabilityBased` Sampler MUST NOT modify a valid `tracestate`. -##### Requirement 3 +##### Requirement: ParentConsistentProbabilityBased and ParentBased make identical decisions The `ParentConsistentProbabilityBased` Sampler MUST make the same -decision specified for the `ParentBased` sampler when the context is -valid. +decision specified for the `ParentBased` sampler. #### ConsistentProbabilityBased sampler @@ -350,34 +349,28 @@ two probabilistically. For example, 5% sampling can be achieved by selecting 1/16 sampling 60% of the time and 1/32 sampling 40% of the time. -##### Requirement 1 +##### Requirement: TraceIdRatioBased API compatibility The `ConsistentProbabilityBased` Sampler MUST have the same constructor signature as the built-in `TraceIdRatioBased` sampler in each OpenTelemetry SDK. -##### Requirement 2 +##### Requirement: ConsistentProbabilityBased sampler sets r for root span The `ConsistentProbabilityBased` Sampler MUST set `r` when it makes a root sampling decision. -##### Requirement 3 +##### Requirement: ConsistentProbabilityBased sampler unsets p when not sampling The `ConsistentProbabilityBased` Sampler MUST unset `p` from the `tracestate` when it decides not to sample. -##### Requirement 4 +##### Requirement: ConsistentProbabilityBased sampler sets p when sampling The `ConsistentProbabilityBased` Sampler MUST set `p` when it decides to sample. -##### Requirement 5 - -When it decides to sample, the `ConsistentProbabilityBased` Sampler -MUST set `p` to the negative base-2 logarithm of a power-of-two -sampling probability. - -##### Requirement 6 +##### Requirement: ConsistentProbabilityBased sampler sets p using an unbiased algorithm The `ConsistentProbabilityBased` Sampler MUST set `p` so that the adjusted count interpreted from the `tracestate` is an unbiased @@ -385,13 +378,14 @@ estimate of the number of representative spans in the population. A test specification for this requirement is given in the appendix. -##### Requirement 7 +##### Requirement: ConsistentProbabilityBased sampler sets r for non-root span If `r` is not set on the input `tracecontext` and the Span is not a root span, `ConsistentProbabilityBased` SHOULD set `r` as if it were a -root span. +root span and MAY warn the user that a potentially inconsistent trace +is being produced. -##### Requirement 8 +##### Requirement: ConsistentProbabilityBased sampler acts like AlwaysOff sampler for probabilities less than 2**-62 If the configured sampling probability is less than `2**-62`, the Sampler should round down to zero probability and make the same @@ -418,33 +412,39 @@ effect signifying zero adjusted count. #### List of requirements -##### Requirement 1 +##### Requirement: Combining multiple sampling decisions using logical `or` + +When multiple samplers are combined using composition, the sampling +decision is to sample if at least of the combined samplers decies to +sample. + +##### Requirement: Combine multiple consistent probability samplers using the minimum p-value When combining Sampler decisions for multiple consistent probability Samplers and at least one decides to sample, the minimum of the "yes" decision `p` values MUST be set in the `tracestate`. -##### Requirement 2 +##### Requirement: Unset p when multiple consistent probability samplers decide not to sample When combining Sampler decisions for multiple consistent probability Samplers and none decides to sample, p-value MUST be unset in the `tracestate`. -##### Requirement 3 +##### Requirement: Use probability sampler p-value when its decision to sample is combined with non-probability samplers When combining Sampler decisions for a consistent probability Sampler and a non-probability Sampler, and the probability Sampler decides to sample, its p-value MUST be set in the `tracestate` regardless of the non-probability Sampler decision. -##### Requirement 4 +##### Requirement: Use p-value 63 when a probability sampler decision not to sample when combined with non-probability sampler decision to sample When combining Sampler decisions for a consistent probability Sampler and a non-probability Sampler, and the probability Sampler decides not to sample but the non-probability does sample, p-value 63 MUST be set in the `tracestate`. -### Consumer-only requirements +### Consumer recommendations #### Trace consumers @@ -457,7 +457,7 @@ means the trace was not correctly sampled at the root for probability sampling. While the adjusted count of each span is correct in this scenario, it may be impossible to detect complete traces. -##### Requirement 1 +##### Recommendation: Recognize inconsistent r-values When a single trace contains spans with `tracestate` values containing more than one distinct value for `r`, the consumer SHOULD recognize @@ -538,7 +538,7 @@ For each probability in the table above, the test is required to demonstrate a seed that produces exactly one ChiSquared value less than 0.102587. -##### Requirement 1 +##### Requirement: Pass 15 non-power-of-two statistical tests For the test with 20 trials and 1 million spans each, the test MUST demonstrate a random number generator seed such that the ChiSquared @@ -570,7 +570,7 @@ For each probability in the table above, the test is required to demonstrate a seed that produces exactly one ChiSquared value less than 0.003932. -##### Requirement 1 +##### Requirement: Pass 5 power-of-two statistical tests For the teset with 20 trials and 1 million spans each, the test MUST demonstrate a random number generator seed such that the ChiSquared From 14dac185294a761429995e89748c67dc2c59afd8 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 2 Nov 2021 14:10:11 -0700 Subject: [PATCH 12/45] add test guidance --- .../trace/tracestate-probability-sampling.md | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 57bb6f4d30e..e40b975464d 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -576,3 +576,44 @@ For the teset with 20 trials and 1 million spans each, the test MUST demonstrate a random number generator seed such that the ChiSquared test statistic is below 0.003932 exactly 1 out of 20 times. +#### Test implementation + +The recommended structure for this test uses a table listing the 20 +probability values, the expected p-values, whether the ChiSquared +statistic has one or two degrees of freedom, and the index into the +predetermined list of seeds. + +``` + for _, test := range []testCase{ + // Non-powers of two + {0.90000, 1, twoDegrees, 5}, + {0.60000, 1, twoDegrees, 14}, + {0.33000, 2, twoDegrees, 3}, + {0.13000, 3, twoDegrees, 2}, + {0.10000, 4, twoDegrees, 0}, + {0.05000, 5, twoDegrees, 0}, + {0.01700, 6, twoDegrees, 2}, + {0.01000, 7, twoDegrees, 3}, + {0.00500, 8, twoDegrees, 1}, + {0.00290, 9, twoDegrees, 1}, + {0.00100, 10, twoDegrees, 5}, + {0.00050, 11, twoDegrees, 1}, + {0.00026, 12, twoDegrees, 3}, + {0.00023, 13, twoDegrees, 0}, + {0.00010, 14, twoDegrees, 2}, + + // Powers of two + {0x1p-1, 1, oneDegree, 0}, + {0x1p-4, 4, oneDegree, 2}, + {0x1p-7, 7, oneDegree, 3}, + {0x1p-10, 10, oneDegree, 0}, + {0x1p-13, 13, oneDegree, 1}, + } { +``` + +Note that seed indexes in the example above have what appears to be +the correct distribution. The five 0s, four 1s, four 2s, four 3s, and +two 5s demonstrate that it is relatively easy to find examples where +there is exactly one failure. Seed index 14, for probability 0.6 in +this case, is a reminder that outliers exist. Further significance +testing of this distribution is not recommended. From 30b4f347733eaa4ed3da659298a5d948f15961e8 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 4 Nov 2021 11:38:34 -0700 Subject: [PATCH 13/45] table origin=1 --- .../trace/tracestate-probability-sampling.md | 44 +++++++++---------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index e40b975464d..47c2997845f 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -508,21 +508,21 @@ The following table summarizes the test parameters. | Test case | Sampling probability | Lower, Upper p-value when sampled | Expectlower | Expectupper | Expectunsampled | | --- | --- | --- | --- | --- | --- | -| 0 | 0.900000 | 0, 1 | 100000 | 800000 | 100000 | -| 1 | 0.600000 | 0, 1 | 400000 | 200000 | 400000 | -| 2 | 0.330000 | 1, 2 | 170000 | 160000 | 670000 | -| 3 | 0.130000 | 2, 3 | 120000 | 10000 | 870000 | -| 4 | 0.100000 | 3, 4 | 25000 | 75000 | 900000 | -| 5 | 0.050000 | 4, 5 | 12500 | 37500 | 950000 | -| 6 | 0.017000 | 5, 6 | 14250 | 2750 | 983000 | -| 7 | 0.010000 | 6, 7 | 5625 | 4375 | 990000 | -| 8 | 0.005000 | 7, 8 | 2812.5 | 2187.5 | 995000 | -| 9 | 0.002900 | 8, 9 | 1006.25 | 1893.75 | 997100 | -| 10 | 0.001000 | 9, 10 | 953.125 | 46.875 | 999000 | -| 11 | 0.000500 | 10, 11 | 476.5625 | 23.4375 | 999500 | -| 12 | 0.000260 | 11, 12 | 228.28125 | 31.71875 | 999740 | -| 13 | 0.000230 | 12, 13 | 14.140625 | 215.859375 | 999770 | -| 14 | 0.000100 | 13, 14 | 22.0703125 | 77.9296875 | 999900 | +| 1 | 0.900000 | 0, 1 | 100000 | 800000 | 100000 | +| 2 | 0.600000 | 0, 1 | 400000 | 200000 | 400000 | +| 3 | 0.330000 | 1, 2 | 170000 | 160000 | 670000 | +| 4 | 0.130000 | 2, 3 | 120000 | 10000 | 870000 | +| 5 | 0.100000 | 3, 4 | 25000 | 75000 | 900000 | +| 6 | 0.050000 | 4, 5 | 12500 | 37500 | 950000 | +| 7 | 0.017000 | 5, 6 | 14250 | 2750 | 983000 | +| 8 | 0.010000 | 6, 7 | 5625 | 4375 | 990000 | +| 9 | 0.005000 | 7, 8 | 2812.5 | 2187.5 | 995000 | +| 10 | 0.002900 | 8, 9 | 1006.25 | 1893.75 | 997100 | +| 11 | 0.001000 | 9, 10 | 953.125 | 46.875 | 999000 | +| 12 | 0.000500 | 10, 11 | 476.5625 | 23.4375 | 999500 | +| 13 | 0.000260 | 11, 12 | 228.28125 | 31.71875 | 999740 | +| 14 | 0.000230 | 12, 13 | 14.140625 | 215.859375 | 999770 | +| 15 | 0.000100 | 13, 14 | 22.0703125 | 77.9296875 | 999900 | The formula for computing Chi-Squared in this case is: @@ -549,13 +549,13 @@ test statistic is below 0.102587 exactly 1 out of 20 times. In this case there is one degree of freedom for the Chi-Squared test. The following table summarizes the test parameters. -| Test case | Sampling probability | P-value when sampled | Expectsampled | Expectunsampled | -| --- | --- | --- | --- | --- | -| 15 | 0x1p-01 (0.500000) | 1 | 500000 | 500000 | -| 16 | 0x1p-04 (0.062500) | 4 | 62500 | 937500 | -| 17 | 0x1p-07 (0.007812) | 7 | 7812.5 | 992187.5 | -| 18 | 0x1p-10 (0.000977) | 10 | 976.5625 | 999023.4375 | -| 19 | 0x1p-13 (0.000122) | 13 | 122.0703125 | 999877.9297 | +| Test case | Sampling probability | P-value when sampled | Expectsampled | Expectunsampled | | +| --- | --- | --- | --- | --- | | +| 16 | 0x1p-01 (0.500000) | 1 | 500000 | n/a | 500000 | +| 17 | 0x1p-04 (0.062500) | 4 | 62500 | n/a | 937500 | +| 18 | 0x1p-07 (0.007812) | 7 | 7812.5 | n/a | 992187.5 | +| 19 | 0x1p-10 (0.000977) | 10 | 976.5625 | n/a | 999023.4375 | +| 20 | 0x1p-13 (0.000122) | 13 | 122.0703125 | n/a | 999877.9297 | The formula for computing Chi-Squared in this case is: From 67dd0d6fb11d3afb0b5b798834524f67a1eebe10 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 4 Nov 2021 12:04:51 -0700 Subject: [PATCH 14/45] remove the probability-of-r-value column, it causes confusion --- .../trace/tracestate-probability-sampling.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 47c2997845f..6c8dc6bb934 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -259,7 +259,7 @@ Zero adjusted count is represented by the special p-value 63, otherwise the p-value is set to the negative base-2 logarithm of sampling probability: -| p-value | Parent Probability | Adjusted count | +| P-value | Parent Probability | Adjusted count | | ----- | ----------- | -- | | 0 | 1 | 1 | | 1 | 1/2 | 2 | @@ -287,19 +287,19 @@ to participate in consistent probability sampling. R-value determines which sampling probabilities and will not sample for spans of a given trace, as follows: -| r-value | Probability of r-value | Implied sampling probabilities | -| ---------------- | ------------------------ | ---------------------- | -| 0 | 1/2 | 1 | -| 1 | 1/4 | 1/2 and above | -| 2 | 1/8 | 1/4 and above | -| 3 | 1/16 | 1/8 and above | -| ... | ... | ... | -| 0 <= r <= 61 | 1/(2**(-r-1)) | 2**(-r) and above | -| ... | ... | ... | -| 59 | 2**-60 | 2**-59 and above | -| 60 | 2**-61 | 2**-60 and above | -| 61 | 2**-62 | 2**-61 and above | -| 62 | 2**-62 | 2**-62 and above | +| R-value | Implied sampling probabilities | +| -------------- | ---------------------- | +| 0 | 1 | +| 1 | 1/2 and above | +| 2 | 1/4 and above | +| 3 | 1/8 and above | +| ... | ... | +| 0 <= r <= 62 | 2**-r and above | +| ... | ... | +| 59 | 2**-59 and above | +| 60 | 2**-60 and above | +| 61 | 2**-61 and above | +| 62 | 2**-62 and above | ##### Requirement: Out-of-range r-values unset both p and r From 1c510d7cae134e0db4b4601276f4b02524e9c084 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 4 Nov 2021 12:51:09 -0700 Subject: [PATCH 15/45] Reformat probability of r-value; add r-value requirement --- .../trace/tracestate-probability-sampling.md | 42 ++++++++++++------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 6c8dc6bb934..d74e00ef7e7 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -214,7 +214,7 @@ fields are required to meet the behavioral requirements stated for the ConsistentProbabilityBased sampler, to ensure statistically valid outcomes, using a test suite included in this specification, so that users and consumers of OpenTelemetry `tracestate` can be assured of -accurate span counts in a span-to-metrics pipeline. +accurate span counts in a Span-to-metrics pipeline. ### Context invariants @@ -240,7 +240,7 @@ Probability sampling uses additional information to enable consistent decision making and to record the adjusted count of sampled spans. When both values are defined and in the specified range, the invariant between r-value and p-value and the `sampled` trace flag states that -`sampled` is equivalent to the expression `p <= r || p == 63`. +`((p <= r) == sampled) OR (sampled AND (p == 63)) == TRUE`. The invariant between `sampled`, `p`, and `r` only applies when both `p` and `r` are present. When the invariant is violated, the @@ -259,7 +259,7 @@ Zero adjusted count is represented by the special p-value 63, otherwise the p-value is set to the negative base-2 logarithm of sampling probability: -| P-value | Parent Probability | Adjusted count | +| p-value | Parent Probability | Adjusted count | | ----- | ----------- | -- | | 0 | 1 | 1 | | 1 | 1/2 | 2 | @@ -287,19 +287,24 @@ to participate in consistent probability sampling. R-value determines which sampling probabilities and will not sample for spans of a given trace, as follows: -| R-value | Implied sampling probabilities | -| -------------- | ---------------------- | -| 0 | 1 | -| 1 | 1/2 and above | -| 2 | 1/4 and above | -| 3 | 1/8 and above | -| ... | ... | -| 0 <= r <= 62 | 2**-r and above | -| ... | ... | -| 59 | 2**-59 and above | -| 60 | 2**-60 and above | -| 61 | 2**-61 and above | -| 62 | 2**-62 and above | +| r-value | Probability of r-value | Implied sampling probabilities | +| ---------------- | ------------------------ | ---------------------- | +| 0 | 1/2 | 1 | +| 1 | 1/4 | 1/2 and above | +| 2 | 1/8 | 1/4 and above | +| 3 | 1/16 | 1/8 and above | +| ... | ... | ... | +| 0 <= r <= 61 | 2**-(r+1) | 2**-r and above | +| ... | ... | ... | +| 59 | 2**-60 | 2**-59 and above | +| 60 | 2**-61 | 2**-60 and above | +| 61 | 2**-62 | 2**-61 and above | +| 62 | 2**-62 | 2**-62 and above | + +These probabilities are specified to ensure that conforming Sampler +implementations record spans with correct adjusted counts. The +recommended method of generating r-values is to count the number of +leading 0s in a string of 62 random bits. ##### Requirement: Out-of-range r-values unset both p and r @@ -307,6 +312,11 @@ Samplers SHOULD unset both `r` and `p` from the `tracestate` if the unsigned value is of `r` is greater than 62 before using the `tracestate` to make a sampling decision. +##### Requirement: R-value is generated with the correct probabilities + +Samplers MUST generate r-values using a randomized scheme that +produces each value with the specified probability. + ### Samplers #### ParentConsistentProbabilityBased sampler From b04c56de3a65dd9408887c1a101a1ad53bf08dd1 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 4 Nov 2021 14:02:51 -0700 Subject: [PATCH 16/45] reorganize (partly) as suggested by yuri --- .../trace/tracestate-probability-sampling.md | 44 ++++++++++--------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index d74e00ef7e7..1db9dd4d848 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -114,19 +114,11 @@ Zero adjusted count is defined in a way that supports composition of probability and non-probability sampling. Zero is assigned as the adjusted count when a probability sampler does not select a span. -#### Power-of-two random sampling +#### Simple random sampling -A simple sampling scheme can be implemented using a random bit string -as the input. This scheme is limited to power-of-two sampling -probabilities, as follows. - -1. Express the sampling probability as `2**-s`. For example, 25% - equals `2**-2` with `s=2` -2. Count `r`, the number of consecutive zero bits in the input string -3. If `s <= r`, select the item with adjusted count `2**s`. - -This algorithm is the basis of the consistent probability sampling -approach used in OpenTelemetry, defined in greater detail below. +Simple random sampling is a scheme for selecting items that chooses +items independently with fixed probability `prob`. The adjusted count +of each selected item is set to `1 / prob`. #### Sampler @@ -185,7 +177,7 @@ effectively unrepresented. A non-probability sampler is a Sampler that makes its decisions not based on chance, but instead uses arbitrary logic and internal state. -## Probability sampling +## Consistent Probability sampling The consistent sampling scheme adopted by OpenTelemetry propagates two values via the context, termed "p-value" and "r-value": @@ -196,9 +188,19 @@ values via the context, termed "p-value" and "r-value": Both fields are propagated via the OpenTelemetry `tracestate` under the `ot` vendor tag using the rules for [tracestate handling](tracestate-handling.md). Both fields are represented as -unsigned integers requiring at most 6 bits of information. An -invariant will be stated that connects the `sampled` trace flag found -in `traceparent` context to the r-value and p-value found in +unsigned integers requiring at most 6 bits of information. + +Unlike the simple random sampling algorithm define above, this scheme +selects items from among a fixed set of 63 distinct probability +values, those being the integer powers of two between 1 and 2**-62. + +R-value encodes which among the 63 possibilities will decide to sample +for a given trace. P-value encodes the adjusted count for child +contexts (i.e., consumers of `tracestate`) and consumers of sampled +spans. + +An invariant will be stated that connects the `sampled` trace flag +found in `traceparent` context to the r-value and p-value found in `tracestate` context. ### Conformance @@ -211,10 +213,10 @@ interpreting span adjusted counts. Producers of OpenTelemetry `tracestate` containing p-value and r-value fields are required to meet the behavioral requirements stated for the -ConsistentProbabilityBased sampler, to ensure statistically valid -outcomes, using a test suite included in this specification, so that +ConsistentProbabilityBased sampler and to ensure statistically valid +outcomes. A test suite is included in this specification so that users and consumers of OpenTelemetry `tracestate` can be assured of -accurate span counts in a Span-to-metrics pipeline. +accuracy in Span-to-metrics pipelines. ### Context invariants @@ -378,9 +380,9 @@ The `ConsistentProbabilityBased` Sampler MUST unset `p` from the ##### Requirement: ConsistentProbabilityBased sampler sets p when sampling The `ConsistentProbabilityBased` Sampler MUST set `p` when it decides -to sample. +to sample according to its configured sampling probability. -##### Requirement: ConsistentProbabilityBased sampler sets p using an unbiased algorithm +##### Requirement: ConsistentProbabilityBased records unbiased adjusted counts The `ConsistentProbabilityBased` Sampler MUST set `p` so that the adjusted count interpreted from the `tracestate` is an unbiased From cf6c2af136ec2a94e6320d5266938a7dc9341529 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 4 Nov 2021 15:28:38 -0700 Subject: [PATCH 17/45] reorder sections --- .../trace/tracestate-probability-sampling.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 1db9dd4d848..5f555efccf2 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -7,7 +7,7 @@ * [Definitions](#definitions) + [Sampling](#sampling) + [Adjusted count](#adjusted-count) - + [Power-of-two random sampling](#power-of-two-random-sampling) + + [Simple random sampling](#simple-random-sampling) + [Sampler](#sampler) + [Parent-based sampler](#parent-based-sampler) + [Probability sampler](#probability-sampler) @@ -15,7 +15,7 @@ + [Always-on sampler](#always-on-sampler) + [Always-off sampler](#always-off-sampler) + [Non-probability sampler](#non-probability-sampler) -- [Probability sampling](#probability-sampling) +- [Consistent Probability sampling](#consistent-probability-sampling) * [Conformance](#conformance) * [Context invariants](#context-invariants) + [Sampled flag](#sampled-flag) @@ -24,6 +24,7 @@ - [Requirement: Out-of-range p-values are unset](#requirement-out-of-range-p-values-are-unset) + [R-value](#r-value) - [Requirement: Out-of-range r-values unset both p and r](#requirement-out-of-range-r-values-unset-both-p-and-r) + - [Requirement: R-value is generated with the correct probabilities](#requirement-r-value-is-generated-with-the-correct-probabilities) * [Samplers](#samplers) + [ParentConsistentProbabilityBased sampler](#parentconsistentprobabilitybased-sampler) - [Requirement: ParentBased API compatibility](#requirement-parentbased-api-compatibility) @@ -34,7 +35,7 @@ - [Requirement: ConsistentProbabilityBased sampler sets r for root span](#requirement-consistentprobabilitybased-sampler-sets-r-for-root-span) - [Requirement: ConsistentProbabilityBased sampler unsets p when not sampling](#requirement-consistentprobabilitybased-sampler-unsets-p-when-not-sampling) - [Requirement: ConsistentProbabilityBased sampler sets p when sampling](#requirement-consistentprobabilitybased-sampler-sets-p-when-sampling) - - [Requirement: ConsistentProbabilityBased sampler sets p using an unbiased algorithm](#requirement-consistentprobabilitybased-sampler-sets-p-using-an-unbiased-algorithm) + - [Requirement: ConsistentProbabilityBased records unbiased adjusted counts](#requirement-consistentprobabilitybased-records-unbiased-adjusted-counts) - [Requirement: ConsistentProbabilityBased sampler sets r for non-root span](#requirement-consistentprobabilitybased-sampler-sets-r-for-non-root-span) - [Requirement: ConsistentProbabilityBased sampler acts like AlwaysOff sampler for probabilities less than 2**-62](#requirement-consistentprobabilitybased-sampler-acts-like-alwaysoff-sampler-for-probabilities-less-than-2-62) * [Composition rules](#composition-rules) @@ -52,6 +53,7 @@ - [Requirement: Pass 15 non-power-of-two statistical tests](#requirement-pass-15-non-power-of-two-statistical-tests) + [Test procedure: exact powers of two](#test-procedure-exact-powers-of-two) - [Requirement: Pass 5 power-of-two statistical tests](#requirement-pass-5-power-of-two-statistical-tests) + + [Test implementation](#test-implementation) @@ -180,10 +182,7 @@ based on chance, but instead uses arbitrary logic and internal state. ## Consistent Probability sampling The consistent sampling scheme adopted by OpenTelemetry propagates two -values via the context, termed "p-value" and "r-value": - -1. p-value: the "parent probability" value can be set independently by any span in the trace, for its children, and informs child parent-based Samplers of their adjusted count -2. r-value: the "randomness" value is determined and propagated from the root to all spans in the trace and serves to make sampling decisions consistent +values via the context, termed "p-value" and "r-value". Both fields are propagated via the OpenTelemetry `tracestate` under the `ot` vendor tag using the rules for [tracestate @@ -194,10 +193,10 @@ Unlike the simple random sampling algorithm define above, this scheme selects items from among a fixed set of 63 distinct probability values, those being the integer powers of two between 1 and 2**-62. -R-value encodes which among the 63 possibilities will decide to sample -for a given trace. P-value encodes the adjusted count for child -contexts (i.e., consumers of `tracestate`) and consumers of sampled -spans. +R-value encodes which among the 63 possibilities will consistently +decide to sample for a given trace. P-value encodes the adjusted count +for child contexts (i.e., consumers of `tracestate`) and consumers of +sampled spans to record for use in Span-to-metrics pipelines. An invariant will be stated that connects the `sampled` trace flag found in `traceparent` context to the r-value and p-value found in @@ -306,7 +305,8 @@ for spans of a given trace, as follows: These probabilities are specified to ensure that conforming Sampler implementations record spans with correct adjusted counts. The recommended method of generating r-values is to count the number of -leading 0s in a string of 62 random bits. +leading 0s in a string of 62 random bits, however it is not required +to use this approach. ##### Requirement: Out-of-range r-values unset both p and r From 11fdaae2c3622fe9ef4f4b770c8740f59424b96e Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 4 Nov 2021 15:29:51 -0700 Subject: [PATCH 18/45] typeo --- specification/trace/tracestate-probability-sampling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 5f555efccf2..dcb734f688f 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -449,7 +449,7 @@ and a non-probability Sampler, and the probability Sampler decides to sample, its p-value MUST be set in the `tracestate` regardless of the non-probability Sampler decision. -##### Requirement: Use p-value 63 when a probability sampler decision not to sample when combined with non-probability sampler decision to sample +##### Requirement: Use p-value 63 when a probability sampler decision not to sample is combined with a non-probability sampler decision to sample When combining Sampler decisions for a consistent probability Sampler and a non-probability Sampler, and the probability Sampler decides not From 961fbef0d4809b16e107899a771492c0c08df5d2 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 4 Nov 2021 15:54:26 -0700 Subject: [PATCH 19/45] revise power-of-two sampling intro --- .../trace/tracestate-probability-sampling.md | 27 ++++++++++--------- 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index dcb734f688f..525129f292e 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -116,12 +116,6 @@ Zero adjusted count is defined in a way that supports composition of probability and non-probability sampling. Zero is assigned as the adjusted count when a probability sampler does not select a span. -#### Simple random sampling - -Simple random sampling is a scheme for selecting items that chooses -items independently with fixed probability `prob`. The adjusted count -of each selected item is set to `1 / prob`. - #### Sampler A Sampler provides configurable logic, used by the SDK, for selecting @@ -189,14 +183,23 @@ the `ot` vendor tag using the rules for [tracestate handling](tracestate-handling.md). Both fields are represented as unsigned integers requiring at most 6 bits of information. -Unlike the simple random sampling algorithm define above, this scheme -selects items from among a fixed set of 63 distinct probability -values, those being the integer powers of two between 1 and 2**-62. +This sampling scheme selects items from among a fixed set of 63 +distinct probability values. The set of supported probabilities +includes the integer powers of two between 1 and 2**-62. R-value encodes which among the 63 possibilities will consistently -decide to sample for a given trace. P-value encodes the adjusted count -for child contexts (i.e., consumers of `tracestate`) and consumers of -sampled spans to record for use in Span-to-metrics pipelines. +decide to sample for a given trace. Specifically, r-value specifies +the smallest probability that will decide to sample a given trace in +terms of the corresponding p-value. For example, a trace with r-value +0 will sample spans configured for 100% sampling, while r-value 1 will +sample spans configured for 50% or 100% sampling, and so on through +r-value 62 which will sample every span. + +P-value encodes the adjusted count for child contexts (i.e., consumers +of `tracestate`) and consumers of sampled spans to record for use in +Span-to-metrics pipelines. A special p-value of 63 is defined to mean +zero adjusted count, which helps define composition rules for +non-probability samplers. An invariant will be stated that connects the `sampled` trace flag found in `traceparent` context to the r-value and p-value found in From e3f9f0b612056eb040ae294d51cd3e089340ec83 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 4 Nov 2021 15:54:41 -0700 Subject: [PATCH 20/45] toc --- specification/trace/tracestate-probability-sampling.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 525129f292e..c2475001a26 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -7,7 +7,6 @@ * [Definitions](#definitions) + [Sampling](#sampling) + [Adjusted count](#adjusted-count) - + [Simple random sampling](#simple-random-sampling) + [Sampler](#sampler) + [Parent-based sampler](#parent-based-sampler) + [Probability sampler](#probability-sampler) @@ -44,7 +43,7 @@ - [Requirement: Combine multiple consistent probability samplers using the minimum p-value](#requirement-combine-multiple-consistent-probability-samplers-using-the-minimum-p-value) - [Requirement: Unset p when multiple consistent probability samplers decide not to sample](#requirement-unset-p-when-multiple-consistent-probability-samplers-decide-not-to-sample) - [Requirement: Use probability sampler p-value when its decision to sample is combined with non-probability samplers](#requirement-use-probability-sampler-p-value-when-its-decision-to-sample-is-combined-with-non-probability-samplers) - - [Requirement: Use p-value 63 when a probability sampler decision not to sample when combined with non-probability sampler decision to sample](#requirement-use-p-value-63-when-a-probability-sampler-decision-not-to-sample-when-combined-with-non-probability-sampler-decision-to-sample) + - [Requirement: Use p-value 63 when a probability sampler decision not to sample is combined with a non-probability sampler decision to sample](#requirement-use-p-value-63-when-a-probability-sampler-decision-not-to-sample-is-combined-with-a-non-probability-sampler-decision-to-sample) * [Consumer recommendations](#consumer-recommendations) + [Trace consumers](#trace-consumers) - [Recommendation: Recognize inconsistent r-values](#recommendation-recognize-inconsistent-r-values) From 8a01692529e106dc468294b0dbaf7a9788ba479a Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Fri, 5 Nov 2021 00:19:21 -0700 Subject: [PATCH 21/45] clarify producer recommendations re: use of non-powers-of-two --- .../trace/tracestate-probability-sampling.md | 42 +++++++++++++++---- 1 file changed, 34 insertions(+), 8 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index c2475001a26..6ad945b0546 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -147,13 +147,15 @@ probability is treated as a special, non-probabilistic case. #### Consistent probability sampler -A consistent probability sampler is a Sampler that supports independent -sampling decisions at each span in a trace while maintaining that -traces will be complete with probability equal to the minimum sampling -probability across the trace. Consistent probability sampling requires that -for any span in a given trace, if a Sampler with lesser sampling probability -selects the span for sampling, then the span would also be selected by a -Sampler configured with greater sampling probability. +A consistent probability sampler is a Sampler that supports +independent sampling decisions at each span in a trace while +maintaining that traces will be complete with a certain minimum +probability across the trace. + +Consistent probability sampling requires that for any span in a given +trace, if a Sampler with lesser sampling probability selects the span +for sampling, then the span would also be selected by a Sampler +configured with greater sampling probability. #### Always-on sampler @@ -219,6 +221,21 @@ outcomes. A test suite is included in this specification so that users and consumers of OpenTelemetry `tracestate` can be assured of accuracy in Span-to-metrics pipelines. +### Completeness guarantee + +This specification defines consistent sampling for power-of-two +sampling probabilities. When a sampler is configured with a +non-power-of-two sampling probability, the sampler will +probabilistically choose between the nearest powers of two, and as a +result, trace completeness is only ensured at the smallest power of +two greater than or equal to the minimum sampling rate across the +trace, with one exception. + +The root span in a trace can be configured for arbitrary sampling +probability without risking incomplete traces, therefore +non-power-of-two sampling probability is recomended for use only at +trace roots. + ### Context invariants The W3C `traceparent` (version 0) contains three fields of @@ -458,7 +475,16 @@ and a non-probability Sampler, and the probability Sampler decides not to sample but the non-probability does sample, p-value 63 MUST be set in the `tracestate`. -### Consumer recommendations +### Producer and consumer recommendations + +#### Trace producers + +As stated in the [completeness guarantee](#completeness-guarantee), +traces will be possibly incomplete when the sampling probability is +not a power of two for non-root spans. When complete traces are +desired, users are recommended to configure non-root spans for +sampling by a parent-based sampler or an exact power-of-two consistent +probability sampler. #### Trace consumers From a38e321c282f42f2df932d91af62074fbe841e48 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 8 Nov 2021 12:21:00 -0800 Subject: [PATCH 22/45] Rephrase consistency guarantees --- .../trace/tracestate-probability-sampling.md | 104 +++++++++++++----- 1 file changed, 76 insertions(+), 28 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 6ad945b0546..bd5e75137aa 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -157,22 +157,50 @@ trace, if a Sampler with lesser sampling probability selects the span for sampling, then the span would also be selected by a Sampler configured with greater sampling probability. -#### Always-on sampler +#### Trace completeness -An always-on sampler is another name for a consistent probability -sampler with probability equal to one. +A trace is said to be complete when all of the spans belonging to the +trace are collected. When at least one span is collected but not all +spans are collected, the trace is considered incomplete. -#### Always-off sampler +Trace incompleteness may happen on purpose (e.g., through sampling +configuration), or by accident (e.g., through collection errors). The +OpenTelemetry trace data model supports a _one-way_ test for +incompleteness: for any non-root span, the trace is definitely +incomplete if the span's parent span was not collected. -An always-off Sampler has the effect of disabling a span completely, -effectively excluding it from the population. This is not defined as -a probability sampler with zero probability, because these spans are -effectively unrepresented. +Incomplete traces that result from sampling configuration (i.e., on +purpose) are known as partial traces. An important subset of the +partial traces are those which are also complete subtraces. A +complete subtrace is defined at span S when: + +- S is not a root span +- S's parent was not collected +- S either has no children or all the children of S are also complete +subtraces. + +Since the test for an incompleteness is one-way, it is important to +know which sampling configurations may lead to incomplete traces. +Sampling configurations that lead naturally to complete traces and +complete subtraces are [discussed below](#trace-producers). #### Non-probability sampler A non-probability sampler is a Sampler that makes its decisions not based on chance, but instead uses arbitrary logic and internal state. +The adjusted count of spans sampled by a non-probabiliy is unknown. + +#### Always-on consistent probability sampler + +An always-on sampler is another name for a consistent probability +sampler with probability equal to one. + +#### Always-off sampler + +An always-off Sampler has the effect of disabling a span completely, +effectively excluding it from the population. This is defined as a +non-probability sampler, not a zero-percent probability sampler, +because the spans are effectively unrepresented. ## Consistent Probability sampling @@ -216,7 +244,7 @@ interpreting span adjusted counts. Producers of OpenTelemetry `tracestate` containing p-value and r-value fields are required to meet the behavioral requirements stated for the -ConsistentProbabilityBased sampler and to ensure statistically valid +`ConsistentProbabilityBased` sampler and to ensure statistically valid outcomes. A test suite is included in this specification so that users and consumers of OpenTelemetry `tracestate` can be assured of accuracy in Span-to-metrics pipelines. @@ -226,15 +254,17 @@ accuracy in Span-to-metrics pipelines. This specification defines consistent sampling for power-of-two sampling probabilities. When a sampler is configured with a non-power-of-two sampling probability, the sampler will -probabilistically choose between the nearest powers of two, and as a -result, trace completeness is only ensured at the smallest power of -two greater than or equal to the minimum sampling rate across the -trace, with one exception. +probabilistically choose between the nearest powers of two + +When a single consistent probability sampler is used at the root of a +trace, the resulting traces are always complete (ignoring collection +errors). This property holds even for non-power-of-two sampling +probabilities. -The root span in a trace can be configured for arbitrary sampling -probability without risking incomplete traces, therefore -non-power-of-two sampling probability is recomended for use only at -trace roots. +When multiple consistent probability samplers are used in the same +trace, in general, trace completeness is ensured at the smallest power +of two greater than or equal to the minimum sampling probability +across the trace. ### Context invariants @@ -480,22 +510,40 @@ in the `tracestate`. #### Trace producers As stated in the [completeness guarantee](#completeness-guarantee), -traces will be possibly incomplete when the sampling probability is -not a power of two for non-root spans. When complete traces are -desired, users are recommended to configure non-root spans for -sampling by a parent-based sampler or an exact power-of-two consistent -probability sampler. +traces will be possibly incomplete when configuring multiple +consistent probability samplers in the same trace. One way to avoid +producing incomplete traces is to use parent-based samplers except for +root spans. + +There is a simple test for trace incompleteness, but it is a one-way +test and does not detect when child spans are uncollected. One way to +avoid producing incomplete traces is to avoid configuring +non-power-of-two sampling probabilities for non-root spans, because +completeness is not guaranteed for non-power-of-two sampling +probabilities. + +Complete subtraces will be produced when the sequence of sampling +probabilities from the root of a trace to its leaves is +non-descending. To ensure complete sub-traces are produced, configure +a child-span's sampler with probability greater than or equal to the +parent span's sampling probability. #### Trace consumers +Trace consumers are expected to apply the simple one-way test for +incompleteness. When non-root spans are configured with independent +sampling probabilities, traces may be complete in a way that cannot be +detected. Because of the one-way test, consumers wanting to ensure +complete traces are expected to know the minimum sampling probability +across the system. + Due to the `ConsistentProbabilityBased` Sampler requirement about setting `r` when it is unset for a non-root span, trace consumers are -advised to check traces for r-value consistency. - -When a single trace contains more than a single distinct `r` value, it -means the trace was not correctly sampled at the root for probability -sampling. While the adjusted count of each span is correct in this -scenario, it may be impossible to detect complete traces. +advised to check traces for r-value consistency. When a single trace +contains more than a single distinct `r` value, it means the trace was +not correctly sampled at the root for probability sampling. While the +adjusted count of each span is correct in this scenario, it may be +impossible to detect complete traces. ##### Recommendation: Recognize inconsistent r-values From 514545a910688f8a9fea84e05c43dc3bc7e95163 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 16 Nov 2021 10:30:53 -0800 Subject: [PATCH 23/45] Apply suggestions from code review Co-authored-by: Diego Hurtado Co-authored-by: J. Kalyana Sundaram --- specification/trace/tracestate-probability-sampling.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index bd5e75137aa..5d41a070f94 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -449,7 +449,7 @@ is being produced. ##### Requirement: ConsistentProbabilityBased sampler acts like AlwaysOff sampler for probabilities less than 2**-62 If the configured sampling probability is less than `2**-62`, the -Sampler should round down to zero probability and make the same +Sampler SHOULD round down to zero probability and make the same sampling decision as the builtin `AlwaysOff` sampler would. ### Composition rules @@ -476,7 +476,7 @@ effect signifying zero adjusted count. ##### Requirement: Combining multiple sampling decisions using logical `or` When multiple samplers are combined using composition, the sampling -decision is to sample if at least of the combined samplers decies to +decision is to sample if at least one of the combined samplers decides to sample. ##### Requirement: Combine multiple consistent probability samplers using the minimum p-value From 93a9b37b2c05fae9b3b9b56d0122c8a534f897c3 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 16 Nov 2021 10:36:53 -0800 Subject: [PATCH 24/45] one keyword --- specification/trace/tracestate-probability-sampling.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 5d41a070f94..03fa915379b 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -476,8 +476,8 @@ effect signifying zero adjusted count. ##### Requirement: Combining multiple sampling decisions using logical `or` When multiple samplers are combined using composition, the sampling -decision is to sample if at least one of the combined samplers decides to -sample. +decision MUST be to sample if at least one of the combined samplers +decides to sample. ##### Requirement: Combine multiple consistent probability samplers using the minimum p-value From c497172a2106b34d36f786fab8da8a78057e832e Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 16 Nov 2021 10:50:07 -0800 Subject: [PATCH 25/45] minor rephrasing --- .../trace/tracestate-probability-sampling.md | 23 ++++++++++--------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 03fa915379b..26e3e75ee45 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -254,12 +254,12 @@ accuracy in Span-to-metrics pipelines. This specification defines consistent sampling for power-of-two sampling probabilities. When a sampler is configured with a non-power-of-two sampling probability, the sampler will -probabilistically choose between the nearest powers of two +probabilistically choose between the nearest powers of two. When a single consistent probability sampler is used at the root of a -trace, the resulting traces are always complete (ignoring collection -errors). This property holds even for non-power-of-two sampling -probabilities. +trace and all other spans use a parent-based sampler, the resulting +traces are always complete (ignoring collection errors). This +property holds even for non-power-of-two sampling probabilities. When multiple consistent probability samplers are used in the same trace, in general, trace completeness is ensured at the smallest power @@ -431,7 +431,7 @@ The `ConsistentProbabilityBased` Sampler MUST unset `p` from the The `ConsistentProbabilityBased` Sampler MUST set `p` when it decides to sample according to its configured sampling probability. -##### Requirement: ConsistentProbabilityBased records unbiased adjusted counts +##### Requirement: ConsistentProbabilityBased sampler records unbiased adjusted counts The `ConsistentProbabilityBased` Sampler MUST set `p` so that the adjusted count interpreted from the `tracestate` is an unbiased @@ -512,8 +512,8 @@ in the `tracestate`. As stated in the [completeness guarantee](#completeness-guarantee), traces will be possibly incomplete when configuring multiple consistent probability samplers in the same trace. One way to avoid -producing incomplete traces is to use parent-based samplers except for -root spans. +producing incomplete traces is to use parent-based samplers except forr +oot spans. There is a simple test for trace incompleteness, but it is a one-way test and does not detect when child spans are uncollected. One way to @@ -523,10 +523,11 @@ completeness is not guaranteed for non-power-of-two sampling probabilities. Complete subtraces will be produced when the sequence of sampling -probabilities from the root of a trace to its leaves is -non-descending. To ensure complete sub-traces are produced, configure -a child-span's sampler with probability greater than or equal to the -parent span's sampling probability. +probabilities from the root of a trace to its leaves consists of +non-descending powers of two. To ensure complete sub-traces are +produced, configure a child-span's sampler to a power-of-two +probability greater than or equal to the parent span's sampling +probability. #### Trace consumers From 85c8939a25acea995c7d15836a3985e00d34f73b Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 16 Nov 2021 10:57:10 -0800 Subject: [PATCH 26/45] one paragraph from Peter F and correct the definition of complete subtrace --- .../trace/tracestate-probability-sampling.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 26e3e75ee45..c5d5ed12e06 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -172,12 +172,8 @@ incomplete if the span's parent span was not collected. Incomplete traces that result from sampling configuration (i.e., on purpose) are known as partial traces. An important subset of the partial traces are those which are also complete subtraces. A -complete subtrace is defined at span S when: - -- S is not a root span -- S's parent was not collected -- S either has no children or all the children of S are also complete -subtraces. +complete subtrace is defined at span S when every descendent span is +collected. Since the test for an incompleteness is one-way, it is important to know which sampling configurations may lead to incomplete traces. @@ -538,6 +534,11 @@ detected. Because of the one-way test, consumers wanting to ensure complete traces are expected to know the minimum sampling probability across the system. +Ignoring accidental data loss, a trace will be complete if all its +spans are sampled with consistent probability samplers and the trace's +r-value is larger than the corresponding smallest power of two greater +than or equal to the minimum sampling probability across the trace. + Due to the `ConsistentProbabilityBased` Sampler requirement about setting `r` when it is unset for a non-root span, trace consumers are advised to check traces for r-value consistency. When a single trace From cbc3852b834e498a1dcf75038236502ccb1c1ad5 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Wed, 17 Nov 2021 08:32:14 -0800 Subject: [PATCH 27/45] Apply suggestions from code review Co-authored-by: J. Kalyana Sundaram --- specification/trace/tracestate-probability-sampling.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index c5d5ed12e06..7d7377b2cb4 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -508,8 +508,8 @@ in the `tracestate`. As stated in the [completeness guarantee](#completeness-guarantee), traces will be possibly incomplete when configuring multiple consistent probability samplers in the same trace. One way to avoid -producing incomplete traces is to use parent-based samplers except forr -oot spans. +producing incomplete traces is to use parent-based samplers except for +root spans. There is a simple test for trace incompleteness, but it is a one-way test and does not detect when child spans are uncollected. One way to From 2d9d7ffa0b06dddd3fa2c7f88ec0a7b5d194a957 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 30 Nov 2021 00:02:31 -0800 Subject: [PATCH 28/45] more producer/consumer recommendations --- .../trace/tracestate-probability-sampling.md | 108 ++++++++++++++++-- 1 file changed, 99 insertions(+), 9 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 7d7377b2cb4..ed2cafccd10 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -11,11 +11,13 @@ + [Parent-based sampler](#parent-based-sampler) + [Probability sampler](#probability-sampler) + [Consistent probability sampler](#consistent-probability-sampler) - + [Always-on sampler](#always-on-sampler) - + [Always-off sampler](#always-off-sampler) + + [Trace completeness](#trace-completeness) + [Non-probability sampler](#non-probability-sampler) + + [Always-on consistent probability sampler](#always-on-consistent-probability-sampler) + + [Always-off sampler](#always-off-sampler) - [Consistent Probability sampling](#consistent-probability-sampling) * [Conformance](#conformance) + * [Completeness guarantee](#completeness-guarantee) * [Context invariants](#context-invariants) + [Sampled flag](#sampled-flag) - [Requirement: Inconsistent p-values are unset](#requirement-inconsistent-p-values-are-unset) @@ -34,7 +36,7 @@ - [Requirement: ConsistentProbabilityBased sampler sets r for root span](#requirement-consistentprobabilitybased-sampler-sets-r-for-root-span) - [Requirement: ConsistentProbabilityBased sampler unsets p when not sampling](#requirement-consistentprobabilitybased-sampler-unsets-p-when-not-sampling) - [Requirement: ConsistentProbabilityBased sampler sets p when sampling](#requirement-consistentprobabilitybased-sampler-sets-p-when-sampling) - - [Requirement: ConsistentProbabilityBased records unbiased adjusted counts](#requirement-consistentprobabilitybased-records-unbiased-adjusted-counts) + - [Requirement: ConsistentProbabilityBased sampler records unbiased adjusted counts](#requirement-consistentprobabilitybased-sampler-records-unbiased-adjusted-counts) - [Requirement: ConsistentProbabilityBased sampler sets r for non-root span](#requirement-consistentprobabilitybased-sampler-sets-r-for-non-root-span) - [Requirement: ConsistentProbabilityBased sampler acts like AlwaysOff sampler for probabilities less than 2**-62](#requirement-consistentprobabilitybased-sampler-acts-like-alwaysoff-sampler-for-probabilities-less-than-2-62) * [Composition rules](#composition-rules) @@ -44,8 +46,14 @@ - [Requirement: Unset p when multiple consistent probability samplers decide not to sample](#requirement-unset-p-when-multiple-consistent-probability-samplers-decide-not-to-sample) - [Requirement: Use probability sampler p-value when its decision to sample is combined with non-probability samplers](#requirement-use-probability-sampler-p-value-when-its-decision-to-sample-is-combined-with-non-probability-samplers) - [Requirement: Use p-value 63 when a probability sampler decision not to sample is combined with a non-probability sampler decision to sample](#requirement-use-p-value-63-when-a-probability-sampler-decision-not-to-sample-is-combined-with-a-non-probability-sampler-decision-to-sample) - * [Consumer recommendations](#consumer-recommendations) - + [Trace consumers](#trace-consumers) + * [Producer and consumer recommendations](#producer-and-consumer-recommendations) + + [Trace producer: completeness](#trace-producer-completeness) + - [Recommenendation: use non-descending power-of-two probabilities](#recommenendation-use-non-descending-power-of-two-probabilities) + + [Trace producer: correctness](#trace-producer-correctness) + - [Recommenendation: sampler delegation](#recommenendation-sampler-delegation) + + [Trace producer: interoperability with `ParentBased` sampler](#trace-producer-interoperability-with-parentbased-sampler) + + [Trace producer: interoperability with `TraceIDRatioBased` sampler](#trace-producer-interoperability-with-traceidratiobased-sampler) + + [Trace consumer](#trace-consumer) - [Recommendation: Recognize inconsistent r-values](#recommendation-recognize-inconsistent-r-values) * [Appendix: Statistical test requirements](#appendix-statistical-test-requirements) + [Test procedure: non-powers of two](#test-procedure-non-powers-of-two) @@ -184,7 +192,8 @@ complete subtraces are [discussed below](#trace-producers). A non-probability sampler is a Sampler that makes its decisions not based on chance, but instead uses arbitrary logic and internal state. -The adjusted count of spans sampled by a non-probabiliy is unknown. +The adjusted count of spans sampled by a non-probability sampler is +unknown. #### Always-on consistent probability sampler @@ -503,7 +512,7 @@ in the `tracestate`. ### Producer and consumer recommendations -#### Trace producers +#### Trace producer: completeness As stated in the [completeness guarantee](#completeness-guarantee), traces will be possibly incomplete when configuring multiple @@ -518,14 +527,95 @@ non-power-of-two sampling probabilities for non-root spans, because completeness is not guaranteed for non-power-of-two sampling probabilities. +##### Recommenendation: use non-descending power-of-two probabilities + Complete subtraces will be produced when the sequence of sampling probabilities from the root of a trace to its leaves consists of non-descending powers of two. To ensure complete sub-traces are -produced, configure a child-span's sampler to a power-of-two +produced, child samplers SHOULD be configured with a power-of-two probability greater than or equal to the parent span's sampling probability. -#### Trace consumers +#### Trace producer: correctness + +The use of tracestate to convey adjusted count information rests upon +trust between participants in a trace. Users are advised not to use a +Span-to-metrics pipeline when the parent sampling decision's +corresponding adjusted count is untrustworthy. + +The `ConsistentProbabilityBased` and +`ParentConsistentProbabilityBased` samplers can be used as delegates +of another sampler, for conditioning the choice of sampler on span and +other fixed attributes. However, for adjusted counts to be +trustworthy, the choice of non-root sampler cannot be conditioned on +the parent's sampled trace flag or the OpenTelemetry tracestate +r-value and p-value, as these decisions would lead to incorrect +adjusted counts. + +For example, the built-in [`ParentBased` sampler](sdk.md#parentbased) +supports configuring the delegated sampler based on whether the parent +context is remote or non-remote, sampled or unsampled. If a +`ParentBased` sampler delegates to a `ConsistentProbabilityBased` +sampler only for unsampled contexts, the resulting Span-to-metrics +pipeline will (probably) overcount spans. + +##### Recommenendation: sampler delegation + +For non-root spans, composite samplers SHOULD NOT condition the choice +of delegating sampler based on the parent's sampled flag or +OpenTelemetry tracestate. + +#### Trace producer: interoperability with `ParentBased` sampler + +The OpenTelemetry built-in `ParentBased` sampler is interoperable with +the `ConsistentProbabilityBased` sampler, provided that the +delegated-to sampler does not change the decision that determined its +selection. For example, it is safe to configure an alternate +`ParentBased` sampler delegate for unsampled spans, provided the +decision does not change to sampled. + +Because the `ParentBased` sampler honors the sampled trace flag, and +OpenTelemetry SDKs include the tracestate in the `Span` data, which +means a system can be upgraded to probability sampling by just +replacing `TraceIDRatioBased` samplers with conforming +`ConsistentProbabilityBased` samplers everywhere in the trace. + +#### Trace producer: interoperability with `TraceIDRatioBased` sampler + +The [`TraceIDRatioBased` specification](sdk.md#traceidratiobased) +includes a RECOMMENDATION against being used for non-root spans +because it does not specify how to make the sampler decision +consistent across the trace. A `TraceIDRatioBased` sampler at the +root span is interoperable with a `ConsistentParentProbabilityBased` +sampler in terms of completeness, although the resulting spans will +have unknown adjusted count. + +When a `TraceIDRatioBased` sampler is configured for a non-root span, +several cases arise where an incorrect OpenTelemetry tracestate can be +generated. Consider for example a trace with three spans where the +root (R) has a `ConsistentProbabilityBased` sampler, the root's child +(P) has a `TraceIDRatioBased` sampler, and the grand-child (C) has a +`ParentBased` sampler. Because the `TraceIDRatioBased` sampler change +the intermediate sampled flag without updating the OpenTelemetry +tracestate, we have the following cases: + +1. If `TraceIDRatioBased` does not change P's decision, the trace is + complete and all spans' adjusted counts are correct. +2. If `TraceIDRatioBased` changes P's decision from no to yes, the + consumer will observe a (definitely) incomplete trace containing P + and C. Both spans will have invalid OpenTelemetry tracestate, + leading to unknown adjusted count in this case. +3. If `TraceIDRatioBased` changes the sampling decision from yes to + no, the consumer will observe singleton trace with correct adjusted + count. The consumer cannot determine that R has two unsampled + descendents. + +As these cases demonstrate, users can expect incompleteness and +unknown adjusted count when using `TraceIDRatioBased` samplers for +non-root spans, but this goes against the originally specified +warning. + +#### Trace consumer Trace consumers are expected to apply the simple one-way test for incompleteness. When non-root spans are configured with independent From 3c0dfd81de194d49fd9426c56377a0e45ebd4f43 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 30 Nov 2021 00:09:11 -0800 Subject: [PATCH 29/45] lint --- .../trace/tracestate-probability-sampling.md | 58 +++++++++---------- 1 file changed, 29 insertions(+), 29 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index ed2cafccd10..b387a9e9008 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -279,7 +279,7 @@ information: the TraceId, the SpanId, and the trace flags. The sample the context. The [Sampler API](sdk.md#sampler) is responsible for setting the -`sampled` flag and the `tracestate`. +`sampled` flag and the `tracestate`. P-value and r-value are set in the OpenTelemetry `tracestate`, under the vendor tag `ot`, using the identifiers `p` and `r`. P-value is an @@ -517,7 +517,7 @@ in the `tracestate`. As stated in the [completeness guarantee](#completeness-guarantee), traces will be possibly incomplete when configuring multiple consistent probability samplers in the same trace. One way to avoid -producing incomplete traces is to use parent-based samplers except for +producing incomplete traces is to use parent-based samplers except for root spans. There is a simple test for trace incompleteness, but it is a one-way @@ -663,7 +663,7 @@ this a strict test for random behavior, we take the following approach: - Generate a pre-determined list of 20 random seeds - Use fixed values for significance level (5%) and trials (20) - Use a population size of one million spans -- For each trial, simulate the population and compute ChiSquared +- For each trial, simulate the population and compute ChiSquared test statistic - Locate the first seed value in the ordered list such that the Chi-Squared significance test fails exactly once out of 20 trials @@ -756,7 +756,7 @@ For the teset with 20 trials and 1 million spans each, the test MUST demonstrate a random number generator seed such that the ChiSquared test statistic is below 0.003932 exactly 1 out of 20 times. -#### Test implementation +#### Test implementation The recommended structure for this test uses a table listing the 20 probability values, the expected p-values, whether the ChiSquared @@ -764,31 +764,31 @@ statistic has one or two degrees of freedom, and the index into the predetermined list of seeds. ``` - for _, test := range []testCase{ - // Non-powers of two - {0.90000, 1, twoDegrees, 5}, - {0.60000, 1, twoDegrees, 14}, - {0.33000, 2, twoDegrees, 3}, - {0.13000, 3, twoDegrees, 2}, - {0.10000, 4, twoDegrees, 0}, - {0.05000, 5, twoDegrees, 0}, - {0.01700, 6, twoDegrees, 2}, - {0.01000, 7, twoDegrees, 3}, - {0.00500, 8, twoDegrees, 1}, - {0.00290, 9, twoDegrees, 1}, - {0.00100, 10, twoDegrees, 5}, - {0.00050, 11, twoDegrees, 1}, - {0.00026, 12, twoDegrees, 3}, - {0.00023, 13, twoDegrees, 0}, - {0.00010, 14, twoDegrees, 2}, - - // Powers of two - {0x1p-1, 1, oneDegree, 0}, - {0x1p-4, 4, oneDegree, 2}, - {0x1p-7, 7, oneDegree, 3}, - {0x1p-10, 10, oneDegree, 0}, - {0x1p-13, 13, oneDegree, 1}, - } { + for _, test := range []testCase{ + // Non-powers of two + {0.90000, 1, twoDegrees, 5}, + {0.60000, 1, twoDegrees, 14}, + {0.33000, 2, twoDegrees, 3}, + {0.13000, 3, twoDegrees, 2}, + {0.10000, 4, twoDegrees, 0}, + {0.05000, 5, twoDegrees, 0}, + {0.01700, 6, twoDegrees, 2}, + {0.01000, 7, twoDegrees, 3}, + {0.00500, 8, twoDegrees, 1}, + {0.00290, 9, twoDegrees, 1}, + {0.00100, 10, twoDegrees, 5}, + {0.00050, 11, twoDegrees, 1}, + {0.00026, 12, twoDegrees, 3}, + {0.00023, 13, twoDegrees, 0}, + {0.00010, 14, twoDegrees, 2}, + + // Powers of two + {0x1p-1, 1, oneDegree, 0}, + {0x1p-4, 4, oneDegree, 2}, + {0x1p-7, 7, oneDegree, 3}, + {0x1p-10, 10, oneDegree, 0}, + {0x1p-13, 13, oneDegree, 1}, + } { ``` Note that seed indexes in the example above have what appears to be From ddf4fdfa99f7e68b869dd4cfd24f2ef692b5580b Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 30 Nov 2021 00:14:44 -0800 Subject: [PATCH 30/45] lint --- specification/trace/tracestate-probability-sampling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index b387a9e9008..cab70d6eed8 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -186,7 +186,7 @@ collected. Since the test for an incompleteness is one-way, it is important to know which sampling configurations may lead to incomplete traces. Sampling configurations that lead naturally to complete traces and -complete subtraces are [discussed below](#trace-producers). +complete subtraces are [discussed below](#trace-producer-completeness). #### Non-probability sampler From 86f50dda5a623295ca94fff9896ff70c61f088cb Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 30 Nov 2021 14:07:50 -0800 Subject: [PATCH 31/45] eliminate API compatibility with ParentBased --- .../trace/tracestate-probability-sampling.md | 29 ++++++++++++------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index cab70d6eed8..c258f63b37d 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -28,9 +28,10 @@ - [Requirement: R-value is generated with the correct probabilities](#requirement-r-value-is-generated-with-the-correct-probabilities) * [Samplers](#samplers) + [ParentConsistentProbabilityBased sampler](#parentconsistentprobabilitybased-sampler) - - [Requirement: ParentBased API compatibility](#requirement-parentbased-api-compatibility) + - [Requirement: ParentConsistentProbabilityBased API](#requirement-parentconsistentprobabilitybased-api) - [Requirement: ParentConsistentProbabilityBased does not modify valid tracestate](#requirement-parentconsistentprobabilitybased-does-not-modify-valid-tracestate) - - [Requirement: ParentConsistentProbabilityBased and ParentBased make identical decisions](#requirement-parentconsistentprobabilitybased-and-parentbased-make-identical-decisions) + - [Requirement: ParentConsistentProbabilityBased calls the configured root sampler for root spans](#requirement-parentconsistentprobabilitybased-calls-the-configured-root-sampler-for-root-spans) + - [Requirement: ParentConsistentProbabilityBased respects the sampled flag for non-root spans](#requirement-parentconsistentprobabilitybased-respects-the-sampled-flag-for-non-root-spans) + [ConsistentProbabilityBased sampler](#consistentprobabilitybased-sampler) - [Requirement: TraceIdRatioBased API compatibility](#requirement-traceidratiobased-api-compatibility) - [Requirement: ConsistentProbabilityBased sampler sets r for root span](#requirement-consistentprobabilitybased-sampler-sets-r-for-root-span) @@ -379,24 +380,30 @@ produces each value with the specified probability. The `ParentConsistentProbabilityBased` sampler is meant as an optional replacement for the [`ParentBased` Sampler](sdk.md#parentbased). It is -required to first validate the `tracestate` and then behave as the -`ParentBased` sampler would. +required to first validate the `tracestate` and then respect the +`sampled` flag in the W3C traceparent. -##### Requirement: ParentBased API compatibility +##### Requirement: ParentConsistentProbabilityBased API -The `ParentConsistentProbabilityBased` Sampler MUST have the same -constructor signature as the built-in `ParentBased` sampler in each -OpenTelemetry SDK. +The `ParentConsistentProbabilityBased` Sampler constructor SHOULD take +a single Sampler argument, which is the Sampler to use in case the +`ParentConsistentProbabilityBased` Sampler is called for a root span. ##### Requirement: ParentConsistentProbabilityBased does not modify valid tracestate The `ParentConsistentProbabilityBased` Sampler MUST NOT modify a valid `tracestate`. -##### Requirement: ParentConsistentProbabilityBased and ParentBased make identical decisions +##### Requirement: ParentConsistentProbabilityBased calls the configured root sampler for root spans -The `ParentConsistentProbabilityBased` Sampler MUST make the same -decision specified for the `ParentBased` sampler. +The `ParentConsistentProbabilityBased` Sampler MUST delegate to the +configured root Sampler when there is not a valid parent trace context. + +##### Requirement: ParentConsistentProbabilityBased respects the sampled flag for non-root spans + +The `ParentConsistentProbabilityBased` Sampler MUST decide to sample +the span according to the value of the `sampled` flag in the W3C +traceparent header. #### ConsistentProbabilityBased sampler From 0c2b1cd8fc2d5ad782c3e72a410e2c020f86e762 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 6 Dec 2021 16:06:40 -0800 Subject: [PATCH 32/45] add examples --- .../trace/tracestate-probability-sampling.md | 57 +++++++++++++++++-- 1 file changed, 52 insertions(+), 5 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index c258f63b37d..853536097d9 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -216,7 +216,7 @@ values via the context, termed "p-value" and "r-value". Both fields are propagated via the OpenTelemetry `tracestate` under the `ot` vendor tag using the rules for [tracestate handling](tracestate-handling.md). Both fields are represented as -unsigned integers requiring at most 6 bits of information. +unsigned decimal integers requiring at most 6 bits of information. This sampling scheme selects items from among a fixed set of 63 distinct probability values. The set of supported probabilities @@ -329,9 +329,9 @@ sampling probability: ##### Requirement: Out-of-range p-values are unset -Consumers SHOULD unset `p` from the `tracestate` if the unsigned value -is greater than 63 before using the `tracestate` to make a sampling -decision or interpret adjusted count. +Consumers SHOULD unset `p` from the `tracestate` if the unsigned +decimal value is greater than 63 before using the `tracestate` to make +a sampling decision or interpret adjusted count. #### R-value @@ -366,7 +366,7 @@ to use this approach. ##### Requirement: Out-of-range r-values unset both p and r Samplers SHOULD unset both `r` and `p` from the `tracestate` if the -unsigned value is of `r` is greater than 62 before using the +unsigned decimal value of `r` is greater than 62 before using the `tracestate` to make a sampling decision. ##### Requirement: R-value is generated with the correct probabilities @@ -374,6 +374,53 @@ unsigned value is of `r` is greater than 62 before using the Samplers MUST generate r-values using a randomized scheme that produces each value with the specified probability. +#### Examples + +Consider a trace context with the following headers: + +``` +traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 +tracestate: ot=r:3;p:2 +``` + +The `traceparent` contents in this example example are repeated from +the [W3C specification](https://www.w3.org/TR/trace-context/#examples-of-http-traceparent-headers)) +and have the following base64-encoded field values: + +``` +base16(version) = 00 +base16(trace-id) = 4bf92f3577b34da6a3ce929d0e0e4736 +base16(parent-id) = 00f067aa0ba902b7 +base16(trace-flags) = 01 // (i.e., sampled) +``` + +The `tracestate` header contains OpenTelemetry string `r:3;p:2`, +containing decimal-encoded p-value and r-value: + +``` +base10(r) = 3 +base10(p) = 2 +``` + +Here, r-value 3 indicates that a consistent probability sampler +configured with probability 12.5% (i.e., 1-in-2**3) or greater will +sample the trace. The p-value 2 indicates that the parent that set +the `sampled` flag was configured to sample at 25% (i.e., 1-in-2**2). +This trace context is consistent because `p <= r` is true and the +`sampled` flag is set. + +Now, an example with an unsampled context where only the r-value is +set. + +``` +traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00 +tracestate: ot=r:3 +``` + +This supports consistent probability sampling in child contexts by +virtue of having an r-value, although p-value is not set, consistent +with an unsampled context. + ### Samplers #### ParentConsistentProbabilityBased sampler From 318bdd2ece1d8de60d60dbe68ea3eb53754bfa4c Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 6 Dec 2021 16:14:12 -0800 Subject: [PATCH 33/45] categorize and summarize adjusted count --- specification/trace/tracestate-probability-sampling.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 853536097d9..268c16ef5a2 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -124,6 +124,14 @@ Zero adjusted count is defined in a way that supports composition of probability and non-probability sampling. Zero is assigned as the adjusted count when a probability sampler does not select a span. +Thus, there are three meaningfully distinct categories of adjusted count: + +| Adjusted count is | Interpretation | +| -- | -- | +| _Unknown_ | The adjusted count is not known, possibly as a result of a non-probability sampler. Items in this category should not be counted. | +| _Zero_ | The adjusted count is known; the effective count of the item is zero. | +| _Non-zero_ | The adjusted count is known; the effective count of the item is greater than zero. | + #### Sampler A Sampler provides configurable logic, used by the SDK, for selecting From 5879cfa78b157ab18540385227811d75eb841bb4 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 6 Dec 2021 16:30:20 -0800 Subject: [PATCH 34/45] move table of r-value probabilities to an appendix --- .../trace/tracestate-probability-sampling.md | 93 ++++++++++++++++--- 1 file changed, 79 insertions(+), 14 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 268c16ef5a2..967be7a5e3a 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -26,6 +26,7 @@ + [R-value](#r-value) - [Requirement: Out-of-range r-values unset both p and r](#requirement-out-of-range-r-values-unset-both-p-and-r) - [Requirement: R-value is generated with the correct probabilities](#requirement-r-value-is-generated-with-the-correct-probabilities) + + [Examples](#examples) * [Samplers](#samplers) + [ParentConsistentProbabilityBased sampler](#parentconsistentprobabilitybased-sampler) - [Requirement: ParentConsistentProbabilityBased API](#requirement-parentconsistentprobabilitybased-api) @@ -62,6 +63,8 @@ + [Test procedure: exact powers of two](#test-procedure-exact-powers-of-two) - [Requirement: Pass 5 power-of-two statistical tests](#requirement-pass-5-power-of-two-statistical-tests) + [Test implementation](#test-implementation) +- [Appendix](#appendix) + * [Methods for generating R-values](#methods-for-generating-r-values) @@ -351,19 +354,19 @@ to participate in consistent probability sampling. R-value determines which sampling probabilities and will not sample for spans of a given trace, as follows: -| r-value | Probability of r-value | Implied sampling probabilities | -| ---------------- | ------------------------ | ---------------------- | -| 0 | 1/2 | 1 | -| 1 | 1/4 | 1/2 and above | -| 2 | 1/8 | 1/4 and above | -| 3 | 1/16 | 1/8 and above | -| ... | ... | ... | -| 0 <= r <= 61 | 2**-(r+1) | 2**-r and above | -| ... | ... | ... | -| 59 | 2**-60 | 2**-59 and above | -| 60 | 2**-61 | 2**-60 and above | -| 61 | 2**-62 | 2**-61 and above | -| 62 | 2**-62 | 2**-62 and above | +| r-value | Implied sampling probabilities | +| ---------------- | ---------------------- | +| 0 | 1 | +| 1 | 1/2 and above | +| 2 | 1/4 and above | +| 3 | 1/8 and above | +| ... | ... | +| 0 <= r <= 61 | 2**-r and above | +| ... | ... | +| 59 | 2**-59 and above | +| 60 | 2**-60 and above | +| 61 | 2**-61 and above | +| 62 | 2**-62 and above | These probabilities are specified to ensure that conforming Sampler implementations record spans with correct adjusted counts. The @@ -380,7 +383,9 @@ unsigned decimal value of `r` is greater than 62 before using the ##### Requirement: R-value is generated with the correct probabilities Samplers MUST generate r-values using a randomized scheme that -produces each value with the specified probability. +produces each value with the probabilities equivalent to those +produced by counting the number of leading 0s in a string of 62 random +bits. See the appendix for details. #### Examples @@ -859,3 +864,63 @@ two 5s demonstrate that it is relatively easy to find examples where there is exactly one failure. Seed index 14, for probability 0.6 in this case, is a reminder that outliers exist. Further significance testing of this distribution is not recommended. + +## Appendix + +### Methods for generating R-values + +The method used for generating r-values is not specified, in order to +leave the implementation freedom to optimize. Typically, when the +TraceId is known to contain at a 62-bit substring of random bits, +R-values can be derived directly from the 62 random bits of TraceId +by: + +1. Count the leading zeros +2. Count the leading ones +3. Count the trailing zeros +4. Count the trailing ones. + +```golang +import ( + "math/rand" + "math/bits" +) + +func nextRValueLeading() int { + x := uint64(rand.Int63()) // 63 least-significant bits are random + y := x << 1 | 0x3 // 62 most-significant bits are random + return bits.LeadingZeros64(y) +} +``` + +If the TraceId contains unknown or insufficient randomness, another +approach is to generate random bits until the first true or false +value. + +``` +func nextRValueGenerated() int { + for r := 0; r < 62; r++ { + if rand.Bool() == true { + return r + } + } + return 62 +} +``` + +Any scheme that produces r-values shown in the following table is +considered conforming. + +| r-value | Probability of r-value | +| ---------------- | ------------------------ | +| 0 | 1/2 | +| 1 | 1/4 | +| 2 | 1/8 | +| 3 | 1/16 | +| ... | ... | +| 0 <= r <= 61 | 2**-(r+1) | +| ... | ... | +| 59 | 2**-60 | +| 60 | 2**-61 | +| 61 | 2**-62 | +| 62 | 2**-62 | From 0da323db757e662c1e792131889013d7572dd8d6 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 7 Dec 2021 21:44:34 -0800 Subject: [PATCH 35/45] add composition examples --- .../trace/tracestate-probability-sampling.md | 62 ++++++++++++++++--- 1 file changed, 54 insertions(+), 8 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 967be7a5e3a..28ad4309297 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -26,7 +26,9 @@ + [R-value](#r-value) - [Requirement: Out-of-range r-values unset both p and r](#requirement-out-of-range-r-values-unset-both-p-and-r) - [Requirement: R-value is generated with the correct probabilities](#requirement-r-value-is-generated-with-the-correct-probabilities) - + [Examples](#examples) + + [Examples: Context invariants](#examples-context-invariants) + - [Example: Probability sampled context](#example-probability-sampled-context) + - [Example: Probability unsampled](#example-probability-unsampled) * [Samplers](#samplers) + [ParentConsistentProbabilityBased sampler](#parentconsistentprobabilitybased-sampler) - [Requirement: ParentConsistentProbabilityBased API](#requirement-parentconsistentprobabilitybased-api) @@ -48,6 +50,9 @@ - [Requirement: Unset p when multiple consistent probability samplers decide not to sample](#requirement-unset-p-when-multiple-consistent-probability-samplers-decide-not-to-sample) - [Requirement: Use probability sampler p-value when its decision to sample is combined with non-probability samplers](#requirement-use-probability-sampler-p-value-when-its-decision-to-sample-is-combined-with-non-probability-samplers) - [Requirement: Use p-value 63 when a probability sampler decision not to sample is combined with a non-probability sampler decision to sample](#requirement-use-p-value-63-when-a-probability-sampler-decision-not-to-sample-is-combined-with-a-non-probability-sampler-decision-to-sample) + + [Examples: Composition](#examples-composition) + - [Example: Probability and non-probability sampler in a root context](#example-probability-and-non-probability-sampler-in-a-root-context) + - [Example: Two consistent probability samplers](#example-two-consistent-probability-samplers) * [Producer and consumer recommendations](#producer-and-consumer-recommendations) + [Trace producer: completeness](#trace-producer-completeness) - [Recommenendation: use non-descending power-of-two probabilities](#recommenendation-use-non-descending-power-of-two-probabilities) @@ -387,7 +392,9 @@ produces each value with the probabilities equivalent to those produced by counting the number of leading 0s in a string of 62 random bits. See the appendix for details. -#### Examples +#### Examples: Context invariants + +##### Example: Probability sampled context Consider a trace context with the following headers: @@ -416,14 +423,15 @@ base10(p) = 2 ``` Here, r-value 3 indicates that a consistent probability sampler -configured with probability 12.5% (i.e., 1-in-2**3) or greater will +configured with probability 12.5% (i.e., 1-in-8) or greater will sample the trace. The p-value 2 indicates that the parent that set -the `sampled` flag was configured to sample at 25% (i.e., 1-in-2**2). +the `sampled` flag was configured to sample at 25% (i.e., 1-in-4). This trace context is consistent because `p <= r` is true and the `sampled` flag is set. -Now, an example with an unsampled context where only the r-value is -set. +##### Example: Probability unsampled + +This example has an unsampled context where only the r-value is set. ``` traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00 @@ -431,8 +439,8 @@ tracestate: ot=r:3 ``` This supports consistent probability sampling in child contexts by -virtue of having an r-value, although p-value is not set, consistent -with an unsampled context. +virtue of having an r-value. P-value is not set, consistent with an +unsampled context. ### Samplers @@ -577,6 +585,44 @@ and a non-probability Sampler, and the probability Sampler decides not to sample but the non-probability does sample, p-value 63 MUST be set in the `tracestate`. +#### Examples: Composition + +##### Example: Probability and non-probability sampler in a root context + +In a new root context, a consistent probability sampler decides not to +set the sampled flag, adds `r:4` indicating that the trace is +consistently sampled at 6.5% (i.e., 1-in-16) and larger probabilities. + +The probability sampler decision is composed with a non-probability +sampler that decides to sample the context. Setting `sampled` when +the probability sampler has not sampled requires setting `p:63`, +indicating zero adjusted count. + +The resulting context: + +``` +traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 +tracestate: ot=r:3;p:63 +``` + +##### Example: Two consistent probability samplers + +Whether a root or non-root, if multiple consistent probability +samplers make a decision to sample a given context, the minimum +p-value is output in the tracestate. + +If a root context, the first of the samplers generates `r:15` and its +own p-value `p:10` (i.e., adjusted count 1024). The second of the two +probability samplers outputs a smaller adjusted count `p:8` (i.e., +adjusted count 256). + +The resulting context takes the smaller p-value: + +``` +traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 +tracestate: ot=r:15;p:8 +``` + ### Producer and consumer recommendations #### Trace producer: completeness From 7c374c2d7aacce98678adcc03bcfa06925024e22 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 9 Dec 2021 13:57:13 -0800 Subject: [PATCH 36/45] clarify the zero special case --- .../trace/tracestate-probability-sampling.md | 31 +++++++++++++------ 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 28ad4309297..6848e0f3e6c 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -1,5 +1,10 @@ # TraceState: Probability Sampling +**Status**: [Experimental](../document-status.md) + +
+Table of Contents + - [Approach used in this document](#approach-used-in-this-document) @@ -73,7 +78,7 @@ -**Status**: [Experimental](../document-status.md) +
Probability sampling allows OpenTelemetry tracing users to lower their collection costs with the use of randomized sampling techniques. @@ -236,7 +241,11 @@ unsigned decimal integers requiring at most 6 bits of information. This sampling scheme selects items from among a fixed set of 63 distinct probability values. The set of supported probabilities -includes the integer powers of two between 1 and 2**-62. +includes the integer powers of two between 1 and 2**-62. Zero +probability and probabilities smaller than 2**-62 are treated as a +special case of "ConsistentAlwaysOff" sampler, just as unit +probability (i.e., 100%) describes a special case of +"ConsistentAlwaysOn" sampler. R-value encodes which among the 63 possibilities will consistently decide to sample for a given trace. Specifically, r-value specifies @@ -244,7 +253,9 @@ the smallest probability that will decide to sample a given trace in terms of the corresponding p-value. For example, a trace with r-value 0 will sample spans configured for 100% sampling, while r-value 1 will sample spans configured for 50% or 100% sampling, and so on through -r-value 62 which will sample every span. +r-value 62, for which a consistent probability sampler will decide +"yes" at every supported probability (i.e., greater than or equal to +2**-62). P-value encodes the adjusted count for child contexts (i.e., consumers of `tracestate`) and consumers of sampled spans to record for use in @@ -513,9 +524,10 @@ to sample according to its configured sampling probability. ##### Requirement: ConsistentProbabilityBased sampler records unbiased adjusted counts -The `ConsistentProbabilityBased` Sampler MUST set `p` so that the -adjusted count interpreted from the `tracestate` is an unbiased -estimate of the number of representative spans in the population. +The `ConsistentProbabilityBased` Sampler with non-zero probability +MUST set `p` so that the adjusted count interpreted from the +`tracestate` is an unbiased estimate of the number of representative +spans in the population. A test specification for this requirement is given in the appendix. @@ -526,11 +538,10 @@ root span, `ConsistentProbabilityBased` SHOULD set `r` as if it were a root span and MAY warn the user that a potentially inconsistent trace is being produced. -##### Requirement: ConsistentProbabilityBased sampler acts like AlwaysOff sampler for probabilities less than 2**-62 +##### Requirement: ConsistentProbabilityBased sampler decides not to sample for probabilities less than 2**-62 -If the configured sampling probability is less than `2**-62`, the -Sampler SHOULD round down to zero probability and make the same -sampling decision as the builtin `AlwaysOff` sampler would. +If the configured sampling probability is in the interval `[0, +2**-62)`, the Sampler MUST decide not to sample. ### Composition rules From 81a6b07a3fae0c2edac94d85335e74e17fa65a01 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 9 Dec 2021 14:52:17 -0800 Subject: [PATCH 37/45] sampler examples --- .../trace/tracestate-probability-sampling.md | 80 +++++++++++++++++-- 1 file changed, 75 insertions(+), 5 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 6848e0f3e6c..364337c57f1 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -524,9 +524,9 @@ to sample according to its configured sampling probability. ##### Requirement: ConsistentProbabilityBased sampler records unbiased adjusted counts -The `ConsistentProbabilityBased` Sampler with non-zero probability -MUST set `p` so that the adjusted count interpreted from the -`tracestate` is an unbiased estimate of the number of representative +The `ConsistentProbabilityBased` Sampler with non-zero probability +MUST set `p` so that the adjusted count interpreted from the +`tracestate` is an unbiased estimate of the number of representative spans in the population. A test specification for this requirement is given in the appendix. @@ -543,6 +543,76 @@ is being produced. If the configured sampling probability is in the interval `[0, 2**-62)`, the Sampler MUST decide not to sample. +#### Examples: Consistent probability samplers + +##### Example: Setting R-value for a root span + +A new root span is sampled by a consistent probability sampler at 25%. +A new r-value should be generated (see the appendix for suitable +methods), in this example r-value 5 is used which happens 1.5625% of +the time and indicates to sample: + +``` +tracestate: ot=r:5;p:2 +``` + +The span would be sampled because p-value 2 is less than or equal to +r-value 5. An example `tracestate` where r-value 1 indicates not to +sample at 25%: + +``` +tracestate: ot=r:1 +``` + +This span would not be sampled because p-value 2 (corresponding with +25% sampling) is greater than r-value 1. + +##### Example: Handling inconsistent P-value + +When either the consistent probability sampler or the parent-based +consistent probability sampler receives a sampled context but +invalid p-value, for example, + +``` +tracestate: ot=r:4;p:73 +``` + +the `tracestate` will have its p-value stripped. The r-value is kept, +and the sampler should act as if the following had been received: + +``` +tracestate: ot=r:4 +``` + +The consistent probability sampler will make its own (consistent) +decision using the r-value that was received. + +The parent-based consistent probability sampler will in this case +follow the `sampled` flag. If the context is sampled, the resulting +span will have an r-value without a p-value, which indicates unknown +adjusted count. + +##### Example: Handling corrupt R-value + +A non-root span receives: + +``` +tracestate: ot=r:100;p:10 +``` + +where the r-value is out of its valid range. The r-value and p-value +are stripped during validation, according to the invariants. In this +case, the sampler will act as though no `tracestate` were received. + +The parent-based consistent probability sampler will sample or not +sample based on the `sampled` flag, in this case. If the context is +sampled, the recorded span will have an r-value without a p-value, +which indicates unknown adjusted count. + +The consistent probability sampler will generate a new r-value and +make a new sampling decision while warning the user of a corrupt and +potentially inconsistent r-value. + ### Composition rules When more than one Sampler participates in the decision to sample a @@ -677,7 +747,7 @@ r-value and p-value, as these decisions would lead to incorrect adjusted counts. For example, the built-in [`ParentBased` sampler](sdk.md#parentbased) -supports configuring the delegated sampler based on whether the parent +supports configuring the delegated-to sampler based on whether the parent context is remote or non-remote, sampled or unsampled. If a `ParentBased` sampler delegates to a `ConsistentProbabilityBased` sampler only for unsampled contexts, the resulting Span-to-metrics @@ -686,7 +756,7 @@ pipeline will (probably) overcount spans. ##### Recommenendation: sampler delegation For non-root spans, composite samplers SHOULD NOT condition the choice -of delegating sampler based on the parent's sampled flag or +of delegated-to sampler based on the parent's sampled flag or OpenTelemetry tracestate. #### Trace producer: interoperability with `ParentBased` sampler From 9f951de4b072b96716f3df86968ca93c32953f0f Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 9 Dec 2021 14:52:32 -0800 Subject: [PATCH 38/45] TOC --- specification/trace/tracestate-probability-sampling.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 364337c57f1..3c11e31b68e 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -47,7 +47,11 @@ - [Requirement: ConsistentProbabilityBased sampler sets p when sampling](#requirement-consistentprobabilitybased-sampler-sets-p-when-sampling) - [Requirement: ConsistentProbabilityBased sampler records unbiased adjusted counts](#requirement-consistentprobabilitybased-sampler-records-unbiased-adjusted-counts) - [Requirement: ConsistentProbabilityBased sampler sets r for non-root span](#requirement-consistentprobabilitybased-sampler-sets-r-for-non-root-span) - - [Requirement: ConsistentProbabilityBased sampler acts like AlwaysOff sampler for probabilities less than 2**-62](#requirement-consistentprobabilitybased-sampler-acts-like-alwaysoff-sampler-for-probabilities-less-than-2-62) + - [Requirement: ConsistentProbabilityBased sampler decides not to sample for probabilities less than 2**-62](#requirement-consistentprobabilitybased-sampler-decides-not-to-sample-for-probabilities-less-than-2-62) + + [Examples: Consistent probability samplers](#examples-consistent-probability-samplers) + - [Example: Setting R-value for a root span](#example-setting-r-value-for-a-root-span) + - [Example: Handling inconsistent P-value](#example-handling-inconsistent-p-value) + - [Example: Handling corrupt R-value](#example-handling-corrupt-r-value) * [Composition rules](#composition-rules) + [List of requirements](#list-of-requirements) - [Requirement: Combining multiple sampling decisions using logical `or`](#requirement-combining-multiple-sampling-decisions-using-logical-or) From eb645ef1f3919c71decf2ca384da3a8603e178fb Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Thu, 9 Dec 2021 14:53:47 -0800 Subject: [PATCH 39/45] Changelog --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 748e270406d..e79d9824249 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,9 @@ release. - Clarify `StartSpan` returning the parent as a non-recording Span when no SDK is in use ([#2121](https://github.com/open-telemetry/opentelemetry-specification/pull/2121)) +- Add support for probability sampling in the OpenTelemetry `tracestate` entry and + add optional specification for consistent probability sampling. + ([#2047](https://github.com/open-telemetry/opentelemetry-specification/pull/2047)) ### Metrics From 5a78017dd81077462246a8eb69e147836389a9ef Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 4 Jan 2022 10:28:42 -0800 Subject: [PATCH 40/45] rewrite top matter using input from @bdarfler --- .../trace/tracestate-probability-sampling.md | 50 ++++++++++++------- 1 file changed, 31 insertions(+), 19 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 3c11e31b68e..1b884661e08 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -7,8 +7,7 @@ -- [Approach used in this document](#approach-used-in-this-document) - * [Objective](#objective) +- [Overview](#overview) * [Definitions](#definitions) + [Sampling](#sampling) + [Adjusted count](#adjusted-count) @@ -84,27 +83,40 @@ -Probability sampling allows OpenTelemetry tracing users to lower their -collection costs with the use of randomized sampling techniques. -OpenTelemetry specifies how to convey and record the results of -probability sampling using the W3C `tracestate` in a way that allows -Span-to-Metrics pipelines to be built that accurately count sampled -spans. +## Overview -## Approach used in this document +Probability sampling allows OpenTelemetry tracing users to lower span +collection costs by the use of randomized sampling techniques. The +objectives are: -### Objective +- Compatible with the existing W3C trace context `sampled` flag +- Spans can be accurately counted using a Span-to-metrics pipeline +- Traces tend to be complete, even though spans may make independent sampling decisions. -This document specifies two `tracestate` fields, known as "r-value" -and "p-value" meant to support interoperable Sampler implementations. -Rules are given for creating, validating, interpreting, and mutating -these fields in an OpenTelemetry [context](../context/context.md). +This document specifies an approach based on an "r-value" and a +"p-value". At a very high level, r-value is a source of randomness +and p-value encodes the sampling probability. A context is sampled +when `p <= r`. -Two Samplers are specified for optional inclusion in OpenTelemetry -tracing SDKs, named `ParentConsistentProbabilitityBased` and -`ConsistentProbabilityBased`. These are meant as optional -replacements for the built-in `ParentBased` and `TraceIdRatioBased` -Samplers. +Significantly, by including the r-value and p-value in the +OpenTelemetry `tracestate`, these two values automatically propagate +through the context and are recorded on every Span. This allows Trace +consumers to correctly count spans simply by interpreting the p-value +on a given span. + +For efficiency, the supported sampling probabilities are limited to +powers of two. P-value is derived from sampling probability, which +equals `2**-p`, thus p-value is encoded using an unsigned integer. + +For example, a p-value of 3 indicates a sampling probability of 1/8. + +Since the W3C trace context does not specify that any of the 128 bits +in a TraceID are true uniform-distributed random bits, the r-value is +introduced as an additional source of randomness. + +The recommended method of generating an "r-value" is to count the +number of leading 0s in a string of 62 random bits, however, it is not +required to use this approach. ### Definitions From 7b903846058ee875821e4f56467971c73a9fec2e Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 4 Jan 2022 10:32:43 -0800 Subject: [PATCH 41/45] sampling->sampled --- specification/trace/tracestate-probability-sampling.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 1b884661e08..18da3542cba 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -42,8 +42,8 @@ + [ConsistentProbabilityBased sampler](#consistentprobabilitybased-sampler) - [Requirement: TraceIdRatioBased API compatibility](#requirement-traceidratiobased-api-compatibility) - [Requirement: ConsistentProbabilityBased sampler sets r for root span](#requirement-consistentprobabilitybased-sampler-sets-r-for-root-span) - - [Requirement: ConsistentProbabilityBased sampler unsets p when not sampling](#requirement-consistentprobabilitybased-sampler-unsets-p-when-not-sampling) - - [Requirement: ConsistentProbabilityBased sampler sets p when sampling](#requirement-consistentprobabilitybased-sampler-sets-p-when-sampling) + - [Requirement: ConsistentProbabilityBased sampler unsets p when not sampled](#requirement-consistentprobabilitybased-sampler-unsets-p-when-not-sampled) + - [Requirement: ConsistentProbabilityBased sampler sets p when sampled](#requirement-consistentprobabilitybased-sampler-sets-p-when-sampled) - [Requirement: ConsistentProbabilityBased sampler records unbiased adjusted counts](#requirement-consistentprobabilitybased-sampler-records-unbiased-adjusted-counts) - [Requirement: ConsistentProbabilityBased sampler sets r for non-root span](#requirement-consistentprobabilitybased-sampler-sets-r-for-non-root-span) - [Requirement: ConsistentProbabilityBased sampler decides not to sample for probabilities less than 2**-62](#requirement-consistentprobabilitybased-sampler-decides-not-to-sample-for-probabilities-less-than-2-62) @@ -528,12 +528,12 @@ each OpenTelemetry SDK. The `ConsistentProbabilityBased` Sampler MUST set `r` when it makes a root sampling decision. -##### Requirement: ConsistentProbabilityBased sampler unsets p when not sampling +##### Requirement: ConsistentProbabilityBased sampler unsets p when not sampled The `ConsistentProbabilityBased` Sampler MUST unset `p` from the `tracestate` when it decides not to sample. -##### Requirement: ConsistentProbabilityBased sampler sets p when sampling +##### Requirement: ConsistentProbabilityBased sampler sets p when sampled The `ConsistentProbabilityBased` Sampler MUST set `p` when it decides to sample according to its configured sampling probability. From 6de220eef39d3b5fe276687aeef205486747defc Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 4 Jan 2022 10:49:35 -0800 Subject: [PATCH 42/45] typo --- specification/trace/tracestate-probability-sampling.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index 18da3542cba..eea7b36f87c 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -383,8 +383,8 @@ trace, in order to support consistent probability sampling. When the value is omitted or not present, child spans in the trace are not able to participate in consistent probability sampling. -R-value determines which sampling probabilities and will not sample -for spans of a given trace, as follows: +R-value determines which sampling probabilities will and will not +sample for spans of a given trace, as follows: | r-value | Implied sampling probabilities | | ---------------- | ---------------------- | From f568ed4ce25a645b6f186629428254f58e1f6ef1 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 4 Jan 2022 10:52:03 -0800 Subject: [PATCH 43/45] rephrase --- specification/trace/tracestate-probability-sampling.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index eea7b36f87c..bcb50449c28 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -383,8 +383,8 @@ trace, in order to support consistent probability sampling. When the value is omitted or not present, child spans in the trace are not able to participate in consistent probability sampling. -R-value determines which sampling probabilities will and will not -sample for spans of a given trace, as follows: +R-value determines which sampling probabilities will decide to sample +or not decide to sample for spans of a given trace, as follows: | r-value | Implied sampling probabilities | | ---------------- | ---------------------- | From 0fc9a3e12f85c53f6c31b00b6c7f625bed6fe64d Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Wed, 5 Jan 2022 14:19:30 -0800 Subject: [PATCH 44/45] Update specification/trace/tracestate-probability-sampling.md Co-authored-by: Bogdan Drutu --- specification/trace/tracestate-probability-sampling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index bcb50449c28..ced236ef54b 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -699,7 +699,7 @@ The resulting context: ``` traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 -tracestate: ot=r:3;p:63 +tracestate: ot=r:4;p:63 ``` ##### Example: Two consistent probability samplers From 055b9215ec065c690954715f67e965a094c0d954 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Tue, 18 Jan 2022 09:46:20 -0800 Subject: [PATCH 45/45] conform to syntax of https://github.com/open-telemetry/opentelemetry-specification/pull/2003 --- specification/trace/tracestate-probability-sampling.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/specification/trace/tracestate-probability-sampling.md b/specification/trace/tracestate-probability-sampling.md index bcb50449c28..bbc41830eb2 100644 --- a/specification/trace/tracestate-probability-sampling.md +++ b/specification/trace/tracestate-probability-sampling.md @@ -417,7 +417,7 @@ unsigned decimal value of `r` is greater than 62 before using the Samplers MUST generate r-values using a randomized scheme that produces each value with the probabilities equivalent to those produced by counting the number of leading 0s in a string of 62 random -bits. See the appendix for details. +bits. #### Examples: Context invariants @@ -545,13 +545,11 @@ MUST set `p` so that the adjusted count interpreted from the `tracestate` is an unbiased estimate of the number of representative spans in the population. -A test specification for this requirement is given in the appendix. - ##### Requirement: ConsistentProbabilityBased sampler sets r for non-root span If `r` is not set on the input `tracecontext` and the Span is not a root span, `ConsistentProbabilityBased` SHOULD set `r` as if it were a -root span and MAY warn the user that a potentially inconsistent trace +root span and warn the user that a potentially inconsistent trace is being produced. ##### Requirement: ConsistentProbabilityBased sampler decides not to sample for probabilities less than 2**-62