Broader test scenarios

[kylesharp-optimizely]

I suggest that we should broaden the test scenarios.

As of this writing, we have the following scenarios covered:

1. Web page delivered through edge worker with no changes applied
2. Web page delivered through edge worker with a mix of pre-UA and on-UA changes

In order to better understand the performance impacts of this, we should add the following at minimum:

3. Web page delivered straight from origin to browser with no changes applied
4. Web page delivered through edge worker with just pre-UA changes
5. Web page delivered through edge worker with just on-UA changes
6. Web page delivered straight from origin with a script tag that will apply all changes via JS, which can be delivered from origin as well (to replicate how existing client-side A/B testing offerings work)

This will give us a comprehensive view of how this performs in multiple scenarios, with a proper set of baselines to compare against.

[alexnj]

Currently the prototype is built around the idea of using Edge as an integration point, that does both injection of the Client A/B script into HEAD of the document, as well as performing any pre-UA transforms along with it. To avoid comparing actual origin latency differences vs. CF cache performance and to focus on any latency introduced by the A/B method itself, I moved both control and and test to base on the same origin (CF) — control just proxies to actual origin and variant does proxy + transforms, thus giving us a fair baseline to compare the A/B spec against. I tend to agree that we should expand the test scenarios / dimensions. Would the following be the correct understanding of the intent of each test?

3. Web page delivered straight from origin to browser with no changes applied

This would establish the baseline performance of the origin page with no A/B tests implemented (Control).

4. Web page delivered through edge worker with just pre-UA changes

I'm assuming this would help us to measure the performance if we rely only on Edge side transforms? Would it be practical to expect only pre-UA transforms to meet the need — static page A/B tests may be? Or would the intent be to rule out latencies, if any, added by the client side implementation of the spec?

5. Web page delivered through edge worker with just on-UA changes

Cost (or improvement in case being cached) of fronting with a compute capable Edge/CDN and using it for integration only, i.e., no transform logic performed (other than injection of script to HEAD of the document) thus avoiding latency due to compute at Edge.

While the prototype is built around Edge as an integration point, we could also perform the same operations that Edge does today, right at Origin, thus making Origin the integration point between a publisher and A/B provider. This was also one of the asks that came up at W3PerfWG. It would be good to add that scenario as a test if once we have an Origin-based pre-UA+injection implementation is available.

[kylesharp-optimizely]

3 would be our absolute control, yes, to determine the impact any approach would have on a site. 6 would be a similar baseline to compare against current client side testing methodology. Though, thinking more about it, each current product offering of client side testing is so different, maybe it's meaningless to provide that in the context of this prototype and instead would be more help to make it easy to use an arbitrary experimentation tool run in the same environment. Or simply leave it up to anyone who wants a direct test to implement it on their own.

4 and 5 are just for performance granularity, like you said. On-UA changes are going to need to be applied at the browser, but pre-UA changes can be applied virtually anywhere: origin, edge worker, even the browser itself if someone wanted to, so 4 tells us what an edge worker-based approach can yield without noise from the browser's on-UA changes and 5 tells us what the minimum line would be regardless of pre-UA changes implementation for changes that contain both pre- and on-UA.

[micahheineckoptimizely]

Here are results from experiments that Optimizely ran to test out some of these scenarios.

The different variants tested were:

• no-snippet (test scenario 1): No transformations are applied to the page. ​The Control HTML is returned.​
• pre-ua (test scenario 4): Transformations are applied to the Control at the Edge. ​The transformed HTML is returned.​
• on-ua (test scenario 5): The Control HTML, along with On-UA transforms and applicator code, is returned. ​Transformations are applied on the browser. ​
• edge-transforms: The control HTML is returned. ​The browser then requests from a different Edge worker, which returns the transforms and transform the applicator code. The Transformations are then applied on the browser.​
• web-snippet (similar to test scenario 6): The Control HTML is returned.​ The browser requests a dynamically generated Optimizely Web snippet from the Edge.​ The snippet is executed on the browser (applying the transformations).​

These all go through a Cloudflare edge worker. The code from https://github.com/WICG/ab-worker-prototype was used to to apply the transformations pre-ua and on-ua. These were simulations run using a headless Chrome, and each trial was executed 500 times.

This table shows the results from executing the above variants, each applying 0, 10, or 1000 transformations, and their respective median load event end time (in ms) over the 500 simulations.

Variant	Transforms	P50 (Load Event End)
no-snippet	0	59.6​
on-ua​	0	78​
pre-ua​	0	79.6​
pre-ua​	10	81.5​
on-ua​	10	85.5​
edge-transforms​	0	89.7​
edge-transforms​	10	92.2​
edge-transforms​	1000	105.6​
pre-ua​	1000	177.1​
web-snippet​	0	189.7​
web-snippet​	10	210.9​
on-ua​​	1000	246.1​
web-snippet​	1000	8930.6​

[alexnj]

Thanks for sharing the test results, @micahheineckoptimizely!