Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmark for interaction (zoom) #73

Merged

Conversation

ianthomas23
Copy link
Collaborator

This adds a benchmark for zooming an existing Bokeh figure, so records the time taken for re-rendering the figure (and any recalculation prior to that required for the re-render). It is achieved by creating a button whose callback zooms the figure so the Playwright functionality is to locate the button and click it in the same way as already performed for the latency benchmark. All of the important work occurs in JavaScript in the browser (and possibly GPU), except for some constant-ish overhead of sending the zoom request from the Python side and waiting for the render completion message to arrive on the Python side.

This is part of issue #66.

Summary of changes:

  1. There is now support for multiple Bokeh figures per benchmark. This isn't actually used yet but is a better approach that will be needed in future.
  2. Base benchmark sets the number and warmup_time of benchmark runs. This is really important here as the default asv behaviour assumes that the results of the setup stage are not affected by the actual benchmark runs whereas here that is not the case.
  3. Renaming the benchmarks from Timeseries to BokehExample.
  4. Some refactoring, pushing functionality down into the Base class so that the actual benchmark classes are as simple as possible.
  5. I have removed the n = 10_000_000 runs as this sometimes gives problems with timeouts that I have yet to solve.

On my M1 mac (no dedicated GPU) I obtain the following:

$ asv run -e
· Creating environments
· Discovering benchmarks
· Running 2 total benchmarks (1 commits * 1 environments * 2 benchmarks)
[  0.00%] · For hvneuro commit 0388952c <main>:
[  0.00%] ·· Benchmarking virtualenv-py3.11-playwright
[ 25.00%] ··· Running (bokeh_example.BokehExampleLatency.time_latency--).
[ 50.00%] ··· Running (bokeh_example.BokehExampleZoom.time_zoom--).
[ 75.00%] ··· bokeh_example.BokehExampleLatency.time_latency                                                                                                          ok
[ 75.00%] ··· ========= ============ ============
              --              output_backend     
              --------- -------------------------
                  n        canvas       webgl    
              ========= ============ ============
                 1000     84.7±5ms     90.4±5ms  
                10000     103±8ms      111±7ms   
                100000    238±4ms      240±4ms   
               1000000   1.71±0.01s   1.69±0.01s 
              ========= ============ ============

[100.00%] ··· bokeh_example.BokehExampleZoom.time_zoom                                                                                                                ok
[100.00%] ··· ========= ========== ============
              --             output_backend    
              --------- -----------------------
                  n       canvas      webgl    
              ========= ========== ============
                 1000    29.0±2ms    37.5±3ms  
                10000    30.5±1ms   33.0±0.7ms 
                100000   32.0±2ms    40.2±3ms  
               1000000   220±8ms     162±9ms   
              ========= ========== ============

@droumis
Copy link
Collaborator

droumis commented Aug 8, 2023

noice

 asv run -e
· Creating environments...
· Discovering benchmarks.
·· Uninstalling from virtualenv-py3.11-playwright
·· Building 359a708a <main> for virtualenv-py3.11-playwright.....
·· Installing 359a708a <main> into virtualenv-py3.11-playwright....................
· Running 2 total benchmarks (1 commits * 1 environments * 2 benchmarks)
[  0.00%] · For hvneuro commit 359a708a <main>:
[  0.00%] ·· Benchmarking virtualenv-py3.11-playwright
[ 25.00%] ··· Running (bokeh_example.BokehExampleLatency.time_latency--).
[ 50.00%] ··· Running (bokeh_example.BokehExampleZoom.time_zoom--).
[ 75.00%] ··· bokeh_example.BokehExampleLatency.time_latency                                                                                                                                               ok
[ 75.00%] ··· ========= ============ ============
              --              output_backend     
              --------- -------------------------
                  n        canvas       webgl    
              ========= ============ ============
                 1000     70.1±3ms     79.2±3ms  
                10000     89.6±1ms     104±3ms   
                100000    226±4ms      230±6ms   
               1000000   1.79±0.02s   1.72±0.02s 
              ========= ============ ============

[100.00%] ··· bokeh_example.BokehExampleZoom.time_zoom                                                                                                                                                     ok
[100.00%] ··· ========= ========== ==========
              --            output_backend   
              --------- ---------------------
                  n       canvas     webgl   
              ========= ========== ==========
                 1000    52.9±3ms   54.3±4ms 
                10000    50.6±4ms   47.2±4ms 
                100000   45.0±4ms   48.3±6ms 
               1000000   214±7ms    165±3ms  
              ========= ========== ==========

Copy link
Collaborator

@droumis droumis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @ianthomas23 , this looks/works great. I left a couple (for the future) comments but I am happy to merge now. One thing I wanted to get your opinion on was changing the zoom interaction as described in my comment..

benchmarks/benchmarks/base.py Show resolved Hide resolved
benchmarks/benchmarks/base.py Show resolved Hide resolved
benchmarks/benchmarks/bokeh_example.py Outdated Show resolved Hide resolved
@droumis
Copy link
Collaborator

droumis commented Aug 9, 2023

Checking the variation:

asv run -b time_zoom --record-samples
and then see results using something like
asv show <hash> --details
where the hash comes from the output of asv show

image

image

zoom out to mimic common user interaction
@droumis droumis merged commit 40bc762 into holoviz-topics:main Sep 25, 2023
@ianthomas23 ianthomas23 deleted the multiple_benchmarks_per_scenario branch October 26, 2023 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: WIP
Development

Successfully merging this pull request may close these issues.

2 participants