Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add benchmarking CI #399

Merged
merged 6 commits into from
Nov 4, 2021
Merged

Add benchmarking CI #399

merged 6 commits into from
Nov 4, 2021

Conversation

theogf
Copy link
Member

@theogf theogf commented Nov 4, 2021

Summary

Following #387 (and #2) We need some benchmarking to see the effect of our changes!
Using BenchmarkCI.jl we should be able to trigger a Benchmark CI when the PR is tagged with performance critical.
What BenchmarkCI does is to run tests both on master and the current commit and produces a report.
This PR is also to try things and see how they run.

This PR also introduce a very short benchmark suite to test things are working. This can be extended in a future PR.

What alternatives have you considered?

None

Breaking changes

Nope

@theogf theogf added the performance critical Triggers benchmarking CI label Nov 4, 2021
github-actions[bot]
github-actions bot reviewed Nov 4, 2021
View reviewed changes
benchmark/benchmarks.jl Outdated Show resolved Hide resolved
benchmark/benchmarks.jl Outdated Show resolved Hide resolved
benchmark/benchmarks.jl Outdated Show resolved Hide resolved
benchmark/benchmarks.jl Outdated Show resolved Hide resolved
theogf and others added 2 commits November 4, 2021 14:00
@theogf @github-actions
Apply suggestions from code review
f4cd28a 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@theogf
Missing end of line
d41b6e0
@theogf
Copy link
Member Author

theogf commented Nov 4, 2021

Things seem to work, except that since benchmarks.jl is completely different in master and does not have a proper Project.toml

@codecov
Copy link

codecov bot commented Nov 4, 2021 •
edited
Loading

Codecov Report

Merging #399 (d41b6e0) into master (360ce10) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #399   +/-   ##
=======================================
  Coverage   89.23%   89.23%           
=======================================
  Files          52       52           
  Lines        1198     1198           
=======================================
  Hits         1069     1069           
  Misses        129      129

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 360ce10...d41b6e0. Read the comment docs.

st--
st-- approved these changes Nov 4, 2021
View reviewed changes
Copy link
Member

@st-- st-- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's see if it works: )

@devmotion
Copy link
Member

It errors: https://github.com/JuliaGaussianProcesses/KernelFunctions.jl/runs/4105200042?check_suite_focus=true#step:5:373

@theogf
Copy link
Member Author

theogf commented Nov 4, 2021

@devmotion See my comment above, once merged, we can really check if it works.

@theogf
Copy link
Member Author

theogf commented Nov 4, 2021

I can always revert if it does not work!

@theogf theogf merged commit 33d64d1 into master Nov 4, 2021
@theogf theogf deleted the tgf/benchmark2 branch November 4, 2021 13:23
Crown421 added a commit that referenced this pull request Jan 28, 2022
@Crown421
Squashed commit of the following:
98c6913 
commit f9bbd84
Author: st-- <[email protected]>
Date:   Fri Jan 28 09:11:50 2022 +0100

    make nystrom work with AbstractVector (#427)

    * make nystrom work with AbstractVector

    * add test

    * Update test/approximations/nystrom.jl

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    * patch bump

    * Update test/approximations/nystrom.jl

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    * Apply suggestions from code review

    Co-authored-by: David Widmann <[email protected]>

    * Apply suggestions from code review

    * deprecate

    * Apply suggestions from code review

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    * Apply suggestions from code review

    Co-authored-by: David Widmann <[email protected]>

    * Apply suggestions from code review

    Co-authored-by: Théo Galy-Fajou <[email protected]>

    * Update src/approximations/nystrom.jl

    Co-authored-by: Théo Galy-Fajou <[email protected]>

    * Update src/approximations/nystrom.jl

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
    Co-authored-by: David Widmann <[email protected]>
    Co-authored-by: Théo Galy-Fajou <[email protected]>

commit d1c68a9
Author: st-- <[email protected]>
Date:   Thu Jan 13 22:33:43 2022 +0100

    fix Distances compat (#423)

    * CompatHelper: bump compat for Distances to 0.10 for package test, (keep existing compat)

    * try out Theo's fix

    * fix test compat

    * use ForwardDiff for chain rule test of SqMahalanobis

    * test on 1.4 instead of 1.3 - see if the chainrules test passes there

    * revert version branch

    * revert to 1.3

    * test_broken for older Julia versions

    Co-authored-by: CompatHelper Julia <[email protected]>

commit 93d33c2
Author: st-- <[email protected]>
Date:   Wed Jan 12 14:11:14 2022 +0100

    fix figure & cleanup (#422)

    * fix figure & cleanup

    * bump LIBSVM compat & Manifest

    * improve writing, replaces #321

commit 40cb59e
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date:   Wed Jan 12 09:39:01 2022 +0200

    CompatHelper: bump compat for Kronecker to 0.5 for package docs, (keep existing compat) (#367)

    * CompatHelper: bump compat for Kronecker to 0.5 for package docs, (keep existing compat)

    * ] up

    Co-authored-by: CompatHelper Julia <[email protected]>
    Co-authored-by: st-- <[email protected]>

commit 7204529
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date:   Tue Jan 11 18:37:23 2022 +0200

    CompatHelper: bump compat for Kronecker to 0.5 for package test, (keep existing compat) (#366)

    Co-authored-by: CompatHelper Julia <[email protected]>
    Co-authored-by: st-- <[email protected]>

commit 924925d
Author: st-- <[email protected]>
Date:   Tue Jan 11 16:26:02 2022 +0100

    switch SVM example to half-moon dataset (#421)

commit 992b665
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date:   Fri Dec 24 12:18:56 2021 +0200

    CompatHelper: bump compat for SpecialFunctions to 2 for package test, (keep existing compat) (#412)

    Co-authored-by: CompatHelper Julia <[email protected]>
    Co-authored-by: Théo Galy-Fajou <[email protected]>
    Co-authored-by: st-- <[email protected]>

commit 04fa7f7
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date:   Thu Dec 23 13:33:59 2021 +0200

    CompatHelper: bump compat for SpecialFunctions to 2, (keep existing compat) (#411)

    Co-authored-by: CompatHelper Julia <[email protected]>
    Co-authored-by: Théo Galy-Fajou <[email protected]>
    Co-authored-by: st-- <[email protected]>

commit c0fc3e1
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date:   Thu Dec 23 01:10:40 2021 +0200

    CompatHelper: add new compat entry for Compat at version 3 for package test, (keep existing compat) (#418)

    Co-authored-by: CompatHelper Julia <[email protected]>

commit 05fe340
Author: st-- <[email protected]>
Date:   Tue Dec 21 00:49:37 2021 +0200

    use only() instead of first() (#403)

    * use only() instead of first() for 1-"vectors" that were for the benefit of Flux

    * fix one test that should not have worked as it was

    * add missing scalar Sinus constructor

commit 2d17212
Author: st-- <[email protected]>
Date:   Sat Dec 18 23:43:30 2021 +0200

    Zygote AD failure workarounds & test cleanup (#414)

    Zygote AD failures:

    * revert #409 (test_utils workaround for broken Zygote - now working again)

    * disable broken Zygote AD test for ChainTransform

    Improved tests:

    * finer-grained testsets

    * add missing test cases to test_AD

    * replace test_FiniteDiff with test_AD(..., :FiniteDiff, ...)

    * remove code duplication

commit 3c49949
Author: Théo Galy-Fajou <[email protected]>
Date:   Wed Nov 24 18:32:19 2021 +0100

    Fix typo in valid_inputs error (#408)

    * Fix typo in valid_inputs error

    * Update src/utils.jl

    Co-authored-by: David Widmann <[email protected]>

    Co-authored-by: David Widmann <[email protected]>

commit 9955044
Author: st-- <[email protected]>
Date:   Wed Nov 24 18:55:18 2021 +0200

    Fix for Zygote 0.6.30 breaking our tests (#409)

    * restrict Zygote to <0.6.30

    * revert Zygote test restriction and add finer-grained testset

    * Update test/utils.jl

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    * revert testset

    * mark test_broken

    * Use `@test_throws` instead of `@test_broken`

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
    Co-authored-by: David Widmann <[email protected]>

commit 33d64d1
Author: Théo Galy-Fajou <[email protected]>
Date:   Thu Nov 4 14:23:57 2021 +0100

    Add benchmarking CI (#399)

    * Add benchmark file

    * delete old benchmarks

    * Add github action

    * Add Project

    * Apply suggestions from code review

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    * Missing end of line

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

commit 360ce10
Author: David Widmann <[email protected]>
Date:   Tue Nov 2 11:09:58 2021 +0100

    Update docstring of `GibbsKernel` (#395)
@devmotion devmotion mentioned this pull request Mar 1, 2022
Closed
st-- added a commit that referenced this pull request Mar 21, 2022
@Crown421 @st-- @github-actions
Updated Train- kernel-parameters example (#430)
6033b56 
* initial version of training kernel parameters example from st/examples (#234)

* update script

* Apply suggestions from code review

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix out-of-domain initial value

* initial version of training kernel parameters example from st/examples (#234)

* update script

* Apply suggestions from code review

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix out-of-domain initial value

* Add ParameterHandling

* Extend example

* Failed attempts with default Flux opt

* Add Flux.destructure example

* Add some nothing

* Squashed commit of the following:

commit f9bbd84
Author: st-- <[email protected]>
Date:   Fri Jan 28 09:11:50 2022 +0100

    make nystrom work with AbstractVector (#427)

    * make nystrom work with AbstractVector

    * add test

    * Update test/approximations/nystrom.jl

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    * patch bump

    * Update test/approximations/nystrom.jl

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    * Apply suggestions from code review

    Co-authored-by: David Widmann <[email protected]>

    * Apply suggestions from code review

    * deprecate

    * Apply suggestions from code review

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    * Apply suggestions from code review

    Co-authored-by: David Widmann <[email protected]>

    * Apply suggestions from code review

    Co-authored-by: Théo Galy-Fajou <[email protected]>

    * Update src/approximations/nystrom.jl

    Co-authored-by: Théo Galy-Fajou <[email protected]>

    * Update src/approximations/nystrom.jl

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
    Co-authored-by: David Widmann <[email protected]>
    Co-authored-by: Théo Galy-Fajou <[email protected]>

commit d1c68a9
Author: st-- <[email protected]>
Date:   Thu Jan 13 22:33:43 2022 +0100

    fix Distances compat (#423)

    * CompatHelper: bump compat for Distances to 0.10 for package test, (keep existing compat)

    * try out Theo's fix

    * fix test compat

    * use ForwardDiff for chain rule test of SqMahalanobis

    * test on 1.4 instead of 1.3 - see if the chainrules test passes there

    * revert version branch

    * revert to 1.3

    * test_broken for older Julia versions

    Co-authored-by: CompatHelper Julia <[email protected]>

commit 93d33c2
Author: st-- <[email protected]>
Date:   Wed Jan 12 14:11:14 2022 +0100

    fix figure & cleanup (#422)

    * fix figure & cleanup

    * bump LIBSVM compat & Manifest

    * improve writing, replaces #321

commit 40cb59e
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date:   Wed Jan 12 09:39:01 2022 +0200

    CompatHelper: bump compat for Kronecker to 0.5 for package docs, (keep existing compat) (#367)

    * CompatHelper: bump compat for Kronecker to 0.5 for package docs, (keep existing compat)

    * ] up

    Co-authored-by: CompatHelper Julia <[email protected]>
    Co-authored-by: st-- <[email protected]>

commit 7204529
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date:   Tue Jan 11 18:37:23 2022 +0200

    CompatHelper: bump compat for Kronecker to 0.5 for package test, (keep existing compat) (#366)

    Co-authored-by: CompatHelper Julia <[email protected]>
    Co-authored-by: st-- <[email protected]>

commit 924925d
Author: st-- <[email protected]>
Date:   Tue Jan 11 16:26:02 2022 +0100

    switch SVM example to half-moon dataset (#421)

commit 992b665
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date:   Fri Dec 24 12:18:56 2021 +0200

    CompatHelper: bump compat for SpecialFunctions to 2 for package test, (keep existing compat) (#412)

    Co-authored-by: CompatHelper Julia <[email protected]>
    Co-authored-by: Théo Galy-Fajou <[email protected]>
    Co-authored-by: st-- <[email protected]>

commit 04fa7f7
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date:   Thu Dec 23 13:33:59 2021 +0200

    CompatHelper: bump compat for SpecialFunctions to 2, (keep existing compat) (#411)

    Co-authored-by: CompatHelper Julia <[email protected]>
    Co-authored-by: Théo Galy-Fajou <[email protected]>
    Co-authored-by: st-- <[email protected]>

commit c0fc3e1
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date:   Thu Dec 23 01:10:40 2021 +0200

    CompatHelper: add new compat entry for Compat at version 3 for package test, (keep existing compat) (#418)

    Co-authored-by: CompatHelper Julia <[email protected]>

commit 05fe340
Author: st-- <[email protected]>
Date:   Tue Dec 21 00:49:37 2021 +0200

    use only() instead of first() (#403)

    * use only() instead of first() for 1-"vectors" that were for the benefit of Flux

    * fix one test that should not have worked as it was

    * add missing scalar Sinus constructor

commit 2d17212
Author: st-- <[email protected]>
Date:   Sat Dec 18 23:43:30 2021 +0200

    Zygote AD failure workarounds & test cleanup (#414)

    Zygote AD failures:

    * revert #409 (test_utils workaround for broken Zygote - now working again)

    * disable broken Zygote AD test for ChainTransform

    Improved tests:

    * finer-grained testsets

    * add missing test cases to test_AD

    * replace test_FiniteDiff with test_AD(..., :FiniteDiff, ...)

    * remove code duplication

commit 3c49949
Author: Théo Galy-Fajou <[email protected]>
Date:   Wed Nov 24 18:32:19 2021 +0100

    Fix typo in valid_inputs error (#408)

    * Fix typo in valid_inputs error

    * Update src/utils.jl

    Co-authored-by: David Widmann <[email protected]>

    Co-authored-by: David Widmann <[email protected]>

commit 9955044
Author: st-- <[email protected]>
Date:   Wed Nov 24 18:55:18 2021 +0200

    Fix for Zygote 0.6.30 breaking our tests (#409)

    * restrict Zygote to <0.6.30

    * revert Zygote test restriction and add finer-grained testset

    * Update test/utils.jl

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    * revert testset

    * mark test_broken

    * Use `@test_throws` instead of `@test_broken`

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
    Co-authored-by: David Widmann <[email protected]>

commit 33d64d1
Author: Théo Galy-Fajou <[email protected]>
Date:   Thu Nov 4 14:23:57 2021 +0100

    Add benchmarking CI (#399)

    * Add benchmark file

    * delete old benchmarks

    * Add github action

    * Add Project

    * Apply suggestions from code review

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

    * Missing end of line

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

commit 360ce10
Author: David Widmann <[email protected]>
Date:   Tue Nov 2 11:09:58 2021 +0100

    Update docstring of `GibbsKernel` (#395)

* change title

* Update manifest

* Formatter

* Some small updates

* Add BenchmarkTools

* Move Benchmarks to the correct manifest

* Formatter

* Fix missed comment

* Add ParameterHandling too

* Change gif embed

* Forgot #

* Formatter

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Compat and gif kwargs

* , for ;

* different commit msg

* Some more clarification

* Change display

* Final changes

* Apply suggestions from code review

Co-authored-by: st-- <[email protected]>

* Apply suggestions from code review

Co-authored-by: st-- <[email protected]>

* Address comments

* Missed one

* Manifest issue

* Apply suggestions from formatter

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Local formatter

* Delete loss description.

Co-authored-by: st-- <[email protected]>

* format

Co-authored-by: ST John <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@st-- st-- st-- approved these changes

Assignees
No one assigned
Labels
performance critical Triggers benchmarking CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@theogf @devmotion @st--