Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add NNlib extension #85

Merged
merged 7 commits into from
May 22, 2024
Merged

Add NNlib extension #85

merged 7 commits into from
May 22, 2024

Conversation

adrhill
Copy link
Owner

@adrhill adrhill commented May 22, 2024 •
edited
Loading

Adds NNlib activation functions, as discussed in this comment in #56.

@adrhill adrhill requested a review from gdalle May 22, 2024 14:02
@adrhill adrhill added the run benchmark Run benchmarks in CI label May 22, 2024
@codecov-commenter
Copy link

codecov-commenter commented May 22, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 77.27273% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 79.36%. Comparing base (57def1e) to head (d299074).

Files Patch % Lines
ext/SparseConnectivityTracerNNlibExt.jl 76.19% 5 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main      #85      +/-   ##
==========================================
- Coverage   79.58%   79.36%   -0.22%     
==========================================
  Files          16       17       +1     
  Lines         676      698      +22     
==========================================
+ Hits          538      554      +16     
- Misses        138      144       +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions GitHub Actions
Copy link

Benchmark result

Judge result

Benchmark Report for /home/runner/work/SparseConnectivityTracer.jl/SparseConnectivityTracer.jl

Job Properties

  • Time of benchmarks:
    • Target: 22 May 2024 - 14:15
    • Baseline: 22 May 2024 - 14:19
  • Package commits:
    • Target: 08735e
    • Baseline: 57def1
  • Julia commits:
    • Target: 0b4590
    • Baseline: 0b4590
  • Julia command flags:
    • Target: None
    • Baseline: None
  • Environment variables:
    • Target: None
    • Baseline: None

Results

A ratio greater than 1.0 denotes a possible regression (marked with ❌), while a ratio less
than 1.0 denotes a possible improvement (marked with ✅). Only significant results - results
that indicate possible regressions or improvements - are shown below (thus, an empty table means that all
benchmark results remained invariant between builds).

ID time ratio memory ratio
["Jacobian", "Global", "conv", "size=128x128x3", "Set{UInt64}"] 1.14 (5%) ❌ 1.00 (1%)
["Jacobian", "Global", "conv", "size=128x128x3", "SortedVector{UInt64}"] 1.10 (5%) ❌ 1.00 (1%)
["Jacobian", "Global", "sparse_matmul", "sparsity=0.01", "BitSet"] 1.05 (5%) 1.03 (1%) ❌
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "BitSet"] 1.08 (5%) ❌ 0.98 (1%) ✅
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "DuplicateVector{UInt64}"] 1.16 (5%) ❌ 1.18 (1%) ❌
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "Set{UInt64}"] 1.07 (5%) ❌ 1.04 (1%) ❌
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "SortedVector{UInt64}"] 1.15 (5%) ❌ 1.07 (1%) ❌
["Jacobian", "Global", "sparse_matmul", "sparsity=0.1", "Set{UInt64}"] 1.03 (5%) 1.05 (1%) ❌
["Jacobian", "Global", "sparse_matmul", "sparsity=0.1", "SortedVector{UInt64}"] 1.00 (5%) 0.98 (1%) ✅
["Jacobian", "Global", "sparse_matmul", "sparsity=0.25", "DuplicateVector{UInt64}"] 1.01 (5%) 1.04 (1%) ❌
["Jacobian", "Global", "sparse_matmul", "sparsity=0.25", "SortedVector{UInt64}"] 1.07 (5%) ❌ 1.00 (1%)

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["Jacobian", "Global", "brusselator", "N=100"]
  • ["Jacobian", "Global", "brusselator", "N=24"]
  • ["Jacobian", "Global", "brusselator", "N=6"]
  • ["Jacobian", "Global", "brusselator_ode_solve", "N=100"]
  • ["Jacobian", "Global", "brusselator_ode_solve", "N=24"]
  • ["Jacobian", "Global", "brusselator_ode_solve", "N=6"]
  • ["Jacobian", "Global", "conv", "size=128x128x3"]
  • ["Jacobian", "Global", "conv", "size=28x28x3"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.01"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.05"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.1"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.25"]

Julia versioninfo

Target

Julia Version 1.10.3
Commit 0b4590a5507 (2024-04-30 10:59 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
      Ubuntu 22.04.4 LTS
  uname: Linux 6.5.0-1021-azure #22~22.04.1-Ubuntu SMP Tue Apr 30 16:08:18 UTC 2024 x86_64 x86_64
  CPU: AMD EPYC 7763 64-Core Processor: 
              speed         user         nice          sys         idle          irq
       #1  3082 MHz       2182 s          0 s        174 s       5799 s          0 s
       #2  3154 MHz       2699 s          0 s        174 s       5288 s          0 s
       #3  3246 MHz       2558 s          0 s        201 s       5390 s          0 s
       #4  3243 MHz       1529 s          0 s        174 s       6447 s          0 s
  Memory: 15.606502532958984 GB (13318.68359375 MB free)
  Uptime: 819.73 sec
  Load Avg:  1.0  1.03  0.74
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

Baseline

Julia Version 1.10.3
Commit 0b4590a5507 (2024-04-30 10:59 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
      Ubuntu 22.04.4 LTS
  uname: Linux 6.5.0-1021-azure #22~22.04.1-Ubuntu SMP Tue Apr 30 16:08:18 UTC 2024 x86_64 x86_64
  CPU: AMD EPYC 7763 64-Core Processor: 
              speed         user         nice          sys         idle          irq
       #1  2445 MHz       3113 s          0 s        205 s       7269 s          0 s
       #2  3218 MHz       4002 s          0 s        207 s       6385 s          0 s
       #3  3243 MHz       2712 s          0 s        220 s       7646 s          0 s
       #4  3254 MHz       1532 s          0 s        192 s       8855 s          0 s
  Memory: 15.606502532958984 GB (13711.36328125 MB free)
  Uptime: 1063.33 sec
  Load Avg:  1.0  1.01  0.82
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

Target result

Benchmark Report for /home/runner/work/SparseConnectivityTracer.jl/SparseConnectivityTracer.jl

Job Properties

  • Time of benchmark: 22 May 2024 - 14:15
  • Package commit: 08735e
  • Julia commit: 0b4590
  • Julia command flags: None
  • Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID time GC time memory allocations
["Jacobian", "Global", "brusselator", "N=100", "BitSet"] 46.193 ms (5%) 5.628 ms 145.98 MiB (1%) 426018
["Jacobian", "Global", "brusselator", "N=100", "DuplicateVector{UInt64}"] 65.859 ms (5%) 4.678 ms 75.53 MiB (1%) 740057
["Jacobian", "Global", "brusselator", "N=100", "Set{UInt64}"] 44.718 ms (5%) 69.12 MiB (1%) 660060
["Jacobian", "Global", "brusselator", "N=100", "SortedVector{UInt64}"] 10.663 ms (5%) 23.19 MiB (1%) 200057
["Jacobian", "Global", "brusselator", "N=24", "BitSet"] 906.170 μs (5%) 2.10 MiB (1%) 23418
["Jacobian", "Global", "brusselator", "N=24", "DuplicateVector{UInt64}"] 3.459 ms (5%) 4.38 MiB (1%) 42665
["Jacobian", "Global", "brusselator", "N=24", "Set{UInt64}"] 2.237 ms (5%) 4.01 MiB (1%) 38060
["Jacobian", "Global", "brusselator", "N=24", "SortedVector{UInt64}"] 593.998 μs (5%) 1.37 MiB (1%) 11561
["Jacobian", "Global", "brusselator", "N=6", "BitSet"] 49.312 μs (5%) 104.75 KiB (1%) 1254
["Jacobian", "Global", "brusselator", "N=6", "DuplicateVector{UInt64}"] 213.128 μs (5%) 281.31 KiB (1%) 2693
["Jacobian", "Global", "brusselator", "N=6", "Set{UInt64}"] 113.992 μs (5%) 258.02 KiB (1%) 2408
["Jacobian", "Global", "brusselator", "N=6", "SortedVector{UInt64}"] 38.432 μs (5%) 88.38 KiB (1%) 749
["Jacobian", "Global", "brusselator_ode_solve", "N=100", "BitSet"] 205.586 ms (5%) 48.598 ms 800.89 MiB (1%) 1225764
["Jacobian", "Global", "brusselator_ode_solve", "N=100", "DuplicateVector{UInt64}"] 385.206 ms (5%) 32.172 ms 531.22 MiB (1%) 3100171
["Jacobian", "Global", "brusselator_ode_solve", "N=100", "Set{UInt64}"] 318.483 ms (5%) 54.858 ms 360.10 MiB (1%) 2480179
["Jacobian", "Global", "brusselator_ode_solve", "N=100", "SortedVector{UInt64}"] 37.373 ms (5%) 3.724 ms 80.25 MiB (1%) 460171
["Jacobian", "Global", "brusselator_ode_solve", "N=24", "BitSet"] 2.616 ms (5%) 6.32 MiB (1%) 67431
["Jacobian", "Global", "brusselator_ode_solve", "N=24", "DuplicateVector{UInt64}"] 17.288 ms (5%) 31.01 MiB (1%) 178708
["Jacobian", "Global", "brusselator_ode_solve", "N=24", "Set{UInt64}"] 12.785 ms (5%) 20.76 MiB (1%) 143004
["Jacobian", "Global", "brusselator_ode_solve", "N=24", "SortedVector{UInt64}"] 1.812 ms (5%) 4.64 MiB (1%) 26644
["Jacobian", "Global", "brusselator_ode_solve", "N=6", "BitSet"] 140.792 μs (5%) 275.11 KiB (1%) 3233
["Jacobian", "Global", "brusselator_ode_solve", "N=6", "DuplicateVector{UInt64}"] 1.066 ms (5%) 1.95 MiB (1%) 11295
["Jacobian", "Global", "brusselator_ode_solve", "N=6", "Set{UInt64}"] 731.555 μs (5%) 1.31 MiB (1%) 9071
["Jacobian", "Global", "brusselator_ode_solve", "N=6", "SortedVector{UInt64}"] 130.663 μs (5%) 305.86 KiB (1%) 1791
["Jacobian", "Global", "conv", "size=128x128x3", "BitSet"] 1.943 s (5%) 385.702 ms 19.15 GiB (1%) 16384623
["Jacobian", "Global", "conv", "size=128x128x3", "DuplicateVector{UInt64}"] 1.144 s (5%) 86.584 ms 1.20 GiB (1%) 7109869
["Jacobian", "Global", "conv", "size=128x128x3", "Set{UInt64}"] 3.675 s (5%) 816.293 ms 4.65 GiB (1%) 34480063
["Jacobian", "Global", "conv", "size=128x128x3", "SortedVector{UInt64}"] 698.712 ms (5%) 89.906 ms 1.20 GiB (1%) 7159021
["Jacobian", "Global", "conv", "size=28x28x3", "BitSet"] 25.451 ms (5%) 2.021 ms 74.83 MiB (1%) 598714
["Jacobian", "Global", "conv", "size=28x28x3", "DuplicateVector{UInt64}"] 36.962 ms (5%) 48.51 MiB (1%) 267454
["Jacobian", "Global", "conv", "size=28x28x3", "Set{UInt64}"] 111.038 ms (5%) 10.262 ms 181.66 MiB (1%) 1294798
["Jacobian", "Global", "conv", "size=28x28x3", "SortedVector{UInt64}"] 20.520 ms (5%) 48.65 MiB (1%) 269806
["Jacobian", "Global", "sparse_matmul", "sparsity=0.01", "BitSet"] 16.100 μs (5%) 31.59 KiB (1%) 377
["Jacobian", "Global", "sparse_matmul", "sparsity=0.01", "DuplicateVector{UInt64}"] 13.185 μs (5%) 22.52 KiB (1%) 230
["Jacobian", "Global", "sparse_matmul", "sparsity=0.01", "Set{UInt64}"] 25.207 μs (5%) 70.62 KiB (1%) 671
["Jacobian", "Global", "sparse_matmul", "sparsity=0.01", "SortedVector{UInt64}"] 14.427 μs (5%) 25.25 KiB (1%) 276
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "BitSet"] 86.752 μs (5%) 159.38 KiB (1%) 1283
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "DuplicateVector{UInt64}"] 931.399 μs (5%) 1.29 MiB (1%) 7697
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "Set{UInt64}"] 294.369 μs (5%) 487.98 KiB (1%) 3186
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "SortedVector{UInt64}"] 84.838 μs (5%) 169.25 KiB (1%) 753
["Jacobian", "Global", "sparse_matmul", "sparsity=0.1", "BitSet"] 166.781 μs (5%) 340.52 KiB (1%) 2503
["Jacobian", "Global", "sparse_matmul", "sparsity=0.1", "DuplicateVector{UInt64}"] 5.386 ms (5%) 12.08 MiB (1%) 24957
["Jacobian", "Global", "sparse_matmul", "sparsity=0.1", "Set{UInt64}"] 1.315 ms (5%) 2.20 MiB (1%) 8563
["Jacobian", "Global", "sparse_matmul", "sparsity=0.1", "SortedVector{UInt64}"] 221.493 μs (5%) 643.03 KiB (1%) 1316
["Jacobian", "Global", "sparse_matmul", "sparsity=0.25", "BitSet"] 346.627 μs (5%) 597.56 KiB (1%) 6093
["Jacobian", "Global", "sparse_matmul", "sparsity=0.25", "DuplicateVector{UInt64}"] 328.993 ms (5%) 165.683 ms 804.28 MiB (1%) 29945
["Jacobian", "Global", "sparse_matmul", "sparsity=0.25", "Set{UInt64}"] 4.403 ms (5%) 8.26 MiB (1%) 25187
["Jacobian", "Global", "sparse_matmul", "sparsity=0.25", "SortedVector{UInt64}"] 605.300 μs (5%) 1.98 MiB (1%) 3180

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["Jacobian", "Global", "brusselator", "N=100"]
  • ["Jacobian", "Global", "brusselator", "N=24"]
  • ["Jacobian", "Global", "brusselator", "N=6"]
  • ["Jacobian", "Global", "brusselator_ode_solve", "N=100"]
  • ["Jacobian", "Global", "brusselator_ode_solve", "N=24"]
  • ["Jacobian", "Global", "brusselator_ode_solve", "N=6"]
  • ["Jacobian", "Global", "conv", "size=128x128x3"]
  • ["Jacobian", "Global", "conv", "size=28x28x3"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.01"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.05"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.1"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.25"]

Julia versioninfo

Julia Version 1.10.3
Commit 0b4590a5507 (2024-04-30 10:59 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
      Ubuntu 22.04.4 LTS
  uname: Linux 6.5.0-1021-azure #22~22.04.1-Ubuntu SMP Tue Apr 30 16:08:18 UTC 2024 x86_64 x86_64
  CPU: AMD EPYC 7763 64-Core Processor: 
              speed         user         nice          sys         idle          irq
       #1  3082 MHz       2182 s          0 s        174 s       5799 s          0 s
       #2  3154 MHz       2699 s          0 s        174 s       5288 s          0 s
       #3  3246 MHz       2558 s          0 s        201 s       5390 s          0 s
       #4  3243 MHz       1529 s          0 s        174 s       6447 s          0 s
  Memory: 15.606502532958984 GB (13318.68359375 MB free)
  Uptime: 819.73 sec
  Load Avg:  1.0  1.03  0.74
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

Baseline result

Benchmark Report for /home/runner/work/SparseConnectivityTracer.jl/SparseConnectivityTracer.jl

Job Properties

  • Time of benchmark: 22 May 2024 - 14:19
  • Package commit: 57def1
  • Julia commit: 0b4590
  • Julia command flags: None
  • Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID time GC time memory allocations
["Jacobian", "Global", "brusselator", "N=100", "BitSet"] 46.130 ms (5%) 10.655 ms 145.98 MiB (1%) 426018
["Jacobian", "Global", "brusselator", "N=100", "DuplicateVector{UInt64}"] 66.731 ms (5%) 4.829 ms 75.53 MiB (1%) 740057
["Jacobian", "Global", "brusselator", "N=100", "Set{UInt64}"] 43.450 ms (5%) 69.12 MiB (1%) 660060
["Jacobian", "Global", "brusselator", "N=100", "SortedVector{UInt64}"] 10.588 ms (5%) 23.19 MiB (1%) 200057
["Jacobian", "Global", "brusselator", "N=24", "BitSet"] 915.149 μs (5%) 2.10 MiB (1%) 23418
["Jacobian", "Global", "brusselator", "N=24", "DuplicateVector{UInt64}"] 3.469 ms (5%) 4.38 MiB (1%) 42665
["Jacobian", "Global", "brusselator", "N=24", "Set{UInt64}"] 2.237 ms (5%) 4.01 MiB (1%) 38060
["Jacobian", "Global", "brusselator", "N=24", "SortedVector{UInt64}"] 592.546 μs (5%) 1.37 MiB (1%) 11561
["Jacobian", "Global", "brusselator", "N=6", "BitSet"] 48.220 μs (5%) 104.75 KiB (1%) 1254
["Jacobian", "Global", "brusselator", "N=6", "DuplicateVector{UInt64}"] 212.296 μs (5%) 281.31 KiB (1%) 2693
["Jacobian", "Global", "brusselator", "N=6", "Set{UInt64}"] 113.371 μs (5%) 258.02 KiB (1%) 2408
["Jacobian", "Global", "brusselator", "N=6", "SortedVector{UInt64}"] 37.881 μs (5%) 88.38 KiB (1%) 749
["Jacobian", "Global", "brusselator_ode_solve", "N=100", "BitSet"] 211.818 ms (5%) 49.926 ms 800.89 MiB (1%) 1225764
["Jacobian", "Global", "brusselator_ode_solve", "N=100", "DuplicateVector{UInt64}"] 381.023 ms (5%) 34.746 ms 531.22 MiB (1%) 3100171
["Jacobian", "Global", "brusselator_ode_solve", "N=100", "Set{UInt64}"] 318.764 ms (5%) 45.306 ms 360.10 MiB (1%) 2480179
["Jacobian", "Global", "brusselator_ode_solve", "N=100", "SortedVector{UInt64}"] 36.939 ms (5%) 3.328 ms 80.25 MiB (1%) 460171
["Jacobian", "Global", "brusselator_ode_solve", "N=24", "BitSet"] 2.580 ms (5%) 6.32 MiB (1%) 67431
["Jacobian", "Global", "brusselator_ode_solve", "N=24", "DuplicateVector{UInt64}"] 17.519 ms (5%) 31.01 MiB (1%) 178708
["Jacobian", "Global", "brusselator_ode_solve", "N=24", "Set{UInt64}"] 12.957 ms (5%) 20.76 MiB (1%) 143004
["Jacobian", "Global", "brusselator_ode_solve", "N=24", "SortedVector{UInt64}"] 1.811 ms (5%) 4.64 MiB (1%) 26644
["Jacobian", "Global", "brusselator_ode_solve", "N=6", "BitSet"] 138.910 μs (5%) 275.11 KiB (1%) 3233
["Jacobian", "Global", "brusselator_ode_solve", "N=6", "DuplicateVector{UInt64}"] 1.076 ms (5%) 1.95 MiB (1%) 11295
["Jacobian", "Global", "brusselator_ode_solve", "N=6", "Set{UInt64}"] 724.613 μs (5%) 1.31 MiB (1%) 9071
["Jacobian", "Global", "brusselator_ode_solve", "N=6", "SortedVector{UInt64}"] 130.965 μs (5%) 305.86 KiB (1%) 1791
["Jacobian", "Global", "conv", "size=128x128x3", "BitSet"] 2.025 s (5%) 391.529 ms 19.15 GiB (1%) 16384623
["Jacobian", "Global", "conv", "size=128x128x3", "DuplicateVector{UInt64}"] 1.111 s (5%) 60.864 ms 1.20 GiB (1%) 7109869
["Jacobian", "Global", "conv", "size=128x128x3", "Set{UInt64}"] 3.230 s (5%) 522.739 ms 4.65 GiB (1%) 34480063
["Jacobian", "Global", "conv", "size=128x128x3", "SortedVector{UInt64}"] 633.668 ms (5%) 57.984 ms 1.20 GiB (1%) 7159021
["Jacobian", "Global", "conv", "size=28x28x3", "BitSet"] 25.307 ms (5%) 2.410 ms 74.83 MiB (1%) 598714
["Jacobian", "Global", "conv", "size=28x28x3", "DuplicateVector{UInt64}"] 36.276 ms (5%) 48.51 MiB (1%) 267454
["Jacobian", "Global", "conv", "size=28x28x3", "Set{UInt64}"] 113.451 ms (5%) 12.891 ms 181.66 MiB (1%) 1294798
["Jacobian", "Global", "conv", "size=28x28x3", "SortedVector{UInt64}"] 20.308 ms (5%) 48.65 MiB (1%) 269806
["Jacobian", "Global", "sparse_matmul", "sparsity=0.01", "BitSet"] 15.359 μs (5%) 30.66 KiB (1%) 363
["Jacobian", "Global", "sparse_matmul", "sparsity=0.01", "DuplicateVector{UInt64}"] 12.965 μs (5%) 22.48 KiB (1%) 231
["Jacobian", "Global", "sparse_matmul", "sparsity=0.01", "Set{UInt64}"] 25.888 μs (5%) 70.23 KiB (1%) 667
["Jacobian", "Global", "sparse_matmul", "sparsity=0.01", "SortedVector{UInt64}"] 14.377 μs (5%) 25.31 KiB (1%) 277
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "BitSet"] 80.501 μs (5%) 162.38 KiB (1%) 1303
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "DuplicateVector{UInt64}"] 802.798 μs (5%) 1.09 MiB (1%) 6624
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "Set{UInt64}"] 275.705 μs (5%) 470.09 KiB (1%) 3067
["Jacobian", "Global", "sparse_matmul", "sparsity=0.05", "SortedVector{UInt64}"] 73.758 μs (5%) 157.97 KiB (1%) 730
["Jacobian", "Global", "sparse_matmul", "sparsity=0.1", "BitSet"] 161.872 μs (5%) 341.17 KiB (1%) 2483
["Jacobian", "Global", "sparse_matmul", "sparsity=0.1", "DuplicateVector{UInt64}"] 5.519 ms (5%) 12.16 MiB (1%) 24964
["Jacobian", "Global", "sparse_matmul", "sparsity=0.1", "Set{UInt64}"] 1.274 ms (5%) 2.11 MiB (1%) 8564
["Jacobian", "Global", "sparse_matmul", "sparsity=0.1", "SortedVector{UInt64}"] 220.983 μs (5%) 659.38 KiB (1%) 1334
["Jacobian", "Global", "sparse_matmul", "sparsity=0.25", "BitSet"] 335.937 μs (5%) 594.56 KiB (1%) 6047
["Jacobian", "Global", "sparse_matmul", "sparsity=0.25", "DuplicateVector{UInt64}"] 326.839 ms (5%) 161.280 ms 770.52 MiB (1%) 29963
["Jacobian", "Global", "sparse_matmul", "sparsity=0.25", "Set{UInt64}"] 4.327 ms (5%) 8.29 MiB (1%) 25390
["Jacobian", "Global", "sparse_matmul", "sparsity=0.25", "SortedVector{UInt64}"] 565.245 μs (5%) 1.98 MiB (1%) 3160

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

  • ["Jacobian", "Global", "brusselator", "N=100"]
  • ["Jacobian", "Global", "brusselator", "N=24"]
  • ["Jacobian", "Global", "brusselator", "N=6"]
  • ["Jacobian", "Global", "brusselator_ode_solve", "N=100"]
  • ["Jacobian", "Global", "brusselator_ode_solve", "N=24"]
  • ["Jacobian", "Global", "brusselator_ode_solve", "N=6"]
  • ["Jacobian", "Global", "conv", "size=128x128x3"]
  • ["Jacobian", "Global", "conv", "size=28x28x3"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.01"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.05"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.1"]
  • ["Jacobian", "Global", "sparse_matmul", "sparsity=0.25"]

Julia versioninfo

Julia Version 1.10.3
Commit 0b4590a5507 (2024-04-30 10:59 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
      Ubuntu 22.04.4 LTS
  uname: Linux 6.5.0-1021-azure #22~22.04.1-Ubuntu SMP Tue Apr 30 16:08:18 UTC 2024 x86_64 x86_64
  CPU: AMD EPYC 7763 64-Core Processor: 
              speed         user         nice          sys         idle          irq
       #1  2445 MHz       3113 s          0 s        205 s       7269 s          0 s
       #2  3218 MHz       4002 s          0 s        207 s       6385 s          0 s
       #3  3243 MHz       2712 s          0 s        220 s       7646 s          0 s
       #4  3254 MHz       1532 s          0 s        192 s       8855 s          0 s
  Memory: 15.606502532958984 GB (13711.36328125 MB free)
  Uptime: 1063.33 sec
  Load Avg:  1.0  1.01  0.82
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

Runtime information

Runtime Info
BLAS #threads 2
BLAS.vendor() lbt
Sys.CPU_THREADS 4

lscpu output:

Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             4
On-line CPU(s) list:                0-3
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7763 64-Core Processor
CPU family:                         25
Model:                              1
Thread(s) per core:                 2
Core(s) per socket:                 2
Socket(s):                          1
Stepping:                           1
BogoMIPS:                           4890.85
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload umip vaes vpclmulqdq rdpid fsrm
Virtualization:                     AMD-V
Hypervisor vendor:                  Microsoft
Virtualization type:                full
L1d cache:                          64 KiB (2 instances)
L1i cache:                          64 KiB (2 instances)
L2 cache:                           1 MiB (2 instances)
L3 cache:                           32 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-3
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Cpu Property Value
Brand AMD EPYC 7763 64-Core Processor
Vendor :AMD
Architecture :Unknown
Model Family: 0xaf, Model: 0x01, Stepping: 0x01, Type: 0x00
Cores 16 physical cores, 16 logical cores (on executing CPU)
No Hyperthreading hardware capability detected
Clock Frequencies Not supported by CPU
Data Cache Level 1:3 : (32, 512, 32768) kbytes
64 byte cache line size
Address Size 48 bits virtual, 48 bits physical
SIMD 256 bit = 32 byte max. SIMD vector size
Time Stamp Counter TSC is accessible via rdtsc
TSC runs at constant rate (invariant from clock frequency)
Perf. Monitoring Performance Monitoring Counters (PMC) are not supported
Hypervisor Yes, Microsoft

gdalle
gdalle reviewed May 22, 2024
View reviewed changes
Copy link
Collaborator

@gdalle gdalle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing a few second order tests but otherwise lgtm

As a side note, NNlib activation functions are implemented from Base functions, so if we supported the following properly it would be enough:

  • ifelse
  • oftype
  • clamp
  • muladd
  • evalpoly
  • @fastmath (not sure what it takes to support it)

Of course the local sparsity rules would not kick in, or not as well

ext/SparseConnectivityTracerNNlibExt.jl Show resolved Hide resolved
@adrhill
Copy link
Owner Author

adrhill commented May 22, 2024

As a side note, NNlib activation functions are implemented from Base functions

To some degree, the argument that everything is made up from Base functions can be applied to any pure Julia code. We're trying to skip some computations.

I was hoping for a visible speed-up in the benchmarks, but they only test global tracing on Conv layers with identity activation functions.

Benchmarking locally, the difference is very minor on relu (2 allocations saved):

julia> @benchmark local_jacobian_pattern(LAYER_RELU, $INPUT_FLUX)
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  16.458 μs …  2.417 ms  ┊ GC (min … max): 0.00% … 98.49%
 Time  (median):     18.083 μs              ┊ GC (median):    0.00%
 Time  (mean ± σ):   18.862 μs ± 38.831 μs  ┊ GC (mean ± σ):  3.96% ±  1.95%

    ▁▄▄▃▁       ▁▂▄▅▇▆█▇▆▅▃▂▁                                 ▂
  ▃▅███████▆▇▇█████████████████▇████████▇▅██▇▇▇▇▆▆▆▂▆▅▅▆▆▄▆▆▆ █
  16.5 μs      Histogram: log(frequency) by time      21.1 μs <

 Memory estimate: 23.91 KiB, allocs estimate: 355.
julia> @benchmark local_jacobian_pattern(LAYER_RELU, $INPUT_FLUX)
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  16.625 μs …  2.329 ms  ┊ GC (min … max): 0.00% … 98.01%
 Time  (median):     18.000 μs              ┊ GC (median):    0.00%
 Time  (mean ± σ):   18.891 μs ± 37.138 μs  ┊ GC (mean ± σ):  3.76% ±  1.95%

            ▁▂▄▅▅▇██▇▆▄▃▂▁▁▁▁▁  ▁ ▁▁▁▁                        ▂
  ▃▁▁▁▃▆▅▇▇████████████████████▇████████████▇█▆▇▇▆▇▆▆▇▆▆▇▆▆▆▅ █
  16.6 μs      Histogram: log(frequency) by time      21.4 μs <

 Memory estimate: 23.78 KiB, allocs estimate: 353.

For other activation functions that are more involved than relu, an increase in performance should be much more noticeable.

@adrhill
Copy link
Owner Author

adrhill commented May 22, 2024

so if we supported the following properly it would be enough

We already support and test on ifelse:

SparseConnectivityTracer.jl/test/second_order.jl

Lines 190 to 197 in 57def1e

f2(x) = ifelse(x[2] < x[3], x[1] * x[2], x[3] * x[4])
h = hessian_sparsity(f2, [1 2 3 4], method)
@test h ≈ [
0 1 0 0
1 0 0 0
0 0 0 0
0 0 0 0
]

Adding clamp, muladd and friends would open the can of worms that are three argument 3_to_1 functions. These have even more cases of differentiability than the already complicated 2_to_1 functions, in case we want to support them in a generic fashion.

If we ever support them, then probably just by computing the most conservative estimate of the Jacobian/Hessian. Or we simply add the minimum required code for overloads on clamp and muladd specifically.

@gdalle
Copy link
Collaborator

gdalle commented May 22, 2024

I'm not against the extension, I think it's a good idea. Just using the opportunity to highlight that we're missing some stuff from Base, but #74 is perfect to track it. Feel free to merge!

@adrhill adrhill merged commit 10c6961 into main May 22, 2024
5 checks passed
@adrhill adrhill deleted the ah/nnlib-ext branch May 22, 2024 14:58
@adrhill
Copy link
Owner Author

adrhill commented May 22, 2024 •
edited
Loading

I just realized another upside of adding extensions: it allows non-dual tracers to run through functions that otherwise call ifelse, max etc.!

@gdalle
Copy link
Collaborator

gdalle commented May 22, 2024

They should already be able to flow through ifelse and max

@adrhill
Copy link
Owner Author

adrhill commented May 22, 2024

You're right about max, but ifelse requires a boolean primal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gdalle gdalle gdalle left review comments

Assignees
No one assigned
Labels
run benchmark Run benchmarks in CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@adrhill @codecov-commenter @gdalle