Add support for batches #46

adrhill · 2022-03-28T22:05:35Z

Closes #39.

Support and tests for analyzer calls on input batches.
Analyzers still return Explanations. However, if the batch dimension (=last dim.) of the explanation is not equal to one, heatmap will output a Vector of images.

As a default, this vector is unpacked for "singleton batches". This can be turned off by setting the heatmap kwarg unpack_singleton=false.

This required refactoring of the neuron selection, calls to gradient_wrt_input and an additional Tullio Einsum index for LRP rules on Dense layers.

codecov-commenter · 2022-03-29T11:25:07Z

Codecov Report

Merging #46 (087a545) into master (715c5a8) will increase coverage by 1.86%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #46      +/-   ##
==========================================
+ Coverage   94.52%   96.39%   +1.86%     
==========================================
  Files           9        9              
  Lines         201      222      +21     
==========================================
+ Hits          190      214      +24     
+ Misses         11        8       -3

Impacted Files	Coverage Δ
src/lrp_checks.jl	`100.00% <ø> (+3.70%)`	⬆️
src/analyze_api.jl	`100.00% <100.00%> (ø)`
src/flux.jl	`100.00% <100.00%> (+3.70%)`	⬆️
src/gradient.jl	`100.00% <100.00%> (ø)`
src/heatmap.jl	`100.00% <100.00%> (+3.44%)`	⬆️
src/lrp.jl	`96.29% <100.00%> (+0.14%)`	⬆️
src/lrp_rules.jl	`100.00% <100.00%> (ø)`
src/neuron_selection.jl	`100.00% <100.00%> (ø)`
src/utils.jl	`56.25% <100.00%> (ø)`
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 312c060...087a545. Read the comment docs.

github-actions · 2022-03-29T11:32:43Z

Benchmark result

Judge result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Time of benchmarks:
- Target: 29 Mar 2022 - 11:27
- Baseline: 29 Mar 2022 - 11:32
Package commits:
- Target: 30e9c8
- Baseline: 312c06
Julia commits:
- Target: bf5349
- Baseline: bf5349
Julia command flags:
- Target: None
- Baseline: None
Environment variables:
- Target: None
- Baseline: None

Results

A ratio greater than 1.0 denotes a possible regression (marked with ❌), while a ratio less
than 1.0 denotes a possible improvement (marked with ✅). Only significant results - results
that indicate possible regressions or improvements - are shown below (thus, an empty table means that all
benchmark results remained invariant between builds).

ID	time ratio	memory ratio
`["Layer", "Conv", "EpsilonRule"]`	0.89 (5%) ✅	1.00 (1%)
`["Layer", "Conv", "ZeroRule"]`	1.07 (5%) ❌	1.00 (1%)
`["Layer", "Dense", "EpsilonRule"]`	2.80 (5%) ❌	1.01 (1%)
`["Layer", "Dense", "GammaRule"]`	1.21 (5%) ❌	1.00 (1%)
`["Layer", "Dense", "ZBoxRule"]`	2.32 (5%) ❌	1.00 (1%)
`["Layer", "Dense", "ZeroRule"]`	2.64 (5%) ❌	1.01 (1%)
`["Layer", "MaxPool", "EpsilonRule"]`	0.91 (5%) ✅	1.00 (1%)
`["Layer", "MaxPool", "GammaRule"]`	1.06 (5%) ❌	1.00 (1%)
`["Layer", "WrappedDense", "EpsilonRule"]`	2.32 (5%) ❌	1.00 (1%)
`["Layer", "WrappedDense", "GammaRule"]`	1.62 (5%) ❌	1.00 (1%)
`["Layer", "WrappedDense", "ZBoxRule"]`	2.02 (5%) ❌	1.00 (1%)
`["Layer", "WrappedDense", "ZeroRule"]`	2.73 (5%) ❌	1.00 (1%)
`["VGG", "Gradient", "construct analyzer"]`	0.75 (5%) ✅	1.00 (1%)
`["VGG", "InputTimesGradient", "analyze"]`	0.05 (5%) ✅	0.59 (1%) ✅
`["VGG", "InputTimesGradient", "construct analyzer"]`	1.79 (5%) ❌	1.00 (1%)
`["VGG", "LRPCustom", "analyze"]`	0.21 (5%) ✅	0.78 (1%) ✅
`["VGG", "LRPCustom", "construct analyzer"]`	0.31 (5%) ✅	1.00 (1%)
`["VGG", "LRPZero", "construct analyzer"]`	1.12 (5%) ❌	1.00 (1%)

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

["Layer", "Conv"]
["Layer", "Dense"]
["Layer", "MaxPool"]
["Layer", "WrappedDense"]
["VGG", "Gradient"]
["VGG", "InputTimesGradient"]
["VGG", "LRPCustom"]
["VGG", "LRPZero"]

Julia versioninfo

Target

Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      Ubuntu 20.04.4 LTS
  uname: Linux 5.11.0-1028-azure #31~20.04.2-Ubuntu SMP Tue Jan 18 08:46:15 UTC 2022 x86_64 x86_64
  CPU: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz: 
              speed         user         nice          sys         idle          irq
       #1  2294 MHz       5061 s          2 s        408 s       2359 s          0 s
       #2  2294 MHz       3778 s          1 s        383 s       3609 s          0 s
       
  Memory: 6.7845458984375 GB (661.12890625 MB free)
  Uptime: 790.56 sec
  Load Avg:  1.16  1.18  0.81
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, broadwell)

Baseline

Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      Ubuntu 20.04.4 LTS
  uname: Linux 5.11.0-1028-azure #31~20.04.2-Ubuntu SMP Tue Jan 18 08:46:15 UTC 2022 x86_64 x86_64
  CPU: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz: 
              speed         user         nice          sys         idle          irq
       #1  2294 MHz       7227 s          2 s        460 s       2976 s          0 s
       #2  2294 MHz       4607 s          1 s        524 s       5470 s          0 s
       
  Memory: 6.7845458984375 GB (1603.68359375 MB free)
  Uptime: 1074.4 sec
  Load Avg:  1.19  1.17  0.91
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, broadwell)

Target result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Time of benchmark: 29 Mar 2022 - 11:27
Package commit: 30e9c8
Julia commit: bf5349
Julia command flags: None
Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID	time	GC time	memory	allocations
`["Layer", "Conv", "EpsilonRule"]`	1.040 ms (5%)		1.54 MiB (1%)	361
`["Layer", "Conv", "GammaRule"]`	1.151 ms (5%)		1.54 MiB (1%)	373
`["Layer", "Conv", "ZBoxRule"]`	3.715 ms (5%)		4.77 MiB (1%)	1047
`["Layer", "Conv", "ZeroRule"]`	1.183 ms (5%)		1.54 MiB (1%)	361
`["Layer", "Dense", "EpsilonRule"]`	78.801 μs (5%)		3.03 KiB (1%)	8
`["Layer", "Dense", "GammaRule"]`	174.801 μs (5%)		590.56 KiB (1%)	17
`["Layer", "Dense", "ZBoxRule"]`	714.604 μs (5%)		1.01 MiB (1%)	416
`["Layer", "Dense", "ZeroRule"]`	74.501 μs (5%)		3.03 KiB (1%)	8
`["Layer", "MaxPool", "EpsilonRule"]`	112.301 μs (5%)		107.23 KiB (1%)	70
`["Layer", "MaxPool", "GammaRule"]`	114.101 μs (5%)		107.23 KiB (1%)	70
`["Layer", "MaxPool", "ZBoxRule"]`	111.601 μs (5%)		106.52 KiB (1%)	70
`["Layer", "MaxPool", "ZeroRule"]`	118.201 μs (5%)		106.52 KiB (1%)	70
`["Layer", "WrappedDense", "EpsilonRule"]`	193.701 μs (5%)		212.08 KiB (1%)	153
`["Layer", "WrappedDense", "GammaRule"]`	300.001 μs (5%)		799.61 KiB (1%)	162
`["Layer", "WrappedDense", "ZBoxRule"]`	644.803 μs (5%)		1.01 MiB (1%)	416
`["Layer", "WrappedDense", "ZeroRule"]`	211.401 μs (5%)		211.31 KiB (1%)	153
`["VGG", "Gradient", "analyze"]`	1.498 s (5%)	18.498 ms	1.10 GiB (1%)	3708
`["VGG", "Gradient", "construct analyzer"]`	16.900 μs (5%)		27.41 KiB (1%)	205
`["VGG", "InputTimesGradient", "analyze"]`	1.471 s (5%)	18.289 ms	1.10 GiB (1%)	3713
`["VGG", "InputTimesGradient", "construct analyzer"]`	19.200 μs (5%)		27.41 KiB (1%)	205
`["VGG", "LRPCustom", "analyze"]`	2.710 s (5%)	36.151 ms	2.65 GiB (1%)	4717
`["VGG", "LRPCustom", "construct analyzer"]`	27.400 μs (5%)		45.52 KiB (1%)	155
`["VGG", "LRPZero", "analyze"]`	1.493 s (5%)	23.372 ms	933.87 MiB (1%)	3908
`["VGG", "LRPZero", "construct analyzer"]`	21.600 μs (5%)		43.88 KiB (1%)	82

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

["Layer", "Conv"]
["Layer", "Dense"]
["Layer", "MaxPool"]
["Layer", "WrappedDense"]
["VGG", "Gradient"]
["VGG", "InputTimesGradient"]
["VGG", "LRPCustom"]
["VGG", "LRPZero"]

Julia versioninfo

Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      Ubuntu 20.04.4 LTS
  uname: Linux 5.11.0-1028-azure #31~20.04.2-Ubuntu SMP Tue Jan 18 08:46:15 UTC 2022 x86_64 x86_64
  CPU: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz: 
              speed         user         nice          sys         idle          irq
       #1  2294 MHz       5061 s          2 s        408 s       2359 s          0 s
       #2  2294 MHz       3778 s          1 s        383 s       3609 s          0 s
       
  Memory: 6.7845458984375 GB (661.12890625 MB free)
  Uptime: 790.56 sec
  Load Avg:  1.16  1.18  0.81
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, broadwell)

Baseline result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Time of benchmark: 29 Mar 2022 - 11:32
Package commit: 312c06
Julia commit: bf5349
Julia command flags: None
Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID	time	GC time	memory	allocations
`["Layer", "Conv", "EpsilonRule"]`	1.172 ms (5%)		1.54 MiB (1%)	361
`["Layer", "Conv", "GammaRule"]`	1.160 ms (5%)		1.54 MiB (1%)	373
`["Layer", "Conv", "ZBoxRule"]`	3.722 ms (5%)		4.77 MiB (1%)	1047
`["Layer", "Conv", "ZeroRule"]`	1.110 ms (5%)		1.54 MiB (1%)	361
`["Layer", "Dense", "EpsilonRule"]`	28.100 μs (5%)		3.02 KiB (1%)	8
`["Layer", "Dense", "GammaRule"]`	143.901 μs (5%)		590.55 KiB (1%)	17
`["Layer", "Dense", "ZBoxRule"]`	308.102 μs (5%)		1.01 MiB (1%)	413
`["Layer", "Dense", "ZeroRule"]`	28.200 μs (5%)		3.02 KiB (1%)	8
`["Layer", "MaxPool", "EpsilonRule"]`	123.201 μs (5%)		107.23 KiB (1%)	70
`["Layer", "MaxPool", "GammaRule"]`	107.501 μs (5%)		107.23 KiB (1%)	70
`["Layer", "MaxPool", "ZBoxRule"]`	114.301 μs (5%)		106.52 KiB (1%)	70
`["Layer", "MaxPool", "ZeroRule"]`	115.801 μs (5%)		106.52 KiB (1%)	70
`["Layer", "WrappedDense", "EpsilonRule"]`	83.400 μs (5%)		211.66 KiB (1%)	152
`["Layer", "WrappedDense", "GammaRule"]`	184.701 μs (5%)		799.19 KiB (1%)	161
`["Layer", "WrappedDense", "ZBoxRule"]`	318.802 μs (5%)		1.01 MiB (1%)	413
`["Layer", "WrappedDense", "ZeroRule"]`	77.501 μs (5%)		211.05 KiB (1%)	152
`["VGG", "Gradient", "analyze"]`	1.467 s (5%)	17.089 ms	1.10 GiB (1%)	3697
`["VGG", "Gradient", "construct analyzer"]`	22.601 μs (5%)		27.41 KiB (1%)	205
`["VGG", "InputTimesGradient", "analyze"]`	32.397 s (5%)	347.427 ms	1.89 GiB (1%)	15132653
`["VGG", "InputTimesGradient", "construct analyzer"]`	10.700 μs (5%)		27.41 KiB (1%)	205
`["VGG", "LRPCustom", "analyze"]`	12.841 s (5%)	332.839 ms	3.41 GiB (1%)	16530196
`["VGG", "LRPCustom", "construct analyzer"]`	89.301 μs (5%)		45.52 KiB (1%)	155
`["VGG", "LRPZero", "analyze"]`	1.463 s (5%)	15.903 ms	933.87 MiB (1%)	3908
`["VGG", "LRPZero", "construct analyzer"]`	19.300 μs (5%)		43.88 KiB (1%)	82

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

["Layer", "Conv"]
["Layer", "Dense"]
["Layer", "MaxPool"]
["Layer", "WrappedDense"]
["VGG", "Gradient"]
["VGG", "InputTimesGradient"]
["VGG", "LRPCustom"]
["VGG", "LRPZero"]

Julia versioninfo

Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      Ubuntu 20.04.4 LTS
  uname: Linux 5.11.0-1028-azure #31~20.04.2-Ubuntu SMP Tue Jan 18 08:46:15 UTC 2022 x86_64 x86_64
  CPU: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz: 
              speed         user         nice          sys         idle          irq
       #1  2294 MHz       7227 s          2 s        460 s       2976 s          0 s
       #2  2294 MHz       4607 s          1 s        524 s       5470 s          0 s
       
  Memory: 6.7845458984375 GB (1603.68359375 MB free)
  Uptime: 1074.4 sec
  Load Avg:  1.19  1.17  0.91
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, broadwell)

Runtime information

Runtime Info
BLAS #threads	2
`BLAS.vendor()`	`openblas64`
`Sys.CPU_THREADS`	2

lscpu output:

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           79
Model name:                      Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Stepping:                        1
CPU MHz:                         2294.685
BogoMIPS:                        4589.37
Hypervisor vendor:               Microsoft
Virtualization type:             full
L1d cache:                       64 KiB
L1i cache:                       64 KiB
L2 cache:                        512 KiB
L3 cache:                        50 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     KVM: Mitigation: VMX unsupported
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT Host state unknown
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt md_clear

Cpu Property	Value
Brand	Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Vendor	:Intel
Architecture	:Broadwell
Model	Family: 0x06, Model: 0x4f, Stepping: 0x01, Type: 0x00
Cores	2 physical cores, 2 logical cores (on executing CPU)
	No Hyperthreading hardware capability detected
Clock Frequencies	Not supported by CPU
Data Cache	Level 1:3 : (32, 256, 51200) kbytes
	64 byte cache line size
Address Size	48 bits virtual, 46 bits physical
SIMD	256 bit = 32 byte max. SIMD vector size
Time Stamp Counter	TSC is accessible via `rdtsc`
	TSC increased at every clock cycle (non-invariant TSC)
Perf. Monitoring	Performance Monitoring Counters (PMC) are not supported
Hypervisor	Yes, Microsoft

github-actions · 2022-03-29T15:15:22Z

Benchmark result

Judge result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Time of benchmarks:
- Target: 29 Mar 2022 - 15:11
- Baseline: 29 Mar 2022 - 15:15
Package commits:
- Target: 323678
- Baseline: 312c06
Julia commits:
- Target: bf5349
- Baseline: bf5349
Julia command flags:
- Target: None
- Baseline: None
Environment variables:
- Target: None
- Baseline: None

Results

A ratio greater than 1.0 denotes a possible regression (marked with ❌), while a ratio less
than 1.0 denotes a possible improvement (marked with ✅). Only significant results - results
that indicate possible regressions or improvements - are shown below (thus, an empty table means that all
benchmark results remained invariant between builds).

ID	time ratio	memory ratio
`["Layer", "Dense", "EpsilonRule"]`	2.06 (5%) ❌	1.01 (1%)
`["Layer", "Dense", "GammaRule"]`	1.22 (5%) ❌	1.00 (1%)
`["Layer", "Dense", "ZBoxRule"]`	1.65 (5%) ❌	1.00 (1%)
`["Layer", "Dense", "ZeroRule"]`	2.06 (5%) ❌	1.01 (1%)
`["Layer", "WrappedDense", "EpsilonRule"]`	2.36 (5%) ❌	1.00 (1%)
`["Layer", "WrappedDense", "GammaRule"]`	1.41 (5%) ❌	1.00 (1%)
`["Layer", "WrappedDense", "ZBoxRule"]`	1.73 (5%) ❌	1.00 (1%)
`["Layer", "WrappedDense", "ZeroRule"]`	1.92 (5%) ❌	1.00 (1%)
`["VGG", "Gradient", "construct analyzer"]`	0.86 (5%) ✅	1.00 (1%)
`["VGG", "InputTimesGradient", "analyze"]`	0.04 (5%) ✅	0.59 (1%) ✅
`["VGG", "InputTimesGradient", "construct analyzer"]`	1.22 (5%) ❌	1.00 (1%)
`["VGG", "LRPCustom", "analyze"]`	0.21 (5%) ✅	0.78 (1%) ✅
`["VGG", "LRPCustom", "construct analyzer"]`	0.28 (5%) ✅	1.00 (1%)

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

["Layer", "Conv"]
["Layer", "Dense"]
["Layer", "MaxPool"]
["Layer", "WrappedDense"]
["VGG", "Gradient"]
["VGG", "InputTimesGradient"]
["VGG", "LRPCustom"]
["VGG", "LRPZero"]

Julia versioninfo

Target

Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      Ubuntu 20.04.4 LTS
  uname: Linux 5.11.0-1028-azure #31~20.04.2-Ubuntu SMP Tue Jan 18 08:46:15 UTC 2022 x86_64 x86_64
  CPU: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz: 
              speed         user         nice          sys         idle          irq
       #1  2593 MHz       4620 s          2 s        336 s       3195 s          0 s
       #2  2593 MHz       2293 s          0 s        319 s       5554 s          0 s
       
  Memory: 6.7845458984375 GB (1741.5234375 MB free)
  Uptime: 820.61 sec
  Load Avg:  1.17  1.15  0.73
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake-avx512)

Baseline

Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      Ubuntu 20.04.4 LTS
  uname: Linux 5.11.0-1028-azure #31~20.04.2-Ubuntu SMP Tue Jan 18 08:46:15 UTC 2022 x86_64 x86_64
  CPU: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz: 
              speed         user         nice          sys         idle          irq
       #1  2593 MHz       6714 s          2 s        400 s       3261 s          0 s
       #2  2593 MHz       2546 s          0 s        426 s       7414 s          0 s
       
  Memory: 6.7845458984375 GB (1686.8125 MB free)
  Uptime: 1043.1 sec
  Load Avg:  1.23  1.17  0.84
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake-avx512)

Target result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Time of benchmark: 29 Mar 2022 - 15:11
Package commit: 323678
Julia commit: bf5349
Julia command flags: None
Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID	time	GC time	memory	allocations
`["Layer", "Conv", "EpsilonRule"]`	811.504 μs (5%)		1.54 MiB (1%)	361
`["Layer", "Conv", "GammaRule"]`	813.804 μs (5%)		1.54 MiB (1%)	373
`["Layer", "Conv", "ZBoxRule"]`	2.597 ms (5%)		4.77 MiB (1%)	1047
`["Layer", "Conv", "ZeroRule"]`	813.704 μs (5%)		1.54 MiB (1%)	361
`["Layer", "Dense", "EpsilonRule"]`	42.201 μs (5%)		3.03 KiB (1%)	8
`["Layer", "Dense", "GammaRule"]`	117.301 μs (5%)		590.56 KiB (1%)	17
`["Layer", "Dense", "ZBoxRule"]`	423.102 μs (5%)		1.01 MiB (1%)	416
`["Layer", "Dense", "ZeroRule"]`	42.200 μs (5%)		3.03 KiB (1%)	8
`["Layer", "MaxPool", "EpsilonRule"]`	82.800 μs (5%)		107.23 KiB (1%)	70
`["Layer", "MaxPool", "GammaRule"]`	82.600 μs (5%)		107.23 KiB (1%)	70
`["Layer", "MaxPool", "ZBoxRule"]`	82.501 μs (5%)		106.52 KiB (1%)	70
`["Layer", "MaxPool", "ZeroRule"]`	82.500 μs (5%)		106.52 KiB (1%)	70
`["Layer", "WrappedDense", "EpsilonRule"]`	110.600 μs (5%)		212.08 KiB (1%)	153
`["Layer", "WrappedDense", "GammaRule"]`	200.001 μs (5%)		799.61 KiB (1%)	162
`["Layer", "WrappedDense", "ZBoxRule"]`	427.802 μs (5%)		1.01 MiB (1%)	416
`["Layer", "WrappedDense", "ZeroRule"]`	94.900 μs (5%)		211.31 KiB (1%)	153
`["VGG", "Gradient", "analyze"]`	1.065 s (5%)	15.043 ms	1.10 GiB (1%)	3708
`["VGG", "Gradient", "construct analyzer"]`	16.501 μs (5%)		27.41 KiB (1%)	205
`["VGG", "InputTimesGradient", "analyze"]`	1.061 s (5%)	14.944 ms	1.10 GiB (1%)	3713
`["VGG", "InputTimesGradient", "construct analyzer"]`	14.301 μs (5%)		27.41 KiB (1%)	205
`["VGG", "LRPCustom", "analyze"]`	1.934 s (5%)	36.323 ms	2.65 GiB (1%)	4717
`["VGG", "LRPCustom", "construct analyzer"]`	19.400 μs (5%)		45.52 KiB (1%)	155
`["VGG", "LRPZero", "analyze"]`	1.121 s (5%)	12.563 ms	933.87 MiB (1%)	3908
`["VGG", "LRPZero", "construct analyzer"]`	13.400 μs (5%)		43.88 KiB (1%)	82

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

["Layer", "Conv"]
["Layer", "Dense"]
["Layer", "MaxPool"]
["Layer", "WrappedDense"]
["VGG", "Gradient"]
["VGG", "InputTimesGradient"]
["VGG", "LRPCustom"]
["VGG", "LRPZero"]

Julia versioninfo

Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      Ubuntu 20.04.4 LTS
  uname: Linux 5.11.0-1028-azure #31~20.04.2-Ubuntu SMP Tue Jan 18 08:46:15 UTC 2022 x86_64 x86_64
  CPU: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz: 
              speed         user         nice          sys         idle          irq
       #1  2593 MHz       4620 s          2 s        336 s       3195 s          0 s
       #2  2593 MHz       2293 s          0 s        319 s       5554 s          0 s
       
  Memory: 6.7845458984375 GB (1741.5234375 MB free)
  Uptime: 820.61 sec
  Load Avg:  1.17  1.15  0.73
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake-avx512)

Baseline result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Time of benchmark: 29 Mar 2022 - 15:15
Package commit: 312c06
Julia commit: bf5349
Julia command flags: None
Environment variables: None

Results

Below is a table of this job's results, obtained by running the benchmarks.
The values listed in the ID column have the structure [parent_group, child_group, ..., key], and can be used to
index into the BaseBenchmarks suite to retrieve the corresponding benchmarks.
The percentages accompanying time and memory values in the below table are noise tolerances. The "true"
time/memory value for a given benchmark is expected to fall within this percentage of the reported value.
An empty cell means that the value was zero.

ID	time	GC time	memory	allocations
`["Layer", "Conv", "EpsilonRule"]`	799.104 μs (5%)		1.54 MiB (1%)	361
`["Layer", "Conv", "GammaRule"]`	794.604 μs (5%)		1.54 MiB (1%)	373
`["Layer", "Conv", "ZBoxRule"]`	2.490 ms (5%)		4.77 MiB (1%)	1047
`["Layer", "Conv", "ZeroRule"]`	793.604 μs (5%)		1.54 MiB (1%)	361
`["Layer", "Dense", "EpsilonRule"]`	20.500 μs (5%)		3.02 KiB (1%)	8
`["Layer", "Dense", "GammaRule"]`	96.101 μs (5%)		590.55 KiB (1%)	17
`["Layer", "Dense", "ZBoxRule"]`	256.802 μs (5%)		1.01 MiB (1%)	413
`["Layer", "Dense", "ZeroRule"]`	20.500 μs (5%)		3.02 KiB (1%)	8
`["Layer", "MaxPool", "EpsilonRule"]`	86.201 μs (5%)		107.23 KiB (1%)	70
`["Layer", "MaxPool", "GammaRule"]`	85.700 μs (5%)		107.23 KiB (1%)	70
`["Layer", "MaxPool", "ZBoxRule"]`	85.901 μs (5%)		106.52 KiB (1%)	70
`["Layer", "MaxPool", "ZeroRule"]`	85.900 μs (5%)		106.52 KiB (1%)	70
`["Layer", "WrappedDense", "EpsilonRule"]`	46.800 μs (5%)		211.66 KiB (1%)	152
`["Layer", "WrappedDense", "GammaRule"]`	141.401 μs (5%)		799.19 KiB (1%)	161
`["Layer", "WrappedDense", "ZBoxRule"]`	246.801 μs (5%)		1.01 MiB (1%)	413
`["Layer", "WrappedDense", "ZeroRule"]`	49.500 μs (5%)		211.05 KiB (1%)	152
`["VGG", "Gradient", "analyze"]`	1.052 s (5%)	13.853 ms	1.10 GiB (1%)	3697
`["VGG", "Gradient", "construct analyzer"]`	19.100 μs (5%)		27.41 KiB (1%)	205
`["VGG", "InputTimesGradient", "analyze"]`	25.053 s (5%)	319.760 ms	1.89 GiB (1%)	15170800
`["VGG", "InputTimesGradient", "construct analyzer"]`	11.700 μs (5%)		27.41 KiB (1%)	205
`["VGG", "LRPCustom", "analyze"]`	9.290 s (5%)	462.112 ms	3.42 GiB (1%)	16537219
`["VGG", "LRPCustom", "construct analyzer"]`	69.000 μs (5%)		45.52 KiB (1%)	155
`["VGG", "LRPZero", "analyze"]`	1.122 s (5%)	12.655 ms	933.87 MiB (1%)	3908
`["VGG", "LRPZero", "construct analyzer"]`	12.800 μs (5%)		43.88 KiB (1%)	82

Benchmark Group List

Here's a list of all the benchmark groups executed by this job:

["Layer", "Conv"]
["Layer", "Dense"]
["Layer", "MaxPool"]
["Layer", "WrappedDense"]
["VGG", "Gradient"]
["VGG", "InputTimesGradient"]
["VGG", "LRPCustom"]
["VGG", "LRPZero"]

Julia versioninfo

Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      Ubuntu 20.04.4 LTS
  uname: Linux 5.11.0-1028-azure #31~20.04.2-Ubuntu SMP Tue Jan 18 08:46:15 UTC 2022 x86_64 x86_64
  CPU: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz: 
              speed         user         nice          sys         idle          irq
       #1  2593 MHz       6714 s          2 s        400 s       3261 s          0 s
       #2  2593 MHz       2546 s          0 s        426 s       7414 s          0 s
       
  Memory: 6.7845458984375 GB (1686.8125 MB free)
  Uptime: 1043.1 sec
  Load Avg:  1.23  1.17  0.84
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake-avx512)

Runtime information

Runtime Info
BLAS #threads	2
`BLAS.vendor()`	`openblas64`
`Sys.CPU_THREADS`	2

lscpu output:

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Stepping:                        7
CPU MHz:                         2593.907
BogoMIPS:                        5187.81
Hypervisor vendor:               Microsoft
Virtualization type:             full
L1d cache:                       64 KiB
L1i cache:                       64 KiB
L2 cache:                        2 MiB
L3 cache:                        35.8 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     KVM: Mitigation: VMX unsupported
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT Host state unknown
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear

Cpu Property	Value
Brand	Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Vendor	:Intel
Architecture	:Skylake
Model	Family: 0x06, Model: 0x55, Stepping: 0x07, Type: 0x00
Cores	2 physical cores, 2 logical cores (on executing CPU)
	No Hyperthreading hardware capability detected
Clock Frequencies	Not supported by CPU
Data Cache	Level 1:3 : (32, 1024, 36608) kbytes
	64 byte cache line size
Address Size	48 bits virtual, 46 bits physical
SIMD	512 bit = 64 byte max. SIMD vector size
Time Stamp Counter	TSC is accessible via `rdtsc`
	TSC increased at every clock cycle (non-invariant TSC)
Perf. Monitoring	Performance Monitoring Counters (PMC) are not supported
Hypervisor	Yes, Microsoft

adrhill added 7 commits March 28, 2022 16:48

Refactor and test neuron selection

e12bcae

Add test for batches

4803275

Fix gradient methods by adding gradients_wrt_batch

977bdbb

Add batch dimension to LRP einsum

32b70cb

Add batch dim to LRP rules tests

54b3591

Add heatmapping support for batches

6488ea6

Test heatmaps on batches

e1bd6b9

adrhill added the run benchmark Run benchmark CI action label Mar 28, 2022

adrhill added 4 commits March 29, 2022 00:12

Replace verbose comment with link to new Advanced LRP docs

f0a7fbb

Fix for single color channel

9922bff

Small updates to utils

586def2

Attempt to fix benchmarks

f8db22a

adrhill added 2 commits March 29, 2022 17:01

Fix typo in check_output_softmax

c01b520

Increase code coverage

087a545

adrhill merged commit c74f50d into master Mar 29, 2022

adrhill deleted the ah/batches branch March 29, 2022 15:33

adrhill mentioned this pull request Mar 29, 2022

Document support for batches #48

Merged

Add support for batches #46

Add support for batches #46

Conversation

adrhill commented Mar 28, 2022

codecov-commenter commented Mar 29, 2022 • edited Loading

Codecov Report

github-actions bot commented Mar 29, 2022

Judge result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Results

Benchmark Group List

Julia versioninfo

Target

Baseline

Target result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Results

Benchmark Group List

Julia versioninfo

Baseline result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Results

Benchmark Group List

Julia versioninfo

Runtime information

github-actions bot commented Mar 29, 2022

Judge result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Results

Benchmark Group List

Julia versioninfo

Target

Baseline

Target result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Results

Benchmark Group List

Julia versioninfo

Baseline result

Benchmark Report for /home/runner/work/ExplainableAI.jl/ExplainableAI.jl

Job Properties

Results

Benchmark Group List

Julia versioninfo

Runtime information

codecov-commenter commented Mar 29, 2022 •

edited

Loading