Gamma family function support #321

xukai92 · 2019-04-17T17:06:04Z

Porting codes from CuGammaFuns.jll; see xukai92/CuGammaFuns.jl#1

Currently I

add a new folder called special and put gamma.jl inside
defines the new diff. rules in forwarddiff.jl
link implemented gamma functions in broadcast.

Open to make any changes!

xukai92 · 2019-04-17T17:08:09Z

@MikeInnes I need some help from you here. I cannot run tests on my local and saw the error message below. What did I do wrong?

WARNING: could not import NNlib.spatial_dims into CUDNN
ERROR: LoadError: LoadError: LoadError: UndefVarError: DenseConvDims not defined
Stacktrace:
 [1] top-level scope at none:0
 [2] include at ./boot.jl:326 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1038
 [4] include at ./sysimg.jl:29 [inlined]
 [5] include(::String) at /home/kai/projects/CuArrays.jl/src/dnn/CUDNN.jl:1
 [6] top-level scope at none:0
 [7] include at ./boot.jl:326 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:1038
 [9] include at ./sysimg.jl:29 [inlined]
 [10] include(::String) at /home/kai/projects/CuArrays.jl/src/CuArrays.jl:3
 [11] top-level scope at none:0
 [12] include at ./boot.jl:326 [inlined]
 [13] include_relative(::Module, ::String) at ./loading.jl:1038
 [14] include(::Module, ::String) at ./sysimg.jl:29
 [15] top-level scope at none:2
 [16] eval at ./boot.jl:328 [inlined]
 [17] eval(::Expr) at ./client.jl:404
 [18] top-level scope at ./none:3
in expression starting at /home/kai/projects/CuArrays.jl/src/dnn/libcudnn.jl:265
in expression starting at /home/kai/projects/CuArrays.jl/src/dnn/CUDNN.jl:35
in expression starting at /home/kai/projects/CuArrays.jl/src/CuArrays.jl:53
ERROR: LoadError: Failed to precompile CuArrays [3a865a2d-5b23-5a0f-bc46-62713ec82fae] to /home/kai/.julia/compiled/v1.1/CuArrays/7YFE0.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1197
 [3] _require(::Base.PkgId) at ./loading.jl:960
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853
 [6] include at ./boot.jl:326 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1038
 [8] include(::Module, ::String) at ./sysimg.jl:29
 [9] exec_options(::Base.JLOptions) at ./client.jl:267
 [10] _start() at ./client.jl:436
in expression starting at /home/kai/projects/CuArrays.jl/test/special.jl:4

My package status:

    Status `~/.julia/environments/v1.1/Project.toml`
  [3ad36a84] AdvancedHMC v0.1.0 [`~/.julia/dev/AdvancedHMC`]
  [c52e3926] Atom v0.8.2
  [6710c13c] AutoGrad v1.1.3
  [3895d2a7] CUDAapi v0.6.4 #master (https://github.com/JuliaGPU/CUDAapi.jl.git)
  [c5f51814] CUDAdrv v3.0.0 #master (https://github.com/JuliaGPU/CUDAdrv.jl.git)
  [be33ccc6] CUDAnative v2.1.0 #master (https://github.com/JuliaGPU/CUDAnative.jl.git)
  [d29381fa] Catsay v0.0.0 #master ([email protected]:xukai92/Catsay.jl.git)
  [944b1d66] CodecZlib v0.5.2
  [3a865a2d] CuArrays v1.0.2 [`~/projects/CuArrays.jl`]
  [31c24e10] Distributions v0.17.0
  [bbc10e6e] DynamicHMC v1.0.4
  [587475ba] Flux v0.8.1
  [f6369f11] ForwardDiff v0.10.3
  [0c68f7d7] GPUArrays v0.7.0 #master (https://github.com/JuliaGPU/GPUArrays.jl.git)
  [e5e0dc1b] Juno v0.7.0
  [1902f260] Knet v1.2.1+ #master (https://github.com/denizyuret/Knet.jl.git)
  [b964fa9f] LaTeXStrings v1.0.3
  [519e820e] MLToolkit v0.1.0 [`~/projects/MLToolkit.jl`]
  [d330b81b] PyPlot v2.8.0
  [276daf66] SpecialFunctions v0.7.2
  [8188c328] Stheno v0.1.1
  [fce5fe82] Turing v0.6.15+ [`~/.julia/dev/Turing`]

MikeInnes · 2019-04-25T11:27:22Z

You may just need to add NNlib master, and make sure CuArrays is up to date with master as well.

xukai92 · 2019-07-09T01:29:03Z

@MikeInnes This works on my local now.

maleadt · 2019-07-09T09:01:09Z

Thanks! Should be fine for now, but we'll probably be able to just reuse the upstream SpecialFunctions.jl definitions once we have JuliaGPU/CUDAnative.jl#334

bors try

bors · 2019-07-09T09:20:07Z

try

Build failed

ci/gitlab/trying

xukai92 · 2019-07-09T10:48:39Z

Looking forward to JuliaGPU/CUDAnative.jl#334

try

Build failed

ci/gitlab/trying

It seems that some tests not related to this PR fails.

xukai92 · 2019-07-26T21:55:47Z

@maleadt Any idea what I should do for this PR?

MikeInnes · 2019-11-04T14:31:00Z

bors try

bors · 2019-11-04T15:02:54Z

try

Build failed

ci/gitlab/trying

MikeInnes · 2019-11-04T15:15:51Z

Looks like you need to update the manifest to include SpecialFunctions. You should see a diff in git and can just push that.

torfjelde · 2021-01-03T21:32:54Z

Bump:)
What is needed for this to make it into CUDA.jl? Is it just a matter of moving the code to that package or is there anything else required?

maleadt · 2021-01-04T08:14:02Z

The next version of CUDA.jl should have some form of method substitution capabilities, so I'd rather wait and see if these reimplementations are still required then.

torfjelde · 2021-01-04T10:49:08Z

The next version of CUDA.jl should have some form of method substitution capabilities, so I'd rather wait and see if these reimplementations are still required then.

Are you referring to JuliaGPU/CUDAnative.jl#334, which doesn't seem quite "active"? Or is there somewhere else I can follow the progress?

If there's not a PR that is essentially a "done deal", adding this and just bumping the minor version would be super-nice (and I'd be happy to help!). Support for SpecialFunctions.jl is really a big bottleneck to getting Bayesian inference + GPU, and for someone not familiar with CUDA.jl it can be really confusing figuring out what's going wrong (I started out thinking I needed to define custom adjoints that worked on the GPU for all these methods). Of course at this point I know how to fix it, but at it took me reading up on Julia's broadcasting mechanism and familiarizing myself quite a bit with CUDA.jl before getting here. On the bright side I now understand Julia's broadcasting mechanism better 🙃

torfjelde · 2021-01-04T10:59:30Z

src/special/gamma.jl

+  ψ += CUDAnative.log(x) - 0.5 * t
+  t *= t # 1/z^2
+  # the coefficients here are Float64(bernoulli[2:9] .// (2*(1:8)))
+  ψ -= t * @evalpoly(t,0.08333333333333333,-0.008333333333333333,0.003968253968253968,-0.004166666666666667,0.007575757575757576,-0.021092796092796094,0.08333333333333333,-0.4432598039215686)


Btw, this will convert into Float64; maybe it ought to be dependent on the input or use Float32 by default?

maleadt · 2021-01-04T12:45:16Z

src/CuArrays.jl

@@ -31,6 +31,7 @@ include("array.jl")
 include("subarray.jl")
 include("utils.jl")
 include("indexing.jl")
+include("special/gamma.jl")


Just make it special.jl, no need for directories with single source files.

maleadt · 2021-01-04T12:46:16Z

Are you referring to JuliaGPU/CUDAnative.jl#334, which doesn't seem quite "active"? Or is there somewhere else I can follow the progress?

This is now happening in JuliaGPU/GPUCompiler.jl#122.

If there's not a PR that is essentially a "done deal", adding this and just bumping the minor version would be super-nice (and I'd be happy to help!). Support for SpecialFunctions.jl is really a big bottleneck to getting Bayesian inference + GPU, and for someone not familiar with CUDA.jl it can be really confusing figuring out what's going wrong (I started out thinking I needed to define custom adjoints that worked on the GPU for all these methods). Of course at this point I know how to fix it, but at it took me reading up on Julia's broadcasting mechanism and familiarizing myself quite a bit with CUDA.jl before getting here. On the bright side I now understand Julia's broadcasting mechanism better upside_down_face

Sure, but I don't plan to create a CuArrays.jl release. If you want to port this to CUDA.jl, I'm happy to create a minor release containing it though.

xukai92 · 2021-01-04T14:15:43Z

Thanks for bumping this up @torfjelde.
It would be great if you can port this PR to CUDA.jl - let me know if you need any help.

torfjelde · 2021-01-06T14:30:24Z

So just an update: these methods are just a small piece of the pie that we want, so I'm instead trying to improve the CUDA.@cufunc macro to do more than what it does now, e.g. extract diffrules from DiffRules.diffrule and define for corresponding cufunc.

port codes from CuGammaFuns.jl

034a52c

xukai92 changed the title ~~Gamma family function Support~~ Gamma family function support Apr 17, 2019

xukai92 added 2 commits July 9, 2019 02:09

merge with master

87733ac

bugfix

61c66f4

Merge branch 'master' into kx/cugammafuns

3fc544e

maleadt added the enhancement label Jul 9, 2019

bors bot added a commit that referenced this pull request Jul 9, 2019

Try #321:

d92300d

bors bot added a commit that referenced this pull request Nov 4, 2019

Try #321:

073c836

maleadt force-pushed the master branch from 2e25b28 to 16ebad8 Compare December 20, 2019 16:03

maleadt force-pushed the master branch 2 times, most recently from fc487fd to fced436 Compare January 22, 2020 15:17

maleadt force-pushed the master branch from 2c08630 to 6ca9b4f Compare February 7, 2020 11:23

maleadt force-pushed the master branch from 8259597 to 1fe1883 Compare March 10, 2020 11:42

torfjelde reviewed Jan 4, 2021

View reviewed changes

maleadt reviewed Jan 4, 2021

View reviewed changes

torfjelde mentioned this pull request Jan 5, 2021

To the GPU we go cambridge-mlg/Covid19#5

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gamma family function support #321

Gamma family function support #321

xukai92 commented Apr 17, 2019 •

edited

Loading

xukai92 commented Apr 17, 2019

MikeInnes commented Apr 25, 2019

xukai92 commented Jul 9, 2019

maleadt commented Jul 9, 2019

bors bot commented Jul 9, 2019

xukai92 commented Jul 9, 2019

try

Build failed

xukai92 commented Jul 26, 2019

MikeInnes commented Nov 4, 2019

bors bot commented Nov 4, 2019

MikeInnes commented Nov 4, 2019

torfjelde commented Jan 3, 2021

maleadt commented Jan 4, 2021

torfjelde commented Jan 4, 2021

torfjelde Jan 4, 2021

maleadt Jan 4, 2021

maleadt commented Jan 4, 2021

xukai92 commented Jan 4, 2021

torfjelde commented Jan 6, 2021

Gamma family function support #321

Are you sure you want to change the base?

Gamma family function support #321

Conversation

xukai92 commented Apr 17, 2019 • edited Loading

xukai92 commented Apr 17, 2019

MikeInnes commented Apr 25, 2019

xukai92 commented Jul 9, 2019

maleadt commented Jul 9, 2019

bors bot commented Jul 9, 2019

try

Build failed

xukai92 commented Jul 9, 2019

try

Build failed

xukai92 commented Jul 26, 2019

MikeInnes commented Nov 4, 2019

bors bot commented Nov 4, 2019

try

Build failed

MikeInnes commented Nov 4, 2019

torfjelde commented Jan 3, 2021

maleadt commented Jan 4, 2021

torfjelde commented Jan 4, 2021

torfjelde Jan 4, 2021

Choose a reason for hiding this comment

maleadt Jan 4, 2021

Choose a reason for hiding this comment

maleadt commented Jan 4, 2021

xukai92 commented Jan 4, 2021

torfjelde commented Jan 6, 2021

xukai92 commented Apr 17, 2019 •

edited

Loading