GPU usage and console messages

[JLDC]

Hello,

when using GPU arrays in Pluto (e.g. such as for training Flux models). The console output gets spammed by messages which I do not really understand. These messages did not appear in previous versions (I cannot say with which version they started appearing however). These messages are not printed when running the code in the REPL.

I have attached my Pluto notebook as a .txt file for reproduction as well as a screenshot of the console output (these messages are repeated multiple times (as you can see from the scrolling bar size on the right).

Thanks for your time!

mnist-test.txt

[fonsp]

@c42f Is this because of our logger level?

[c42f]

This doesn't look familiar to me. Perhaps the GPU package is writing to stdout?

[fonsp]

Right, this isn't a log 😅

@JLDC Thank you for bringing this up, but we need more information to be able to fix it. For example, you could run an old version of pluto (instructions) to narrow down the problem.

We changed some of the logging behaviour in Pluto 0.12.0, so you could test 0.11.14.

(It could also be caused by an update in another package.)

[JLDC]

Don't mention it, I'm glad to help this amazing package! 👍

I don't think the package (Flux) prints to stdout, since the messages do not appear when running the code in the REPL but perhaps I do not understand this fully.

Running the same code with Pluto 0.11.14 does not produce the messages. The only difference between what I did above and what I did here is that I changed the version 0.11.14 of Pluto using the second method on the link you provided (package manager).

So, in short, it works as it should with 0.11.14. I have now also tried it with 0.12.12 and it has the same output as 0.12.10 (my first post was on 0.12.10).

0.11.14

0.12.12

[fonsp]

Can you use Pkg.status() check whether any other packages changed version? Changing Pluto's version could still change other packages. Can you share the notebook?

[fonsp]

Can you run this version of Pluto and check if the problem is still there?

https://github.com/fonsp/Pluto.jl/tree/debug-685-1

[JLDC]

I just tried the debug-685-1 branch and the output messages from the GPU disappear 👍. Great!

Let me know if you need any other infos.

[fonsp]

It looks like CUDA.jl checks whether a Logger shows debug messages:

https://github.com/JuliaGPU/CUDA.jl/blob/c99ae6ffdd8ec6c668c98ee4053836bc4ebb1f5c/src/debug.jl#L11-L12

And if so, they enable verbose logging:

https://github.com/JuliaGPU/CUDA.jl/blob/a14344b609358df38f5af511e8c25102ca8d27e6/lib/cublas/CUBLAS.jl#L150-L156

Running this check inside current Pluto:

and the REPL:

@c42f Do you understand this first code snippet? Is there a way for us to still enable debug-level logging inside the notebook without triggering that check?

[fonsp]

I have created a PR that disables debug logging which fixes this issue, at the cost of losing debug log messages inside the notebook:

#723

Let's do this unless there is a more creative solution that still enables debug logging inside the notebook?

[c42f]

It looks like CUDA.jl checks whether a Logger shows debug messages

Yeah, seems like quite a hack — nobody is really supposed to call those CoreLogging functions... But it may be that the cublas logging API isn't really compatible with ours, leading to such hacks.

[fonsp]

@c42f Thanks for taking a look! What do you suggest I do next?

I have been working on the logging GUI recently and I think it will be really cool! @debug not working there might be a greater loss than verbose logging by CUDA.jl ?

[c42f]

Yeah, I think it's CUDA.jl which should be tweaked here — if there's a desire to enable debug logging directly from CUDA (which is incompatible with the Julia logging API), it should probably be controlled by a separate switch like an environment variable. Not by looking at the julia logger state.

[fonsp]

Okay! @JLDC feel free to continue the discussion on the CUDA.jl side