A fine debugger

[oxinabox]

This combines #14 and #19
I've just done the merge on this branch rather than master as neither felt complete without the other.

When this is complete one will have a debugger that works with line-numbers.

[oxinabox]

If anyone would care to give this a quick review, that would be appreciated.

It is basically working,
I am just chasing down some errors that seem to occur only on travis
(https://travis-ci.com/oxinabox/MagneticReadHead.jl/jobs/185076830#L263)
and I probably want to write some more "interactive" tests, where I drive the debugger with preprogrammed commands.

It would be good to merge this, rather than continuing to add features to thiis PR.

[timholy]

Very intriguing, @oxinabox! I can't pretend I understand this yet, but it looks like an interesting approach. Doing this kind of thing was the first thing I thought of when I first heard about Cassette, but I still haven't taken the plunge yet.

How's the run-time performance of the instrumented code? Over at JuliaInterpreter we're a lot slower than compiled code; I'm sure this is slower too, but it would be interesting to compare the magnitudes. My naive guess is this might be slower to compile but faster to run? (We have also thought about creating "compiled Frames," but that's in the future.)

We're gearing up for a release announcement of our debugger stack. How close is this? Would you be interested in making a discourse announcement at around the same time? Or is this still too much of a research project?

[timholy]

I'd be happy to work with you on this. Do you have a reproducible trigger?

[oxinabox]

Not yet, I added that when I was seeing Nothing related errors. I think on CI.
I'll let you know when I do.

[oxinabox]

it may have been a Standard Library.
As of the commit i just made we now attempt to do Revise.track(mod)
if pkgfiles comes up with nothing

[timholy]

Can you use Base.loaded_modules?

[oxinabox]

I'll look into it

[oxinabox]

Problem is that includes standard libraries, which are not tracked by default.
And that is probably a wise choice by Revise?

[oxinabox]

How's the run-time performance of the instrumented code?

I've only just started some basic timings (using Optimum solving Rossenbock).
And I need to setup a better timing system since I am just using time_ns() right now. And am running twice.

Not great, I can get maybe down to 2 orders of magnitude of the original code, if I set_nodebug!() on all modules, barring the one I am interested in debugging.
If I don't knock out things with set_nodebug then I think it is like 4+ orders if magnitude, because it instruments everything.
But that seems too long, the runtime cost should only be 1 order of magnitude.

I have a feeling I am breaking compilation caching, since it is no faster the second time I run anything.

Also that I should implement my own cache for the instrumentation step.

[oxinabox]

We're gearing up for a release announcement of our debugger stack. How close is this? Would you be interested in making a discourse announcement at around the same time? Or is this still too much of a research project?

When this PR is merged, I think it will be time for an announcement.
And all that is really hard blocking this from being merged is fixing the tests for filename+line number breakpoints.
It will still be beta, but i will want user feedback to help discover issues.

There are some other thing I want to do that I don't want to put in this PR, like the ability to go back-up the callstack and read variables. Which shouldn't be too hard.
But those can come later.

Long term, I hope this can be complentairy to your work on Debugger.jl.
I like the notion that one should be able to switch between "compiled-mode" and "interpreted-mode".
Also I would like to be sharing some of the UI stuff, probably before too long,as I don't fancy writing a text-ui thing that can show position in callstack on one side, and position in file on the other.
And the interface to Juno of course should be the same.

I really need to catch up on the state of Debugger.jl

[timholy]

I really need to catch up on the state of Debugger.jl

The real guts are in JuliaInterpreter.jl; as of this morning there are some fairly decent docs, though a bit forward-looking since none of Debugger/Rebugger/Juno have released & tagged versions that support it (but Debugger.jl has it on master, and Rebugger on timholy/Rebugger.jl#61).

Debugger.jl can be thought of as "just a REPL mode," i.e., the UI part. Rebugger.jl has (on that branch) a meta-i key binding that's "just a different REPL mode" to interface with JuliaInterpreter. These two REPL-modes have different philosophies: Debugger is more like gdb, Rebugger more like an IDE. Juno will be a "real IDE" (see JunoLab/Juno.jl#257).

[timholy]

One option would be for you to implement the debug_command API of JuliaInterpreter & then maybe you could use Debugger.jl for your UI?

[oxinabox]

I have a feeling I am breaking compilation caching, since it is no faster the second time I run anything.

Further timing actually shows I am

Normally calling:

1.318972 seconds (4.85 M allocations: 244.061 MiB, 9.40% gc time)
0.000079 seconds (535 allocations: 21.297 KiB)

With using @iron_debug but with basically maximum set_nodebug on, so very little should be getting instrumented:

5.116838 seconds (17.96 M allocations: 921.936 MiB, 5.17% gc time)
0.016707 seconds (39.39 k allocations: 2.019 MiB)

So it is indeed faster after compiling.

Also that I should implement my own cache for the instrumentation step.

I tried this. It didn't work.

[oxinabox]

Ok, that last cruicial feature is in. Thanks to @timholy's fixes for random files outside of packages in CodeTracking 0.4.0.
Which means I can make a Beta testing announcement when you make an announcement for Debugger.jl