How python debugging works

This page talks about how the python extension debugs a local python script from the point of view of an extension developer. It's being discussed here because it's really the basis for the rest of debugging in the Jupyter extension.

If you want to know how to use the python debugger, go here

Pieces involved

(Borrowed from https://code.visualstudio.com/api/extension-guides/debugger-extension)

In order to hook into that workflow, the Python extension calls registerDebugAdapterDescriptorFactory with a type of python. (See here for a description of the registerDebugAdapterDescriptorFactory API)

This sets up VS code to call the Python extension whenever a debug launch of type python occurs.

This would look like so in a launch.json:

    {
      "name": "Python: Current File",
      "type": "python", // Notice 'python' here. It corresponds to the registered factory
      "request": "launch",
      "program": "${file}",
      "console": "integratedTerminal"
    },

ONn launch, VS code will use the launch.json entry descriptors and call into the DebugAdapterDescriptorFactory::createDebugAdapterDescriptor for the registered type.

The python extension would then return a structure describing an executable to run. That executable is a 'debug' server that is expected to communicate with VS code over stdio. That server handles 'debugging' the python file.

The structure describing the server would look something like so:

{
   executable: "c:\\users\\rich\\miniconda\\envs\\myenv\\scripts\\python.exe"
   args: [
      "'c:\\Users\\rich\\.vscode\\extensions\\ms-python.python-2022.10.1\\pythonFiles\\lib\\python\\debugpy\\launcher'",
      "57624",
      "--",
      "foo.py"
   ]
}

The 'debugpy' launcher creates a server that will sit and listen on stdin for messages.

DAP

What sort of messages does it listen to?

It uses the Debug Adapter Protocol (or DAP for short).

The DAP has messages for stuff like:

• Launching a new process
• Setting breakpoints
• Querying for variables, stack frames, and threads
• Send step into/step out/step over commands
• Responding with break states from breakpoints or stepping or exceptions

Messages on startup

What happens when a breakpoint hits

Looking at messages yourself

Debugpy supports logging all of the DAP messages by setting a few environment variables:

• PYDEVD_DEBUG=1
• DEBUGPY_LOG_DIR=
• PYDEVD_DEBUG_FILE=

There's a batch script to set these before starting VS code here.

This generates logs with data like so:

D+00000.094: Client[1] --> {
                 "seq": 1,
                 "type": "request",
                 "command": "initialize",
                 "arguments": {
                     "clientID": "vscode",
                     "clientName": "Visual Studio Code",
                     "adapterID": "python",
                     "pathFormat": "path",
                     "linesStartAt1": true,
                     "columnsStartAt1": true,
                     "supportsVariableType": true,
                     "supportsVariablePaging": true,
                     "supportsRunInTerminalRequest": true,
                     "locale": "en-us",
                     "supportsProgressReporting": true,
                     "supportsInvalidatedEvent": true,
                     "supportsMemoryReferences": true
                 }
             }

D+00000.094: /handling #1 request "initialize" from Client[1]/
             Capabilities: {
                 "supportsVariableType": true,
                 "supportsVariablePaging": true,
                 "supportsRunInTerminalRequest": true,
                 "supportsMemoryReferences": true
             }

D+00000.094: /handling #1 request "initialize" from Client[1]/
             Expectations: {
                 "locale": "en-us",
                 "linesStartAt1": true,
                 "columnsStartAt1": true,
                 "pathFormat": "path"
             }

D+00000.094: /handling #1 request "initialize" from Client[1]/
             Client[1] <-- {
                 "seq": 3,
                 "type": "response",
                 "request_seq": 1,
                 "success": true,
                 "command": "initialize",
                 "body": {
                     "supportsCompletionsRequest": true,
                     "supportsConditionalBreakpoints": true,
                     "supportsConfigurationDoneRequest": true,
                     "supportsDebuggerProperties": true,
                     "supportsDelayedStackTraceLoading": true,
                     "supportsEvaluateForHovers": true,
                     "supportsExceptionInfoRequest": true,
                     "supportsExceptionOptions": true,
                     "supportsFunctionBreakpoints": true,
                     "supportsHitConditionalBreakpoints": true,
                     "supportsLogPoints": true,
                     "supportsModulesRequest": true,
                     "supportsSetExpression": true,
                     "supportsSetVariable": true,
                     "supportsValueFormattingOptions": true,
                     "supportsTerminateDebuggee": true,
                     "supportsGotoTargetsRequest": true,
                     "supportsClipboardContext": true,
                     "exceptionBreakpointFilters": [
                         {
                             "filter": "raised",
                             "label": "Raised Exceptions",
                             "default": false,
                             "description": "Break whenever any exception is raised."
                         },
                         {
                             "filter": "uncaught",
                             "label": "Uncaught Exceptions",
                             "default": true,
                             "description": "Break when the process is exiting due to unhandled exception."
                         },
                         {
                             "filter": "userUnhandled",
                             "label": "User Uncaught Exceptions",
                             "default": false,
                             "description": "Break when exception escapes into library code."
                         }
                     ],
                     "supportsStepInTargetsRequest": true
                 }
             }

This is the debugpy.adapter log and shows the DAP messages and their responses.