Hydra does not work with pdb

[ethanabrooks]

🐛 Bug

Description

When running pdb as in

python -m pdb myscript.py

the debugger does not allow me to inspect local variables upon exception.

To reproduce

# myscript.py
import hydra

@hydra.main()
def app(cfg) -> None:
    x = 0
    assert False

if __name__ == "__main__":
    app()

Run:

❯ python -m pdb -c continue myscript.py
Traceback (most recent call last):
  File "/Users/ethanbrooks/ppo/myscript.py", line 8, in app
    assert False
AssertionError

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
The program exited via sys.exit(). Exit status: 1
> /Users/ethanbrooks/ppo/myscript.py(2)<module>()
-> import hydra
(Pdb)

At the prompt, type p x

(Pdb) p x
*** NameError: name 'x' is not defined

Expected Behavior

pdb should work as it does when not using hydra:

def app() -> None:
    x = 0
    assert False

if __name__ == "__main__":
    app()

❯ python -m pdb -c continue myscript.py
Traceback (most recent call last):
  File "/Users/ethanbrooks/miniconda3/envs/ppo/lib/python3.8/pdb.py", line 1704, in main
    pdb._runscript(mainpyfile)
  File "/Users/ethanbrooks/miniconda3/envs/ppo/lib/python3.8/pdb.py", line 1573, in _runscript
    self.run(statement)
  File "/Users/ethanbrooks/miniconda3/envs/ppo/lib/python3.8/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/Users/ethanbrooks/ppo/myscript.py", line 1, in <module>
    def app() -> None:
  File "/Users/ethanbrooks/ppo/myscript.py", line 3, in app
    assert False
AssertionError
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /Users/ethanbrooks/ppo/myscript.py(3)app()
-> assert False
(Pdb) p x
0

System information

• Hydra Version : 1.0.4
• Python version : 3.8
• Virtual environment type and version : conda 4.9.2
• Operating system : BigSur 11.1

[omry]

Thanks for reporting.

This works:

HYDRA_FULL_ERROR=1 python -m pdb -c continue 1.py

Hydra is manipulating the exception to provide a more useful stack trace. Run the above without pdb to see the difference from

$ python 1.py 
Traceback (most recent call last):
  File "1.py", line 7, in app
    assert False
AssertionError

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Versus:

$ HYDRA_FULL_ERROR=1 python 1.py 
Traceback (most recent call last):
  File "1.py", line 10, in <module>
    app()
  File "/home/omry/miniconda3/envs/test/lib/python3.8/site-packages/hydra/main.py", line 32, in decorated_main
    _run_hydra(
  File "/home/omry/miniconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra
    run_and_report(
  File "/home/omry/miniconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 201, in run_and_report
    raise ex
  File "/home/omry/miniconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/home/omry/miniconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 347, in <lambda>
    lambda: hydra.run(
  File "/home/omry/miniconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 107, in run
    return run_job(
  File "/home/omry/miniconda3/envs/test/lib/python3.8/site-packages/hydra/core/utils.py", line 125, in run_job
    ret.return_value = task_function(task_cfg)
  File "1.py", line 7, in app
    assert False
AssertionError

If there is a way to detect that we are running under pdb this can be fixed by not manipulating the exception in that case.
Would you like to work on such a solution (or on any alternative you have in mind)?

[omry]

Alright, #1239 is fixing it and will be released with 1.1.
In the meantime, set HYDRA_FULL_ERROR=1 when you run under pdb.

once #1239 is merged you are very welcome to try it out from master and report any issues.

[ethanabrooks]

Beautiful! Thank you.

[LouChao98]

This pr #1465 breaks the patch #1239. In #1465, HydraJobException is always raised if the job raise an exception.
I think test _is_env_set("HYDRA_FULL_ERROR") in JobReturn.return_value can fix this.

@dataclass
class JobReturn:
    ....

    @property
    def return_value(self) -> Any:
        assert self.status != JobStatus.UNKNOWN, "return_value not yet available"
        if self.status == JobStatus.COMPLETED:
            return self._return_value
        else:
            if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger():  # add this
                raise self._return_value
            raise HydraJobException(
                f"Error executing job with overrides: {self.overrides}"
            ) from self._return_value

[omry]

Thanks for reporting.
Please file a new issue with a minimal repro.

[LouChao98]

Goto #1613