[Bug] pdb support is broken

[LouChao98]

🐛 Bug

Description

In patch #1239, debugger can work with hydra by setting HYDRA_FULL_ERROR=1 , but pr #1465 breaks it. In #1465, HydraJobException is always raised if the job raise an exception, such that debugger will catch HydraJobException instead of the original exceptions.

Checklist

• I checked on the latest version of Hydra
• I created a minimal repro (See this for tips).

To reproduce

** Minimal Code/Config snippet to reproduce **

# mini.py
import hydra

@hydra.main()
def foo(cfg):
    raise Exception

foo()

then run HYDRA_FULL_ERROR=1 python -m pdb -c continue mini.py in shell.

pdb catch HydraJobException in hydra/core/utils.py

Expected Behavior

pdb catch Exception in foo.

System information

• Hydra Version : 1.1.0dev6
• Python version : 3.8.8
• Virtual environment type and version :
• Operating system : linux 5.9.10

Additional context

I think add if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger(): in JobReturn.return_value can temporarily fix this.

@dataclass
class JobReturn:
    ....

    @property
    def return_value(self) -> Any:
        assert self.status != JobStatus.UNKNOWN, "return_value not yet available"
        if self.status == JobStatus.COMPLETED:
            return self._return_value
        else:
            if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger():  # add this
                raise self._return_value
            raise HydraJobException(
                f"Error executing job with overrides: {self.overrides}"
            ) from self._return_value

[omry]

In patch #1239, debugger can work with hydra by setting HYDRA_FULL_ERROR=1

Thanks for reporting.
#1239 enables pdb support without needing to set HYDRA_FULL_ERROR.
What happens if you do not set it?

[LouChao98]

Run python mini.py has no problem. But if run python -m pdb -c continue mini.py or HYDRA_FULL_ERROR=1 python -m pdb -c continue mini.py, pdb will always catch a HydraJobException instead of the Exception raised in mini.py.

This is the output of pdb

/home/louchao/code/struct_vat2/mini.py:1: UserWarning: 
config_path is not specified in @hydra.main().
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/changes_to_hydra_main_config_path for more information.
  import hydra
Traceback (most recent call last):
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/core/utils.py", line 137, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/louchao/code/struct_vat2/mini.py", line 6, in foo
    raise Exception
Exception

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/louchao/miniconda3/lib/python3.8/pdb.py", line 1705, in main
    pdb._runscript(mainpyfile)
  File "/home/louchao/miniconda3/lib/python3.8/pdb.py", line 1573, in _runscript
    self.run(statement)
  File "/home/louchao/miniconda3/lib/python3.8/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/louchao/code/struct_vat2/mini.py", line 1, in <module>
    import hydra
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/main.py", line 49, in decorated_main
    _run_hydra(
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 370, in _run_hydra
    run_and_report(
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 215, in run_and_report
    raise ex
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 212, in run_and_report
    return func()
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 371, in <lambda>
    lambda: hydra.run(
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 107, in run
    _ = ret.return_value
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/core/utils.py", line 212, in return_value
    raise HydraJobException(
hydra.errors.HydraJobException: Error executing job with overrides: []
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /home/louchao/miniconda3/lib/python3.8/site-packages/hydra/core/utils.py(212)return_value()
-> raise HydraJobException(
(Pdb) 

In #1239, _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger() is checked, if True, run_and_report will throw the original exception directly. This is OK.

In #1465, the following change makes exception stacktrace changed before #1239. Let's focus on line 110 of hydra/_internal/hydra.py.

hydra/hydra/_internal/hydra.py

Lines 105 to 112 in 3f74e8f

	configure_logging=with_log_configuration,
	)
	callbacks.on_run_end(config=cfg, config_name=config_name, job_return=ret)
	# access the result to trigger an exception in case the job failed.
	_ = ret.return_value
	return ret

which is calling

hydra/hydra/core/utils.py

Lines 226 to 234 in 3f74e8f

	@property
	def return_value(self) -> Any:
	assert self.status != JobStatus.UNKNOWN, "return_value not yet available"
	if self.status == JobStatus.COMPLETED:
	return self._return_value
	else:
	raise HydraJobException(
	f"Error executing job with overrides: {self.overrides}"
	) from self._return_value

Here Hydra raise HydraJobException instead of the original exception even if under a debugger.

[omry]

Thanks. #1614 is supposed to fix it.