[Bug] %run magic with jupyter notebook

[noklam]

🐛 Bug

Expect to run hydra in a notebook cell and get realtime output like in Terminal.

Two ways of running a python file in Jupyter
!python (calling the terminal directly) -> but cannot get real time output
%run (Jupyter way of running python file, but does not seems compatible with Hydra directly)

I am expecting this can run %run runner.py exp_name=abc

To reproduce

hydra_tqdm_notebook.py

import hydra
@hydra.main(config_path="configs/config.yaml")
def experiment_runner(cfg) -> None:
    from tqdm import tqdm_notebook as tqdm


    for i in tqdm(range(5)):
        import time
        time.sleep(0.02)
        print('helloworld')

if __name__ == "__main__":
    experiment_runner()

** Stack trace/error message **

Expected Behavior

The only difference between test_hydra_tqdm_notebook.py and test_tqdm_notebook.py is I wrapped it inside the Hydra runner. I expect the output to be the same.

System information

• Hydra Version :
• Python version :
• Virtual environment type and version :
• Operating system :

Additional context

test_tqdm_notebook.py

from tqdm import tqdm_notebook as tqdm
for i in tqdm(range(100)):
    import time
    time.sleep(0.02)
    print('helloworld')

[omry]

Please edit your bug report to a minimal example to reproduce the problem, will full source code of the runner code and the cell itself.

Also, explain what is it you are trying to do.
I have never used %run.

[noklam]

jupyter/notebook#4048 (This issue shows why I want to use %run because I need to get realtime terminal output)

For the first image (expected output), I want the progress bar can be print realtime in notebook. (This example without using hydra yet)

The problem is when I try to use %run, it does not play well with hydra.

with tqdm/tqdm_notebook, it does not matter much, the point is, I need to use %run to get the correct output from terminal. If I use !python, the output does not return until the entire program is finished

[omry]

I will take a look at this, but this is really not a supported mode now.
See the Compose API page which is the supported Hydra API for notebooks.

[omry]

Your example is a bit convoluted. can you try to minimize it a bit? I am not sure I understand the role of each of the 3 files.

[noklam]

(I clean up the issue a bit, let's focus only for tqdm_notebook)
To give u some more information. I try to comment out the part that is checking is it calling from notebook. It works when I run for the first time, if I then run again it will throw errors.

The first run is the expected output.

[omry]

Thanks for updating your example and for providing additional context.
It is not likely that I will support %run, but I will take a look when I have some time.

Take a look at the compose API I linked above. with some slight refactoring you can use it to compose the config in the Notebook instead of relying on @hydra.main() in the notebook.

[noklam]

Sure, I will try using the notebook API instead, thanks!

I like to use a notebook even just for running terminal commands because it cache output and I can log some high-level information for my experiment naturally inside just one notebook. If I need to use the notebook API, all logical code with stay with the config, and I may need 1 notebook for 1 experiment

What I would like to do is something like

Experiment 1

%run runner exp_name=1
output: accuracy=0.9

Experiment 2

%run runner exp_name=2
accuracy=0.92

[omry]

I understand, but Hydra was never tested using %run, and clearly it does not work.
Hydra is detecting that it's running from within a notebook and behaves differently, however @hydra.main() is designed to run in a regular python process.
there is a lot of magic going on to allow Hydra to auto detect where the config is.

If you really want to run Hydra as a process, use the Python subprocess module.
This not do what you want though. I really think the best thing for you to do is to just use the compose API.

train.py:

def train(cfg):
  ...

@hydra.main(config_path="config.yaml")
def my_app(_: DictConfig) -> None:
  train(cfg)

if __name__ == "__main__":
    my_app()

notebook:

import hydra.experimental
from train import train

hydra.experimental.initialize()
cfg = hydra.experimental.compose("config.yaml", overrides=["model=resnet50", "lr=0.1"])

train(cfg) # call your train function with the composed cfg

[Khoa-NT]

Hi,
It also happened in The Jupyter Lab Terminal:

[omry]

Hi,
It also happened in The Jupyter Lab Terminal:

I have a fix coming for this issue (%run).
If this does not solve the other problem file a different issue and explain how to reproduce. I have never used Jupyter Lab Terminal.