GPL tutorial - add GPU header and open in colab button

[vblagoje]

Related Issue(s): #2632, tutorial update

Proposed changes:
-Adds Colab GPU setup header and "open in colab" button to GPL tutorial

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[vblagoje]

Any idea why this PR fails now @ZanSara @bogdankostic and it passed before (2-3 weeks ago)?

[julian-risch]

The tutorial might just need a pip install datasets.

[vblagoje]

The tutorial might just need a pip install datasets.

How would you add it @julian-risch I thought adding it in the setup cells of the notebook (the very top) is enough.

[julian-risch]

@vblagoje for the tests of the tutorials, we need it here as well: https://github.com/deepset-ai/haystack/blob/master/.github/workflows/tutorials.yml and here: https://github.com/deepset-ai/haystack/blob/master/.github/workflows/tutorials_nightly.yml or add it here

haystack/setup.cfg

Line 212 in 8d65bc5

all =

for the python script (not the jupyter notebook).
What fails is the execution of https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial18_GPL.py if I am not mistaken.

[julian-risch]

Btw, here you can see how the tutorial tests are executed: https://github.com/deepset-ai/haystack/blob/master/.github/utils/tutorials.sh

[ZanSara]

To add to @julian-risch answer, if GLP needs additional dependencies it should have its own dependency group. So we should add something like:

gpl = 
   datasets

in setup.cfg, and then add the gpl group to the all list. You can have a look how it was done for the audio group (b4fbb85 and 1641248). I can help you @vblagoje if you're unsure about something 🙂

[vblagoje]

Thanks @julian-risch @ZanSara Tutorial 18 takes about 20-25 mins to run. Is that ok or should we add it to the no-run list?

[ZanSara]

I think 25 minutes is still manageable 👍 Nightly runs can take as long as 6 hours and the current one takes less than two, so we have plenty of room still.

The no-run list is for tutorials that can't possibly run on GH, for example the ones that that take several hours on CPU, or need too much ram.

[vblagoje]

I think 25 minutes is still manageable 👍 Nightly runs can take as long as 6 hours and the current one takes less than two, so we have plenty of room still.

The no-run list is for tutorials that can't possibly run on GH, for example the ones that that take several hours on CPU, or need too much ram.

Oh, I thought we have a GPU for these nightly runs. Then there is no chance for this tutorial to finish. Training alone takes 15 min on GPU.

[vblagoje]

Thanks a bunch @ZanSara @julian-risch for your assistance! I think it should be gtg now.

[julian-risch]

LGTM! 👍 I tried it out on Google colab and it worked. I was just surprised that the installation command is slightly different than in our other tutorials. Here it is !pip install -q git+https://github.com/deepset/haystack.git, whereas all other tutorials use !pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]. So it seems that the colab extra is not needed anymore. We could remove that from all the other tutorials in a separate PR then.

[vblagoje]

LGTM! 👍 I tried it out on Google colab and it worked. I was just surprised that the installation command is slightly different than in our other tutorials. Here it is !pip install -q git+https://github.com/deepset/haystack.git, whereas all other tutorials use !pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]. So it seems that the colab extra is not needed anymore. We could remove that from all the other tutorials in a separate PR then.

Thanks for trying it out @julian-risch I wanted to see the bare minimum of deps that would work and it seems that it stuck - the raw repo. Not sure tbh what is the best option here...

[ZanSara]

LGTM! +1 I tried it out on Google colab and it worked. I was just surprised that the installation command is slightly different than in our other tutorials. Here it is !pip install -q git+https://github.com/deepset/haystack.git, whereas all other tutorials use !pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]. So it seems that the colab extra is not needed anymore. We could remove that from all the other tutorials in a separate PR then.

I am very happy if we can remove the colab extra, but let's test it well before doing it. It also might be a good idea to keep it in haystack even if unused, because when Colab decides to upgrade or downgrade the version of grpcio again, we at least have an easy way to patch it quickly.

[ZanSara]

I think 25 minutes is still manageable +1 Nightly runs can take as long as 6 hours and the current one takes less than two, so we have plenty of room still.
The no-run list is for tutorials that can't possibly run on GH, for example the ones that that take several hours on CPU, or need too much ram.

Oh, I thought we have a GPU for these nightly runs. Then there is no chance for this tutorial to finish. Training alone takes 15 min on GPU.

Ok I thought 20-25 minutes was the time on CPU! Ok then it goes in the no-run list for now. We should spin up some self-hosted runners for these 4 tutorials anyway...

[vblagoje]

@ZanSara Ok, in that case, I should undo the last commit of this PR which changes setup.cfg and add a commit of adding this tutorial to no-run list. WDYT?