[KED-1456] Docker environment for developers

[neomatrix369]

Description

I had issues building and running kedro tests on my MacOSX environment. Haven't had these conflicts in dependencies when working with other projects or packages. It's possible it's an isolated case but there is a chance it might happen to someone else's development environment.

Context

Instead of immediately trying to debug this issue, I decided to create a docker environment for development and testing purposes. It takes care of managing python dependencies without conflicting with your host machine local environment python environments. It's also a good alternative way to run kedro (with or without our changes) in an isolated environment for sanity checking.

You run the shell script to start the docker environment, it has the relevant mappings to the python dependencies (isolated) environment and the kedro sources.

(Happy to share these scripts via a PR, if there isn't one already out there)

[ghost]

Is this related to/a duplicate of: #254? :)

[neomatrix369]

Is this related to/a duplicate of: #254? :)

It's going to have additional functionality than just running tests in the docker environment.

You get an interactive environment i.e. a prompt inside the docker container where you can run all your commands from. For e.g. run the build-docs or any other make command from inside this container while having access to the kedro sources on your host machine:

kedro@3febd7d5d069:~/kedro$
kedro@3febd7d5d069:~/kedro$ make build-docs
./docs/build-docs.sh
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/kedro/kedro
  Installing build dependencies ... |
...
Collecting sphinx<2.0,>=1.8.4; extra == "docs"
  Downloading Sphinx-1.8.5-py2.py3-none-any.whl (3.1 MB)
     |████████████████████████████████| 3.1 MB 1.8 MB/s
Collecting tornado<6.0,>=4.2; extra == "docs"
  Downloading tornado-5.1.1.tar.gz (516 kB)
     |████████████████████████████████| 516 kB 2.2 MB/s
Collecting nbsphinx==0.4.2; extra == "docs"
  Downloading nbsphinx-0.4.2-py2.py3-none-any.whl (20 kB)
...

[neomatrix369]

@ZainPatelQB I now have a PR - so you can see how this will be different from #254 - although both serve similar purposes and could co-exist

[neomatrix369]

I did read that there is some work going on wrt Kedro docker (#57 (comment)), happy to align with that work if its available in the open. I have some experience in this area and happy to share it with this project.

[mzjp2]

As mentioned in PR #268

Hey!

Thanks for raising this PR and sorry for the delay in getting back to you. We've had an internal discussion about this and decided that we'd instead focus on making the development requirements more minimal, rather than introducing a docker image. You can now run tests and get started on contributing without Spark installed, which was a major impediment before.

I'll go ahead and close this, but thank you again for the PR and discussion. :)

[neomatrix369]

As mentioned in PR #268

Hey!
Thanks for raising this PR and sorry for the delay in getting back to you. We've had an internal discussion about this and decided that we'd instead focus on making the development requirements more minimal, rather than introducing a docker image. You can now run tests and get started on contributing without Spark installed, which was a major impediment before.
I'll go ahead and close this, but thank you again for the PR and discussion. :)

So this does not solve the problem, it only pushes it into another direction and encapsulates it.

As a developer, I would still like to build and run the whole test suite. Also, docker gives great isolation and also easy to share images and build on top of it. As we know in the world of python each one of us has our own environments and wouldn't change it for another tool/package.

When one has errors or issues, it could be so many reasons, while docker would bring a standard way to reproduce it and fix it. I see more uses, advantages and pluses having it than not. But the final decision is of course yours.