[setup] Using pip-tools to create requirements.txt

[john-bodley]

@mistercrunch this is an approach we could take for automatically ensuring that the build was deterministic by autogenerating requirements.txt from setup.py.

Note this approach pins all sub-modules (in effect the output of pip freeze) which wasn't viable when we supported both Python 2 and Python 3.

[codecov-io]

Codecov Report

Merging #6054 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6054   +/-   ##
=======================================
  Coverage   77.81%   77.81%           
=======================================
  Files          46       46           
  Lines        9446     9446           
=======================================
  Hits         7350     7350           
  Misses       2096     2096

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 395359f...83dea5f. Read the comment docs.

[victornoel]

@john-bodley why not use pipenv directly? It is meant to solve this situation I think…

[mistercrunch]

@victornoel this article seems to be leaning towards pip-tools over pipenv and is fairly recent from what I gather: https://twigstechtips.blogspot.com/2018/05/python-comparison-of-pipenv-vs-pip-tools.html

[mistercrunch]

BTW it's ideal to have deterministic builds, but in some cases it prevents us from ever knowing about issues that occur when people simply pip install superset (which is based purely on what's in the setup.py, which is not pinned). I wonder if we should have an extra slot in the build matrix that would pip install -e . and not pip install -r requirements.txt to let us know that something is up with the vanilla build.

[mistercrunch]

LGTM

[victornoel]

@mistercrunch thanks for the link :)