Specify library version in Pipfile

[mwobensmith]

We should make sure that we lock down compatible library versions to known good releases, to avoid introducing regressions when a library has a bug.

We will then periodically run the project with current versions, and update this file when we are confident that there are no issues.

[wizofe]

I've noticed that you don't have an integrated CI mechanism. Do you have plans to do that? This would definitely improve code quality as well as issues with libraries (at least the ones you can easily catch...).

[mwobensmith]

@wizofe We agree. :)

In the meantime, we have daily runs on hardware, which isn't perfect but is the best we can do at the moment.

[wizofe]

Well, it sounds like an extremely difficult task to be honest. Sounds like that you need to write another GUI Automating to test the GUI Automating tool 🤣

In all seriousness though, I mostly meant to test the whole bootstrap process and the library dependencies, in the lowest operating level possible, to ensure that everything works as expected and verify new versions.

[mwobensmith]

@wizofe That's a good idea and is probably achievable, since the problems we encounter happen later when the actual tests are running.

[AndreiH]

I am encountering some errors when modifying the Pipfile and trying to do a pipenv install.

Pipfile.lock (9e6cac) out of date, updating to (7e8cbb)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
thon2.7/dist-packages/pipenv/patched/piptools/repositories/pypi.py", line 297, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/resolve.py", line 263, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/resolve.py", line 213, in _get_abstract_dist_for
    self.require_hashes
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 294, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 127, in prep_for_dist
    self.req.run_egg_info()
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 636, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/utils/misc.py", line 701, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pipenv.patched.notpip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/tmpdRuKgsbuild/pyobjc-framework-security/
/usr/local/lib/python2.7/dist-packages/pipenv/_compat.py:113: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/pipenv-daTiCf-requirements'>
  warnings.warn(warn_message, ResourceWarning)

It seems to be a Critical Bug on Pipenv side: pypa/pipenv#2320
This is strange that is happening when I modify the versions of the libraries.
I will investigate this further.

Update:
Actually the command that produces this error is pipenv lock which is executed when Pipfile is modified, to update the Pipfile.lock

[AndreiH]

2nd Update:

After updating Pipenv to the latest version that has the fix for the error previously posted, we receive another error that has to do with all pyobjc libraries.
They are failing to be locked on other Platforms than MacOS.
If I run pipenv install on Linux or Windows, after I modified the Pipfile, we experience another error:

File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 114, in main
    parsed.requirements_dir, parsed.packages)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 85, in _main
    requirements_dir=requirements_dir,
  File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 69, in resolve
    req_dir=requirements_dir
  File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 597, in resolve_deps
    req_dir=req_dir,
  File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 450, in actually_resolve_deps
    resolved_tree = resolver.resolve()
  File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 407, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 206, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 301, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/repositories/pypi.py", line 234, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/repositories/pypi.py", line 426, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(download_dir, ireq, wheel_cache)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/repositories/pypi.py", line 297, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/resolve.py", line 260, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/resolve.py", line 213, in _get_abstract_dist_for
    self.require_hashes
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 294, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 127, in prep_for_dist
    self.req.run_egg_info()
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 474, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/_internal/utils/misc.py", line 705, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pipenv.patched.notpip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/tmpuhlNXzbuild/pyobjc-framework-cocoa/

[AndreiH]

I opened an issue on Pipenv side: pypa/pipenv#3187

[AndreiH]

Following up after the issue that was posted on Pipenv side (pypa/pipenv#3187), it seems that the errors have to do with the objective c wrappers (pyobjc libraries), and they suggested to open an issue on Pyonbjc side. (comment: pypa/pipenv#3187 (comment))

On my side I came up with a more clean solution that I think would benefit the project.
Given the fact that we now are locking our libraries to Pipfile, I don't see the need to push the Pipfile.lock to the git repo (for locking the dependencies versions), therefore I deleted it from the repository and added it to .gitignore.
This will benefit us because it's a more clean solution being the fact that we use so many platforms on testing Iris. Depending on the Platform, Pipfile.lock can update differently on each one.

And regarding the Pyobjc libraries errors when trying to be installed from different platforms than Mac, I added Pyobjc to be installed in the OSX bootstrap, and also added it to be checked for in tools/platform_check.sh.

I cleaned up the Pipfile and now it installs more faster on each platform, and there is no more additional and unclear noise in the terminal when installing the dependencies.

@mwobensmith let me know what you think about the changes. They are ready for review on branch issue_1320. Thanks!

[mwobensmith]

Hi @AndreiH ! This is approved but I'm running into issues with the merge. Can you rebase to the latest dev? Once you get a successful rebase, pleas ask @icrisan to land it. Thank you!

[AndreiH]

Thanks Matt ! Updated the branch and created a PR: https://github.com/mozilla/iris/pull/1454

[AndreiH]

Update: Changes were merged !

[AndreiH]

Adding a quick fix: https://github.com/mozilla/iris/pull/1455/files

[silviuFlorin]

Code approved