Disable Test Modules

[jjsheridan]

What happened?

After building a Python image, our scanner flagged private keys contained in the image. @arjun024 states that these come from the dependency archive released by python. He mentioned there's an option to set --disable-test-modules that could possibly remove the keys. Our Security team requires our containers to not have any private keys stored.

• What were you attempting to do?
  Build a Python image.
• What did you expect to happen?
  Container without private keys.
• What was the actual behavior? Please provide log output, if possible.

Build Configuration

We were just using the Python sample code from this git site.

• What platform (pack, kpack, tekton buildpacks plugin, etc.) are you
  using? Please include a version.
  Pack

• What buildpacks are you using? Please include versions.
  No buildpacks were specified.

• What builder are you using? If custom, can you provide the output from pack inspect-builder <builder>?
  Paketo base builder

• Can you provide a sample app or relevant configuration (buildpack.yml,
  nginx.conf, etc.)?

Checklist

• I have included log output.
• The log output includes an error message.
• I have included steps for reproduction.
  You can refer to this conversation in Slack - https://paketobuildpacks.slack.com/archives/CULAS8ACD/p1637103874047800

[sophiewigmore]

@ryanmoran without a huge amount of context, I had recommended this issue be filed on a Python buildpack, since we want to evaluate the effect of setting this flag, before we go in and just rebuild our dependencies. Do you think this should be approached differently?

[ryanmoran]

I don't think we should just rebuild the Python dependency without discussion from the @paketo-buildpacks/python-maintainers, but I think the issue still belongs here given that this is ultimately where the dependency lives and the work would be tracked.

[arjun024]

--disable-test-modules seems to skip installing the test package that's generally part of the standard library. I'd recommend that a proposal to do this should have an RFC doing a cost-benefit analysis of disabling test modules. Another option would be to simply remove the keys from the dependency if they are just test artifacts that have no value post-build.

[jjsheridan]

Hi, Is this still under consideration?

[sophiewigmore]

Hey @jjsheridan, it's still under consideration, we just haven't had the time to prioritize this yet. Is this blocking for you, or have you been able to find a workaround? Contributions are always welcome; although, in this case I recognize that the dependency compilation code quite difficult to navigate (which is a problem), and it's not the easiest thing to test in the full buildpack workflow.

For reference, here are the steps I'd follow if I were to pick up this issue:

• making modifications to the compilation code, and recompiling the dependency locally following @arjun024's suggestions.
• Record the difference between the newly compiled dependency, and the one we ship now.
• Using the local dependency in the Cpython buildpack, and plugging the new Cpython buildpack in the language family Python buildpack, and then running the integration suite, and also possibly use the newly generated Python buildpack to build the Paketo Python sample apps.
• Record findings here for discussion.

[jjsheridan]

Thanks @sophiewigmore. We haven't been able to come up with a work-around for this, so yes, this is a blocker. Our Security team won't accept containers with key files.

[sophiewigmore]

Got it, I'll try to take a look at this soon.

[sophiewigmore]

I'm still working on this investigation. To document what I've seen so far:

• I have discovered that the --disable-test-modules flag was introduced in Python v3.10, so we could only include this for versions going forward. We wouldn't backport old versions to remove test modules from the dependencies.
• When I tried to build a pip app with the version pinned to v3.10.1 using the binary without test modules, the build failed on installing pip:

[builder]     Installing Pip 21.3.1
[builder] failed to configure pip:
[builder] /usr/bin/python: No module named pip

Building with v.3.10.1 with the test modules failed on the pip install step:

[builder]     Running 'pip install --requirement requirements.txt --exists-action=w --cache-dir=/layers/paketo-buildpacks_pip-install/cache --compile --user --disable-pip-version-check'
[builder] pip install failed:
[builder] WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
[builder] WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/flask/
...
[builder] ERROR: Could not find a version that satisfies the requirement Flask==0.12.3 (from versions: none)
[builder] ERROR: No matching distribution found for Flask==0.12.3

I need to do more investigation to understand the issue with Pip and this version of Python, since it's likely just a problem with the new version of Python, and not the lack of test module availability

• Python test module docs state “The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.” Pending investigation, this gives me reason to believe that releasing the dependency without the test modules should in theory be fine.

[jjsheridan]

Thanks for the update, @sophiewigmore. Were you able to determine what caused the pip install to fail?

[sophiewigmore]

@jjsheridan this is as far as I've gotten with the exploration so far. Not sure yet 👎

[sophiewigmore]

After recompiling the 3.10.1 dependency, it works in builds! I think it was compiled with an out of date version of the compilation code.

[sophiewigmore]

When v3.10.1 is compiled with --disable-test-modules, the builds fail on the installation of pip in the Pip Buildpack step because the ctypes modules needs libffi-dev, which isn't there since it's a development module.

Installing libffi-dev during dependency compilation (added it as an additional package we install here) resolves failures, and results in a successful build with Python v3.10.1 as far as I can tell.

However, when actually running the built container, I am still seeing failures:

  File "/layers/paketo-buildpacks_pip-install/packages/lib/python3.10/site-packages/jinja2/_compat.py", line 113, in <module>
    from collections import Mapping as mapping_types
ImportError: cannot import name 'Mapping' from 'collections' (/layers/paketo-buildpacks_cpython/cpython/lib/python3.10/collections/__init__.py)

From a cursory search, this seems like a common issue with Python 3.10 in general. Common suggestions on Stackoverflow in response to this error were to downgrade the Python version. This error occurs with and without --disable-test-modules set.

Conclusion

From my investigation, I think it's viable to compile cpython v3.10.* and beyond with the --disable-test-modules flag, and install libffi-dev during compilation.

At this point, some work on the Python side of things is needed to support version 3.10 in general, outside of introducing the --disable-test-modules flag. Support for v3.10 is out of scope for this specific issue. Once that's addressed on the Python side of things, I would be willing to put an RFC in and facilitate discussions around this flag.
cc @paketo-buildpacks/python-maintainers @jjsheridan

[jjsheridan]

Thanks for putting the time into this, @sophiewigmore.