Document the behavior of --cert

[wiml]

Environment

• pip version: 19.2dev0 (and earlier)
• Python version: n/a
• OS: n/a

Description

The documentation for the --cert option currently just says "Path to alternate CA bundle."

This leaves the user wondering: does the specified bundle replace the default set of trust roots, or augment it?

(Also, it would be nice if --trusted-host had a mention of --cert since many people use the former when they could more securely be using the latter.)

[pradyunsg]

This is likely a good first issue too -- someone can check what the actual behavior is and document it properly in the help string for the option. :)

[brainwane]

@serhii73 would you like to work on this?

[serhii73]

@serhii73 would you like to work on this?

Yes, I can.
But need more information. Not full understand.
Please, tell me more what needs to see, what read and I to do it.

[brainwane]

@serhii73 You should start with https://pip.pypa.io/en/stable/development/getting-started/ -- then I suggest you skim this rough bit of documentation #6637 , play around with the existing help for the pip options on the command line, use git grep to learn more about where those bits of text live in the repository, and start learning what the --cert option does.

[serhii73]

@brainwane I'm on it.
Thank you very much!

[serhii73]

• https://pip.pypa.io/en/stable/development/getting-started/

• Add architecture overview. #6637

• play around with the existing help for the pip options on the command line

• use git grep to learn more about where those bits of text live in the repository

• --cert

[pradyunsg]

@serhii73 Please feel free to ask questions if you have any. :)

[serhii73]

1. | Manages the building of packages (offloads package building to a
   backend) when that’s necessary (a source distribution package --
   this is not necessary if the package is a wheel).

What does it mean?
Can you tell me more or show me a link where I can more read about it?

Manages the building of packages

This is I got it. But what next is no.

[serhii73]

@serhii73 You should start with https://pip.pypa.io/en/stable/development/getting-started/ -- then I suggest you skim this rough bit of documentation #6637 , play around with the existing help for the pip options on the command line, use git grep to learn more about where those bits of text live in the repository, and start learning what the --cert option does.

Do I need to read PEP 517 too?

[webknjaz]

@pradyunsg @brainwane plz assign this issue to @serhii73 on the GitHub UI. TIA.

[webknjaz]

@serhii73 PEP517/518 is unrelated to this issue

[brainwane]

@serhii73 how is this going? If you are working on this issue and have questions, please feel free to ask them here, in #pypa-dev on Freenode IRC, or on the pypa-dev mailing list. The live chat in Freenode IRC may be best because then you can have a live conversation and ask several questions in a row.

[serhii73]

Thank you very much, Sumana!

[brainwane]

@serhii73 https://freenode.net/kb/answer/registration , then use https://webchat.freenode.net/?channels=%23pypa-dev

[serhii73]

replace the default set of trust roots, or augment it

Sorry, I don't understand what function or class to do it in our package.

cert = partial(
    Option,
    '--cert',
    dest='cert',
    type='str',
    metavar='path',
    help="Path to alternate CA bundle.",
)  # type: Callable[..., Option]

Ok, need to find cert in the pckage, right?
cert exist in _build_session, cert_verify, _match_hostname
But i don't understand what function need if I want see

replace the default set of trust roots, or augment it

Do you know?
Thank you.

[pradyunsg]

Hi @serhii73! Thanks for continuing to work on this!

pip's option handling infrastructure passes the return value of optparse.parse_args to BaseCommand.run (i.e. base class of a CLI command, like 'install' or 'freeze').

The "cert" attribute on options, is the value given to pip via --cert (that's defined by the dest="cert" in the lines you mentioned above). Thus, to get a better understanding of how pip's using that value, you'd want to follow options.cert on the commands that have the --cert argument.

[serhii73]

options.cert exist only here.

[serhii73]

first - dest='cert' and then - _build_session ?

[serhii73]

Not fully understand how it working.

first - dest='cert' and then - _build_session ?

In src/pip/_internal/cli/base_command.py run def run after type user command?

[serhii73]

For debugging with pdb need a real certificate?
How without certificate understand the task?
Or exist a simple way where to get a certificate?

[serhii73]

I seem stuck in this task.
Can you help me @pradyunsg, please?

[webknjaz]

Run openssl version -d to find the default the CA bundle dir path on your computer. Then, create some test dir and copy some certificate there. After this, try using that dir (path) with --cert option.

[pradyunsg]

Right, since options.cert is being assigned to session.verify, you'd wanna check what session.verify affects/means. If you look at the value/type of session, you'll notice it's a PipSession (a class defined within pip). That's inheriting from a vendored copy of requests.sessions.Session. Thus, the next step would be to see what the verify attribute does on a requests.Session object.

Googling for "python requests session" lead me to https://2.python-requests.org/en/master/user/advanced/#ssl-cert-verification, which basically describes verify attribute is a path to a CA bundle, used for verifying SSL Certificates, that replaces the default bundle.

There's also https://2.python-requests.org/en/master/api/#requests.Session.verify but that doesn't contain much detail.

In any case, this information answers OP's (original post/poster) question -- it needs to be a path to a CA bundle, and that it replaces the use of any pre-existing bundle.

With that answered, our next steps are to check whether the existing documentation could be augmented and the explanation from OP points out two such locations.

With all this information, our next step would be to mention in the help text of --cert, that it replaces the default CA bundle.

@serhii73 does this help? Lemme know if something in my description above seems odd or not clear and I'll be happy to elaborate. :)

(I'm on mobile, apologies if there's typos and for the lack of use of markdown syntax)

[webknjaz]

There's also 2.python-requests.org/en/master/api/#requests.Session.verify but that doesn't contain much detail.

I bet this loosely corresponds to the TLS verification mode (https://docs.python.org/3/library/ssl.html#ssl.SSLContext.verify_mode) allowing the user to ignore untrusted certificates.
It looks like if it's set, the check is enforced unconditionally.

[pradyunsg]

allowing the user to ignore untrusted certificates

Yep yep -- that's --trusted-host in pip -- though Idk if that's how we handle that option.

[brainwane]

@serhii73 I'm unassigning you since I think this isn't the best first issue for you -- I suggest you look at one of the other "good first issue" bugs.

[brainwane]

Thanks for your work on this!

[chrahunt]

Some information on the current state of pip (a6b0605) with respect to SSL verification:

1. pip vendors requests and certifi (the versions specified here), and does not patch them in any way to change their default cert-handling behavior. As a result, the default behavior when --cert is not provided is the default behavior of requests.
   1. The default behavior of requests is to use the path returned by certifi.where() if verify is not set to an alternate path (link)
      1. requests itself sets verify to the contents of REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE if not explicitly set (link)
      2. By default certifi.where() returns the path to the CA certificate bundle cacert.pem which is distributed in certifi itself (link)
      3. On Debian, which debundles pip, certifi.where() returns /etc/ssl/certs/ca-certificates.crt (link) - likewise for Ubuntu
      4. On Fedora, which keeps pip bundled but carries a few patches, certifi.where() returns /etc/pki/tls/certs/ca-bundle.crt (link) - probably similar to RedHat and CentOS
2. if --cert is provided, pip sets its session's verify property to the provided path. As shown in the requests sources above, this completely overrides the default CA certificate bundle. Also as shown above, this can be a file path or a directory.
3. The format of the file/directory provided via --cert should be as described for ssl.SSLContext.load_verify_locations. An example of the file form is in Certificate chains and the directory form is in SSL_CTX_load_verify_locations (in the description of the CApath argument)
4. Previously, pip supported a globally-managed version of certifi being used in preference to the vendored version, but this was reverted by Upgrade our Vendored Dependencies #4712 (10.0.0b1) (I don't see this in NEWS)

Which I think would translate into the following actions for this issue:

1. Update the pip_install/SSL Certificate Verification docs, mentioning that by default pip uses a bundled CA certificate store provided by certifi (with a link to certifi), but that this may be overridden by package-manager-managed pip.
2. Update the description of --cert. I would leave out the possibility of using a directory and instead just say something like "PEM-encoded CA certificate bundle. If provided, overrides the default." and optionally:
   1. when generating documentation, we could also include a link to the SSL Certificate Verification section mentioned above
   2. when being invoked as a command, we may mention to reference the user guide for details

[mdebi]

I would like to work on this.

[brainwane]

@mdebi please go ahead! I suggest you start a work-in-progress ("WIP") pull request as soon as you start.

[chrahunt]

Thanks @mdebi! The info in my comment is still accurate, so the two items I mentioned should act as a guide for tackling this. Please let us know if we can help in any way!

[mdebi]

Sure @brainwane @chrahunt . Thank you.

[jflemer-ndp]

As assumed above, on CentOS/RHEL, certifi.where() returns /etc/pki/tls/certs/ca-bundle.crt (link).

[bbodenmiller]

...
3. On Debian, which debundles pip, certifi.where() returns /etc/ssl/certs/ca-certificates.crt (link) - likewise for Ubuntu
4. On Fedora, which keeps pip bundled but carries a few patches, certifi.where() returns /etc/pki/tls/certs/ca-bundle.crt (link) - probably similar to RedHat and CentOS

@chrahunt where can one find these versions for installation on Debian and Fedora? In the versions I've found they are still using bundled cert rather than these system certs.