Added doc for header and json format for pip show #8008

deveshks · 2020-04-09T16:47:44Z

Towards #5261

As per #7967 (comment) , this serves as a companion PR for #7967 by providing explanation of different fields for pip show in both header and json format.

cc @pradyunsg @sbidoul @xavfernandez

sbidoul

Thanks for tackling this @deveshks.

I'm not sure we should couple the human readable output and the json output, and take the opportunity that json output is almost absent today to design it from a blank page. So I personally would not take the current output as a basis.

Since the human readable output is not documented today, it is maybe not worth the trouble documenting it, to not encourage tools authors to parse it. Of course we should not change it in case some people depend on it.

For the new json output, how about a structure with the following keys:

metadata: a dictionary with the core metadata fields, lower cased, with dashes replaced by underscores (https://packaging.python.org/specifications/core-metadata/)
direct_url: the content of direct_url.json, if present (PEP 610)
installer: the content of INSTALLER if present (PEP 376)
record: a list of {path, hash, size} from RECORD (PEP 376)
requested: a boolean (true if REQUESTED is present) (PEP 376)
required_by: a list of canonicalized distribution names that depend on this entry
location: this probably requires some investigation (it's different if the install is editable or not)

Also do we need a flag for editables? Probably for legacy ediables. For modern editables (which generate .dist-info metadata), this is covered by direct_url.

docs/html/reference/pip_show.rst

deveshks · 2020-04-10T06:23:03Z

Hi @sbidoul

Thanks for tackling this @deveshks.

I'm not sure we should couple the human readable output and the json output, and take the opportunity that json output is almost absent today to design it from a blank page. So I personally would not take the current output as a basis.

Agreed that we can have a newer json output with different fields as you suggested. I have added the new keys according to your suggestion, and instead of expanding on each individual sub-keys, I have referred to the corresponding URL for PEP or packaging. Please take a look

Also do we need a flag for editables? Probably for legacy ediables. For modern editables (which generate .dist-info metadata), this is covered by direct_url.

Could you expand on what do you mean when you say a flag for editables. Do you mean a flag which will cause the output to change depending on whether the package is installed in editable mode using the -e flag? Also what outputs might change/be present or not present based on that?

I think that once we agree on the content of json output based on this document, I will resume my work on the code PR to add these fields, and the respective test cases.

McSinyx · 2020-04-10T07:00:28Z

docs/html/reference/pip_show.rst

+- **metadata:** A dictionary with the core metadata fields present in the METADATA file, as defined in the
+  `Core metadata specifications`_. The fields are lower cased, with dashes replaced by underscores.
+- **direct_url:** A dictionary containing the content of direct_url.json, if present, as specified in `PEP 610`_
+- **installer:** A string containing the content of INSTALLER, if present, as specified in `PEP 376`_
+- **record:** A list of {file_path, file_content_hash, file_size|, representing the content of RECORD,
+  if present, as specified in `PEP 376`_ .
+- **requested:** A boolean, set to True if REQUESTED is present, as specified in `PEP 376`_
+- **required_by:** A list of canonicalized distribution names that depend on the distribution in question
+- **location:** A string containing the path where the distribution is installed.


Hi, I think definition list markup might fit this part better.

Hi @McSinyx

Did you mean something like this?

- **metadata:** A dictionary with the core metadata fields present in the METADATA file, as defined in the `Core metadata specifications`_. The fields are lower cased, with dashes replaced by underscores. - **direct_url:** A dictionary containing the content of direct_url.json, if present, as specified in `PEP 610`_ ....

It looks like this when rendered

I meant something line this

Restructured Text

metadata A dictionary with the core metadata fields present in the ``METADATA`` file, as defined in the `Core metadata specifications`_. The fields are lower cased, with dashes replaced by underscores. direct_url A dictionary containing the content of ``direct_url.json``, if present, as specified in :pep`610`. installer A string containing the content of ``INSTALLER``, if present, as specified in :pep:`376`. record A list of ``[file_path, file_content_hash, file_size]``, representing the content of ``RECORD``, if present, as specified in :pep:`376`. requested A boolean, set to True if ``REQUESTED`` is present, as specified in :pep:`376`. required_by A list of canonicalized distribution names that depend on the queried distribution. location A string containing the path where the distribution is installed. .. _`Core metadata specifications`: https://packaging.python.org/specifications/core-metadata/

which renders
Screenshot

I nit a few spots, e.g. typos and PEP references, though I have no strong opinion on the bold part.

Edit, seemingly field list would be a better choice if bold is desired. My point is to make the source easier to read.

Thanks @McSinyx ,

That looks much more readable than what I had. . I have italized the fieldnames instead of making them bold. Also TIL that :pep:`376`. automatically creates the PEP link.

Hi @sbidoul

I have reformatted the content according to the suggestions with some help from @McSinyx . Please take a look at the same and let me know if further changes are needed

deveshks · 2020-04-10T09:37:27Z

Seems like the travis-ci job timed out in the latest commit (https://travis-ci.org/github/pypa/pip/jobs/673344968#L276-L277) . I think it should probably get fixed in further commits to this PR.

Edit: Closing and Reopening fixed the issue. All checks pass now.

sbidoul · 2020-04-13T14:16:01Z

Noting that PEP 566 has a section explaining how to convert metadata to json. https://www.python.org/dev/peps/pep-0566/#json-compatible-metadata

deveshks · 2020-04-13T14:25:55Z

Noting that PEP 566 has a section explaining how to convert metadata to json. https://www.python.org/dev/peps/pep-0566/#json-compatible-metadata

Thanks @sbidoul ,

So given this information and this PR where I have documented the metadata according to your suggestions, can I start implementing some of the parsing logic in the original PR #7967 ?

Also if you have other suggestions about the document, please let me know the same, so that I can incorporate them to both the PRs 😊

sbidoul · 2020-04-19T10:09:52Z

A few topics to discuss and resolve:

Add a reference to https://www.python.org/dev/peps/pep-0566/#json-compatible-metadata as the way to convert core metadata to json
Do we want a requires field (a list of canonicalized distribution names this distribution depends on)? This is redundant with requires_dist in metadata but since it would not contain version specifiers, this could help higher level tools for simple use cases
Do we want metadata as a separate entry, or all core metadata fields as top level. I prefer a separate entry to avoid name clashes, but I note pip list already has a json output with name and version as top level entry. We could leave them as is and not documented and document metadata
location must be documented more precisely. Currently (what pip list does) this is the parent directory of the metadata (.dist-info or .egg-info) directory. This is also true for legacy editable installs (so for instance, in case of pip installed in editable mode, this is the pip/src directory).

deveshks · 2020-04-19T10:15:39Z

Thanks for those topics. It seems like some of the topics will affect how a particular thing will be implemented, and how will it then be implemented.

Also how do we generally drive such a discussion, as mentioned in the task list above in a PR?

deveshks · 2020-05-01T08:18:48Z

@sbidoul , Please find my opinions on the points raised to discuss below. Feel free to expand and correct these, and to add appropriate labels to the PR so that other people can chime in too 😊

Add a reference to https://www.python.org/dev/peps/pep-0566/#json-compatible-metadata as the way to convert core metadata to json

So this will go under metadata description as Please note that we follow https://www.python.org/dev/peps/pep-0566/#json-compatible-metadata to read METADATA headers and a json object ?

Do we want a requires field (a list of canonicalized distribution names this distribution depends on)? This is redundant with requires_dist in metadata but since it would not contain version specifiers, this could help higher level tools for simple use cases

I think we can keep both fields. But note that the output of the two fields as observed below is different. requires field only lists down the package names which are installed and are present in the dependency map of the package, whereas requires_dist reads it from METADATA. So you can observe that colorama and importlib-metadata are missing from Requires

Requires field in the default output of pip show tox:

Requires: six, virtualenv, packaging, filelock, py, pluggy, toml

requires_dist of METADATA:

['packaging (>=14)', 
'pluggy (<1,>=0.12.0)', 
'py (<2,>=1.4.17)', 
'six (<2,>=1.14.0)', 
'virtualenv (!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0)', 
'toml (>=0.9.4)', 
'filelock (<4,>=3.0.0)', 
'colorama (>=0.4.1) ; platform_system == "Windows"', 
'importlib-metadata (<2,>=0.12) ; python_version < "3.8"']

Do we want metadata as a separate entry, or all core metadata fields as top level. I prefer a separate entry to avoid name clashes, but I note pip list already has a json output with name and version as top level entry. We could leave them as is and not documented and document metadata

Agreed that metadata should be it's own key, with the metadata dictionary under it.

location must be documented more precisely. Currently (what pip list does) this is the parent directory of the metadata (.dist-info or .egg-info) directory. This is also true for legacy editable installs (so for instance, in case of pip installed in editable mode, this is the pip/src directory).

And in case of modern editable installs, the location can be derived by reading direct_url.json. I think adding this should cover all scenarios to document how locations are found?

Also the updated pip freeze handles location modern editables differently (reading from direct_url.json( from the default format of pip show (parent directory of METADATA).

So do we also want to change that in the default format as well, or keep it untouched?

deveshks · 2020-05-21T08:16:28Z

Hi @sbidoul

I have gone ahead and added your suggestions made at #8008 (comment) to this PR.

Could you please look at them and add your suggestions if anything else is needed, and accordingly I can close the work on the docs here, and start with the actual implementation.

sbidoul · 2020-05-30T10:47:04Z

docs/html/reference/pip_show.rst

+
+*requested*
+   A boolean, set to True if ``REQUESTED`` is present,
+   as specified in :pep:`376`.


Maybe we should provide the content of REQUESTED as for INSTALLER. This would allow for a future evolution of pep 376 where, e.g., REQUESTED would contain a version specifier.

Added in 02759a3

sbidoul · 2020-05-30T10:48:29Z

docs/html/reference/pip_show.rst

+*location*
+   A string containing the path where the distribution is installed. For legacy installs,
+   this is the parent directory of the metadata (.dist-info or .egg-info) directory.
+   For legacy editable installs, this is the directory where the source is located.


This could be simplified by saying that location is the parent of the metadata (dist-info or egg-info) directory. This holds True for all kinds of installation.

Added in 02759a3

sbidoul · 2020-05-30T10:52:36Z

docs/html/reference/pip_show.rst

+*required_by*
+   A list of canonicalized distribution names that depend
+   on the queried distribution.
+


I would add requires: a list of canonicalized distribution names which this distribution depends on.

Although we'd need to think about how it would work exactly with extras and environment markers.

I would add requires: a list of canonicalized distribution names which this distribution depends on.

Added in 02759a3

Although we'd need to think about how it would work exactly with extras and environment markers.

Just to confirm, I assume that you are mentioning to PEP-508 extras for extras and PEP-496 for environment markers

It would also be helpful if you could take an example of a package, and perhaps outline what would be the expected output json in case of both, and then I can take that and the info from above links to think of a solution.

Another thing I can think of is to get out the basic version of pip show --format=json with what we have right now, and then think about these later. (if these are considered as advanced use cases of show by us, and can be targeted in another PR)

sbidoul · 2020-05-30T10:59:31Z

@deveshks thanks for your work on this. I'd like to have more opinions from other pip maintainers before going forward. We also need to check if the proposed json structure is applicable to pip list (or pip inspect that is in the air, although I'm not too sure myself of how it would look like).

deveshks · 2020-05-30T11:52:38Z

@deveshks thanks for your work on this. I'd like to have more opinions for other pip maintainers before going forward.

Agreed on that, but what would be a good way of getting that opinion?

We also need to check if the proposed json structure is applicable to pip list (or pip inspect that is in the air, although I'm not too sure myself of how it would look like).

IIUC, pip list currently just lists down name, version and location, the first two of which would be covered by metadata, and the last with the location field itself.

Also pip list is applicable to all packages in the system/user space/virtualenv, and pip show focuses on the specific package. How would the json structure of pip show affect pip list in that case?

deveshks · 2020-06-06T21:15:11Z

As per the comment #8008 (comment) , it would be great if the other maintainers can take a look at this PR and share their thoughts, in order to move this forward.

deveshks · 2020-06-25T06:43:17Z

Hi @pradyunsg , @pfmoore , @uranusjr

As per @sbidoul 's comment #8008 (comment) , I would love to have your opinions on his concerns and have them addressed, before moving this PR forward.

xavfernandez · 2020-07-03T21:10:24Z

I'm still on board with Stephane's suggestions with the same question on the (partial) duplication between a possible requires field and its more complete source metadata.requires_dist.
I'd love to also hear about the other @pypa/pip-committers .

deveshks · 2020-07-04T05:33:04Z

I'm still on board with Stephane's suggestions with the same question on the (partial) duplication between a possible requires field and its more complete source metadata.requires_dist.

Thanks a lot for the response @xavfernandez. I believe that to being with, we can keep both field and clearly document where does those fields differ.

I have tried to outline that and the 4th point which are the unresolved ones in #8008 (comment) in #8008 (comment) . I would be glad if you could take a look at them and provide suggestions as well.

BrownTruck · 2021-02-04T16:00:31Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

BrownTruck · 2021-04-03T05:30:04Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

pradyunsg · 2021-09-18T08:44:04Z

@deveshks Would you be interested in updating this PR, to resolve the merge conflicts?

deveshks · 2021-09-19T11:45:36Z

@pradyunsg yes, I will update this PR and will resolve merge conflicts in a day's time, and accordingly work to move this PR forward.

pradyunsg · 2021-09-22T05:14:14Z

A gentle nudge. :)

sbidoul · 2021-09-22T09:23:20Z

@pradyunsg I think this PR can be kept as a placeholder to discuss the extended json output for pip list and pip show, but the implementation of that format is still TODO, and this PR must not be merged before the implementation.

deveshks · 2021-09-22T10:35:43Z

A gentle nudge. :)

Hi @pradyunsg , apologies for the delay, I have resolved the merge conflicts and updated the PR. Please let me know how we can proceed further on this.

deveshks marked this pull request as draft April 9, 2020 16:48

deveshks mentioned this pull request Apr 9, 2020

Add json formatting option to pip show #7967

Closed

deveshks force-pushed the add-show-json-docs branch 2 times, most recently from 0aaa554 to 78b42de Compare April 9, 2020 16:59

sbidoul reviewed Apr 9, 2020

View reviewed changes

docs/html/reference/pip_show.rst Outdated Show resolved Hide resolved

docs/html/reference/pip_show.rst Outdated Show resolved Hide resolved

docs/html/reference/pip_show.rst Outdated Show resolved Hide resolved

deveshks requested a review from sbidoul April 10, 2020 06:24

McSinyx reviewed Apr 10, 2020

View reviewed changes

sbidoul added the state: needs discussion This needs some more discussion label Apr 10, 2020

deveshks force-pushed the add-show-json-docs branch from 2c4c273 to a0a0a5f Compare April 10, 2020 09:28

deveshks closed this Apr 10, 2020

deveshks reopened this Apr 10, 2020

sbidoul mentioned this pull request Apr 19, 2020

Migrate pip show's Requires field to Requires-Dist #7989

Open

deveshks mentioned this pull request May 5, 2020

Add support for outputting a list of extras and their requirements. #4824

Open

deveshks marked this pull request as ready for review May 12, 2020 09:01

deveshks force-pushed the add-show-json-docs branch from a0a0a5f to e49a3ad Compare May 21, 2020 07:54

deveshks force-pushed the add-show-json-docs branch from 898710e to 29c7395 Compare May 21, 2020 11:19

deveshks mentioned this pull request May 30, 2020

Complete type annotations in "pip._internal.{build_env,self_outdated_check,exceptions}" #8296

Merged

sbidoul reviewed May 30, 2020

View reviewed changes

deveshks requested a review from sbidoul May 30, 2020 11:52

deveshks force-pushed the add-show-json-docs branch from 02759a3 to 6f68007 Compare June 17, 2020 21:51

sbidoul mentioned this pull request Jan 6, 2021

Include all resolution conflict fix info on stderr. #9421

Closed

BrownTruck added the needs rebase or merge PR has conflicts with current master label Feb 4, 2021

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Feb 26, 2021

uranusjr approved these changes Feb 28, 2021

View reviewed changes

deveshks force-pushed the add-show-json-docs branch 2 times, most recently from 430f3f2 to e3bc7fb Compare March 1, 2021 03:07

BrownTruck added the needs rebase or merge PR has conflicts with current master label Apr 3, 2021

sbidoul mentioned this pull request Jun 3, 2021

Add --required-by option to list command #10036

Closed

pradyunsg added the S: awaiting response Waiting for a response/more information label Sep 18, 2021

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Sep 22, 2021

deveshks force-pushed the add-show-json-docs branch from 64b992e to 6ccc19e Compare September 22, 2021 10:22

Added doc for header and json format for pip show

aa4b183

deveshks force-pushed the add-show-json-docs branch from 6ccc19e to aa4b183 Compare September 22, 2021 10:32

This was referenced Jul 10, 2022

Add metadata field to pip list json output #11097

Closed

Add pip inspect command #11245

Merged

sbidoul closed this in #11245 Jul 18, 2022

github-actions bot locked as resolved and limited conversation to collaborators Aug 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added doc for header and json format for pip show #8008

Added doc for header and json format for pip show #8008

deveshks commented Apr 9, 2020 •

edited

Loading

sbidoul left a comment

deveshks commented Apr 10, 2020 •

edited

Loading

McSinyx Apr 10, 2020

deveshks Apr 10, 2020 •

edited

Loading

McSinyx Apr 10, 2020 •

edited

Loading

deveshks Apr 10, 2020

deveshks Apr 10, 2020

deveshks commented Apr 10, 2020 •

edited

Loading

sbidoul commented Apr 13, 2020

deveshks commented Apr 13, 2020

sbidoul commented Apr 19, 2020 •

edited

Loading

deveshks commented Apr 19, 2020

deveshks commented May 1, 2020

deveshks commented May 21, 2020

sbidoul May 30, 2020

deveshks May 30, 2020

sbidoul May 30, 2020

deveshks May 30, 2020

sbidoul May 30, 2020 •

edited

Loading

deveshks May 30, 2020

sbidoul commented May 30, 2020 •

edited

Loading

deveshks commented May 30, 2020

deveshks commented Jun 6, 2020

deveshks commented Jun 25, 2020

xavfernandez commented Jul 3, 2020

deveshks commented Jul 4, 2020

BrownTruck commented Feb 4, 2021

BrownTruck commented Apr 3, 2021

pradyunsg commented Sep 18, 2021

deveshks commented Sep 19, 2021

pradyunsg commented Sep 22, 2021

sbidoul commented Sep 22, 2021 •

edited

Loading

deveshks commented Sep 22, 2021

Added doc for header and json format for pip show #8008

Added doc for header and json format for pip show #8008

Conversation

deveshks commented Apr 9, 2020 • edited Loading

sbidoul left a comment

Choose a reason for hiding this comment

deveshks commented Apr 10, 2020 • edited Loading

Choose a reason for hiding this comment

deveshks Apr 10, 2020 • edited Loading

Choose a reason for hiding this comment

McSinyx Apr 10, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deveshks commented Apr 10, 2020 • edited Loading

sbidoul commented Apr 13, 2020

deveshks commented Apr 13, 2020

sbidoul commented Apr 19, 2020 • edited Loading

deveshks commented Apr 19, 2020

deveshks commented May 1, 2020

deveshks commented May 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbidoul May 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbidoul commented May 30, 2020 • edited Loading

deveshks commented May 30, 2020

deveshks commented Jun 6, 2020

deveshks commented Jun 25, 2020

xavfernandez commented Jul 3, 2020

deveshks commented Jul 4, 2020

BrownTruck commented Feb 4, 2021

BrownTruck commented Apr 3, 2021

pradyunsg commented Sep 18, 2021

deveshks commented Sep 19, 2021

pradyunsg commented Sep 22, 2021

sbidoul commented Sep 22, 2021 • edited Loading

deveshks commented Sep 22, 2021

deveshks commented Apr 9, 2020 •

edited

Loading

deveshks commented Apr 10, 2020 •

edited

Loading

deveshks Apr 10, 2020 •

edited

Loading

McSinyx Apr 10, 2020 •

edited

Loading

deveshks commented Apr 10, 2020 •

edited

Loading

sbidoul commented Apr 19, 2020 •

edited

Loading

sbidoul May 30, 2020 •

edited

Loading

sbidoul commented May 30, 2020 •

edited

Loading

sbidoul commented Sep 22, 2021 •

edited

Loading