-
Notifications
You must be signed in to change notification settings - Fork 444
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Other pipx list
output formats
#627
Comments
For #390, my idea is: # freeze list of pkg
pipx list > $DOTFILES_DIR/pkg-pipx.txt
# parses list of pkg and installs each
cat pkg-pipx.txt | pipx_install.sh Right now this second step (piping pkg list file to install script) is fine but would just be easier with more structured output format. If output format that could be directly be read by pipx itself to install, all the better. Also, thanks for putting all this together :) |
@zachvalenta do I understand you correctly that if you had a structured json output, you wouldn't need another kind of |
Yes indeed. I'm looking for something more structured than the status quo but agnostic about format. |
Thanks for pulling together a bunch of threads!
This is all i want, anyway (and i filed #109 ). An ancillary use case that I am now looking forward to is bumping python interpreter versions for my pipx-managed venvs on the same machine. I have many 3.6.8 venvs, and wouldn't mind kicking them all up to 3.8.x or higher without manually redoing all of it. |
@itsayellow It occurs to me that if you do not already know about it, https://pypi.org/project/pipdeptree/ might be extremely useful for inspecting pipx-created environments and generating a list of top-level packages that were injected into them. For a busy example: i have a very full
and running |
Thanks @gvoysey , I am aware of pipdeptree (and use it myself if I accidentally install something into my system python packages!) Luckily in pipx we have a bit of an easier time of it because of our isolated venvs, and our internal metadata. Our internal metadata keeps track of what we intentionally installed, and each venv can be |
@itsayellow oh. well, that's just /cheating/! 😁 |
The problem seems to be in trying to implement something to install from such a json list. Doing a basic job is fairly straightforward, but covering all the edge cases gets a little hairy. For example, how do you handle things in your json file that do not exist on the target install system? Like paths to specific local packages, or paths to specific python interpreters. This might just mean that implementing a json format for list output is doable, but installing from such an output might be DIY. |
This makes sense to me as a fair middle ground btw added convenience for users and not forcing pipx to deal with a hairy problem. |
i noticed when using Flow like that for |
I don't quite parse what you mean by '"stringly" including extras', but I think I get the basic point, that we could just go through the list and try to install everything as described, and just fail on each particular package that doesn't work out, but otherwise keep installing. I suppose specifying an invalid package or python path would just fail to install that one as if one were trying to install something improperly from the command line. |
Here's a UX question. How would a command that outputs json be structured? Possibilities that come to mind:
I'm leaning toward something like 2 or 3, or in general specifying an output file. It seems to me that you'd really never want to stream the json text to the console anyway. And for a command that always wants an output file, it seems weird to me to make the user have to redirect by hand. I suppose the reason to output the json in a stdout stream would be to pipe it to some other command. But would that be useful? Would that be useful if pipx had its own native command to import such json? |
The best reasons I can think of for streaming to stdout would be if
But specifying a file output makes sense to me as well. |
I am much more in favor of streaming to stdout. The user can always redirect to a file if they want to. And you could also pipe the output to pipx list --json \
| jq '.[].env_name' \
| grep 'mypy|pyre-check' \
| xargs -n1 pipx reinstall Somewhat of a contrived example, but I can't see any reason to force the user to output to a file. |
I would pretty naturally spell that as "a subcommand with an optional argument". so |
I'm starting to think the clearest next step for a PR is to implement (as suggested above)
Which just sends the json output to stdout (and only the json output to stdout). I have more experience with macOS and linux--I assume redirecting to a file is just as easy on the various Windows shells? |
The Windows command prompt and Powershell both support |
JSON is fine, but another modern alternative could be TOML which would be a good balance between machine readable and human readable, without pitfalls of YAML. If one loads a JSON file as is and dump it as TOML, you get: pipx_metadata_version = "0.2"
python_version = "Python 3.8.5"
venv_args = []
[injected_packages]
[main_package]
apps = ["xon.sh", "xonsh", "xonsh-cat"]
apps_of_dependencies = []
include_apps = true
include_dependencies = false
package = "xonsh"
package_or_url = "xonsh"
package_version = "0.9.27"
pip_args = []
suffix = ""
[[main_package.app_paths]]
__Path__ = "/home/user/.local/pipx/venvs/xonsh/bin/xon.sh"
__type__ = "Path"
[[main_package.app_paths]]
__Path__ = "/home/user/.local/pipx/venvs/xonsh/bin/xonsh"
__type__ = "Path"
[[main_package.app_paths]]
__Path__ = "/home/user/local/pipx/venvs/xonsh/bin/xonsh-cat"
__type__ = "Path"
[main_package.app_paths_of_dependencies] I made the following sample TOML after manually editing [metadata]
version = 0.2
[package.xonsh]
spec = "xonsh" # Note: instead of package_or_url
version = "0.9.27"
apps = ["xon.sh", "xonsh", "xonsh-cat"]
apps_of_dependencies = []
include_apps = true
include_dependencies = false
package = "xonsh"
pip_args = []
python_version = "Python 3.8.5" # Note: moved down
suffix = ""
venv_args = [] # Note: moved down
[[package.xonsh.app_paths]]
"xon.sh" = "/home/user/.local/pipx/venvs/xonsh/bin/xon.sh"
"xonsh" = "/home/user/.local/pipx/venvs/xonsh/bin/xonsh"
"xonsh-cat" = "/home/user/.local/pipx/venvs/xonsh/bin/xonsh-cat"
[[package.xonsh.app_paths_of_dependencies]]
# continue with more packages ...
|
There's no TOML equivalent of JQ yet, and JSON generally has more support across programming languages. Therefore TOML would singificantly limit the usefulness of this option as something you can pipe into other scripts/tools for downstream processing. Also JSON comes with the Python standard library, which keeps the dependency count down (although I expect that TOML will end up in the stdlib eventually because it's part of PEP 517/518). That said, maybe Pipx could support multiple output formats? pipx list --format=list # default if --format is omitted
pipx list --format=json
pipx list --format=toml
pipx list --format=yaml There are well-supported TOML and YAML libraries on PyPI to do the output formatting. |
I think it is best if this would be combined with an option like Some pros and cons of different options
|
What is the advantage of having a format that is both human friendly and machine readable? I'm aware we could get toml to work, but I'm just not sure why we need it. |
The argument for TOML usually arises in the context of config files, which this isn't. I'm fine with just spitting out JSON, fwiw. |
It is very similar to a requirements.txt, environment.yml, Config.toml, pyproject.toml etc - call it what you will :)
I understand the arguments made for JSON. Here are some reasons for the file to be human readable. In the event when:
it would make it easier for the user to go in and edit the file. That's my take on why TOML would be a better fit. P.S.: I forgot, you can comment out a package too. |
The json I'm producing right now is pretty-printed. So editing a particular path by hand is actually quite reasonable. Also, I don't think I'm eager to have this format be created by hand from scratch, like a config file. I'd like to keep it pipx-generated so that we don't have to support human quirks that might pop up when we parse it. |
For the folks who want a language associated w/ config files like TOML/YAML, you can generate those e.g. Poetry generates a |
Not sure if the linked PR will trigger anyone's notifications so I'll post this: Check out PR #660 and tell me what you think. I made a short blurb describing the format. |
👍 from me; i think this is the output side of #109 and it looks pretty tidy! |
I'm going to close this now because I believe #660 has implemented json list output format, and that's probably the only other format besides text that we want to implement right now. Thanks everybody for the good discussion. |
How would this feature be useful?
Hopefully this will synthesize (and possibly supersede) Issues #390 and #109. It addresses the reason for PRs #572 and #392.
This Issue is to track exactly what users are looking for with different
pipx list
output formats."Import / Export" set of pipx packages case
In #109, it appears that one primary motivator is the ability to export a file that specifies a set of pipx-installable packages that pipx can then install (possibly on another machine) in the future.
How many users desire this or would find it useful?
It would probably best be done by making a file in json format.
Within this there are some further points for those desiring this functionality:
Do people consider it desired (or necessary?) to freeze all versions of all dependencies for this to be useful? If so then this might be very difficult, based on my investigation creating test code for this.
Simply re-creating a pipx environment, and pinning the version of the main package and any injected packages, and any pipx options is very doable. I have a branch of doing this basically working. In this case, the output would be equivalent to what you would end up with on one machine executing
pipx reinstall-all
. You can be sure the main package and any injected packages will stay with their specified versions, but you cannot be sure that the versions of any dependencies that still satisfy all package requirements will be the same.Archive / Documentation case
In #390 it appears that one motivation is simply a concise version of pipx list for "keeping track" or for archival or documentation purposes.
How many users desire this or would find it useful?
Is this type of list output (that is merely more concise) still desired if the "Import / Export" case is available?
What are the goals of this format? What are the use cases specifically? How would one know if the output of this was "successful"?
I must confess that I need this one explained to me, it doesn't naturally occur to me what is desired. If all that's needed is documentation of what is currently installed, then the current
pipx list
seems to cover that. If what is desired is a structured output that would allow one to recreate a pipx environment, then the "Import / Export" json scenario seems to satisfy the need.@comkieffer @gvoysey @SanketDG @zachvalenta
Describe the solution you'd like
Describe alternatives you've considered
The text was updated successfully, but these errors were encountered: