Switch to TOML for configuration files

[pradyunsg]

#3691 means pip will need to vendor some form of TOML library.

In view of consistency, is there any interest in shifting to the TOML format for configuration files?

[pradyunsg]

@dholth Could you label this config?

[RonnyPfannschmidt]

@pradyunsg this would be a massive compatibility breach - its not necessarily worth it

[dholth]

Which configuration files?

Uwsgi solves this problem by supporting all file formats

[agronholm]

Uwsgi is also notorious for its feature bloat.

[dholth]

And I use uwsgi instead of 15 separate programs, saving tons of maintenance work. Notorious!

[dholth]

Closed by mistake

[agronholm]

Not sure what you're trying to say here, but uWSGI's development style comes with a costly maintenance burden. We don't want pip to be another uWSGI.

[dholth]

The answer to the original question is "no" because this has been untouched for six months. The answer to "is there interest" might change to "yes" if @pradyunsg can show how toml improves pip's configuration from the implementation and end-user sides. But it might be difficult to write an implementation that elegantly supports both the old and new file formats. IIUC pip already supports three configuration formats: environment variables, command line arguments, and the file.

@agronholm You mean you don't think pip would be improved by hilarious documentation and sweary log messages? How about modes named after popular video games? I'm disappointed. We don't want pip to be hard to maintain but that is irrelevant to the end user. Without feature bloat there would be no features and no users. We wouldn't have to implement the features in the same style as uWSGI though.

[agronholm]

@dholth There is a middle ground between "no new features" and "features like in uWSGI". That's where we want to be. Given pip's importance to the Python community, we want to add features in a way that maximizes maintainability, and think hard about what to add and what not to.

[pradyunsg]

I'll respond to/randomly type about various comments:

@dholth said:
if @pradyunsg can show how toml improves pip's configuration

The way I look at this is that TOML is already a part of PEP-518 - it is going to be a part of pip. So, it makes sense to switch to TOML for what it is actually meant for: configuration files.

@RonnyPfannschmidt said:
this would be a massive compatibility breach - its not necessarily worth it

I don't think so. massive compatibility breach is overstating the deal. The current pip.ini or pip.conf files will be nearly what the final pip.toml would look like. Anyway, it'll obviously have a deprecation period and all that, so I think it'll be fine. To me, the minor inconvenience it'll cause seems to be worth it.

@dholth said:
Which configuration files?

There's only INI format files which have a OS-dependant (and environment-dependant) path.

@dholth said:
IIUC pip already supports three configuration formats: environment variables, command line arguments, and the file.

Yep! The overriding order is: cli -> environment -> subcommand config -> global config -> option defaults.

@agronholm said:
There is a middle ground between "no new features" and "features like in uWSGI".

I see this as the said middle ground. And then I like TOML enough to want to see it be the Python Packaging Ecosystem as the configuration file. 😃

@dholth said:
Without feature bloat there would be no features and no users.

Feature Bloat != Having features. Feature bloat is useless features that add no value to end user or even affect UX negatively. Anyway, let's not fight on terminology.

Also, I don't think uWSGI is a good example for this anyway. Let's not talk with that as a reference.

Could you label this config?

Yeah @dholth, please add the label config to this issue...

[dstufft]

FTR I think it makes sense to move our configuration over to using TOML, possibly deprecating the old configuration as well.

[pradyunsg]

@dstufft FWIW, #4192 is a start to getting there.

[pradyunsg]

Is this something that would be a nice-to-have in 10.0?

[dstufft]

It'd be nice to have, not sure I care about if it's in 10.0 or not though.

[agronholm]

Is somebody working on this?

[pradyunsg]

As of now, I'm don't think anyone is working on this. It's on my personal roadmap but I don't think I would be getting to this in the next 2 or even 3 months. I guess anyone who wants to take this on can feel free to do so. :)

…

On Sat, Jul 8, 2017, 17:32 Alex Grönholm ***@***.***> wrote: Is somebody working on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3809 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADH7SWojxApJ_r45D0tmYNL4XH0SKANAks5sL2-5gaJpZM4I8qm1> .

[cupen]

BTW: Would you add a sub-command config which use for set/get variables?
e.g.:

$ pip config set global.index-url   https://pypi.tuna.tsinghua.edu.cn/simple
Done.
$ pip config set global.format      columns
Done.

It means the variable global.index-url and global.format will be saved to the configuration file.

$ pip config get global.index-url
https://pypi.tuna.tsinghua.edu.cn/simple
$ pip config get global.format
columns

It is both on human and scripts friendly.

[RonnyPfannschmidt]

that would depend on if there is a vendor-able toml library that supports editing without throwing away meta-data (else a serialize/de-serialize would just throw away comments and other things the user might have added manually

[cupen]

Oh sorry, it's duplicated with #1736

[pradyunsg]

There is a pip config command ready to ship for pip 10. =) And the serializability that Ronny Pfannschmidt points out isn't needed (pip config doesn't have it now) but is definitely something I want to add.

…

On Thu, 8 Mar 2018, 15:53 Ronny Pfannschmidt, ***@***.***> wrote: that would depend on if there is a vendor-able toml library that supports editing without throwing away meta-data (else a serialize/de-serialize would just throw away comments and other things the user might have added manually — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3809 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADH7SV2A9-k6G1Id25wHv_uMn8BAxSLNks5tcQalgaJpZM4I8qm1> .

[pradyunsg]

that would depend on if there is a vendor-able toml library that supports editing without throwing away meta-data (else a serialize/de-serialize would just throw away comments and other things the user might have added manually

This exists now: https://github.com/sdispater/tomlkit

[Shivansh-007]

I have been working on this, I actually started working on it today, along with the guidance of @pradyunsg. I have gone through the discussion of this PR and decided upon the following:

1. The old configparser based on .ini files would stay, but be deprecated. Pip would interact with this only when it finds none of the pip.toml config files at the suggested locations. Once pip finds a suitable .ini config file, it would prompt the user whether they want to go ahead with this config, if not they specify. This is because sometimes you leave out your old config files on your computer and don't clean them, so pip may have chosen it incorrectly, or you may have incorrectly placed your pip.toml config file.

2. Switching to TOML would require migration of .ini files to .toml files for a number of users. I couldn't find any promising tools on the internet/GitHub, so it would be good to have the ability to convert these files for them, this functionality would only exist till it stays deprecated, it would be removed after that. But this would probably be rejected by you since it is pretty out of focus of pip, that's understandable, if that's the case, I could put out a small tool for this conversion.

3. For implementing this, we would need to move from tomli, it has no write API as of now and there are no plans of adding it. Thus, as suggested by pradyun I have changed to tomlkit for the whole of pip, thus removing tomli from vendored libs.

4. After discussing with pradyun on discord, we have also decided to have no extra logic for "yes", "y", "no", etc. for boolean values like we do now. Since INI parser only spews out strings, and the values like quiet and verbose are expected to be strings, therefore these utility functions would also need to be refactored. Quoting him:

   [12:47] pradyunsg: Yes, and we should fail saying "expected a boolean".
   [12:48] pradyunsg: It's better almost always to force the user to be explicit about what exactly they want, especially when starting with a new format (as we are here).
   [12:49] pradyunsg: Computers aren't very good at guessing human intent. Don't try to make them. :)
   [12:51] pradyunsg: See also, https://hitchdev.com/strictyaml/why/implicit-typing-removed/

Let me know if you have some concerns with this approach.

[Shivansh-007]

On second thought, I think it might be better to have a flag for the current configuration (ini/conf). Since I see a number of config files present on my computer with pip.toml without me being aware of them until now.

[pradyunsg]

Since I see a number of config files present on my computer with pip.toml without me being aware of them until now.

Do you mean these files already exist?

[Shivansh-007]

Ok, I was just being dumb and didn't cross-check, I thought the function would give out files which exist, none of them exist, so this can go as planned :D