Provide PEP-685-compliant extras #10481

mattoberle · 2023-10-16T18:11:58Z

Description

pip==23.3.0 introduces a new "canonical" lookup behavior for package extras:

As a result, a command like this:

pip install 'sqlalchemy[postgresql_psycopg2binary]==1.4.47'

Results in a warning like this:

WARNING: sqlalchemy 1.4.47 does not provide the extra 'postgresql-psycopg2binary'

The extra eventually resolves when pip falls back to the original postgresql_psycopg2binary.
This PR proposes the addition of the PEP-685-compliant names to setup.cfg.

Maintains backward compatibility for pip<23.3.0
Avoids reliance on the "fallback" behavior of pip>=23.3.0

Checklist

Note: I'm not sure how to categorize this given the options in the PR template.
This is a packaging change.

This pull request is:

A documentation / typographical / small typing error fix
- Good to go, no issue or tests are needed
A short code fix
- please include the issue number, and create an issue if none exists, which
  must include a complete example of the issue. one line code fixes without an
  issue and demonstration will not be accepted.
- Please include: Fixes: #<issue number> in the commit message
- please include tests. one line code fixes without tests will not be accepted.
A new feature implementation
- please include the issue number, and create an issue if none exists, which must
  include a complete example of how the feature would look.
- Please include: Fixes: #<issue number> in the commit message
- please include tests.

`pip==23.3.0` introduces a new "canonical" lookup behavior for package extras: - https://pip.pypa.io/en/stable/news/#v23-3 - pypa/pip#12002 As a result, a command like this: ``` pip install 'sqlalchemy[postgresql_psycopg2binary]==1.4.47' ``` Results in a warning like this: ``` WARNING: sqlalchemy 1.4.47 does not provide the extra 'postgresql-psycopg2binary' ``` The extra eventually resolves when pip falls back to the original `postgresql_psycopg2binary`.

zzzeek

new pep not even two years ago already generating warnings...OK

here's what I want to do here:

can you confirm postgresql-psycopg2binary is the correct format (e.g. "normalized")
in the change here, the dashed names should be the primary names, as we will eventually remove the underscore names:

mssql-pyodbc = pyodbc
mssql_pyodbc =
    %(mssql-pyodbc)s

hoping the dash in the variable name there is accepted

put all the "legacy" underscore names in a separate batch way at the bottom so they are out of the way:

# legacy pre-pep-685 names
mssql_pymssql =
   %(mssql-pymssql)
mssql_pyodbc =
   %(mssql-pyodbc)
mysql_connector = 
   %(mysql-connector)

# ...

use the new names in our tox.ini file in the "extras" section:

diff --git a/tox.ini b/tox.ini
index ed68dbfcf7..675b80186b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -18,20 +18,20 @@ extras=
      sqlite_file: aiosqlite
      sqlite_file: sqlcipher; python_version < '3.10'
      postgresql: postgresql
-     postgresql: postgresql_asyncpg
-     postgresql: postgresql_pg8000
-     postgresql: postgresql_psycopg
+     postgresql: postgresql-asyncpg
+     postgresql: postgresql-pg8000
+     postgresql: postgresql-psycopg
 
      mysql: mysql
      mysql: pymysql
      mysql: asyncmy
      mysql: aiomysql
-     mysql: mariadb_connector
+     mysql: mariadb-connector
 
      oracle: oracle
      oracle: oracle_oracledb
      py{3,37,38,39,310,311}-mssql: mssql
-     py{3,37,38,39,310,311}-mssql: mssql_pymssql
+     py{3,37,38,39,310,311}-mssql: mssql-pymssql
 
 install_command=
      # TODO: I can find no way to get pip / tox / anyone to have this

with all that we can run through gerrit make sure it all works

thanks!

zzzeek · 2023-10-16T18:54:19Z

ok,lets try it

@CaselIT any comments?

sqla-tester

OK, this is sqla-tester setting up my work on behalf of zzzeek to try to get revision 2431850 of this pull request into gerrit so we can run tests and reviews and stuff

sqla-tester · 2023-10-16T18:54:31Z

New Gerrit review created for change 2431850: https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4906

mattoberle · 2023-10-16T18:57:12Z

Thanks for taking a look!

1. can you confirm `postgresql-psycopg2binary` is the correct format (e.g. "normalized")

The change to pip relies on a function called canonicalize_name:

https://github.com/uranusjr/pip/blob/9ba2bb90fb57e4ee5f624ecd39eade207863c21a/src/pip/_internal/metadata/base.py#L548-L584

Defined here:

https://github.com/pypa/pip/blob/main/src/pip/_vendor/packaging/utils.py#L27-L35

That follows the PEP which defines the canonical name as:

re.sub(r"[-_.]+", "-", name).lower()

2. in the change here, the dashed names should be the primary names, as we will eventually remove the underscore names:
...
hoping the dash in the variable name there is accepted

This was my concern as well, but I made the change and it looks like %(var-with-dash)s is acceptable.

3. put all the "legacy" underscore names in a separate batch way at the bottom so they are out of the way:
...
use the new names in our tox.ini file in the "extras" section:

Done & done.

CaselIT · 2023-10-16T19:04:10Z

k,lets try it

@CaselIT any comments?

we must ensure that the fallback also works when using pyproject, an in the PR here #9324

also 2.1+ only or it's good also for 2.0?

sqla-tester · 2023-10-16T19:41:52Z

Federico Caselli (CaselIT) wrote:

we have to verify that the fallback works with PEP 621 (#9324). If not we have to find an alternative

View this in Gerrit at https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4906

zzzeek · 2023-10-16T19:42:06Z

k,lets try it
@CaselIT any comments?

we must ensure that the fallback also works when using pyproject, an in the PR here #9324

well this is the official syntax apparently, so it has to work in pyproject and everywhere else

also 2.1+ only or it's good also for 2.0?

it looks like 1.4 users are getting this warning already so if this is safe to do for 2.0 at least, that would be good

CaselIT · 2023-10-16T19:47:34Z

k,lets try it
@CaselIT any comments?

we must ensure that the fallback also works when using pyproject, an in the PR here #9324

well this is the official syntax apparently, so it has to work in pyproject and everywhere else

the pyformat thing %(name)s look a lot like something custom of cfg. As far as I know it's not supported in toml files.

what I'm saying is just to verify that this syntax works also when using toml, if not we can think of a workaround (likely just duplicating the actual requirements) and leave a note in the other PR mentioning that it is something to do before merging that one

zzzeek · 2023-10-16T20:05:01Z

wouldnt the toml use quoted names to handle identifier characters?

CaselIT · 2023-10-16T20:13:48Z

wouldnt the toml use quoted names to handle identifier characters?

checked out #9324
trying to use

postgresql-psycopg = ["psycopg>=3.0.7"]
postgresql_psycopg = %(postgresql-psycopg)s

then running pip install -e .[postgresql_psycopg] makes pip raise pip._vendor.tomli.TOMLDecodeError: Invalid value so that does not work.

trying

postgresql-psycopg = ["psycopg>=3.0.7"]
postgresql_psycopg = "%(postgresql-psycopg)s"

makes pip raise configuration error: `project.optional-dependencies.postgresql_psycopg` must be array

trying

postgresql-psycopg = ["psycopg>=3.0.7"]
postgresql_psycopg = [".[postgresql-psycopg]"]

or

postgresql-psycopg = ["psycopg>=3.0.7"]
postgresql_psycopg = ["%(postgresql-psycopg)s"]

raises instead configuration error: `project.optional-dependencies.postgresql_psycopg[0]` must be pep508

suggestions?

zzzeek · 2023-10-16T21:20:23Z

doesnt make much sense if the two peps conflict? there has to be some simple way to do this

CaselIT · 2023-10-16T21:24:36Z

doesnt make much sense if the two peps conflict? there has to be some simple way to do this

the problem is the alias. There isn't any problem is we do

postgresql-psycopg = ["psycopg>=3.0.7"]
postgresql_psycopg = ["psycopg>=3.0.7"]

If we find no way of defining an alias that would work too, but should be noted in the other PR

zzzeek · 2023-10-16T21:27:20Z

OK then we just define the a-b and a_b names direct to the thing they reference

the goal is:

a-b names point to their thing

a_b names are down at the bottom, pointing to whatever

CaselIT · 2023-10-16T22:33:01Z

works for me. will leave a note on the other PR

zzzeek

@CaselIT just to confirm you want to change these names right now even though we're not on pyproject.toml yet, right? maybe we can just wait until that time comes to change it further?

zzzeek · 2023-10-17T13:33:39Z

setup.cfg

@@ -82,6 +82,30 @@ aiosqlite =
 sqlcipher =
    sqlcipher3_binary

+# legacy pre-pep-685 names
+mariadb_connector =


so, towards us being able to move to a pyproject.toml file easily, the request here is to not use the aliases for these legacy names and just point them to their ultimate dependencies directly (yes, duplicating what we have up above).

the goal here is that this batch of names is moved out to the margins where it can be removed at some point (maybe even 2.1, but maybe 2.2, etc) all at once.

What about items like postgresql-asyncpg?
Prior to this PR that was written as:

postgresql_asyncpg = %(asyncio)s asyncpg

Do we want this?

postgresql_asyncpg = %(asyncio)s asyncpg postgresql-asyncpg = %(asyncio)s asyncpg

Or this?

postgresql-asyncpg = greenlet!=0.4.17 asyncpg postgresql_asyncpg = greenlet!=0.4.17 asyncpg

%(asyncio)s is also referenced in 3 "single-word" extras:

aiomysql

asyncmy

aiosqlite

so from my point of view we can delay all the "remove the alias" to the PR that migrates to pyproject. As far as I know the toml version does not support aliases, but maybe there is a way that I'm not aware of

CaselIT · 2023-10-17T14:55:26Z

@CaselIT just to confirm you want to change these names right now even though we're not on pyproject.toml yet, right? maybe we can just wait until that time comes to change it further?

fine either way for me, I've left a comment in the other PR to do that change so we can delay it until then!

CaselIT · 2023-10-17T15:14:43Z

maybe hope is not lost: pypa/setuptools#3214 (comment)

if I understand correctly we can use (I haven't tried it)

postgresql-psycopg = ["psycopg>=3.0.7"]
postgresql_psycopg = ["sqlalchemy[postgresql-psycopg]"]

zzzeek · 2023-10-24T14:41:45Z

I'm still on the fence if we should be looking at this for 2.0. it seems a little soon, we could instead move right to 2.1 with pyproject.toml and just do the whole thing once. thoughts?

CaselIT · 2023-10-24T17:54:27Z

that works too for me

CaselIT · 2023-12-08T11:50:00Z

This change here will be merged together with #9324, since it seems that when defined in pyproject a the extra foo-bar is automatically used when installing [foo_bar] and having only an extra foo_bar does not seem to work, at least in tox

sqla-tester · 2023-12-08T11:55:05Z

Federico Caselli (CaselIT) wrote:

will be merged with https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/5028 since otherwise tox foes not work

View this in Gerrit at https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4906

sqla-tester · 2023-12-08T11:55:06Z

Gerrit review https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4906 has been abandoned. That means that at least for the moment I need to close this pull request. Sorry it didn't work out :(

Move the metadata for `setuptools` into `PEP 621`-compliant `pyproject.toml`. Use PEP-685 extras, keeping the old names for backward compatibility. Closes: sqlalchemy#9324 Closes: sqlalchemy#10481 Pull-request: sqlalchemy#9324 Change-Id: I14170e33a4a7370257d941adea4f96a39e785911

CaselIT · 2024-02-26T07:34:35Z

This was never backported to 2.0

I don't remember if we said it should

zzzeek requested changes Oct 16, 2023

View reviewed changes

mattoberle added 2 commits October 16, 2023 14:48

consolidate "legacy" extras

de01987

use "canonical" extras references for tox

2431850

zzzeek requested a review from sqla-tester October 16, 2023 18:54

sqla-tester reviewed Oct 16, 2023

View reviewed changes

CaselIT mentioned this pull request Oct 16, 2023

Move the metadata for setuptools into PEP 621-compliant pyproject.toml #9324

Closed

2 tasks

zzzeek requested changes Oct 17, 2023

View reviewed changes

CaselIT added this to the 2.1 milestone Oct 24, 2023

Merge branch 'main' into PEP-685

0d0db34

sqla-tester closed this Dec 8, 2023

CaselIT reopened this Dec 8, 2023

sqlalchemy-bot closed this in 765ac5f Jan 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide PEP-685-compliant extras #10481

Provide PEP-685-compliant extras #10481

mattoberle commented Oct 16, 2023

zzzeek left a comment

zzzeek commented Oct 16, 2023

sqla-tester left a comment

sqla-tester commented Oct 16, 2023

mattoberle commented Oct 16, 2023

CaselIT commented Oct 16, 2023

sqla-tester commented Oct 16, 2023

zzzeek commented Oct 16, 2023

CaselIT commented Oct 16, 2023

zzzeek commented Oct 16, 2023

CaselIT commented Oct 16, 2023 •

edited

Loading

zzzeek commented Oct 16, 2023

CaselIT commented Oct 16, 2023

zzzeek commented Oct 16, 2023

CaselIT commented Oct 16, 2023

zzzeek left a comment

zzzeek Oct 17, 2023

mattoberle Oct 17, 2023

CaselIT Oct 17, 2023

CaselIT commented Oct 17, 2023

CaselIT commented Oct 17, 2023 •

edited

Loading

zzzeek commented Oct 24, 2023

CaselIT commented Oct 24, 2023

CaselIT commented Dec 8, 2023

sqla-tester commented Dec 8, 2023

sqla-tester commented Dec 8, 2023

CaselIT commented Feb 26, 2024

Provide PEP-685-compliant extras #10481

Provide PEP-685-compliant extras #10481

Conversation

mattoberle commented Oct 16, 2023

Description

Checklist

zzzeek left a comment

Choose a reason for hiding this comment

zzzeek commented Oct 16, 2023

sqla-tester left a comment

Choose a reason for hiding this comment

sqla-tester commented Oct 16, 2023

mattoberle commented Oct 16, 2023

CaselIT commented Oct 16, 2023

sqla-tester commented Oct 16, 2023

zzzeek commented Oct 16, 2023

CaselIT commented Oct 16, 2023

zzzeek commented Oct 16, 2023

CaselIT commented Oct 16, 2023 • edited Loading

zzzeek commented Oct 16, 2023

CaselIT commented Oct 16, 2023

zzzeek commented Oct 16, 2023

CaselIT commented Oct 16, 2023

zzzeek left a comment

Choose a reason for hiding this comment

zzzeek Oct 17, 2023

Choose a reason for hiding this comment

mattoberle Oct 17, 2023

Choose a reason for hiding this comment

CaselIT Oct 17, 2023

Choose a reason for hiding this comment

CaselIT commented Oct 17, 2023

CaselIT commented Oct 17, 2023 • edited Loading

zzzeek commented Oct 24, 2023

CaselIT commented Oct 24, 2023

CaselIT commented Dec 8, 2023

sqla-tester commented Dec 8, 2023

sqla-tester commented Dec 8, 2023

CaselIT commented Feb 26, 2024

CaselIT commented Oct 16, 2023 •

edited

Loading

CaselIT commented Oct 17, 2023 •

edited

Loading