implement metadata per bind, refactor entire extension

[davidism]

This began as an attempt to fix the longstanding metadata per bind issue. Based on the pervasiveness of that change throughout the code, and on observations of the organization and maintainability of the code, I ended up refactoring the entire project. I've also rewritten all the tests and documentation. This PR addresses or invalidates the majority of current open issues and PRs. Below is a summary and reasoning for most of the changes, but check the changelog as well.

Compatibility

This will be Flask-SQLAlchemy 3.0. While the basic API (config, db.Model, db.session, etc.) remains the same, the rest of the code has changed significantly. I have tried to add in deprecation warnings where it was feasible. I will be making an alpha release based on this PR for users to try out and report any major incompatibilities that cannot be adapted to.

Many things have been moved to submodules and renamed. The signatures of some methods have changed, such as switching to keyword-only arguments. All setup methods that were used to create objects have been renamed with leading underscore, and are no longer considered public API. Therefore, projects that were subclassing or calling those methods to do advanced customization will encounter incompatibilities. Customization of all the objects managed by SQLAlchemy should be possible through arguments to __init__.

Static Typing

I added static type annotations to the entire library. For the most part, mypy --strict src passes. However, both mypy and pyright have behaviors that will frustrate users if I export the types. For example, mypy will not allow subclassing db.Model. Therefore, I am not enabling the annotations for other projects for now. SQLAlchemy 2 is adding their own static types, so I'd be happy to get help with this from the community.

Minimum Versions

• Python minimum version is 3.7. Python 2, 3.4, 3.5, and 3.6 is dropped.
• Flask minimum version is 2.2 to take advantage of new app context tracking and async contexts.
• SQLAlchemy minimum version is 1.4. 1.4 begins to change in preparation for the 2.0 release. Try running your project in SQLAlchemy 2 preview mode.

App Context

The extension no longer stores the app object if it is initialized as SQLAlchemy(app). An active app context is always required when using the session or engines. The session is scoped to the application context, rather than the current thread. By requiring an app context, this ensures that the session is always cleaned up after it is used, whether in a test, request, or CLI command.

Besides the correctness issue, I think a large amount of the confusion over when the app context was needed was because there were two different modes, and users might see an example assuming one mode while some other tutorial they followed used the other. Now, there is only one way to use it: with the context.

On a related note, Flask 2.2 changed how app contexts are cleaned up in debug mode. The old way may have prevented returning connections to the pool in debug mode, it will be interesting to see if users have fewer connection issues.

Metadata per Bind

One of the big issues in Flask-SQLAlchemy was that one metadata was used for all model and tables in all binds. This meant that tables in different engines could not have the same name, and caused other issues where things were expected to be more separated.

The extension now has a db.metadatas dict mapping bind keys to metadata instances. db.metadata is the default metadata like db.engine. When a model or table uses a bind key, it will create a new metadata object if it doesn't exist yet. The default metadata can still be customized with the metadata argument. Other metadatas will copy the default metadata's naming_convention.

db.engines is a dict mapping engines to bind keys, like db.metadatas. get_engine is deprecated, engines and metadatas should be looked up as the keys in the dicts instead if needed. There is no longer a separate "state" object with "connectors", the extension is stored directly in app.extensions["sqlalchemy"] and the engines are stored in a weakref dict for each app on the extension.

Engines are created during init_app instead of on first access. This is safe

This makes the implementation of create_all, delete_all, and reflect much simpler, as it can now just call the appropriate method on each metadata and engine pair, rather than on the same metadata with different engines. It's possible to inspect the tables for a bind by looking at db.metadatas[key].tables.

The bind_key is stored on each metadata.info. The session always looks up the key on the metadata, not on the model or table.

fixes #941

Pagination

• Pagination is iterable. fixes Support __iter__ on Pagination instances. #70
• iter_pages is more efficient. fixes Very poor performance of Pagination.{pages,iter_pages} #622
• Pagination.apply_to_query classmethod can be used instead of paginate.query.
• There is now documentation for using pagination.

Query

• Added query.one_or_404 to match the other 404 methods.
• Query is renamed from BaseQuery.
• db.relationship sets the query class on backrefs as well. fixes modifications to backref made inside _wrap_with_default_query_class(fn, cls) get lost #417

Note that SQLAlchemy 2 considers the query interface legacy. They won't continue to develop it and may deprecate it in the future. They are moving to using session.execute(select()) instead. I don't plan to deprecate db.Query, db.session.query, db.relationship query_class, or Model.query until they do, but I will be making a separate PR to implement the 404 and paginate methods for select() objects.

Session

• The session is scoped to the current application context instead of the thread. Therefore accessing the session requires an active application context. fixes session.remove called too early when there are stacked application contexts #508 fixes scoped_session’s scopefunc uses LocalStack() instead of AppContext() #944
• SignallingSession is renamed to Session
• get_bind more closely matches the base implementation.
• The session class can be customzied by passing session_options={"class_": SessionSubclass}. fixes Specify custom session class #327

Configuration

Init arguments are used to customize objects managed by the extension, which are common to all apps initialized on the extension. App config is used to customize the engines, which are specific to each app. Configuration is only read during init, it cannot be changed after init.

• engine_options argument sets defaults for all engines rather than overriding config.
• SQLALCHEMY_BINDS values can be dicts (instead of only connection strings) to set the engine options for a specific bind. In that case, it must have the "url" key in the options.
• SQLALCHEMY_ENGINE_OPTIONS only applies to the default bind, and overrides anything set by SQLALCHEMY_BINDS[None].
• Connection strings can be strings or SQLAlchemy URL objects.
• SQLALCHEMY_DATABASE_URI overrides a "url" in SQLALCHEMY_ENGINE_OPTIONS.
• SQLALCHEMY_ECHO sets pool_echo in addition to the engine's echo.

Other Changes

• get_debug_queries is renamed to get_recorded_queries to better describe what it returns. It now returns a dataclass instead of a namedtuple. The context attribute is renamed to location. Calculating location is more accurate for apps defined in subpackages.
• Names from sqlalchemy and sqlalchemy.orm are aliased using __getattr__ rather than setting tons of attributes on the extension object.
• Model __repr__ distinguishes between transient and pending objects. fixes Misleading __repr__ #967
• db.Table is a subclass instead of a function.
• The bind argument to some methods was renamed to bind_key to distinguish between the Flask-SQLAlchemy feature and SQLAlchemy's binds.
• Rewrote all the tests to use modern pytest patterns.
• Rewrote all the docs. Focused on the use of Flask-SQLAlchemy itself, rather than trying to be a SQLAlchemy tutorial.

Further Considerations

I'll make some more PRs for some issues this didn't address, since they could be made outside the main refactor. Additionally, I had the following notes on potential further changes. I'll make issues for these, just wanted to introduce them since it's what I noticed during this refactor.

• To support SQLAlchemy 2, we need to support pagination and 404 methods on select. After talking with SQLAlchemy maintainers, I think I'll add them as db.get_or_404, etc. since adding these methods to the session or result objects isn't intended.
• A pattern I use a lot is to write a little app.shell_context_processor to add db and all the model classes to the flask shell.
• Add a CLI command to call create_all. Should we also have drop_all with a confirmation?
• It would be possible to add some global proxies current_db and db_session.
• Should we deprecate everything related Query since it's legacy? Not yet, but I'll watch what SQLAlchemy does.
• We could make it easier to enable SQLAlchemy 2.0 preview mode, but I'm not sure it's worth it. Currently it can be done with engine_options={"future": True}, session_options={"future": True}.
• Pagination: Does it make sense for per_pages to allow 0? Does it make sense for max_per_page to have a default, like 100? Should pages be None not 0 if total is None or 0?
• Stop enabling SQLALCHEMY_RECORD_QUERIES by default in debug and testing mode. It only makes sense to enable it and incur the performance hit if you're using Flask-DebugToolbar or otherwise inspecting the info.

[jeffwidman]

Thanks for all the hard work here @davidism!

[davidism]

A prerelease with these changes, version 3.0.0a1, is now available on PyPI. Please use the prerelease to test your project ahead of the final release.

pip install -U --pre flask-sqlalchemy