diff --git a/CHANGES.rst b/CHANGES.rst index ac15f4fcb0..050acc6c25 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -6,6 +6,12 @@ Version 2.0.0 Unreleased - Drop support for Python 2 and 3.5. +- JSON support no longer uses simplejson. To use another JSON module, + override ``app.json_encoder`` and ``json_decoder``. :issue:`3555` +- The ``encoding`` option to JSON functions is deprecated. :pr:`3562` +- Passing ``script_info`` to app factory functions is deprecated. This + was not portable outside the ``flask`` command. Use + ``click.get_current_context().obj`` if it's needed. :issue:`3552` - Add :meth:`sessions.SessionInterface.get_cookie_name` to allow setting the session cookie name dynamically. :pr:`3369` - Add :meth:`Config.from_file` to load config using arbitrary file @@ -19,6 +25,8 @@ Unreleased 200 OK and an empty file. :issue:`3358` - When using ad-hoc certificates, check for the cryptography library instead of PyOpenSSL. :pr:`3492` +- When specifying a factory function with ``FLASK_APP``, keyword + argument can be passed. :issue:`3553` - When loading a ``.env`` or ``.flaskenv`` file on top level directory, Flask will not change current work directory to the location of dotenv files, in order to prevent potential confusion. :pr:`3560` diff --git a/docs/api.rst b/docs/api.rst index 801a65bd1a..576e44df9f 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -31,40 +31,6 @@ Incoming Request Data :inherited-members: :exclude-members: json_module - .. attribute:: environ - - The underlying WSGI environment. - - .. attribute:: path - .. attribute:: full_path - .. attribute:: script_root - .. attribute:: url - .. attribute:: base_url - .. attribute:: url_root - - Provides different ways to look at the current :rfc:`3987`. - Imagine your application is listening on the following application - root:: - - http://www.example.com/myapplication - - And a user requests the following URI:: - - http://www.example.com/myapplication/%CF%80/page.html?x=y - - In this case the values of the above mentioned attributes would be - the following: - - ============= ====================================================== - `path` ``'/π/page.html'`` - `full_path` ``'/π/page.html?x=y'`` - `script_root` ``'/myapplication'`` - `base_url` ``'http://www.example.com/myapplication/π/page.html'`` - `url` ``'http://www.example.com/myapplication/π/page.html?x=y'`` - `url_root` ``'http://www.example.com/myapplication/'`` - ============= ====================================================== - - .. attribute:: request To access incoming request data, you can use the global `request` @@ -279,58 +245,34 @@ Message Flashing .. autofunction:: get_flashed_messages + JSON Support ------------ .. module:: flask.json -Flask uses ``simplejson`` for the JSON implementation. Since simplejson -is provided by both the standard library as well as extension, Flask will -try simplejson first and then fall back to the stdlib json module. On top -of that it will delegate access to the current application's JSON encoders -and decoders for easier customization. - -So for starters instead of doing:: - - try: - import simplejson as json - except ImportError: - import json - -You can instead just do this:: - - from flask import json +Flask uses the built-in :mod:`json` module for handling JSON. It will +use the current blueprint's or application's JSON encoder and decoder +for easier customization. By default it handles some extra data types: -For usage examples, read the :mod:`json` documentation in the standard -library. The following extensions are by default applied to the stdlib's -JSON module: +- :class:`datetime.datetime` and :class:`datetime.date` are serialized + to :rfc:`822` strings. This is the same as the HTTP date format. +- :class:`uuid.UUID` is serialized to a string. +- :class:`dataclasses.dataclass` is passed to + :func:`dataclasses.asdict`. +- :class:`~markupsafe.Markup` (or any object with a ``__html__`` + method) will call the ``__html__`` method to get a string. -1. ``datetime`` objects are serialized as :rfc:`822` strings. -2. Any object with an ``__html__`` method (like :class:`~flask.Markup`) - will have that method called and then the return value is serialized - as string. - -The :func:`~htmlsafe_dumps` function of this json module is also available -as a filter called ``|tojson`` in Jinja2. Note that in versions of Flask prior -to Flask 0.10, you must disable escaping with ``|safe`` if you intend to use -``|tojson`` output inside ``script`` tags. In Flask 0.10 and above, this -happens automatically (but it's harmless to include ``|safe`` anyway). +:func:`~htmlsafe_dumps` is also available as the ``|tojson`` template +filter. The filter marks the output with ``|safe`` so it can be used +inside ``script`` tags. .. sourcecode:: html+jinja -.. admonition:: Auto-Sort JSON Keys - - The configuration variable :data:`JSON_SORT_KEYS` can be set to - ``False`` to stop Flask from auto-sorting keys. By default sorting - is enabled and outside of the app context sorting is turned on. - - Notice that disabling key sorting can cause issues when using - content based HTTP caches and Python's hash randomization feature. - .. autofunction:: jsonify .. autofunction:: dumps @@ -349,6 +291,7 @@ happens automatically (but it's harmless to include ``|safe`` anyway). .. automodule:: flask.json.tag + Template Rendering ------------------ diff --git a/docs/cli.rst b/docs/cli.rst index d99742a6ae..abcfb7c66d 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -75,12 +75,8 @@ Within the given import, the command looks for an application instance named found, the command looks for a factory function named ``create_app`` or ``make_app`` that returns an instance. -When calling an application factory, if the factory takes an argument named -``script_info``, then the :class:`~cli.ScriptInfo` instance is passed as a -keyword argument. If the application factory takes only one argument and no -parentheses follow the factory name, the :class:`~cli.ScriptInfo` instance -is passed as a positional argument. If parentheses follow the factory name, -their contents are parsed as Python literals and passes as arguments to the +If parentheses follow the factory name, their contents are parsed as +Python literals and passed as arguments and keyword arguments to the function. This means that strings must still be in quotes. diff --git a/docs/installation.rst b/docs/installation.rst index c99d82cf01..e02e111eec 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -39,16 +39,12 @@ These distributions will not be installed automatically. Flask will detect and use them if you install them. * `Blinker`_ provides support for :doc:`signals`. -* `SimpleJSON`_ is a fast JSON implementation that is compatible with - Python's ``json`` module. It is preferred for JSON operations if it is - installed. * `python-dotenv`_ enables support for :ref:`dotenv` when running ``flask`` commands. * `Watchdog`_ provides a faster, more efficient reloader for the development server. .. _Blinker: https://pythonhosted.org/blinker/ -.. _SimpleJSON: https://simplejson.readthedocs.io/ .. _python-dotenv: https://github.com/theskumar/python-dotenv#readme .. _watchdog: https://pythonhosted.org/watchdog/ diff --git a/docs/requirements.txt b/docs/requirements.txt index b8e76e4556..cadd40cd62 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1,5 +1,5 @@ -Sphinx~=2.2.0 -Pallets-Sphinx-Themes~=1.2.2 +Sphinx~=3.0.0 +Pallets-Sphinx-Themes~=1.2.3 sphinxcontrib-log-cabinet~=1.0.1 sphinx-issues~=1.2.0 -packaging~=19.2 +packaging~=20.3 diff --git a/src/flask/cli.py b/src/flask/cli.py index 29baf005ae..caf0dfeaee 100644 --- a/src/flask/cli.py +++ b/src/flask/cli.py @@ -5,6 +5,7 @@ import re import sys import traceback +import warnings from functools import update_wrapper from operator import attrgetter from threading import Lock @@ -85,90 +86,124 @@ def find_best_app(script_info, module): ) -def call_factory(script_info, app_factory, arguments=()): +def call_factory(script_info, app_factory, args=None, kwargs=None): """Takes an app factory, a ``script_info` object and optionally a tuple of arguments. Checks for the existence of a script_info argument and calls the app_factory depending on that and the arguments provided. """ - args_spec = inspect.getfullargspec(app_factory) - arg_names = args_spec.args - arg_defaults = args_spec.defaults + sig = inspect.signature(app_factory) + args = [] if args is None else args + kwargs = {} if kwargs is None else kwargs + + if "script_info" in sig.parameters: + warnings.warn( + "The 'script_info' argument is deprecated and will not be" + " passed to the app factory function in 2.1.", + DeprecationWarning, + ) + kwargs["script_info"] = script_info - if "script_info" in arg_names: - return app_factory(*arguments, script_info=script_info) - elif arguments: - return app_factory(*arguments) - elif not arguments and len(arg_names) == 1 and arg_defaults is None: - return app_factory(script_info) + if ( + not args + and len(sig.parameters) == 1 + and next(iter(sig.parameters.values())).default is inspect.Parameter.empty + ): + warnings.warn( + "Script info is deprecated and will not be passed as the" + " single argument to the app factory function in 2.1.", + DeprecationWarning, + ) + args.append(script_info) - return app_factory() + return app_factory(*args, **kwargs) -def _called_with_wrong_args(factory): +def _called_with_wrong_args(f): """Check whether calling a function raised a ``TypeError`` because the call failed or because something in the factory raised the error. - :param factory: the factory function that was called - :return: true if the call failed + :param f: The function that was called. + :return: ``True`` if the call failed. """ tb = sys.exc_info()[2] try: while tb is not None: - if tb.tb_frame.f_code is factory.__code__: - # in the factory, it was called successfully + if tb.tb_frame.f_code is f.__code__: + # In the function, it was called successfully. return False tb = tb.tb_next - # didn't reach the factory + # Didn't reach the function. return True finally: - # explicitly delete tb as it is circular referenced + # Delete tb to break a circular reference. # https://docs.python.org/2/library/sys.html#sys.exc_info del tb def find_app_by_string(script_info, module, app_name): - """Checks if the given string is a variable name or a function. If it is a - function, it checks for specified arguments and whether it takes a - ``script_info`` argument and calls the function with the appropriate - arguments. + """Check if the given string is a variable name or a function. Call + a function to get the app instance, or return the variable directly. """ from . import Flask - match = re.match(r"^ *([^ ()]+) *(?:\((.*?) *,? *\))? *$", app_name) - - if not match: + # Parse app_name as a single expression to determine if it's a valid + # attribute name or function call. + try: + expr = ast.parse(app_name.strip(), mode="eval").body + except SyntaxError: raise NoAppException( - f"{app_name!r} is not a valid variable name or function expression." + f"Failed to parse {app_name!r} as an attribute name or function call." ) - name, args = match.groups() + if isinstance(expr, ast.Name): + name = expr.id + args = kwargs = None + elif isinstance(expr, ast.Call): + # Ensure the function name is an attribute name only. + if not isinstance(expr.func, ast.Name): + raise NoAppException( + f"Function reference must be a simple name: {app_name!r}." + ) + + name = expr.func.id + + # Parse the positional and keyword arguments as literals. + try: + args = [ast.literal_eval(arg) for arg in expr.args] + kwargs = {kw.arg: ast.literal_eval(kw.value) for kw in expr.keywords} + except ValueError: + # literal_eval gives cryptic error messages, show a generic + # message with the full expression instead. + raise NoAppException( + f"Failed to parse arguments as literal values: {app_name!r}." + ) + else: + raise NoAppException( + f"Failed to parse {app_name!r} as an attribute name or function call." + ) try: attr = getattr(module, name) - except AttributeError as e: - raise NoAppException(e.args[0]) + except AttributeError: + raise NoAppException( + f"Failed to find attribute {name!r} in {module.__name__!r}." + ) + # If the attribute is a function, call it with any args and kwargs + # to get the real application. if inspect.isfunction(attr): - if args: - try: - args = ast.literal_eval(f"({args},)") - except (ValueError, SyntaxError): - raise NoAppException(f"Could not parse the arguments in {app_name!r}.") - else: - args = () - try: - app = call_factory(script_info, attr, args) - except TypeError as e: + app = call_factory(script_info, attr, args, kwargs) + except TypeError: if not _called_with_wrong_args(attr): raise raise NoAppException( - f"{e}\nThe factory {app_name!r} in module" + f"The factory {app_name!r} in module" f" {module.__name__!r} could not be called with the" " specified arguments." ) @@ -355,8 +390,6 @@ def load_app(self): if self._loaded_app is not None: return self._loaded_app - app = None - if self.create_app is not None: app = call_factory(self, self.create_app) else: diff --git a/src/flask/json/__init__.py b/src/flask/json/__init__.py index 5c698ef0ef..6ef22e2658 100644 --- a/src/flask/json/__init__.py +++ b/src/flask/json/__init__.py @@ -1,11 +1,11 @@ -import codecs import io +import json as _json import uuid +import warnings from datetime import date from datetime import datetime -from itsdangerous import json as _json -from jinja2 import Markup +from markupsafe import Markup from werkzeug.http import http_date from ..globals import current_app @@ -17,66 +17,29 @@ # Python < 3.7 dataclasses = None -# Figure out if simplejson escapes slashes. This behavior was changed -# from one version to another without reason. -_slash_escape = "\\/" not in _json.dumps("/") - - -__all__ = [ - "dump", - "dumps", - "load", - "loads", - "htmlsafe_dump", - "htmlsafe_dumps", - "JSONDecoder", - "JSONEncoder", - "jsonify", -] - - -def _wrap_reader_for_text(fp, encoding): - if isinstance(fp.read(0), bytes): - fp = io.TextIOWrapper(io.BufferedReader(fp), encoding) - return fp - - -def _wrap_writer_for_text(fp, encoding): - try: - fp.write("") - except TypeError: - fp = io.TextIOWrapper(fp, encoding) - return fp - class JSONEncoder(_json.JSONEncoder): - """The default Flask JSON encoder. This one extends the default - encoder by also supporting ``datetime``, ``UUID``, ``dataclasses``, - and ``Markup`` objects. - - ``datetime`` objects are serialized as RFC 822 datetime strings. - This is the same as the HTTP date format. - - In order to support more data types, override the :meth:`default` - method. + """The default JSON encoder. Handles extra types compared to the + built-in :class:`json.JSONEncoder`. + + - :class:`datetime.datetime` and :class:`datetime.date` are + serialized to :rfc:`822` strings. This is the same as the HTTP + date format. + - :class:`uuid.UUID` is serialized to a string. + - :class:`dataclasses.dataclass` is passed to + :func:`dataclasses.asdict`. + - :class:`~markupsafe.Markup` (or any object with a ``__html__`` + method) will call the ``__html__`` method to get a string. + + Assign a subclass of this to :attr:`flask.Flask.json_encoder` or + :attr:`flask.Blueprint.json_encoder` to override the default. """ def default(self, o): - """Implement this method in a subclass such that it returns a - serializable object for ``o``, or calls the base implementation (to - raise a :exc:`TypeError`). - - For example, to support arbitrary iterators, you could implement - default like this:: - - def default(self, o): - try: - iterable = iter(o) - except TypeError: - pass - else: - return list(iterable) - return JSONEncoder.default(self, o) + """Convert ``o`` to a JSON serializable type. See + :meth:`json.JSONEncoder.default`. Python does not support + overriding how basic types like ``str`` or ``list`` are + serialized, they are handled before this method. """ if isinstance(o, datetime): return http_date(o.utctimetuple()) @@ -88,14 +51,17 @@ def default(self, o): return dataclasses.asdict(o) if hasattr(o, "__html__"): return str(o.__html__()) - return _json.JSONEncoder.default(self, o) + return super().default(self, o) class JSONDecoder(_json.JSONDecoder): - """The default JSON decoder. This one does not change the behavior from - the default simplejson decoder. Consult the :mod:`json` documentation - for more information. This decoder is not only used for the load - functions of this module but also :attr:`~flask.Request`. + """The default JSON decoder. + + This does not change any behavior from the built-in + :class:`json.JSONDecoder`. + + Assign a subclass of this to :attr:`flask.Flask.json_decoder` or + :attr:`flask.Blueprint.json_decoder` to override the default. """ @@ -106,13 +72,9 @@ def _dump_arg_defaults(kwargs, app=None): if app: bp = app.blueprints.get(request.blueprint) if request else None - kwargs.setdefault( - "cls", bp.json_encoder if bp and bp.json_encoder else app.json_encoder - ) - - if not app.config["JSON_AS_ASCII"]: - kwargs.setdefault("ensure_ascii", False) - + cls = bp.json_encoder if bp and bp.json_encoder else app.json_encoder + kwargs.setdefault("cls", cls) + kwargs.setdefault("ensure_ascii", app.config["JSON_AS_ASCII"]) kwargs.setdefault("sort_keys", app.config["JSON_SORT_KEYS"]) else: kwargs.setdefault("sort_keys", True) @@ -126,223 +88,235 @@ def _load_arg_defaults(kwargs, app=None): if app: bp = app.blueprints.get(request.blueprint) if request else None - kwargs.setdefault( - "cls", bp.json_decoder if bp and bp.json_decoder else app.json_decoder - ) + cls = bp.json_decoder if bp and bp.json_decoder else app.json_decoder + kwargs.setdefault("cls", cls) else: kwargs.setdefault("cls", JSONDecoder) -def detect_encoding(data): - """Detect which UTF codec was used to encode the given bytes. - - The latest JSON standard (:rfc:`8259`) suggests that only UTF-8 is - accepted. Older documents allowed 8, 16, or 32. 16 and 32 can be big - or little endian. Some editors or libraries may prepend a BOM. - - :param data: Bytes in unknown UTF encoding. - :return: UTF encoding name - """ - head = data[:4] - - if head[:3] == codecs.BOM_UTF8: - return "utf-8-sig" - - if b"\x00" not in head: - return "utf-8" - - if head in (codecs.BOM_UTF32_BE, codecs.BOM_UTF32_LE): - return "utf-32" - - if head[:2] in (codecs.BOM_UTF16_BE, codecs.BOM_UTF16_LE): - return "utf-16" - - if len(head) == 4: - if head[:3] == b"\x00\x00\x00": - return "utf-32-be" - - if head[::2] == b"\x00\x00": - return "utf-16-be" - - if head[1:] == b"\x00\x00\x00": - return "utf-32-le" - - if head[1::2] == b"\x00\x00": - return "utf-16-le" - - if len(head) == 2: - return "utf-16-be" if head.startswith(b"\x00") else "utf-16-le" - - return "utf-8" - - def dumps(obj, app=None, **kwargs): - """Serialize ``obj`` to a JSON-formatted string. If there is an - app context pushed, use the current app's configured encoder - (:attr:`~flask.Flask.json_encoder`), or fall back to the default - :class:`JSONEncoder`. + """Serialize an object to a string of JSON. - Takes the same arguments as the built-in :func:`json.dumps`, and - does some extra configuration based on the application. If the - simplejson package is installed, it is preferred. + Takes the same arguments as the built-in :func:`json.dumps`, with + some defaults from application configuration. :param obj: Object to serialize to JSON. - :param app: App instance to use to configure the JSON encoder. - Uses ``current_app`` if not given, and falls back to the default - encoder when not in an app context. - :param kwargs: Extra arguments passed to :func:`json.dumps`. + :param app: Use this app's config instead of the active app context + or defaults. + :param kwargs: Extra arguments passed to func:`json.dumps`. - .. versionchanged:: 1.0.3 + .. versionchanged:: 2.0 + ``encoding`` is deprecated and will be removed in 2.1. + .. versionchanged:: 1.0.3 ``app`` can be passed directly, rather than requiring an app context for configuration. """ _dump_arg_defaults(kwargs, app=app) encoding = kwargs.pop("encoding", None) rv = _json.dumps(obj, **kwargs) - if encoding is not None and isinstance(rv, str): - rv = rv.encode(encoding) + + if encoding is not None: + warnings.warn( + "'encoding' is deprecated and will be removed in 2.1.", + DeprecationWarning, + stacklevel=2, + ) + + if isinstance(rv, str): + return rv.encode(encoding) + return rv def dump(obj, fp, app=None, **kwargs): - """Like :func:`dumps` but writes into a file object.""" + """Serialize an object to JSON written to a file object. + + Takes the same arguments as the built-in :func:`json.dump`, with + some defaults from application configuration. + + :param obj: Object to serialize to JSON. + :param fp: File object to write JSON to. + :param app: Use this app's config instead of the active app context + or defaults. + :param kwargs: Extra arguments passed to func:`json.dump`. + + .. versionchanged:: 2.0 + Writing to a binary file, and the ``encoding`` argument, is + deprecated and will be removed in 2.1. + """ _dump_arg_defaults(kwargs, app=app) encoding = kwargs.pop("encoding", None) - if encoding is not None: - fp = _wrap_writer_for_text(fp, encoding) + show_warning = encoding is not None + + try: + fp.write("") + except TypeError: + show_warning = True + fp = io.TextIOWrapper(fp, encoding or "utf-8") + + if show_warning: + warnings.warn( + "Writing to a binary file, and the 'encoding' argument, is" + " deprecated and will be removed in 2.1.", + DeprecationWarning, + stacklevel=2, + ) + _json.dump(obj, fp, **kwargs) def loads(s, app=None, **kwargs): - """Deserialize an object from a JSON-formatted string ``s``. If - there is an app context pushed, use the current app's configured - decoder (:attr:`~flask.Flask.json_decoder`), or fall back to the - default :class:`JSONDecoder`. + """Deserialize an object from a string of JSON. - Takes the same arguments as the built-in :func:`json.loads`, and - does some extra configuration based on the application. If the - simplejson package is installed, it is preferred. + Takes the same arguments as the built-in :func:`json.loads`, with + some defaults from application configuration. :param s: JSON string to deserialize. - :param app: App instance to use to configure the JSON decoder. - Uses ``current_app`` if not given, and falls back to the default - encoder when not in an app context. - :param kwargs: Extra arguments passed to :func:`json.dumps`. + :param app: Use this app's config instead of the active app context + or defaults. + :param kwargs: Extra arguments passed to func:`json.dump`. - .. versionchanged:: 1.0.3 + .. versionchanged:: 2.0 + ``encoding`` is deprecated and will be removed in 2.1. The data + must be a string or UTF-8 bytes. + .. versionchanged:: 1.0.3 ``app`` can be passed directly, rather than requiring an app context for configuration. """ _load_arg_defaults(kwargs, app=app) - if isinstance(s, bytes): - encoding = kwargs.pop("encoding", None) - if encoding is None: - encoding = detect_encoding(s) - s = s.decode(encoding) + encoding = kwargs.pop("encoding", None) + + if encoding is not None: + warnings.warn( + "'encoding' is deprecated and will be removed in 2.1. The" + " data must be a string or UTF-8 bytes.", + DeprecationWarning, + stacklevel=2, + ) + + if isinstance(s, bytes): + s = s.decode(encoding) + return _json.loads(s, **kwargs) def load(fp, app=None, **kwargs): - """Like :func:`loads` but reads from a file object.""" + """Deserialize an object from JSON read from a file object. + + Takes the same arguments as the built-in :func:`json.load`, with + some defaults from application configuration. + + :param fp: File object to read JSON from. + :param app: Use this app's config instead of the active app context + or defaults. + :param kwargs: Extra arguments passed to func:`json.load`. + + .. versionchanged:: 2.0 + ``encoding`` is deprecated and will be removed in 2.1. The file + must be text mode, or binary mode with UTF-8 bytes. + """ _load_arg_defaults(kwargs, app=app) - fp = _wrap_reader_for_text(fp, kwargs.pop("encoding", None) or "utf-8") + encoding = kwargs.pop("encoding", None) + + if encoding is not None: + warnings.warn( + "'encoding' is deprecated and will be removed in 2.1. The" + " file must be text mode, or binary mode with UTF-8 bytes.", + DeprecationWarning, + stacklevel=2, + ) + + if isinstance(fp.read(0), bytes): + fp = io.TextIOWrapper(fp, encoding) + return _json.load(fp, **kwargs) -def htmlsafe_dumps(obj, **kwargs): - """Works exactly like :func:`dumps` but is safe for use in ``") assert rv == '"\\u003c/script\\u003e"' - assert type(rv) is str rv = render('{{ ""|tojson }}') assert rv == '"\\u003c/script\\u003e"' rv = render('{{ "<\0/script>"|tojson }}') diff --git a/tox.ini b/tox.ini index 6e1a9d95c4..e276c23124 100644 --- a/tox.ini +++ b/tox.ini @@ -1,7 +1,7 @@ [tox] envlist = py{38,37,36,py3} - py38-{simplejson,devel,lowest} + py38-{devel,lowest} style docs skip_missing_interpreters = true @@ -24,8 +24,6 @@ deps = devel: https://github.com/pallets/itsdangerous/archive/master.tar.gz devel: https://github.com/pallets/click/archive/master.tar.gz - simplejson: simplejson - commands = pip install -q -e examples/tutorial[test] pip install -q -e examples/javascript[test]