CHANGELOG.rst

Release Notes

v1.0.0 (2023-01-24)

With this release gjson-py has reached pretty much feature parity with GJSON, with only some inherent minor caveats, differences and limitations due to the different programming languages and technologies used. With this release gjson-py can also be considered stable and production ready.

Minor improvements

• Modifiers: add support for missing upstream GJSON modifiers: @tostr, @fromstr, @group and @join.
• gjson: make __version__ more reliable:
  • Migrate the reading of the version from pkg_resources to importlib as it's more efficient and avoids an additional dependency.
  • If the version is not detectable, as a fallback, attempt to read the SETUPTOOLS_SCM_PRETEND_VERSION environment variable as documented in setuptools_scm.

Bug fixes

• I/O: allow control characters in all JSON inputs, loading the data with strict=False both for the JSON input and when parsing eventual JSON bits present in the query.
• gjson: fix some corner cases:
  • In some corner cases gjson was not parsing correctly the query string.
  • Introduce a new GJSONInvalidSyntaxError exception to distinguish some special parsing errors.
  • Extract the check for sequence objects into a private static method.

Miscellanea

• setup.py: add python_requires to prevent the package to be installed in an unsupported version of Python.
• setup.py: update classifiers.
• setup.py: mark the package as typed according to PEP 561.
• setup.py: mark Python 3.11 as officially supported.
• tox.ini: add Python 3.11 in the test matrix of tox.
• documentation: reduce the maxdepth property for the release notes page to not show all releases to avoid showing all the releases in the documentation home page.
• documentation: clarify limits on multipaths keys, specifying that in gjson-py if a key of a multipath object is given, it must be specified as a JSON string (e.g. "key":query) and that bare words (e.g. key:query) are not accepted although they are in GJSON.

v0.4.0 (2022-11-12)

New features

• Query parsing: add support for the GJSON Literals feature.
• Queries: add support for nested queries. There is no limit on the number of levels queries can be nested.

Bug fixes

• Queries: fix bug on first element query without operator:
  • When performing a query for the first element #(...) and there is no match, an exception should be raised.
  • In the case of a key existence check when there is no operator (e.g. friends.#(nonexistent)) an empty array was erroneously returned instead of raising an exception.

Miscellanea

• documentation: add missing method docstring.
• documentation: add note to modifiers specifying that the @keys and @values modifiers are valid only if applied to a JSON object (mapping).
• Query parsing: simplify internal method to find a matching parentheses for queries and multipaths.

v0.3.0 (2022-11-10)

New features

• Query parsing: add GJSON multipaths support.

Bug fixes

• Query parsing: fix integer mapping keys
  • When after a hash # or a query that returns all items #(...)#, if there is an integer key, return any matching key from the resulting items from the query if they match the integer key (as string, as per JSON specifications).

v0.2.1 (2022-10-25)

Bug fixes

• Query parsing: fix modifier options:
  • Fix a bug that was failing to properly get the modifier name when there were options and the modifier was not the first path in the query.

v0.2.0 (2022-10-24)

New features

• Query parsing: fully rewrite of the query parser:

  • Until now the parsing was mostly relying on a couple of regular expressions with lookbehind assertions to take into account escaped characters. Although they were working fine and also allowed to sensibly speed up the first development of gjson-py, they also had two major limitations:

    • Could not work in all corner cases.
    • Prevented the implementation of the GJSON features still missing in gjson-py.

  • The parsing has been completely refactored using a more standard parser approach, that allows to fine-tune the parsing much more to cover all corner cases and also enables the development of GJSON features still missing.

  • There shouldn't be any difference for normal queries, but some corner cases might now return a proper error.

  • Introduced a new GJSONParseError for parser-specific errors, that inherits from GJSONError and also provides a graphic way to show where the parsing error occurred. Example output:

    GJSONParseError: Invalid or unsupported query part `invalid`.
    Query: name.last.invalid
    -----------------^
    

Minor improvements

• Refactor code splitting it into multiple files:
  • Restructure the gjson code to split it into multiple files for ease of development and maintenance.
  • Keep all the split modules as private except the exceptions one, and re-export everything from the gjson module itself, both to keep backward compatibility and also for simplicity of use by the clients.
• Custom modifiers:
  • Prevent to register custom modifiers with names that contains characters that are used by the GJSON grammair, raising a GJSONError exception.

Miscellanea

• README: clarify naming for nested queries, based on feedback from issue #2. Also fix a typo.

v0.1.0 (2022-10-03)

Minor improvements

• Modifiers: add @top_n modifier (not present in GJSON):

• Add a @top_n modifier that optionally accepts as options the number of top common items to return: @top_n:{"n": 5}
• If no options are provided all items are returned.
• It requires a list of items as input and returns a dictionary with unique items as keys and the count of them as values.

• Modifiers: add @sum_n modifier (not present in GJSON):

• Add a @sum_n modifier that will work on a sequence of objects, grouping the items with the same value for a given grouping key and sum the values of a sum key for each of them.
• The options are mandatory and must specify the key to use for grouping and the key to use for summing: {"group": "somekey", "sum": "anotherkey"}. Optionally specifying the n parameter to just return the top N results based on the summed value: {"group": "somekey", "sum": "anotherkey", "n": 5}
• It requires a list of objects as input and returns a dictionary with unique items as keys and the sum of their values as values.

Bug fixes

• Output: fix unicode handling:
  • Fix the default behaviour ensuring non-ASCII characters are returned as-is.
  • Add a new modifier @ascii, that when set will escape all non-ASCII characters.
• CLI: fix encoding handling:
  • Use the surrogateescape Python mode when reading the input and back when printing the output to prevent failures when parsing the input and reducing the loss of data.

Miscellanea

• documentation: add mention to Debian packaging and the availability of Debian packages for the project.
• Type hints: use native types when possible. Instead of importing from typing use directly the native types when they support the [] syntax added in Python 3.9.
• documentation: refactor the modifiers documentation to clearly split the GJSON modifiers supported by gjson-py and the additional modifiers specific to gjson-py with more detailed explanation and example usage for the additional ones.
• setup.py: mark project as Beta for this v0.1.0 release and add an additional keyword for PyPI indexing.

v0.0.5 (2022-08-05)

New features

• Queries: add support for the tilde operator:
  • When performing queries on arrays, add support for the Go GJSON tilde operator to perform truthy-ness comparison.
  • The comparison is based on Python's definition of truthy-ness, hence the actual results might differ from the ones in the Go package.

Minor improvements

• documentation: add man page for the gjson binary.

v0.0.4 (2022-06-11)

New features

• CLI: improve the JSON Lines support allowing to use the -l/--lines CLI argument and the special query prefix .. syntax together to encapsulate each parsed line in an array to enable filtering using the Queries capabilities.

Minor improvements

• CLI: the input file CLI argument is now optional, defaulting to read from stdin. The equivalent of passing -.
• Modifiers: add support for the upstream Go GJSON modifier @this, that just returns the current object.

Miscellanea

• Documentation: add a section to with examples on how to use the CLI.
• CLI: add a link at the bottom of the help message of the CLI to the online documentation.

v0.0.3 (2022-06-11)

New features

• Add CLI support for JSON Lines:
  • Add a -l/--lines CLI argument to specify that the input file/stream is made of one JSON per line.
  • When used, gjson applies the same query to all lines.
  • Based on the verbosity level the failing lines are completely ignored, an error message is printed to stderr or the execution is interrupted at the first error printing the full traceback.
• Add CLI support for GJSON JSON Lines queries:
  • Add support for the GJSON queries that encapsulates a JSON Lines input in an array when the query starts with .. so that they the data can be queries as if it was an array of objects in the CLI.
• Add support for custom modifiers:
  • Add a ModifierProtocol to describe the interface that custom modifiers callable need to have.
  • Add a register_modifier() method in the GJSON class to register custom modifiers.
  • Allow to pass a dictionary of modifiers to the low-level GJSONObj class constructor.
  • Add a GJSONObj.builtin_modifiers() static method that returns a set with the names of the built-in modifiers.
  • Is not possible to register a custom modifier with the same name of a built-in modifier.
  • Clarify in the documentation that only JSON objects are accepted as modifier arguments.

Bug fixes

• Query parsing: when using the queries GJSON syntax #(...) and #(...)# fix the return value in case of a key matching that doesn't match any element.
• Query parsing fixes/improvements found with the Python fuzzing engine Atheris:
  • If any query parts between delimiters is empty error out with a specific message instead of hitting a generic IndexError.
  • When a query has an integer index on a mapping object, in case the element is not present, raise a GJSONError instead of a KeyError one.
  • When the query has a wildcard matching, ensure that it's applied on a mapping object. Fail with a GJSONError otherwise.
  • Explicitly catch malformed modifier options and raise a GJSONError instead.
  • If the last part of the query is a #, check that the object is actually a sequence like object and fail with a specific message if not.
  • Ensure all the conditions are valid before attempting to extract the inner element of a sequence like object. Ignore both non-mapping like objects inside the sequence or mapping like objects that don't have the specified key.
  • When parsing the query value as JSON catch the eventual decoding error to encapsulate it into a GJSONError one.
  • When using the queries GJSON syntax #(...) and #(...)# accept also an empty query to follow the same behaviour of the upstream Go GJSON.
  • When using the queries GJSON syntax #(...) and #(...)# follow closely the upstream behaviour of Go GJSON for all items queries #(..)# with regex matching.
  • When using the queries GJSON syntax #(...) and #(...)# fix the wildcard matching regular expression when using pattern matching.
  • Fix the regex to match keys in presence of wildcards escaping only the non-wildcards and ensuring to not double-escaping any already escaped wildcard.
  • When using the queries GJSON syntax #(...) and #(...)# ensure any exception raised while comparing incompatible objects is catched and raise as a GJSONError.

Miscellanea

• tests: when matching exception messages always escape the string or use raw strings to avoid false matchings.
• pylint: remove unnecessary comments

v0.0.2 (2022-05-31)

Bug fixes

• @sort modifier: fix the actual sorting.
• tests: ensure that mapping-like objects are compared also in the order of their keys.

Miscellanea

• GitHub actions: add workflow to run tox.
• GitHub actions: fix branch name for pushes
• documentation: include also the @sort modifier that is not present in the GJSON project.
• documentation: fix link to PyPI package.
• documentation: add link to the generated docs.
• documentation: fix section hierarchy and build.

v0.0.1 (2022-05-22)

• Initial version.