TOML parsing fails on add command if using non-inlined tables

[hannu-oura]

• Poetry version: 1.6.1
• Python version: 3.10
• OS version and name: Ubuntu 22.04
• pyproject.toml: https://gist.github.com/hannu-oura/fced16e1c38630a256c8b0e86549eec8

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

It looks like if we use pretty printed TOML (which is confirmed valid by TOML linter and also passes poetry check command) which contains tables that aren't in the inline format that poetry usually adds them in using poetry add to update those requirements can fail if the updated item is not the last item in the list.

Running the command poetry add "pytz==2023.3.post1 will work
Running the command poetry add "redis=^4.5.6" --extras=hiredis will fail with
'Null' object has no attribute '_trivia'

Reason seems to be the order, redis update will work if it's moved as the first item in the list and vice versa.

Full error log with -vvv:

$ poetry add -vvv "redis=^4.5.6" --extras=hiredis

Loading configuration file /home/hannu/.config/pypoetry/config.toml
Loading configuration file /home/hannu/.config/pypoetry/auth.toml
Using virtualenv: /home/hannu/.cache/pypoetry/virtualenvs/poetry-toml-issue-Dd81th1D-py3.10
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for pypi.org
Source (PyPI): 1 packages found for redis >=4.5.6,<5.0.0

  Stack trace:

  13  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  12  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  11  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

  10  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

   9  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   8  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

   7  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/cleo/commands/command.py:62 in execute
        60│ 
        61│         try:
     →  62│             return self.handle()
        63│         except KeyboardInterrupt:
        64│             return 1

   6  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/poetry/console/commands/add.py:233 in handle
       231│             for key in section:
       232│                 if canonicalize_name(key) == canonical_constraint_name:
     → 233│                     section[key] = constraint
       234│                     break
       235│             else:

   5  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/tomlkit/items.py:1518 in __setitem__
       1516│ 
       1517│         is_replace = key in self
     → 1518│         self._value[key] = value
       1519│ 
       1520│         if key is not None:

   4  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/tomlkit/container.py:630 in __setitem__
       628│         if key is not None and key in self:
       629│             old_key = next(filter(lambda k: k == key, self._map))
     → 630│             self._replace(old_key, key, value)
       631│         else:
       632│             self.append(key, value)

   3  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/tomlkit/container.py:649 in _replace
       647│             raise NonExistentKey(key)
       648│ 
     → 649│         self._replace_at(idx, new_key, value)
       650│ 
       651│     def _replace_at(

   2  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/tomlkit/container.py:682 in _replace_at
       680│             for i in range(idx, len(self._body)):
       681│                 if isinstance(self._body[i][1], (AoT, Table)):
     → 682│                     self._insert_at(i, new_key, value)
       683│                     idx = i
       684│                     break

   1  ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/tomlkit/container.py:412 in _insert_at
       410│                 or ends_with_whitespace(previous_item)
       411│                 or isinstance(item, (AoT, Table))
     → 412│                 or "\n" in previous_item.trivia.trail
       413│             ):
       414│                 previous_item.trivia.trail += "\n"

  AttributeError

  'Null' object has no attribute '_trivia'

  at ~/.local/pipx/venvs/poetry@1-6/lib/python3.10/site-packages/tomlkit/items.py:474 in trivia
       470│ 
       471│     @property
       472│     def trivia(self) -> Trivia:
       473│         """The trivia element associated with this item"""
    →  474│         return self._trivia
       475│ 
       476│     @property
       477│     def discriminant(self) -> int:
       478│         raise NotImplementedError()

[dimbleby]

probably wants reporting to tomlkit

[hannu-oura]

probably wants reporting to tomlkit

From the error alone it's not clear to me if this is indeed an issue in tomlkit or in the way poetry attempts to use tomlkit.
I haven't dug deep into the internals of poetry and without that it's very difficult for me to confirm if it's possible to replicate the same issue with just tomlkit alone.

[dimbleby]

From the error alone it's not clear to me if this is indeed an issue in tomlkit or in the way poetry attempts to use tomlkit.

tomlkit maintainers will be better able to provide guidance than poetry will

I haven't dug deep into the internals of poetry and without that it's very difficult for me to confirm if it's possible to replicate the same issue with just tomlkit alone.

I guess it's not that hard. I don't suppose poetry is doing more than (i) read the file (ii) replace the redis entry with a newer version of the same. Anyway if I cared to try and reproduce this with tomlkit only, that's where I'd start.

[dimbleby]

python-poetry/tomlkit#313

[dimbleby]

there has now been a tomlkit release with that fix, there is no reason to keep this open

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.