tpm2_ptool - Undefined sybol: TSS2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal #840

thxrben · 2023-05-25T08:53:00Z

Hey,
I wanted to use my TPM2 module for storing ssh keys, but I received an error from the tpm2-tools suite.

I ran "tpm2_ptool --help" and received the following stacktrace:

Traceback (most recent call last):
  File "/usr/bin/tpm2_ptool", line 5, in <module>
    from tpm2_pkcs11.tpm2_ptool import main
  File "/usr/lib/python3.11/site-packages/tpm2_pkcs11/tpm2_ptool.py", line 6, in <module>
    from .commandlets_store import InitCommand  # pylint: disable=unused-import # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/tpm2_pkcs11/commandlets_store.py", line 13, in <module>
    from .utils import bytes_to_file
  File "/usr/lib/python3.11/site-packages/tpm2_pkcs11/utils.py", line 21, in <module>
    from tpm2_pytss.ESAPI import ESAPI
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/__init__.py", line 2, in <module>
    from .ESAPI import ESAPI
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/ESAPI.py", line 2, in <module>
    from .types import *
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/types.py", line 12, in <module>
    from ._libtpm2_pytss import ffi, lib
ImportError: /usr/lib/python3.11/site-packages/tpm2_pytss/_libtpm2_pytss.abi3.so: undefined symbol: Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal

I added myself to the tss group and running the command using root does nothing else.
However, I did ran tpm2_clear before, tho I suspect that it is unrelated to the issue.

The text was updated successfully, but these errors were encountered:

retpolanne · 2023-06-04T11:48:00Z

Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal seems to exist on _libtpm2_pytss.abi3.so 🤔

I've uninstalled tpm2-pkcs11 and let only python-tpm2-pytss 2.1.0-1 installed on Arch and it seems that the error continues showing up even on the 2.1.0-1 version:

>>> from tpm2_pytss.ESAPI import ESAPI
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/__init__.py", line 2, in <module>
    from .ESAPI import ESAPI
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/ESAPI.py", line 2, in <module>
    from .types import *
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/types.py", line 12, in <module>
    from ._libtpm2_pytss import ffi, lib
ImportError: /usr/lib/python3.11/site-packages/tpm2_pytss/_libtpm2_pytss.abi3.so: undefined symbol: Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal

edit: my bad, it has a U in front of it, which means undefined

Seems related to tpm2-software/tpm2-pytss#496

workaround

Installing from pip fixes it :)

python3 -m pip install tpm2-pytss

However, it seems that, at least on Arch, the tpm2-pkcs11 package bundles the python one, so it overwrites it and breaks.

Fixing on Arch (at least)

Rebuilding python-tpm2-pytss seems to have fixed it.

sudo pacman -Rsn tpm2-pkcs11 python-tpm2-pytss
curl -sS https://gitlab.archlinux.org/archlinux/packaging/packages/tpm2-pkcs11/-/raw/main/keys/pgp/5B482B8E3E19DA7C978E1D016DE2E9078E1F50C1.asc\?inline\=false | gpg --import
git clone https://gitlab.archlinux.org/archlinux/packaging/packages/python-tpm2-pytss.git
cd python-tpm2-pytss
makepkg -si
sudo pacman -S tpm2-pkcs11

JuergenReppSIT · 2023-06-04T18:48:36Z

@whooo can you give advice on how to solve this problem?

whooo · 2023-06-04T20:29:16Z

Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal and Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Unmarshal where removed in tpm2-tss 4.0.0, so it looks like the tpm2-pytss package was built with an older release of tpm2-tss and then the tpm2-tss libraries where upgraded, can you check which version of tpm2-tss you have installed on your systems? How did you install tpm2-pytss?

williamcroberts · 2023-06-12T21:32:21Z

Wrong project, moving it over to tpm2-pkcs11

retpolanne · 2023-06-12T21:55:20Z

@whooo I did install mine from the tpm2-tss package on Arch. I don't recall the version but I can check when I reinstall it.

thxrben · 2023-06-13T08:13:41Z

Sorry, but I cannot test against this issue currently. On Arch it "just works"?!

I had the problems on Fedora, but the distro didnt work out.

But I guess @retpolanne did find a possible solution or workaround :)

williamcroberts · 2023-06-13T15:11:50Z

Also make sure you don't have multiple .so's on the system. I've seen this happens where it builds against the correct headers but links to a library that wasn't expected. ldd on the shared object will give you where its resolving all the dependencies.

paranormal · 2023-06-15T10:28:30Z

Hi, I can confirm the issuer is present on Arch as of now.

ImportError: /usr/lib/python3.11/site-packages/tpm2_pytss/_libtpm2_pytss.abi3.so: undefined symbol: Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal

Searching for leftovers with pacreport --unowned-files on affected system would show none of them.

ldd is fine too

~: ldd /usr/lib/python3.11/site-packages/tpm2_pytss/_libtpm2_pytss.abi3.so
	linux-vdso.so.1 (0x00007ffc4ffef000)
	libtss2-esys.so.0 => /usr/lib/libtss2-esys.so.0 (0x00007f92bce19000)
	libtss2-tctildr.so.0 => /usr/lib/libtss2-tctildr.so.0 (0x00007f92bce0f000)
	libtss2-rc.so.0 => /usr/lib/libtss2-rc.so.0 (0x00007f92bce05000)
	libtss2-mu.so.0 => /usr/lib/libtss2-mu.so.0 (0x00007f92bcdb5000)
	libtss2-fapi.so.1 => /usr/lib/libtss2-fapi.so.1 (0x00007f92bccb3000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f92bcac9000)
	libcrypto.so.3 => /usr/lib/libcrypto.so.3 (0x00007f92bc5ca000)
	libtss2-sys.so.1 => /usr/lib/libtss2-sys.so.1 (0x00007f92bc5a3000)
	/usr/lib64/ld-linux-x86-64.so.2 (0x00007f92bcff2000)
	libjson-c.so.5 => /usr/lib/libjson-c.so.5 (0x00007f92bc590000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0x00007f92bc4e0000)
	libuuid.so.1 => /usr/lib/libuuid.so.1 (0x00007f92bc4d7000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x00007f92bc4aa000)
	libidn2.so.0 => /usr/lib/libidn2.so.0 (0x00007f92bc488000)
	libssh2.so.1 => /usr/lib/libssh2.so.1 (0x00007f92bc446000)
	libpsl.so.5 => /usr/lib/libpsl.so.5 (0x00007f92bc432000)
	libssl.so.3 => /usr/lib/libssl.so.3 (0x00007f92bc392000)
	libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00007f92bc33e000)
	libzstd.so.1 => /usr/lib/libzstd.so.1 (0x00007f92bc269000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0x00007f92bc25b000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f92bc241000)
	libunistring.so.5 => /usr/lib/libunistring.so.5 (0x00007f92bc087000)
	libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00007f92bbfaf000)
	libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00007f92bbf7f000)
	libcom_err.so.2 => /usr/lib/libcom_err.so.2 (0x00007f92bbf79000)
	libkrb5support.so.0 => /usr/lib/libkrb5support.so.0 (0x00007f92bbf6b000)
	libkeyutils.so.1 => /usr/lib/libkeyutils.so.1 (0x00007f92bbf64000)
	libresolv.so.2 => /usr/lib/libresolv.so.2 (0x00007f92bbf52000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0x00007f92bbf2f000)

I could only surmise that the current Arch package is to blame according to what I read above.

whooo · 2023-06-22T18:40:21Z

I can not speak for the issue with Fedora (but I suspect the same).
the tpm2-pytss package gets built against tpm2-tss 3.2.x, later the tpm2-tss package is upgraded to a newer version (4.0.x) which has dropped those two functions as they are deprecated (and nothing really used it).
But due to how the tpm2-pytss module is built it's linked against basically all symbols defined in any of the headers.

So no package is really to blame, rebuilding the package in Arch should be enough (and might include some new extras then as well).

@diabonas, do you have any insight on the Arch parts?

dvzrv · 2023-07-24T08:12:03Z

Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal and Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Unmarshal where removed in tpm2-tss 4.0.0

tpm2-tss has indeed been updated to 4.0.1 (https://gitlab.archlinux.org/archlinux/packaging/packages/tpm2-tss/-/commit/a6479bce838a3a653495704a0bd4419ac4ff6e4d) after updating python-tpm2-pytss to 2.1.0 (https://gitlab.archlinux.org/archlinux/packaging/packages/python-tpm2-pytss/-/commit/9ef963c16c86b4e0ee8b13735c245178531a23dd).

Not sure what exactly happened there, but did we miss a soname change, or was none introduced? (maybe @arojas remembers)

@diabonas, do you have any insight on the Arch parts?

Since he is M.I.A. I have rebuilt our python-tpm2-pytss package.

dvzrv · 2023-07-24T08:41:26Z

To add further information on the tpm2-tss upgrade (3 -> 4): There are no soname changes introduced, yet @whooo mentioned the removal of Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal and Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Unmarshal in #840 (comment). Removal is a breaking change, which not only should increase the major version of the project (see semver), but lead to a soname change in the affected library.

For reference: Using soname changes downstreams are able to detect ABI changes, which then lead to rebuilds of all affected consumers of a library. If there is no soname change, then there is no way of knowing if the ABI has changed (and things break, as they did here).
As tpm2-tss is far past a stable 1.0.0 release, this needs to be considered carefully and soname changes should be introduced whenever there is a backwards incompatible change (e.g. removal).

The below provides the repod-file output for tpm2-tss 3.2.0-3 and 4.0.1-1.
As evidenced by the provides list, there has been no soname change.

repod-file package inspect -Pp /var/cache/pacman/pkg/tpm2-tss-3.2.0-3-x86_64.pkg.tar.zst
{
  "arch": "x86_64",
  "backup": [
    "etc/tpm2-tss/fapi-config.json",
    "etc/tpm2-tss/fapi-profiles/P_ECCP256SHA256.json",
    "etc/tpm2-tss/fapi-profiles/P_RSA2048SHA256.json"
  ],
  "base": "tpm2-tss",
  "builddate": 1667335578,
  "checkdepends": [
    "iproute2",
    "swtpm",
    "uthash"
  ],
  "conflicts": null,
  "depends": [
    "curl",
    "json-c",
    "openssl",
    "libjson-c.so=5-64"
  ],
  "desc": "Implementation of the TCG Trusted Platform Module 2.0 Software Stack (TSS2)",
  "fakeroot_version": "1.29",
  "groups": null,
  "isize": 2999883,
  "license": [
    "BSD"
  ],
  "makedepends": [
    "cmocka",
    "doxygen",
    "libtpms"
  ],
  "makepkg_version": "6.0.2",
  "name": "tpm2-tss",
  "optdepends": null,
  "packager": "Felix Yan <[email protected]>",
  "provides": [
    "libtss2-esys.so=0-64",
    "libtss2-fapi.so=1-64",
    "libtss2-mu.so=0-64",
    "libtss2-rc.so=0-64",
    "libtss2-sys.so=1-64",
    "libtss2-tctildr.so=0-64"
  ],
  "replaces": null,
  "schema_version": 2,
  "url": "https://github.com/tpm2-software/tpm2-tss",
  "version": "3.2.0-3",
  "xdata": []
}

repod-file package inspect -Pp /var/cache/pacman/pkg/tpm2-tss-4.0.1-1-x86_64.pkg.tar.zst
{
  "arch": "x86_64",
  "backup": [
    "etc/tpm2-tss/fapi-config.json",
    "etc/tpm2-tss/fapi-profiles/P_ECCP256SHA256.json",
    "etc/tpm2-tss/fapi-profiles/P_RSA2048SHA256.json"
  ],
  "base": "tpm2-tss",
  "builddate": 1683452210,
  "checkdepends": [
    "iproute2",
    "swtpm",
    "uthash"
  ],
  "conflicts": null,
  "depends": [
    "curl",
    "json-c",
    "openssl",
    "libjson-c.so=5-64"
  ],
  "desc": "Implementation of the TCG Trusted Platform Module 2.0 Software Stack (TSS2)",
  "fakeroot_version": "1.31",
  "groups": null,
  "isize": 3783221,
  "license": [
    "BSD"
  ],
  "makedepends": [
    "cmocka",
    "doxygen",
    "libtpms"
  ],
  "makepkg_version": "6.0.2",
  "name": "tpm2-tss",
  "optdepends": null,
  "packager": "Antonio Rojas <[email protected]>",
  "provides": [
    "libtss2-esys.so=0-64",
    "libtss2-fapi.so=1-64",
    "libtss2-mu.so=0-64",
    "libtss2-rc.so=0-64",
    "libtss2-sys.so=1-64",
    "libtss2-tctildr.so=0-64"
  ],
  "replaces": null,
  "schema_version": 2,
  "url": "https://github.com/tpm2-software/tpm2-tss",
  "version": "4.0.1-1",
  "xdata": []
}

thxrben · 2023-08-05T18:09:30Z

I didn't have time to test the latest changes yet.
I will take a look if it works, but I assume it does.

Thank you very much!

williamcroberts transferred this issue from tpm2-software/tpm2-tools Jun 12, 2023

JuergenReppSIT mentioned this issue Aug 2, 2023

Undefined sybol: TSS2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal tpm2-software/tpm2-tss#2669

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tpm2_ptool - Undefined sybol: TSS2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal #840

tpm2_ptool - Undefined sybol: TSS2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal #840

thxrben commented May 25, 2023

retpolanne commented Jun 4, 2023 •

edited

Loading

JuergenReppSIT commented Jun 4, 2023

whooo commented Jun 4, 2023

williamcroberts commented Jun 12, 2023

retpolanne commented Jun 12, 2023 •

edited

Loading

thxrben commented Jun 13, 2023

williamcroberts commented Jun 13, 2023

paranormal commented Jun 15, 2023

whooo commented Jun 22, 2023

dvzrv commented Jul 24, 2023

dvzrv commented Jul 24, 2023

thxrben commented Aug 5, 2023 •

edited

Loading

tpm2_ptool - Undefined sybol: TSS2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal #840

tpm2_ptool - Undefined sybol: TSS2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal #840

Comments

thxrben commented May 25, 2023

retpolanne commented Jun 4, 2023 • edited Loading

Fixing on Arch (at least)

JuergenReppSIT commented Jun 4, 2023

whooo commented Jun 4, 2023

williamcroberts commented Jun 12, 2023

retpolanne commented Jun 12, 2023 • edited Loading

thxrben commented Jun 13, 2023

williamcroberts commented Jun 13, 2023

paranormal commented Jun 15, 2023

whooo commented Jun 22, 2023

dvzrv commented Jul 24, 2023

dvzrv commented Jul 24, 2023

thxrben commented Aug 5, 2023 • edited Loading

retpolanne commented Jun 4, 2023 •

edited

Loading

retpolanne commented Jun 12, 2023 •

edited

Loading

thxrben commented Aug 5, 2023 •

edited

Loading