Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Specter Python 3.10 compatibility; Ubuntu 22.04 has python 3.10 #1686

Closed
relativisticelectron opened this issue Apr 29, 2022 · 13 comments
Closed

Specter Python 3.10 compatibility; Ubuntu 22.04 has python 3.10 #1686

relativisticelectron opened this issue Apr 29, 2022 · 13 comments
Labels
chore testing, automation, releasing and the like

Comments

@relativisticelectron
Copy link
Collaborator

relativisticelectron commented Apr 29, 2022 •
edited
Loading

With Ubuntu 22.04 python 3.10 is now the default. The Readme instructions
image
are not working. Can Specter be made compatible with python 3.10?

When I set up a conda python 3.10 env and run pip install -r requirements.txt I get
image

To fix the hwi error I use a modified requirements.in:

certifi==2021.10.08
chardet==3.0.4
Click==7.1.2
Flask==1.1.4
Flask-Babel==2.0.0
Flask-Cors==3.0.10
Flask-Login==0.5.0
Flask-RESTful==0.3.9
Flask-HTTPAuth==4.4.0
hwi==2.1.0
data
pyserial
python-dotenv==0.13.0
requests==2.26.0
pysocks==1.7.1
six==1.16.0
stem==1.8.0
embit==0.4.12
psutil==5.7.3
pyopenssl==20.0.1
flask_wtf==0.14.3
pgpy==0.5.4
cbor==1.0.0
mnemonic==0.20
cryptography==3.4.7
Flask-APScheduler==1.12.3
markupsafe==2.0.1
importlib_metadata

I run into the error

running install_scripts
/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: vx.y.z-get-replaced-by-release-script is an invalid version and will not be supported in a future release
  warnings.warn(
Traceback (most recent call last):
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/__main__.py", line 2, in <module>
    from .cli import entry_point
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/cli/__init__.py", line 8, in <module>
    from .cli_ext import ext
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/cli/cli_ext.py", line 15, in <module>
    from ..server import create_app, init_app
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/server.py", line 7, in <module>
    from cryptoadvance.specter.managers.service_manager import ServiceManager
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/managers/service_manager.py", line 13, in <module>
    from cryptoadvance.specter.user import User
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/user.py", line 19, in <module>
    from .managers.device_manager import DeviceManager
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/managers/device_manager.py", line 7, in <module>
    from cryptoadvance.specter.devices.bitcoin_core import BitcoinCoreWatchOnly
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/devices/__init__.py", line 7, in <module>
    from .keepkey import Keepkey
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/devices/keepkey.py", line 3, in <module>
    from .hwi.keepkey import KeepkeyClient
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/devices/hwi/keepkey.py", line 42, in <module>
    class KeepkeyFeatures(Features):  # type: ignore
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/devices/hwi/keepkey.py", line 55, in KeepkeyFeatures
    def get_fields(cls) -> Dict[int, p.FieldInfo]:
AttributeError: module 'hwilib.devices.trezorlib.protobuf' has no attribute 'FieldInfo'

This needs to be fixed.
@relativisticelectron
Copy link
Collaborator Author

These changes https://github.com/cryptoadvance/specter-desktop/compare/master...relativisticelectron:upgrade310?expand=1 together with the conda list --export > requirements.txt (see requirements.txt attached) allows me to start specter successfully

@k9ert
Copy link
Collaborator

k9ert commented Apr 30, 2022 •
edited
Loading

Great, this might be easier than i expected! I'd suggest to create a Draft PR where we're trying to collect all things necessary to migrate to 3.10. I'll then have "something to work with". I might commit myself or do recommendations on how to get progress there?!

There are probably quite some docker-images which will need some refurbishing and migration to 3.10. But almost all of that (apart from the manual build and the push to the repo) can be done in the repo.

@k9ert k9ert added the chore testing, automation, releasing and the like label Apr 30, 2022
@relativisticelectron
Copy link
Collaborator Author

relativisticelectron commented Apr 30, 2022 •
edited
Loading

There is another issue when loading wallets

Traceback (most recent call last):
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/managers/wallet_manager.py", line 201, in _update
    loaded_wallet = self.WalletClass.from_json(
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/wallet.py", line 758, in from_json
    return cls(
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/wallet.py", line 156, in __init__
    self.update()
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/wallet.py", line 611, in update
    self.getdata()
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/wallet.py", line 820, in getdata
    self.fetch_transactions()
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/wallet.py", line 532, in fetch_transactions
    self._transactions.add(txs)
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/txlist.py", line 341, in add
    self._fill_missing(tx)
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/txlist.py", line 374, in _fill_missing
    outputs = [out.to_dict() for out in psbt.outputs]
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/txlist.py", line 374, in <listcomp>
    outputs = [out.to_dict() for out in psbt.outputs]
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/util/psbt.py", line 142, in to_dict
    "change": self.is_change,
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/cryptoadvance/specter/util/psbt.py", line 110, in is_change
    return self.descriptor.branch(1).owns(self.scope)
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/embit/descriptor/descriptor.py", line 169, in owns
    res = k.check_derivation(der)
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/embit/descriptor/arguments.py", line 260, in check_derivation
    if self.my_fingerprint == derivation_path.fingerprint:
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/embit/descriptor/arguments.py", line 175, in my_fingerprint
    return self.key.my_fingerprint
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/embit/bip32.py", line 71, in my_fingerprint
    self._my_fingerprint = hashes.hash160(sec)[:4]
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/site-packages/embit/hashes.py", line 12, in hash160
    return hashlib.new('ripemd160', hashlib.sha256(msg).digest()).digest()
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/hashlib.py", line 166, in __hash_new
    return __get_builtin_constructor(name)(data)
  File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type ripemd160

So weird... the embit version was not changed. And I currently do not find what thing was changed in https://github.com/python/cpython/blob/3.10/Lib/hashlib.py compared to https://github.com/python/cpython/blob/3.8/Lib/hashlib.py

Possible reason: openssl/openssl#16994 (The openssl version was upgraded to 3.0.2 in ubuntu 22.04)

@stepansnigirev : If I understand correctly, ripemd160 was disabled by default in openssl 3. Do you have ideas how embit can handle this reduced functionality in openssl?

Merged
14 tasks
@evliu
Copy link

evliu commented May 3, 2022

These changes https://github.com/cryptoadvance/specter-desktop/compare/master...relativisticelectron:upgrade310?expand=1 together with the conda list --export > requirements.txt (see requirements.txt attached) allows me to start specter successfully

If this is merged successfully, #1635 will also be fixed and allow for new Jade hardware, which requires pyserial 3.5 and hwi 2.1.0 :)

thanks @relativisticelectron ; I'm not a python guy so i couldn't figure out for the life of me how to fix the keepkey file

@k9ert
Copy link
Collaborator

k9ert commented May 3, 2022

For the reference: I've created a PR #1693 for migrating to HWI 2.1.0 . Why do we need to do that for python 3.10 ? I would really like to try hard separating the different issues and tackling them separately (and in the right order). I think we're not clear yet about the dependencies here.

@relativisticelectron
Copy link
Collaborator Author

relativisticelectron commented May 3, 2022 •
edited
Loading

For the reference: I've created a PR #1693 for migrating to HWI 2.1.0 . Why do we need to do that for python 3.10 ? I would really like to try hard separating the different issues and tackling them separately (and in the right order). I think we're not clear yet about the dependencies here.

When I try keeping the version of HWI fixed, pip-compile --generate-hashes requirements.in gives

pip._internal.exceptions.DistributionNotFound: No matching distribution found for hwi==2.0.2 (from -r requirements.in (line 11))

@evliu
Copy link

evliu commented May 3, 2022

I'm pretty sure I had that issue too. I believe 3.10 is the required version for HWI. Isn't the current specter python version pinned because the previous HWI required python ^3.9?

@k9ert
Copy link
Collaborator

k9ert commented May 4, 2022

I believe 3.10 is the required version for HWI.

Yes, indeed. forgot about that:
https://github.com/bitcoin-core/HWI/blob/2.0.2/setup.py#L49

This means we need to upgrade HWI before migrating to python 3.10. Sure, we can do some things in parallel, though.

@Meru852
Copy link

Meru852 commented May 21, 2022

i need help to solve this below problems.
i get an errors time i run my programs what should i do?

Traceback (most recent call last):
File "/usr/lib/python3.10/hashlib.py", line 160, in __hash_new
return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] unsupported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zee52/Desktop/Files/4Bitcoin/4Bitcoin.py", line 49, in
wallet = Wallet(HEX)
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/wallet.py", line 16, in init
self.address = Address.of(self.key)
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/address.py", line 88, in of
address.generate()
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/address.py", line 94, in generate
self.mainnet.generate_publicaddress1()
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/address.py", line 38, in generate_publicaddress1
self.pubaddr1 = self.instance._generate_publicaddress1(Address.MainNet.PREFIX_A,
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/address.py", line 107, in _generate_publicaddress1
m = prefix_a + hash160(p).digest()
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/util.py", line 20, in hash160
return ripemd(hashlib.sha256(v).digest())
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/util.py", line 24, in ripemd
r = hashlib.new('ripemd160')
File "/usr/lib/python3.10/hashlib.py", line 166, in __hash_new
return __get_builtin_constructor(name)(data)
File "/usr/lib/python3.10/hashlib.py", line 123, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type ripemd160

@Meru852
Copy link

Meru852 commented May 21, 2022

i was install with this sudo apt-get install node-ripemd160 but i still get errors like this below review

Traceback (most recent call last):
File "/usr/lib/python3.10/hashlib.py", line 160, in __hash_new
return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] unsupported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zee52/Desktop/Files/4Bitcoin/4Bitcoin.py", line 49, in
wallet = Wallet(HEX)
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/wallet.py", line 16, in init
self.address = Address.of(self.key)
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/address.py", line 88, in of
address.generate()
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/address.py", line 94, in generate
self.mainnet.generate_publicaddress1()
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/address.py", line 38, in generate_publicaddress1
self.pubaddr1 = self.instance._generate_publicaddress1(Address.MainNet.PREFIX_A,
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/address.py", line 107, in _generate_publicaddress1
m = prefix_a + hash160(p).digest()
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/util.py", line 20, in hash160
return ripemd(hashlib.sha256(v).digest())
File "/home/zee52/.local/lib/python3.10/site-packages/bitcoinaddress/util.py", line 24, in ripemd
r = hashlib.new('ripemd160')
File "/usr/lib/python3.10/hashlib.py", line 166, in __hash_new
return __get_builtin_constructor(name)(data)
File "/usr/lib/python3.10/hashlib.py", line 123, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type ripemd160

@relativisticelectron
Copy link
Collaborator Author

relativisticelectron commented May 21, 2022 •
edited
Loading

@Meru852 : While your question is not concerning specter we also encountered that problem. The root cause is that openssl/openssl#16994 removed ripemd160 from openssl3 (which is the default in ubuntu 22.04). A solution is an own implementation of ripemd160 as it was done in diybitcoinhardware/embit#28 (comment)

@relativisticelectron
Copy link
Collaborator Author

Solved by #1688

@Meru852
Copy link

Meru852 commented Jun 20, 2023

okay thank you very much of your help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
chore testing, automation, releasing and the like
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@k9ert @evliu @Meru852 @relativisticelectron