Disable a particular transaction formatter (RSK)

[tenuki]

• Version: 4.9.2
• Python: 3.7.2
• OS: linux
• pip freeze output

aiohttp==3.5.4
aiopubsub==2.1.5
async-timeout==3.0.1
attrdict==2.0.1
attrs==19.1.0
certifi==2019.3.9
chardet==3.0.4
coloredlogs==10.0
cytoolz==0.9.0.1
eth-abi==1.3.0
eth-account==0.3.0
eth-hash==0.2.0
eth-keyfile==0.5.1
eth-keys==0.2.2
eth-rlp==0.1.2
eth-typing==2.1.0
eth-utils==1.5.1
gevent==1.4.0
greenlet==0.4.15
hexbytes==0.1.0
humanfriendly==4.18
idna==2.8
lru-dict==1.1.6
multidict==4.5.2
parsimonious==0.8.1
pycryptodome==3.8.1
requests==2.21.0
rlp==1.1.0
semantic-version==2.6.0
six==1.12.0
toolz==0.9.0
urllib3==1.24.2
web3==4.9.2
websocket==0.2.1
websockets==6.0
yarl==1.3.0

I need to "change" a couple of transaction formatters for rsk nodes

I'm using web3py to connect to a few nodes. In particular I have this problem when connecting to RSK nodes which says to be somehow compatible with geth's RPC-protocol.

Problem is, when a transaction-formater is used, it breaks because some transactions miss "r" and "s" values.

• The code which produced the error

w3 = Web3(HTTPProvider("https://public-node.rsk.co"))
w3.eth.getBlock("0x6c4e755c5cbf736e89097d5e8c8430770cb1ffce18f9f90312417d053d0da784", 
                 True)

• The full output of the error

$ python3
Python 3.7.2 (default, Mar 22 2019, 18:41:14) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> from web3 import Web3, HTTPProvider
>>> w3 = Web3(HTTPProvider("https://public-node.rsk.co"))
>>> hash = "0x6c4e755c5cbf736e89097d5e8c8430770cb1ffce18f9f90312417d053d0da784"

>>> w3.eth.getBlock(hash, True)
Traceback (most recent call last):
  File ".local/lib/python3.7/site-packages/web3/utils/formatters.py", line 68, in apply_formatters_to_dict
    yield key, formatters[key](item)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File ".local/lib/python3.7/site-packages/web3/middleware/pythonic.py", line 71, in to_hexbytes
    raise TypeError("Cannot convert %r to HexBytes" % val)
TypeError: Cannot convert None to HexBytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".local/lib/python3.7/site-packages/web3/utils/formatters.py", line 68, in apply_formatters_to_dict
    yield key, formatters[key](item)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File ".local/lib/python3.7/site-packages/web3/utils/formatters.py", line 86, in apply_one_of_formatters
    return formatter(value)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "local/lib/python3.7/site-packages/eth_utils/functional.py", line 46, in inner
    return callback(fn(*args, **kwargs))
  File ".local/lib/python3.7/site-packages/web3/utils/formatters.py", line 79, in apply_formatter_to_array
    yield formatter(item)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "local/lib/python3.7/site-packages/eth_utils/functional.py", line 46, in inner
    return callback(fn(*args, **kwargs))
  File ".local/lib/python3.7/site-packages/web3/utils/formatters.py", line 70, in apply_formatters_to_dict
    raise type(exc)("Could not format value %r as field %r" % (item, key)) from exc
TypeError: Could not format value None as field 'r'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".local/lib/python3.7/site-packages/web3/eth.py", line 141, in getBlock
    [block_identifier, full_transactions],
  File ".local/lib/python3.7/site-packages/web3/manager.py", line 109, in request_blocking
    response = self._make_request(method, params)
  File ".local/lib/python3.7/site-packages/web3/manager.py", line 92, in _make_request
    return request_func(method, params)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File ".local/lib/python3.7/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
    response = make_request(method, params)
  File ".local/lib/python3.7/site-packages/web3/middleware/gas_price_strategy.py", line 18, in middleware
    return make_request(method, params)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File ".local/lib/python3.7/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
    response = make_request(method, formatted_params)
  File ".local/lib/python3.7/site-packages/web3/middleware/attrdict.py", line 18, in middleware
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File ".local/lib/python3.7/site-packages/web3/middleware/formatting.py", line 57, in apply_formatters
    formatter(response['result']),
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File ".local/lib/python3.7/site-packages/web3/utils/formatters.py", line 57, in apply_formatter_if
    return formatter(value)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "local/lib/python3.7/site-packages/eth_utils/functional.py", line 46, in inner
    return callback(fn(*args, **kwargs))
  File ".local/lib/python3.7/site-packages/web3/utils/formatters.py", line 70, in apply_formatters_to_dict
    raise type(exc)("Could not format value %r as field %r" % (item, key)) from exc
TypeError: Could not format value [{'hash': '0x4e935981d89dd1634416ee1e761eadab6fc71c21e776519f01e4a188b0bf73b6', 
'nonce': '0x02', 
'blockHash': '0x6c4e755c5cbf736e89097d5e8c8430770cb1ffce18f9f90312417d053d0da784', 
'blockNumber': '0x148481', 
'transactionIndex': '0x0', 
'from': '0x6b9939722eed292298001cc0149c7aa2712e3904', 
'to': '0x970536a801eacfaeb756e76990ad49b94a31ebeb', 
'gas': '0x5208', 
'gasPrice': '0x0bebc200', 
'value': '0x09184e72a000', 
'input': '0x00', 
'v': '0x1c', 
'r': '0xa6adeecff2814a86d293798a0243852a6c9fe44cb107039bb1bbe335dc5f1299', 
's': '0x22cf3de4c68e3fbd9698d654e71c9fe3a6d0fdc1f51cbde6278da6f6f181b3a0'}, 
{'hash': '0x580a3f7b6e114e9bb7e02bbb1383edcf8754e44dddad50ac81ea778e132f2280', 'nonce': '0x148480', 
'blockHash': '0x6c4e755c5cbf736e89097d5e8c8430770cb1ffce18f9f90312417d053d0da784', 
'blockNumber': '0x148481', 
'transactionIndex': '0x1', 
'from': '0x0000000000000000000000000000000000000000', 
'to': '0x0000000000000000000000000000000001000008', 
'gas': '0x00', 
'gasPrice': '0x00', 
'value': '0', 
'input': '0x00', 
'v': None, 
'r': None, 
's': None}] as field 'transactions'
>>> 

• What type of node you were connecting to.

RSK node.

How can it be fixed?

I understand this is not the fix for this "problem" because it actually is not a problem when working with ethereum/geth nodes. However, I've included this here, as an extra information about what is going on and where problem is when using RSK nodes. In particular, I'd like to know which is the way to disable or change these formatters properly for a particular web3 connection and not for all that I have.

Index: venv/lib/python3.7/site-packages/web3/middleware/pythonic.py

===================================================================
--- venv/lib/python3.7/site-packages/web3/middleware/pythonic.py	(date 1556912563771)
+++ venv/lib/python3.7/site-packages/web3/middleware/pythonic.py	(date 1556912563771)
@@ -93,9 +93,9 @@
     'value': to_integer_if_hex,
     'from': to_checksum_address,
     'publicKey': apply_formatter_if(is_not_null, to_hexbytes(64)),
-    'r': to_hexbytes(32, variable_length=True),
+    'r': apply_formatter_if(is_not_null, to_hexbytes(32, variable_length=True)),
     'raw': HexBytes,
-    's': to_hexbytes(32, variable_length=True),
+    's': apply_formatter_if(is_not_null, to_hexbytes(32, variable_length=True)),
     'to': apply_formatter_if(is_address, to_checksum_address),
     'hash': to_hexbytes(32),
     'v': apply_formatter_if(is_not_null, to_integer_if_hex),