Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: Jail reason consensus not achieved #1313

Open
cosmoscats opened this issue Oct 25, 2024 · 36 comments
Open

bug: Jail reason consensus not achieved #1313

cosmoscats opened this issue Oct 25, 2024 · 36 comments
Assignees
@maharifu
Labels
bug Something isn't working

Comments

@cosmoscats
Copy link

What is happening?

Section description Provide as much context as you can. Give as much context as you can to make it easier for the developers to figure what is happening.

Our node goes to Jail every day few times a day. According to the latest data, it almost always goes to Jail because of ETH RPC.
I have the opportunity to install 10 different RPC providers for testing.

Paloma and pigeon versions and logs

Section description Write down paloma version. Write down pigeon version. Copy and paste pigeon config file as well as relevant ENV variables.

palomad version
v2.3.2

pigeon version
App version: v2.3.1
Build commit hash: 5f6f4bcaa645e0f9d8530b24cd1a4cd74b79e857

i will attach latest latest jail log
paloma-jail-24-10-2024.txt

How to reproduce?

Section description Please write detailed steps of what you were doing for this bug to appear.

Unjail, wait for sometime and in 24 hours or less node will be jailed

What is the expected behaviour?

Section description If you know, please write down what is the expected behaviour. If you don't know, that's ok. We can have a discussion in comments.
@cosmoscats cosmoscats added the bug Something isn't working label Oct 25, 2024
@taariq
Copy link
Contributor

taariq commented Oct 25, 2024

Logs added

Oct 24 20:27:28 cc1 palomad[2745431]: 8:27PM INF put message into consensus queue message-id=350030 module=x/palomaconsensus msg="hexAddresses:"0x1177806cD88b0BC0eA363bEd3C64f8314361b162" hexAddresses:"0x0DAaBB4FF60423Eb1F14cC6731394e098ad51bcb" hexAddresses:"0xC1009C72cC2519B0DD595C3E500C1293A862Aa6A" hexAddresses:"0x35c1cC4E3CD4624Ac177c167D2427e9C95336535" hexAddresses:"0x377D23948D41579f2c3cA40308e3bdd53f6dA7B2" hexAddresses:"0x1799D68d773f9E03Baca79cd1d836Ba026B4742A" hexAddresses:"0x43E218f96A567DC26C6e65fCabF1fDE26Af69444" hexAddresses:"0x443266026738061972012e62D5Ecd9D98da8B6F4" hexAddresses:"0x4a89f96fdff3C161937CFBC3e22d2E325612aaEc" hexAddresses:"0xa228292447064D5818BbC80b577C91f5212F9355" hexAddresses:"0xDEea5b069208e0eE37b630A3e7672FC50e8fE24a" hexAddresses:"0xAAc74d38c82c367b3dA7482E3Aecf6DD7A512eF2" hexAddresses:"0xeB784b37365C302C97C9Ec8cB0933cE344e6dE42" hexAddresses:"0x2d91F6C502289eA4d7F9253C45788109377b95bD" hexAddresses:"0x732fBb6018F2cB5b844055C9EB567447a3132833" hexAddresses:"0x19f5911E4cA69E30449aD6BB71De341F01F118BB" hexAddresses:"0x1ad90dB98da083E117D5F62a1673fC0f3A5930ca" hexAddresses:"0x63f55bc560E981d53E1f5bb3643e3a96D26fc635" hexAddresses:"0x6dc59EE4bdFa2C791229004f29b08F783491a934" hexAddresses:"0x14Aa448C2C918C4427c5671028b63BC17f6132d5" hexAddresses:"0xEe21301aF1d9562B5cBEdf520077Ea0a9bC9d535" hexAddresses:"0x7Bd1A3270570b65F264895C2F86E631fDc497545" valAddresses:"\001\276s\021\343\227\312Q_\246G*\224\363(\340K\273\350\006" valAddresses:"\002}0\274a\2108\266M\347f4\001<\236Jp\365bx" valAddresses:"\t\007\3044\210\241\355[\370\347\010W15\206\207(-h." valAddresses:"\t\231\212\337\024\375x\026BE\3600>9\322\357\333\362\316\345" valAddresses:"\021R\201_\333\020\021\310\304-\261\234\366\203F\\313\\363\\344\\322\" valAddresses:\"V5{\\265e\\371\\017\\023\\205\\332J\\335\\205\\245\\275\\342\\306q\\205G\" valAddresses:\"[\\\\\\250\\24435\\325\\313E\\262Z\\363\\177g\\244i64\\242}\" valAddresses:\"\\\\8y\\256\\374\\200sQ~\\331\\222\\345\\026+\\303X\\016\\001x\\032\" valAddresses:\"]:W\\321\\311\\303)o\\274S\\214s)\\260\\255\\266%<\\214\\004\" valAddresses:\"e%\\315\\342\\033\\245\\217\\234\\232H\\022S\\345O\\321\\331\\2446\\340\\332\" valAddresses:\"v\\352C\\222\\341\\t\\270\\335J\\216c\\207\\344\\220\\223\\275\\336U\\355\\224\" valAddresses:\"y\\024\\257H\\255yC\\213d\\304\\212\\230\\244b\\331\\034\\213\\327\\333\\031\" valAddresses:\"y~y\\006\\2421P~7+\\235\\350\\007\\\"\\002\\311\\325\\274\\344\\024\" valAddresses:\"y\\360\\311q\\3445\\352\\333\\313\\314$\\3268\\220\\324(\\375t\\326~\" valAddresses:\"\\200\\242)\200>\033\034\252\305\261\264\332\326\007t\372\304\300\212" valAddresses:"\203\273Ka%\307\350\323(\2013\361A\2274\003S1\004\256" valAddresses:"\203\377\331\332\361\215\202<\215_R\247\331\332\203d9r\332\035" valAddresses:"\217!\373\274\212RS9\264\203\211\242\367\001\247\356\203\220]T" valAddresses:"\242\030KV&\253\364\317[\r\250\322\232\001<X7\211\206Q" valAddresses:"\27471u\333\237xB\200\3260\244\r\253aj\236\035\247\251" valAddresses:"\301\370kXOx&_:\001/6rG\247w\3666G\024" valAddresses:"\3459+\360`_Mf-=\002\020\261\321\334\240!\263@P" fromBlockTime:<seconds:1729801647 nanos:91967665 > " queue-type-name=evm/eth-main/validators-balances
Oct 24 20:27:31 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper15gvyk43x406v7kcd4rff5qfutqmcnpj3w9wpnm
Oct 24 20:27:31 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=147956415466737
Oct 24 20:27:33 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=147956415466737
Oct 24 20:27:35 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1hsmnzawmnauy9qxkxzjqm2mpd20pmfaf5uj30x
Oct 24 20:27:35 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper12c6hhdt9ly838pw6ftwctfdautr8rp288atqnf
Oct 24 20:27:35 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=379703560579887
Oct 24 20:27:36 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=379703560579887
Oct 24 20:27:38 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1v5jumcsm5k8eexjgzff72n73mxjrdcx6xu2e9x
Oct 24 20:27:38 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=560504217079887
Oct 24 20:27:39 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=560504217079887
Oct 24 20:27:41 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1qf7np0rp3qutvn08vc6qz0y7ffc02cncwdvdtd
Oct 24 20:27:41 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper13uslh0y22ffnndyr3x30wqd8a6peqh25m8p743
Oct 24 20:27:41 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=700112818385852
Oct 24 20:27:42 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1wm4y8yhppxud6j5wvwr7fyynhh09tmv5fy845g
Oct 24 20:27:42 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=737193758269149
Oct 24 20:27:44 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=737193758269149
Oct 24 20:27:45 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=737193758269149
Oct 24 20:27:46 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1s0lankh33kprer2l22nank5rvsuh9ksafx3mze
Oct 24 20:27:46 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=865190075873055
Oct 24 20:27:48 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=865190075873055
Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1tdw23fpnxh2uk3djtteh7eaydymrfgnaepkpkz
Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1sz3zjcyq8cd3e2k9kx6d44s8wnavfsy2837nje
Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper108cvju0yxh4dhj7vyntr3yx59r7hf4n7kyzhu4
Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1052826268020914
Oct 24 20:27:51 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1052826268020914
Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1pxvc4hc5l4upvsj97qcruwwjaldl9nh9v2k3z8
Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1z9fgzh7mzqgu33pdkxw0dqmqgm9l8exj6rl5wj
Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1u5ujhurqtaxkvtfaqggtr5wu5qsmxszs9hrvxe
Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1pyrugdyg58k4h788pptnzdvxsu5z66pwhu69gs
Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"792824356058927"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1430325117777092
Oct 24 20:27:54 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1swa5kcf9cl5dx2ypx0c5r9e5qdfnzp9w0yj7uv
Oct 24 20:27:54 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"912838586458895"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1550339348177060
Oct 24 20:27:56 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1t5a905wfcv5kl0zn33ejnv9dkcjnerqy9mwutc
Oct 24 20:27:56 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"918121390061517"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1555622151779682
Oct 24 20:27:58 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"918121390061517"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1555622151779682
Oct 24 20:27:59 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1tsu8nthuspe4zlkejtj3v27rtq8qz7q6983zt2
Oct 24 20:27:59 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"985629872353235"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1623130634071400
Oct 24 20:28:01 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"985629872353235"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1623130634071400
Oct 24 20:28:02 cc1 palomad[2745431]: 8:28PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper10y227j9d09pckexy32v2gckerj9a0kcewgf7xy
Oct 24 20:28:02 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1053007719853992"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1690508481572157
Oct 24 20:28:21 cc1 palomad[2745431]: 8:28PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper109l8jp4zx9g8udetnh5qwgsze82meeq5yq06ta
Oct 24 20:28:21 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:28:23 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:28:24 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:28:26 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:28:27 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:28:29 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482
Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF jailing a validator jail-time=2024-10-24T20:37:49Z module=x/valset reason="No evidence supplied for contentious message 350030" val-addr=palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg
Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF jailing a validator jail-time=2024-10-24T20:37:49Z module=x/valset reason="No evidence supplied for contentious message 350030" val-addr=palomavaloper1c8uxkkz00qn97wsp9um8y3a8wlmrv3c546jw4q
Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF Removed message from queue block_height=25891550 module=server msg={"bytesToSign":"o1Pnaq+DEtinxt97XiWwf5B4W7QMLOUZj2xEQeq3Xxg=","errorData":null,"gasEstimate":"0","id":"350030","msg":{"@type":"/palomachain.paloma.evm.ValidatorBalancesAttestation","assignee":"","fromBlockTime":"2024-10-24T20:27:27.091967665Z","hexAddresses":["0x1177806cD88b0BC0eA363bEd3C64f8314361b162","0x0DAaBB4FF60423Eb1F14cC6731394e098ad51bcb","0xC1009C72cC2519B0DD595C3E500C1293A862Aa6A","0x35c1cC4E3CD4624Ac177c167D2427e9C95336535","0x377D23948D41579f2c3cA40308e3bdd53f6dA7B2","0x1799D68d773f9E03Baca79cd1d836Ba026B4742A","0x43E218f96A567DC26C6e65fCabF1fDE26Af69444","0x443266026738061972012e62D5Ecd9D98da8B6F4","0x4a89f96fdff3C161937CFBC3e22d2E325612aaEc","0xa228292447064D5818BbC80b577C91f5212F9355","0xDEea5b069208e0eE37b630A3e7672FC50e8fE24a","0xAAc74d38c82c367b3dA7482E3Aecf6DD7A512eF2","0xeB784b37365C302C97C9Ec8cB0933cE344e6dE42","0x2d91F6C502289eA4d7F9253C45788109377b95bD","0x732fBb6018F2cB5b844055C9EB567447a3132833","0x19f5911E4cA69E30449aD6BB71De341F01F118BB","0x1ad90dB98da083E117D5F62a1673fC0f3A5930ca","0x63f55bc560E981d53E1f5bb3643e3a96D26fc635","0x6dc59EE4bdFa2C791229004f29b08F783491a934","0x14Aa448C2C918C4427c5671028b63BC17f6132d5","0xEe21301aF1d9562B5cBEdf520077Ea0a9bC9d535","0x7Bd1A3270570b65F264895C2F86E631fDc497545"],"valAddresses":["palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg","palomavaloper1qf7np0rp3qutvn08vc6qz0y7ffc02cncwdvdtd","palomavaloper1pyrugdyg58k4h788pptnzdvxsu5z66pwhu69gs","palomavaloper1pxvc4hc5l4upvsj97qcruwwjaldl9nh9v2k3z8","palomavaloper1z9fgzh7mzqgu33pdkxw0dqmqgm9l8exj6rl5wj","palomavaloper12c6hhdt9ly838pw6ftwctfdautr8rp288atqnf","palomavaloper1tdw23fpnxh2uk3djtteh7eaydymrfgnaepkpkz","palomavaloper1tsu8nthuspe4zlkejtj3v27rtq8qz7q6983zt2","palomavaloper1t5a905wfcv5kl0zn33ejnv9dkcjnerqy9mwutc","palomavaloper1v5jumcsm5k8eexjgzff72n73mxjrdcx6xu2e9x","palomavaloper1wm4y8yhppxud6j5wvwr7fyynhh09tmv5fy845g","palomavaloper10y227j9d09pckexy32v2gckerj9a0kcewgf7xy","palomavaloper109l8jp4zx9g8udetnh5qwgsze82meeq5yq06ta","palomavaloper108cvju0yxh4dhj7vyntr3yx59r7hf4n7kyzhu4","palomavaloper1sz3zjcyq8cd3e2k9kx6d44s8wnavfsy2837nje","palomavaloper1swa5kcf9cl5dx2ypx0c5r9e5qdfnzp9w0yj7uv","palomavaloper1s0lankh33kprer2l22nank5rvsuh9ksafx3mze","palomavaloper13uslh0y22ffnndyr3x30wqd8a6peqh25m8p743","palomavaloper15gvyk43x406v7kcd4rff5qfutqmcnpj3w9wpnm","palomavaloper1hsmnzawmnauy9qxkxzjqm2mpd20pmfaf5uj30x","palomavaloper1c8uxkkz00qn97wsp9um8y3a8wlmrv3c546jw4q","palomavaloper1u5ujhurqtaxkvtfaqggtr5wu5qsmxszs9hrvxe"]},"nonce":"AAAAAAAFV04=","publicAccessData":"AQ==","signData":[],"valsetID":"0"} msg-id=350030

@taariq taariq added this to Paloma Oct 25, 2024
@taariq taariq moved this to In Progress in Paloma Oct 25, 2024
@cosmoscats
Copy link
Author

go version go1.21.0 linux/amd64
maybe this can be the issue?
do i need to update go version on server?

@maharifu
Copy link
Contributor

The jail reason is here:
Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF jailing a validator jail-time=2024-10-24T20:37:49Z module=x/valset reason="No evidence supplied for contentious message 350030" val-addr=palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg

One of the issues here is some pigeons are having RPC issues and returning incomplete balances information, which makes consensus harder to achieve. We can see that here distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"918121390061517"}. This means we have two different values for the same request, which in this request means some pigeons failed to get balances for all validators.

However, on the logs, we can see

Oct 24 20:26:50 mainnet-validator palomad[1574145]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"eth-main","msg.args.error":"failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg","time":"2024-10-24T20:26:50Z","message":"error attesting messages"}

So this is an issue of failing to broadcast tx to paloma, a duplicate of VolumeFi#2259

@cosmoscats I don't think the go version is the issue but you should still update it, especially if you are compiling your own binary. Are you using pigeon operator keys (like detailed here)?

@cosmoscats
Copy link
Author

@maharifu - i will try to update go version.

When we starting to use pigeon operator keys - we go straight to the jail much faster, like in 1 hour or so (situation is same like Nodes Guru described in discord - "out of gas"
We have disabled it for now.

@cosmoscats
Copy link
Author

still no info about? VolumeFi#2259

@cosmoscats
Copy link
Author

@maharifu i have installed new version of go

go version
go version go1.22.8 linux/amd64

and i have recompiled paloma & pigeon with that new version and made restart.

palomad version
v2.3.2
paloma@cc1:~/pigeon$ pigeon version
App version: v2.3.1
Build commit hash: 5f6f4bcaa645e0f9d8530b24cd1a4cd74b79e857

@taariq
Copy link
Contributor

taariq commented Oct 25, 2024

@maharifu - i will try to update go version.

When we starting to use pigeon operator keys - we go straight to the jail much faster, like in 1 hour or so (situation is same like Nodes Guru described in discord - "out of gas" We have disabled it for now.

This was resolved as user-issue. Please check this comment here: #1312 (comment)

@taariq taariq closed this as completed Oct 25, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in Paloma Oct 25, 2024
@taariq taariq reopened this Oct 25, 2024
@cosmoscats
Copy link
Author

Looks like the same problem still occurs after go update.
Still going to jail.

@cosmoscats
Copy link
Author

node jailing every day few times in a day
Paloma-jail

@taariq
Copy link
Contributor

taariq commented Oct 31, 2024

We're taking the position that this MAY be hardware related as no other validators are dealing with this issue. Are you able to switch hosts?

@cosmoscats
Copy link
Author

We're taking the position that this MAY be hardware related as no other validators are dealing with this issue. Are you able to switch hosts?

Yes, i can switch to another hardware server tomorrow.

@cosmoscats
Copy link
Author

Today i have moved my Paloma node to another server:

CPU: AMD Ryzen 9 5900X
MB: B550-A PRO (MS-7C56) v: 2.0 (Socket AM4)
Video: GeForce GT 710
RAM: Corsair VENGEANCE® RGB PRO 64GB (2 x 32GB) DDR4 DRAM 3600MHz C18 CMW64GX4M2D3600C18
PSU: Corsair Core GM-650 80 Plus Gold

I will let you know how it goes.

@cosmoscats
Copy link
Author

so, after 24h jailed again on new server.

i will include logs, can you check please?

Paloma-Jail-2-11-2024.txt

@cosmoscats
Copy link
Author

and again reason: No evidence supplied for contentious message 362591

@cosmoscats
Copy link
Author

And again jail, message - 362963

Also please check this jail and let me know if it is related to same problesm as before?

Is it again the same " issue of failing to broadcast tx to paloma" ?
Paloma-Jail-3-11-Night.txt

I need to understand that to dismiss the server problem.
Because we're using a completely different hardware server right now.

@taariq
Copy link
Contributor

taariq commented Nov 3, 2024

@cosmoscats this was a gnosis broadcast problem

Nov 02 22:10:58 server-31-24-56-54 palomad[2946843]: 10:10PM INF added message evidence. chain-reference-id=gnosis-main chain-type=evm message-id=362963 module=x/palomaconsensus queue-type-name=evm/gnosis-main/validators-balances validator=palomavaloper1v5jumcsm5k8eexjgzff72n73mxjrdcx6xu2e9x

Please change your Gnosis endpoint and try again to unjail.

@cosmoscats
Copy link
Author

Please check our another jail. Is it again the same " issue of failing to broadcast tx to paloma" ?

Paloma-Jail-4-11-Night.txt

@taariq
Copy link
Contributor

taariq commented Nov 4, 2024

Please check our another jail. Is it again the same " issue of failing to broadcast tx to paloma" ?

Paloma-Jail-4-11-Night.txt

this is now Arbitrum
Nov 04 02:34:43 server-31-24-56-54 palomad[2946843]: 2:34AM INF added message evidence. chain-reference-id=arbitrum-main chain-type=evm message-id=364542 module=x/palomaconsensus queue-type-name=evm/arbitrum-main/validators-balances validator=palomavaloper1tsu8nthuspe4zlkejtj3v27rtq8qz7q6983zt2.

Will you update your ARB RPC endpoint and unjail again?

@cosmoscats
Copy link
Author

Arb-Paloma-Stats
according to the logs there were no problems with ARB RPC.
Changed from Alchemy to Ankr. I'll check it out

@taariq
Copy link
Contributor

taariq commented Nov 4, 2024

What does palomad q valset get-validator-jail-reason palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg give you?

@cosmoscats
Copy link
Author

cosmoscats commented Nov 4, 2024 •
edited
Loading

If I remember correctly from last time:
"No evidence supplied for contentious message 364542"

@taariq
Copy link
Contributor

taariq commented Nov 4, 2024

If I remember correctly from last time: "No evidence supplied for contentious message 364542"

So that's the Arbitrum endpoint. I would test another RPC provider. Yes.

@cosmoscats
Copy link
Author

also 4 your info:
i cant change gas-prices in config fewer that 0.01ugrain for paloma.
If i will change this value to 0.001 or to 0.009 I'll instantly get a bunch of errors in my pigeon logs.

@taariq
Copy link
Contributor

taariq commented Nov 4, 2024

also 4 your info: i cant change gas-prices in config fewer that 0.01ugrain for paloma. If i will change this value to 0.001 or to 0.009 I'll instantly get a bunch of errors in my pigeon logs.

Will you open up a new ticket with that and the errors please?

@taariq
Copy link
Contributor

taariq commented Nov 4, 2024

We made no changes in the protocol for this so this should not be an issue, but let's see the errors.

@cosmoscats
Copy link
Author

also 4 your info: i cant change gas-prices in config fewer that 0.01ugrain for paloma. If i will change this value to 0.001 or to 0.009 I'll instantly get a bunch of errors in my pigeon logs.

Will you open up a new ticket with that and the errors please?

#1315

@cosmoscats
Copy link
Author

another jail consensus not achieved
reason: No evidence supplied for contentious message 365041

@cosmoscats
Copy link
Author

another jail
reason: No evidence supplied for contentious message 365797
its again related to ETH RPC (?). but i have changed it to Ankr from Alchemy.

paloma@server-31-24-56-54:~/.pigeon$ sudo journalctl -u pigeond | grep '365797'
Nov 05 01:11:48 server-31-24-56-54 pigeon[3301754]: time="2024-11-05T01:11:48Z" level=info msg="attesting 1 messages" action=attest message-ids="[365797]" messages-to-attest="[365797]" queue-name=evm/eth-main/reference-block x-correlation-id=cskmrv5j182pe4jdat2g
Nov 05 01:13:00 server-31-24-56-54 pigeon[3301754]: time="2024-11-05T01:13:00Z" level=error msg="error attesting messages" action=attest error="rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: validator palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg cannot be a pigeon: validator is jailed [cosmos/[email protected]/baseapp/baseapp.go:1023] with gas used: '37828': unknown request" message-ids="[365797]" messages-to-attest="[365797]" queue-name=evm/eth-main/reference-block x-correlation-id=cskmrv5j182pe4jdat2g
Nov 05 01:14:00 server-31-24-56-54 pigeon[3301754]: time="2024-11-05T01:14:00Z" level=error msg="failed to send Paloma status update" action=attest error="failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block" message-ids="[365797]" messages-to-attest="[365797]" queue-name=evm/eth-main/reference-block x-correlation-id=cskmrv5j182pe4jdat2g

@cosmoscats
Copy link
Author

another jail
reason: No evidence supplied for contentious message 368578

its again related to ETH RPC (?)

Nov 07 03:51:06 server-31-24-56-54 pigeon[3301754]: time="2024-11-07T03:51:06Z" level=info msg="attesting 1 messages" action=attest message-ids="[368578]" messages-to-attest="[368578]" queue-name=evm/eth-main/validators-balances x-correlation-id=csm3ialj182pe4jrijk0
Nov 07 03:54:12 server-31-24-56-54 pigeon[3301754]: time="2024-11-07T03:54:12Z" level=error msg="error attesting messages" action=attest error="rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: validator palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg cannot be a pigeon: validator is jailed [cosmos/[email protected]/baseapp/baseapp.go:1023] with gas used: '42094': unknown request" message-ids="[368578]" messages-to-attest="[368578]" queue-name=evm/eth-main/validators-balances x-correlation-id=csm3ialj182pe4jrijk0
Nov 07 03:56:12 server-31-24-56-54 pigeon[3301754]: time="2024-11-07T03:56:12Z" level=error msg="failed to send Paloma status update" action=attest error="failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block" message-ids="[368578]" messages-to-attest="[368578]" queue-name=evm/eth-main/validators-balances x-correlation-id=csm3ialj182pe4jrijk0

@taariq
Copy link
Contributor

taariq commented Nov 7, 2024

Yeah. the message didn't get included in a block and you've updated the gas issue. This is a pigeon issue. We will need some time to get to this. we're blocked for a bit with Paloma L2 work.

@cosmoscats
Copy link
Author

jail 6 times in a day.

today i have seen some strange logs.

can you look at it ?

Nov 15 14:21:37 server-31-24-56-54 pigeon[302394]: time="2024-11-15T14:21:37Z" level=error msg="failed to send paloma status update" chain-reference-id=bnb-main error="failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block" message-id=381788 message-type="*types.Message_SubmitLogicCall" msg-bytes-to-sign="[164 230 4 64 247 15 44 175 246 180 14 69 236 109 112 108 163 66 249 128 179 161 162 12 88 118 186 213 146 62 83 224]" msg-eth-sender=0xEe21301aF1d9562B5cBEdf520077Ea0a9bC9d535 msg-id=381788 msg-msg="turnstoneID:"22497131\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" chainReferenceID:"bnb-main" submitLogicCall:<hexContractAddress:"0xf5A21C45815b2801B00FdB5E7047BFDE97152040" payload:"F-\000e\000\000\000\000\000\000\000\000\000\000\000\000\305\360\367\266gd\366\354\214\215\377{\246\203\020\"\225\341d\t\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\333\325\266/+\311\272\345\024\315\213k\374Z\006\273K\254\003Ty\3612m\215\221\264A\346A\262/" deadline:1731680290 senderAddress:"\333\325\266/+\311\272\345\024\315\213k\374Z\006\273K\254\003Ty\3612m\215\221\264A\346A\262/" contractAddress:"\333\325\266/+\311\272\345\024\315\213k\374Z\006\273K\254\003Ty\3612m\215\221\264A\346A\262/" executionRequirements:<> > assignee:"palomavaloper1c8uxkkz00qn97wsp9um8y3a8wlmrv3c546jw4q" assigned_at_block_height:"27063750" assigneeRemoteAddress:"0xEe21301aF1d9562B5cBEdf520077Ea0a9bC9d535" " msg-nonce="[0 0 0 0 0 5 211 92]" msg-public-access-data="[]" queue-name=evm/bnb-main/evm-turnstone-message queue-type-name=evm/bnb-main/evm-turnstone-message x-correlation-id=csrlfhdj182mmkhl409g
Nov 15 14:21:37 server-31-24-56-54 pigeon[302394]: time="2024-11-15T14:21:37Z" level=warning msg="Failed to estimate messages" error="failed to find assignee eth address: assignee's eth address not found"
Nov 15 14:21:37 server-31-24-56-54 pigeon[302394]: time="2024-11-15T14:21:37Z" level=info msg="No valid estimates, skipping" action=estimate message-ids="[381788 381789 381790 381791 381799 381800 381801]" messages-to-estimate="[381788 381789 381790 381791 381799 381800 381801]" queue-name=evm/bnb-main/evm-turnstone-message

Jail reason:
reason: No evidence supplied for contentious message 381796

@cosmoscats
Copy link
Author

Hello, when i will check this issue?
Our node keep jailing every day

@taariq
Copy link
Contributor

taariq commented Dec 5, 2024

@cosmoscats as this is a one-machine issue, and not validator-wide, we will need to check your setup. We'll attempt to get to that, but are a little behind. Please bear with us.

@cosmoscats
Copy link
Author

@cosmoscats as this is a one-machine issue, and not validator-wide, we will need to check your setup. We'll attempt to get to that, but are a little behind. Please bear with us.

Recovering from prison and finding a solution takes so long that holding a node on our part may not be worthwhile.
We lose both time and our own money buying different paid rpc's with no results in two months.

@taariq
Copy link
Contributor

taariq commented Dec 5, 2024 •
edited
Loading

We agree @cosmoscats, but don't give up yet. We need you on the network and will jump into this issue. Please allow us to follow-up on Monday.

@taariq
Copy link
Contributor

taariq commented Dec 10, 2024

Hey there @cosmoscats. We are still here. We'll follow-up with you by Friday. In the Interim, will you please open up a Discord Support ticket for us to bring the team to help check your setup?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maharifu maharifu

Labels
bug Something isn't working
Projects
Paloma
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@taariq @maharifu @cosmoscats and others