Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Snark worker crashes #6028

Closed
gretzke opened this issue Sep 18, 2020 · 7 comments
Closed

Snark worker crashes #6028

gretzke opened this issue Sep 18, 2020 · 7 comments
Assignees
@mrmr1993

Comments

@gretzke
Copy link

gretzke commented Sep 18, 2020

Snark worker crashes shortly after startup (roughly every 2 minutes)

crash report:

2020-09-18 14:05:18 UTC [Fatal] Snark worker process died with a nonzero error 1
2020-09-18 14:05:18 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
	exn: "(monitor.ml.Error (\"Coda_lib.Snark_worker_error(1)\")\n  (\"Raised at file \\\"src/lib/coda_lib/coda_lib.ml\\\", line 183, characters 12-53\"\n    \"Called from file \\\"src/deferred0.ml\\\", line 56, characters 64-69\"\n    \"Called from file \\\"src/job_queue.ml\\\" (inlined), line 131, characters 2-5\"\n    \"Called from file \\\"src/job_queue.ml\\\", line 171, characters 6-47\"\n    \"Caught by monitor coda\"))"
{"timestamp":"2020-09-18 14:05:18.140078Z","level":"Error","source":{"module":"Snark_worker__Prod","location":"File \"src/lib/snark_worker/prod.ml\", line 65, characters 16-28"},"message":"SNARK worker failed: $error","metadata":{"error":"( \"Snarky.Checked_runner.Runtime_error(_, _, _, _)\\\n \\n\\\n \\nEncountered an error while evaluating the checked computation:\\\n \\n  (Failure\\\n \\n   \\\"Constraint unsatisfied (unreduced):\\\\\\\n \\n  \\\\nChoose_preimage\\\\\\\n \\n  \\\\napply_tagged_transaction: File \\\\\\\"src/lib/transaction_snark/transaction_snark.ml\\\\\\\", line 1049, characters 2-31259\\\\\\\n \\n  \\\\nmain: File \\\\\\\"src/lib/transaction_snark/transaction_snark.ml\\\\\\\", line 1780, characters 2-2140\\\\\\\n \\n  \\\\nrule_main\\\\\\\n \\n  \\\\nstep_main\\\\\\\n \\n  \\\\n\\\\\\\n \\n  \\\\nConstraint:\\\\\\\n \\n  \\\\n(((basic(R1CS(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Scale 9223372036854775808(Var 56978))(Scale 4611686018427387904(Var 56977)))(Scale 2305843009213693952(Var 56976)))(Scale 1152921504606846976(Var 56975)))(Scale 576460752303423488(Var 56974)))(Scale 288230376151711744(Var 56973)))(Scale 144115188075855872(Var 56972)))(Scale 72057594037927936(Var 56971)))(Scale 36028797018963968(Var 56970)))(Scale 18014398509481984(Var 56969)))(Scale 9007199254740992(Var 56968)))(Scale 4503599627370496(Var 56967)))(Scale 2251799813685248(Var 56966)))(Scale 1125899906842624(Var 56965)))(Scale 562949953421312(Var 56964)))(Scale 281474976710656(Var 56963)))(Scale 140737488355328(Var 56962)))(Scale 70368744177664(Var 56961)))(Scale 35184372088832(Var 56960)))(Scale 17592186044416(Var 56959)))(Scale 8796093022208(Var 56958)))(Scale 4398046511104(Var 56957)))(Scale 2199023255552(Var 56956)))(Scale 1099511627776(Var 56955)))(Scale 549755813888(Var 56954)))(Scale 274877906944(Var 56953)))(Scale 137438953472(Var 56952)))(Scale 68719476736(Var 56951)))(Scale 34359738368(Var 56950)))(Scale 17179869184(Var 56949)))(Scale 8589934592(Var 56948)))(Scale 4294967296(Var 56947)))(Scale 2147483648(Var 56946)))(Scale 1073741824(Var 56945)))(Scale 536870912(Var 56944)))(Scale 268435456(Var 56943)))(Scale 134217728(Var 56942)))(Scale 67108864(Var 56941)))(Scale 33554432(Var 56940)))(Scale 16777216(Var 56939)))(Scale 8388608(Var 56938)))(Scale 4194304(Var 56937)))(Scale 2097152(Var 56936)))(Scale 1048576(Var 56935)))(Scale 524288(Var 56934)))(Scale 262144(Var 56933)))(Scale 131072(Var 56932)))(Scale 65536(Var 56931)))(Scale 32768(Var 56930)))(Scale 16384(Var 56929)))(Scale 8192(Var 56928)))(Scale 4096(Var 56927)))(Scale 2048(Var 56926)))(Scale 1024(Var 56925)))(Scale 512(Var 56924)))(Scale 256(Var 56923)))(Scale 128(Var 56922)))(Scale 64(Var 56921)))(Scale 32(Var 56920)))(Scale 16(Var 56919)))(Scale 8(Var 56918)))(Scale 4(Var 56917)))(Scale 2(Var 56916)))(Var 56915))(Constant 1)(Add(Add(Var 1423)(Scale 2(Add(Var 1424)(Scale 2(Add(Var 1425)(Scale 2(Add(Var 1426)(Scale 2(Add(Var 1427)(Scale 2(Add(Var 1428)(Scale 2(Add(Var 1429)(Scale 2(Add(Var 1430)(Scale 2(Add(Var 1431)(Scale 2(Add(Var 1432)(Scale 2(Add(Var 1433)(Scale 2(Add(Var 1434)(Scale 2(Add(Var 1435)(Scale 2(Add(Var 1436)(Scale 2(Add(Var 1437)(Scale 2(Add(Var 1438)(Scale 2(Add(Var 1439)(Scale 2(Add(Var 1440)(Scale 2(Add(Var 1441)(Scale 2(Add(Var 1442)(Scale 2(Add(Var 1443)(Scale 2(Add(Var 1444)(Scale 2(Add(Var 1445)(Scale 2(Add(Var 1446)(Scale 2(Add(Var 1447)(Scale 2(Add(Var 1448)(Scale 2(Add(Var 1449)(Scale 2(Add(Var 1450)(Scale 2(Add(Var 1451)(Scale 2(Add(Var 1452)(Scale 2(Add(Var 1453)(Scale 2(Add(Var 1454)(Scale 2(Add(Var 1455)(Scale 2(Add(Var 1456)(Scale 2(Add(Var 1457)(Scale 2(Add(Var 1458)(Scale 2(Add(Var 1459)(Scale 2(Add(Var 1460)(Scale 2(Add(Var 1461)(Scale 2(Add(Var 1462)(Scale 2(Add(Var 1463)(Scale 2(Add(Var 1464)(Scale 2(Add(Var 1465)(Scale 2(Add(Var 1466)(Scale 2(Add(Var 1467)(Scale 2(Add(Var 1468)(Scale 2(Add(Var 1469)(Scale 2(Add(Var 1470)(Scale 2(Add(Var 1471)(Scale 2(Add(Var 1472)(Scale 2(Add(Var 1473)(Scale 2(Add(Var 1474)(Scale 2(Add(Var 1475)(Scale 2(Add(Var 1476)(Scale 2(Add(Var 1477)(Scale 2(Add(Var 1478)(Scale 2(Add(Var 1479)(Scale 2(Add(Var 1480)(Scale 2(Add(Var 1481)(Scale 2(Add(Var 1482)(Scale 2(Add(Var 1483)(Scale 2(Add(Var 1484)(Scale 2(Add(Var 1485)(Scale 2(Var 1486)))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))(Add(Var 882)(Scale 2(Add(Var 883)(Scale 2(Add(Var 884)(Scale 2(Add(Var 885)(Scale 2(Add(Var 886)(Scale 2(Add(Var 887)(Scale 2(Add(Var 888)(Scale 2(Add(Var 889)(Scale 2(Add(Var 890)(Scale 2(Add(Var 891)(Scale 2(Add(Var 892)(Scale 2(Add(Var 893)(Scale 2(Add(Var 894)(Scale 2(Add(Var 895)(Scale 2(Add(Var 896)(Scale 2(Add(Var 897)(Scale 2(Add(Var 898)(Scale 2(Add(Var 899)(Scale 2(Add(Var 900)(Scale 2(Add(Var 901)(Scale 2(Add(Var 902)(Scale 2(Add(Var 903)(Scale 2(Add(Var 904)(Scale 2(Add(Var 905)(Scale 2(Add(Var 906)(Scale 2(Add(Var 907)(Scale 2(Add(Var 908)(Scale 2(Add(Var 909)(Scale 2(Add(Var 910)(Scale 2(Add(Var 911)(Scale 2(Add(Var 912)(Scale 2(Add(Var 913)(Scale 2(Add(Var 914)(Scale 2(Add(Var 915)(Scale 2(Add(Var 916)(Scale 2(Add(Var 917)(Scale 2(Add(Var 918)(Scale 2(Add(Var 919)(Scale 2(Add(Var 920)(Scale 2(Add(Var 921)(Scale 2(Add(Var 922)(Scale 2(Add(Var 923)(Scale 2(Add(Var 924)(Scale 2(Add(Var 925)(Scale 2(Add(Var 926)(Scale 2(Add(Var 927)(Scale 2(Add(Var 928)(Scale 2(Add(Var 929)(Scale 2(Add(Var 930)(Scale 2(Add(Var 931)(Scale 2(Add(Var 932)(Scale 2(Add(Var 933)(Scale 2(Add(Var 934)(Scale 2(Add(Var 935)(Scale 2(Add(Var 936)(Scale 2(Add(Var 937)(Scale 2(Add(Var 938)(Scale 2(Add(Var 939)(Scale 2(Add(Var 940)(Scale 2(Add(Var 941)(Scale 2(Add(Var 942)(Scale 2(Add(Var 943)(Scale 2(Add(Var 944)(Scale 2(Var 945))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))(annotation(Choose_preimage))))\\\\\\\n \\n  \\\\nData:\\\\\\\n \\n  \\\\nR1CS 99999999 1 18446744073809551615\\\")\\\n \\n\\\n \\nLabel stack trace:\\\n \\napply_tagged_transaction: File \\\"src/lib/transaction_snark/transaction_snark.ml\\\", line 1049, characters 2-31259\\\n \\nmain: File \\\"src/lib/transaction_snark/transaction_snark.ml\\\", line 1780, characters 2-2140\\\n \\nrule_main\\\n \\nstep_main\\\n \\n\\\n \\n\\\n \\nRaised at file \\\"stdlib.ml\\\", line 33, characters 17-33\\\n \\nCalled from file \\\"src/lib/snarky/src/base/checked_runner.ml\\\", line 188, characters 8-329\\\n \\nCalled from file \\\"src/lib/snarky/src/base/checked_runner.ml\\\", line 343, characters 8-12\\\n \\n\")","pid":147,"spec":"(Transition((source 26744230021640435561250188191109816291210225230386908802995841963063874518666)(target 7652866509765560698740885284367264425295442289283273398088252204754408525447)(supply_increase 0)(pending_coinbase_stack_state((source((data 26997344157949065971920142738077924826969668499788300536286958767257216753756)(state((init 7249477346936434202472111470254572440811516970381384443547208351766864108376)(curr 19292898327577347625577770821274423003912221286320981301588314885382082181727)))))(target((data 26997344157949065971920142738077924826969668499788300536286958767257216753756)(state((init 7249477346936434202472111470254572440811516970381384443547208351766864108376)(curr 19292898327577347625577770821274423003912221286320981301588314885382082181727)))))))(fee_excess((fee_token_l 1)(fee_excess_l((magnitude 100000000)(sgn Pos)))(fee_token_r 1)(fee_excess_r((magnitude 0)(sgn Pos)))))(next_available_token_before 25)(next_available_token_after 25)(sok_digest()))(User_command((payload((common((fee 100000000)(fee_token 1)(fee_payer_pk B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(nonce 4)(valid_until 4294967295)(memo\"\\001\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\\000\")))(body(Mint_tokens((token_id 9)(token_owner_pk B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(receiver_pk B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(amount 18446744073709551615))))))(signer B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(signature(17924292211209113585886508831365610873704138891708085085343937325621473266889 425369572189849097611253492584973331221470452335096742458109965399754353867))))((ledger((indexes(((B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT 9)1036)((B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT 9)1036)((B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT 1)576)))(depth 14)(tree(Node 26744230021640435561250188191109816291210225230386908802995841963063874518666(Node 21392059331522142744448452530817365357111402191397357407199387384627426332828(Node 11725098487825380011807068181582138383360686360790259379150254741375703843318(Node 25625728245993514344480483928375878673015082430228388175172998661166385224212(Node 3001498711446953377530325390985150130429970907247437530639223268401307102507(Hash 15670084179661016929514079603480344652245896550983883951534125453924693875638)(Node 23694559679223509853456606013269641188108327892770995022922688905830439560894(Node 28001923514897303156569687877180397557208293043127759606450640017531970758420(Node 26453690441719970114603278039568283856683473470749460512851575023648289165320(Hash 1616463777494839023937933610789444873322734024847147273760345867640520392509)(Node 13007209329759822045354730163999510948130450533006515984627023591100274724662(Node 26580490740612434060133087739396640873344761659221876528439552948599356900525(Node 40029964702826374307398533805591669909350251233240005095209908875400580958(Node 1879857650799324659587130521411315009698495554797630792683768086439106213963(Node 27576427349691215805830300628220428656234577092054823650199782214883003017829(Node 7123984522702397904457435315954693634864201148618424675616760212591364436120(Account((public_key B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(token_id 1)(token_permissions(Not_owned(account_disabled false)))(balance 458684000000)(nonce 4)(receipt_chain_hash 28175345317911251749461387121864597851594616104944175597524453267179600004613)(delegate B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(voting_for 0)(timing Untimed)))(Hash 4353198019888216971268585171223925691076042183079970017534528381492571463032))(Hash 8772338407348629991714469104940120639852473880974730201810040669612962458892))(Hash 9177631864047978984257950941458375048908187914752031123321390138832098063196))(Hash 17657891353680990594382733642987654082977293626288083805223245590481752268891))(Hash 6616878115464086820263525206984382423683103342065126458088869972114564745154))(Hash 13489915987872190350694508069216602011867926255369860025205269711076038927131)))(Hash 28037232368113487990367789584876175328121518753530397752449802041289113501600))(Hash 1386551745938875674738781393214175394244404311178938245360321552705025347103)))(Node 7573112624599697402715627046102608918986505352536719007612273726575743191151(Node 17795177467543709209384312863121020626795964272255652607117569215292513207416(Node 26161327889485389492109891202678203025034753697572049057172933779580161783769(Node 8612656198993461327994743136672908670438169780580254578451168097405056748584(Node 4599049260867662741416675928994502835568066906085122462251587141371730800439(Node 22830703369794733874470684029128608400383258923721111832190418996691931063678(Node 6368585980548479674998567284737302158461296641620443384373589736929089344166(Hash 17228971002727383382043653276833075498780662714138150897041796017610693617828)(Node 14855024936872913081372142109843358743297392425497584128750881132470144051190(Hash 25260854258570044435086427580824115700160033822711177113693760539290960794320)(Node 15225397565999904938629062118139153599473161158674948314814410238732624384888(Node 12908449912910093819238465739070422257588723137533847497114786671385223350695(Account((public_key B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(token_id 9)(token_permissions(Token_owned(disable_new_accounts false)))(balance 0)(nonce 0)(receipt_chain_hash 15652585235832934921583570765124908656262401037373930895195709801952924295710)(delegate B62qiTKpEPjGTSHZrtM8uXiKgn8So916pLmNJKDhKeyBQL9TDb3nvBG)(voting_for 0)(timing Untimed)))(Hash 14583427262248221979180091564232955188000386646287302853332233205378640889909))(Hash 27355720372690962104054819738065723492206208148109801998957901646985321789336))))(Hash 26018236105887470795631393271484765619915439353447700081216046995973292429047))(Hash 24670948897678800850469166264387812345243148731733182814359950532441166009814))(Hash 19037790595742324731008215269424411031039875753226042883761281351432239020735))(Hash 5991895123077538663571569381770353614051750395398743534641041336163769326404))(Hash 20186076215208085838601165213386167974027758767076534863092406101873737175013))(Hash 24033266339169900877351832551071769000684000939585006171903225629998891287808)))(Hash 1900135125302463214394010648604411958151825345055406313393935004355233472345))(Hash 15021775575191935322932253138368222756937311369240198571933653681461524869092))(Hash 1055801623879847096628433511678534132188122629115779973820670796633139586780)))(next_available_token 25)))(protocol_state_body((genesis_state_hash 7972428409254861556390111253648040251978614088739790308121422272802905213246)(blockchain_state((staged_ledger_hash((non_snark((ledger_hash 7550862602858716022515516376856782306727710928041610920722020131337549487234)(aux_hash\"n\\141\\185\\162.\\238|$\\132dN\\220\\226\\138j\\018=\\157>1\\175\\246\\164\\239\\149\\172\\188\\166Z\\011Q\\158\")(pending_coinbase_aux\"/\\130\\229\\251\\141e\\232\\001\\247\\236\\007x\\136\\225\\184!&\\1959w\\223\\157\\172*\\b\\146\\134Y\\248s3\\148\")))(pending_coinbase_hash 17940004587433749251174024167577017711258176173894295915178394667280724543619)))(snarked_ledger_hash 9531901115761935764758534987471432580407825852720663966824029456410684649371)(snarked_next_available_token 2)(timestamp 1600335200000)))(consensus_state((blockchain_length 121)(epoch_count 0)(min_window_density 160)(sub_window_densities(0 20 20 20 20 20 20 20))(last_vrf_output\"\\208\\247JI\\148Rw\\254\\214X\\148g\\162\\\\|\\147\\140\\170\\158[\\179j\\b\\016\\193\\030\\024\\214\\255\\024R\\004\")(total_currency 90982100000001000)(curr_global_slot((slot_number 355)(slots_per_epoch 480)))(staking_epoch_data((ledger((hash 9531901115761935764758534987471432580407825852720663966824029456410684649371)(total_currency 90982100000001000)))(seed 0)(start_checkpoint 0)(lock_checkpoint 0)(epoch_length 1)))(next_epoch_data((ledger((hash 9531901115761935764758534987471432580407825852720663966824029456410684649371)(total_currency 90982100000001000)))(seed 21792316436254650149327858098687852676553342335811966816389267041363881678203)(start_checkpoint 0)(lock_checkpoint 18129034016165038021433722489384175216853726787068625736021978501149574146563)(epoch_length 122)))(has_ancestor_in_same_checkpoint_window true)))(constants((k 20)(delta 3)(genesis_state_timestamp 1600271300000)))))(init_stack((data 26997344157949065971920142738077924826969668499788300536286958767257216753756)(state((init 7249477346936434202472111470254572440811516970381384443547208351766864108376)(curr 13334482185597395990325781415248015616768158918657357382784782605069596255296)))))(status(Applied((fee_payer_account_creation_fee_paid())(receiver_account_creation_fee_paid())(created_token()))))))"}}
(monitor.ml.Error
 ( "Snarky.Checked_runner.Runtime_error(_, _, _, _)\
  \n\
  \nEncountered an error while evaluating the checked computation:\
  \n  (Failure\
  \n   \"Constraint unsatisfied (unreduced):\\\
  \n  \\nChoose_preimage\\\
  \n  \\napply_tagged_transaction: File \\\"src/lib/transaction_snark/transaction_snark.ml\\\", line 1049, characters 2-31259\\\
  \n  \\nmain: File \\\"src/lib/transaction_snark/transaction_snark.ml\\\", line 1780, characters 2-2140\\\
  \n  \\nrule_main\\\
  \n  \\nstep_main\\\
  \n  \\n\\\
  \n  \\nConstraint:\\\
  \n  \\n(((basic(R1CS(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Add(Scale 9223372036854775808(Var 56978))(Scale 4611686018427387904(Var 56977)))(Scale 2305843009213693952(Var 56976)))(Scale 1152921504606846976(Var 56975)))(Scale 576460752303423488(Var 56974)))(Scale 288230376151711744(Var 56973)))(Scale 144115188075855872(Var 56972)))(Scale 72057594037927936(Var 56971)))(Scale 36028797018963968(Var 56970)))(Scale 18014398509481984(Var 56969)))(Scale 9007199254740992(Var 56968)))(Scale 4503599627370496(Var 56967)))(Scale 2251799813685248(Var 56966)))(Scale 1125899906842624(Var 56965)))(Scale 562949953421312(Var 56964)))(Scale 281474976710656(Var 56963)))(Scale 140737488355328(Var 56962)))(Scale 70368744177664(Var 56961)))(Scale 35184372088832(Var 56960)))(Scale 17592186044416(Var 56959)))(Scale 8796093022208(Var 56958)))(Scale 4398046511104(Var 56957)))(Scale 2199023255552(Var 56956)))(Scale 1099511627776(Var 56955)))(Scale 549755813888(Var 56954)))(Scale 274877906944(Var 56953)))(Scale 137438953472(Var 56952)))(Scale 68719476736(Var 56951)))(Scale 34359738368(Var 56950)))(Scale 17179869184(Var 56949)))(Scale 8589934592(Var 56948)))(Scale 4294967296(Var 56947)))(Scale 2147483648(Var 56946)))(Scale 1073741824(Var 56945)))(Scale 536870912(Var 56944)))(Scale 268435456(Var 56943)))(Scale 134217728(Var 56942)))(Scale 67108864(Var 56941)))(Scale 33554432(Var 56940)))(Scale 16777216(Var 56939)))(Scale 8388608(Var 56938)))(Scale 4194304(Var 56937)))(Scale 2097152(Var 56936)))(Scale 1048576(Var 56935)))(Scale 524288(Var 56934)))(Scale 262144(Var 56933)))(Scale 131072(Var 56932)))(Scale 65536(Var 56931)))(Scale 32768(Var 56930)))(Scale 16384(Var 56929)))(Scale 8192(Var 56928)))(Scale 4096(Var 56927)))(Scale 2048(Var 56926)))(Scale 1024(Var 56925)))(Scale 512(Var 56924)))(Scale 256(Var 56923)))(Scale 128(Var 56922)))(Scale 64(Var 56921)))(Scale 32(Var 56920)))(Scale 16(Var 56919)))(Scale 8(Var 56918)))(Scale 4(Var 56917)))(Scale 2(Var 56916)))(Var 56915))(Constant 1)(Add(Add(Var 1423)(Scale 2(Add(Var 1424)(Scale 2(Add(Var 1425)(Scale 2(Add(Var 1426)(Scale 2(Add(Var 1427)(Scale 2(Add(Var 1428)(Scale 2(Add(Var 1429)(Scale 2(Add(Var 1430)(Scale 2(Add(Var 1431)(Scale 2(Add(Var 1432)(Scale 2(Add(Var 1433)(Scale 2(Add(Var 1434)(Scale 2(Add(Var 1435)(Scale 2(Add(Var 1436)(Scale 2(Add(Var 1437)(Scale 2(Add(Var 1438)(Scale 2(Add(Var 1439)(Scale 2(Add(Var 1440)(Scale 2(Add(Var 1441)(Scale 2(Add(Var 1442)(Scale 2(Add(Var 1443)(Scale 2(Add(Var 1444)(Scale 2(Add(Var 1445)(Scale 2(Add(Var 1446)(Scale 2(Add(Var 1447)(Scale 2(Add(Var 1448)(Scale 2(Add(Var 1449)(Scale 2(Add(Var 1450)(Scale 2(Add(Var 1451)(Scale 2(Add(Var 1452)(Scale 2(Add(Var 1453)(Scale 2(Add(Var 1454)(Scale 2(Add(Var 1455)(Scale 2(Add(Var 1456)(Scale 2(Add(Var 1457)(Scale 2(Add(Var 1458)(Scale 2(Add(Var 1459)(Scale 2(Add(Var 1460)(Scale 2(Add(Var 1461)(Scale 2(Add(Var 1462)(Scale 2(Add(Var 1463)(Scale 2(Add(Var 1464)(Scale 2(Add(Var 1465)(Scale 2(Add(Var 1466)(Scale 2(Add(Var 1467)(Scale 2(Add(Var 1468)(Scale 2(Add(Var 1469)(Scale 2(Add(Var 1470)(Scale 2(Add(Var 1471)(Scale 2(Add(Var 1472)(Scale 2(Add(Var 1473)(Scale 2(Add(Var 1474)(Scale 2(Add(Var 1475)(Scale 2(Add(Var 1476)(Scale 2(Add(Var 1477)(Scale 2(Add(Var 1478)(Scale 2(Add(Var 1479)(Scale 2(Add(Var 1480)(Scale 2(Add(Var 1481)(Scale 2(Add(Var 1482)(Scale 2(Add(Var 1483)(Scale 2(Add(Var 1484)(Scale 2(Add(Var 1485)(Scale 2(Var 1486)))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))(Add(Var 882)(Scale 2(Add(Var 883)(Scale 2(Add(Var 884)(Scale 2(Add(Var 885)(Scale 2(Add(Var 886)(Scale 2(Add(Var 887)(Scale 2(Add(Var 888)(Scale 2(Add(Var 889)(Scale 2(Add(Var 890)(Scale 2(Add(Var 891)(Scale 2(Add(Var 892)(Scale 2(Add(Var 893)(Scale 2(Add(Var 894)(Scale 2(Add(Var 895)(Scale 2(Add(Var 896)(Scale 2(Add(Var 897)(Scale 2(Add(Var 898)(Scale 2(Add(Var 899)(Scale 2(Add(Var 900)(Scale 2(Add(Var 901)(Scale 2(Add(Var 902)(Scale 2(Add(Var 903)(Scale 2(Add(Var 904)(Scale 2(Add(Var 905)(Scale 2(Add(Var 906)(Scale 2(Add(Var 907)(Scale 2(Add(Var 908)(Scale 2(Add(Var 909)(Scale 2(Add(Var 910)(Scale 2(Add(Var 911)(Scale 2(Add(Var 912)(Scale 2(Add(Var 913)(Scale 2(Add(Var 914)(Scale 2(Add(Var 915)(Scale 2(Add(Var 916)(Scale 2(Add(Var 917)(Scale 2(Add(Var 918)(Scale 2(Add(Var 919)(Scale 2(Add(Var 920)(Scale 2(Add(Var 921)(Scale 2(Add(Var 922)(Scale 2(Add(Var 923)(Scale 2(Add(Var 924)(Scale 2(Add(Var 925)(Scale 2(Add(Var 926)(Scale 2(Add(Var 927)(Scale 2(Add(Var 928)(Scale 2(Add(Var 929)(Scale 2(Add(Var 930)(Scale 2(Add(Var 931)(Scale 2(Add(Var 932)(Scale 2(Add(Var 933)(Scale 2(Add(Var 934)(Scale 2(Add(Var 935)(Scale 2(Add(Var 936)(Scale 2(Add(Var 937)(Scale 2(Add(Var 938)(Scale 2(Add(Var 939)(Scale 2(Add(Var 940)(Scale 2(Add(Var 941)(Scale 2(Add(Var 942)(Scale 2(Add(Var 943)(Scale 2(Add(Var 944)(Scale 2(Var 945))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))(annotation(Choose_preimage))))\\\
  \n  \\nData:\\\
  \n  \\nR1CS 99999999 1 18446744073809551615\")\
  \n\
  \nLabel stack trace:\
  \napply_tagged_transaction: File \"src/lib/transaction_snark/transaction_snark.ml\", line 1049, characters 2-31259\
  \nmain: File \"src/lib/transaction_snark/transaction_snark.ml\", line 1780, characters 2-2140\
  \nrule_main\
  \nstep_main\
  \n\
  \n\
  \nRaised at file \"stdlib.ml\", line 33, characters 17-33\
  \nCalled from file \"src/lib/snarky/src/base/checked_runner.ml\", line 188, characters 8-329\
  \nCalled from file \"src/lib/snarky/src/base/checked_runner.ml\", line 343, characters 8-12\
  \n")
 ("Raised at file \"src/error.ml\" (inlined), line 9, characters 14-30"
  "Called from file \"src/lib/snark_worker/prod.ml\", line 74, characters 16-29"
  "Called from file \"src/lib/snark_worker/functor.ml\", line 72, characters 10-116"
  "Called from file \"src/lib/one_or_two/one_or_two.ml\", line 83, characters 22-25"
  "Called from file \"src/lib/snark_worker/functor.ml\", line 69, characters 4-355"
  "Called from file \"src/lib/snark_worker/functor.ml\", line 187, characters 16-45"
  "Called from file \"src/deferred0.ml\", line 56, characters 64-69"
  "Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5"
  "Called from file \"src/job_queue.ml\", line 171, characters 6-47"))
@evstigneeff
Copy link

Yeah it keeps crashing for me every 2-3 mins too (using coda-daemon:0.0.16-beta6 docker image).

   2020-09-18 14:48:25 UTC [Fatal] Snark worker process died with a nonzero error 1
   2020-09-18 14:48:25 UTC [Fatal] Unhandled top-level exception: $exn
   Generating crash report
   exn: "(monitor.ml.Error (\"Coda_lib.Snark_worker_error(1)\")\n  (\"Raised at file \\\"src/lib/coda_lib/coda_lib.ml\\\", line 183, characters 12-53\"\n    \"Called from file \\\"src/deferred0.ml\\\", line 56, characters 64-69\"\n    \"Called from file \\\"src/job_queue.ml\\\" (inlined), line 131, characters 2-5\"\n    \"Called from file \\\"src/job_queue.ml\\\", line 171, characters 6-47\"\n    \"Caught by monitor coda\"))"
     + echo 'Coda process exited with status code 1'
    Coda process exited with status code 1
    + sleep 10


  ☠  Coda Daemon crashed.
  The Coda Protocol developers would like to know why!

  Please:
  Open an issue:
    <https://github.com/CodaProtocol/coda/issues/new>

  Briefly describe what you were doing and attach the crash report /root/.coda-config/coda_crash_report_2020-09-18_14-48-26.164172.tar.gz

See below a generated crash report archive.
coda_crash_report_2020-09-18_14-48-26.164172.tar.gz

@mrmr1993 mrmr1993 self-assigned this Sep 18, 2020
@mrmr1993
Copy link
Member

Thanks @gretzke, managed to reproduce locally, attempting to root-cause now.

Instructions to repo:

(Transition((source 26744230021640435561250188191109816291210225230386908802995841963063874518666)(target 7652866509765560698740885284367264425295442289283273398088252204754408525447)(supply_increase 0)(pending_coinbase_stack_state((source((data 26997344157949065971920142738077924826969668499788300536286958767257216753756)(state((init 7249477346936434202472111470254572440811516970381384443547208351766864108376)(curr 19292898327577347625577770821274423003912221286320981301588314885382082181727)))))(target((data 26997344157949065971920142738077924826969668499788300536286958767257216753756)(state((init 7249477346936434202472111470254572440811516970381384443547208351766864108376)(curr 19292898327577347625577770821274423003912221286320981301588314885382082181727)))))))(fee_excess((fee_token_l 1)(fee_excess_l((magnitude 100000000)(sgn Pos)))(fee_token_r 1)(fee_excess_r((magnitude 0)(sgn Pos)))))(next_available_token_before 25)(next_available_token_after 25)(sok_digest()))(User_command((payload((common((fee 100000000)(fee_token 1)(fee_payer_pk B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(nonce 4)(valid_until 4294967295)(memo"\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000")))(body(Mint_tokens((token_id 9)(token_owner_pk B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(receiver_pk B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(amount 18446744073709551615))))))(signer B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(signature(17924292211209113585886508831365610873704138891708085085343937325621473266889 425369572189849097611253492584973331221470452335096742458109965399754353867))))((ledger((indexes(((B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT 9)1036)((B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT 9)1036)((B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT 1)576)))(depth 14)(tree(Node 26744230021640435561250188191109816291210225230386908802995841963063874518666(Node 21392059331522142744448452530817365357111402191397357407199387384627426332828(Node 11725098487825380011807068181582138383360686360790259379150254741375703843318(Node 25625728245993514344480483928375878673015082430228388175172998661166385224212(Node 3001498711446953377530325390985150130429970907247437530639223268401307102507(Hash 15670084179661016929514079603480344652245896550983883951534125453924693875638)(Node 23694559679223509853456606013269641188108327892770995022922688905830439560894(Node 28001923514897303156569687877180397557208293043127759606450640017531970758420(Node 26453690441719970114603278039568283856683473470749460512851575023648289165320(Hash 1616463777494839023937933610789444873322734024847147273760345867640520392509)(Node 13007209329759822045354730163999510948130450533006515984627023591100274724662(Node 26580490740612434060133087739396640873344761659221876528439552948599356900525(Node 40029964702826374307398533805591669909350251233240005095209908875400580958(Node 1879857650799324659587130521411315009698495554797630792683768086439106213963(Node 27576427349691215805830300628220428656234577092054823650199782214883003017829(Node 7123984522702397904457435315954693634864201148618424675616760212591364436120(Account((public_key B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(token_id 1)(token_permissions(Not_owned(account_disabled false)))(balance 458684000000)(nonce 4)(receipt_chain_hash 28175345317911251749461387121864597851594616104944175597524453267179600004613)(delegate B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(voting_for 0)(timing Untimed)))(Hash 4353198019888216971268585171223925691076042183079970017534528381492571463032))(Hash 8772338407348629991714469104940120639852473880974730201810040669612962458892))(Hash 9177631864047978984257950941458375048908187914752031123321390138832098063196))(Hash 17657891353680990594382733642987654082977293626288083805223245590481752268891))(Hash 6616878115464086820263525206984382423683103342065126458088869972114564745154))(Hash 13489915987872190350694508069216602011867926255369860025205269711076038927131)))(Hash 28037232368113487990367789584876175328121518753530397752449802041289113501600))(Hash 1386551745938875674738781393214175394244404311178938245360321552705025347103)))(Node 7573112624599697402715627046102608918986505352536719007612273726575743191151(Node 17795177467543709209384312863121020626795964272255652607117569215292513207416(Node 26161327889485389492109891202678203025034753697572049057172933779580161783769(Node 8612656198993461327994743136672908670438169780580254578451168097405056748584(Node 4599049260867662741416675928994502835568066906085122462251587141371730800439(Node 22830703369794733874470684029128608400383258923721111832190418996691931063678(Node 6368585980548479674998567284737302158461296641620443384373589736929089344166(Hash 17228971002727383382043653276833075498780662714138150897041796017610693617828)(Node 14855024936872913081372142109843358743297392425497584128750881132470144051190(Hash 25260854258570044435086427580824115700160033822711177113693760539290960794320)(Node 15225397565999904938629062118139153599473161158674948314814410238732624384888(Node 12908449912910093819238465739070422257588723137533847497114786671385223350695(Account((public_key B62qpFp3rsV4t7hcsWh5vwjid3VGHmepDyXLSVFZevJ4GU6n4XVpTQT)(token_id 9)(token_permissions(Token_owned(disable_new_accounts false)))(balance 0)(nonce 0)(receipt_chain_hash 15652585235832934921583570765124908656262401037373930895195709801952924295710)(delegate B62qiTKpEPjGTSHZrtM8uXiKgn8So916pLmNJKDhKeyBQL9TDb3nvBG)(voting_for 0)(timing Untimed)))(Hash 14583427262248221979180091564232955188000386646287302853332233205378640889909))(Hash 27355720372690962104054819738065723492206208148109801998957901646985321789336))))(Hash 26018236105887470795631393271484765619915439353447700081216046995973292429047))(Hash 24670948897678800850469166264387812345243148731733182814359950532441166009814))(Hash 19037790595742324731008215269424411031039875753226042883761281351432239020735))(Hash 5991895123077538663571569381770353614051750395398743534641041336163769326404))(Hash 20186076215208085838601165213386167974027758767076534863092406101873737175013))(Hash 24033266339169900877351832551071769000684000939585006171903225629998891287808)))(Hash 1900135125302463214394010648604411958151825345055406313393935004355233472345))(Hash 15021775575191935322932253138368222756937311369240198571933653681461524869092))(Hash 1055801623879847096628433511678534132188122629115779973820670796633139586780)))(next_available_token 25)))(protocol_state_body((genesis_state_hash 7972428409254861556390111253648040251978614088739790308121422272802905213246)(blockchain_state((staged_ledger_hash((non_snark((ledger_hash 7550862602858716022515516376856782306727710928041610920722020131337549487234)(aux_hash"n\141\185\162.\238|$\132dN\220\226\138j\018=\157>1\175\246\164\239\149\172\188\166Z\011Q\158")(pending_coinbase_aux"/\130\229\251\141e\232\001\247\236\007x\136\225\184!&\1959w\223\157\172*\b\146\134Y\248s3\148")))(pending_coinbase_hash 17940004587433749251174024167577017711258176173894295915178394667280724543619)))(snarked_ledger_hash 9531901115761935764758534987471432580407825852720663966824029456410684649371)(snarked_next_available_token 2)(timestamp 1600335200000)))(consensus_state((blockchain_length 121)(epoch_count 0)(min_window_density 160)(sub_window_densities(0 20 20 20 20 20 20 20))(last_vrf_output"\208\247JI\148Rw\254\214X\148g\162\\|\147\140\170\158[\179j\b\016\193\030\024\214\255\024R\004")(total_currency 90982100000001000)(curr_global_slot((slot_number 355)(slots_per_epoch 480)))(staking_epoch_data((ledger((hash 9531901115761935764758534987471432580407825852720663966824029456410684649371)(total_currency 90982100000001000)))(seed 0)(start_checkpoint 0)(lock_checkpoint 0)(epoch_length 1)))(next_epoch_data((ledger((hash 9531901115761935764758534987471432580407825852720663966824029456410684649371)(total_currency 90982100000001000)))(seed 21792316436254650149327858098687852676553342335811966816389267041363881678203)(start_checkpoint 0)(lock_checkpoint 18129034016165038021433722489384175216853726787068625736021978501149574146563)(epoch_length 122)))(has_ancestor_in_same_checkpoint_window true)))(constants((k 20)(delta 3)(genesis_state_timestamp 1600271300000)))))(init_stack((data 26997344157949065971920142738077924826969668499788300536286958767257216753756)(state((init 7249477346936434202472111470254572440811516970381384443547208351766864108376)(curr 13334482185597395990325781415248015616768158918657357382784782605069596255296)))))(status(Applied((fee_payer_account_creation_fee_paid())(receiver_account_creation_fee_paid())(created_token()))))))

@mrmr1993
Copy link
Member

This an overflow here in the transaction snark. Adding the transaction amount (2^64 - 1 to the fee amount 100000000 gives the overflowed number 18446744073809551615.

$ ghci
GHCi, version 8.6.3: http://www.haskell.org/ghc/  :? for help
Prelude> logBase 2 18446744073809551615
64.00000000000782
Prelude> 2 ^ 64 - 18446744073809551615
-99999999

@mrmr1993 mrmr1993 mentioned this issue Sep 18, 2020
Merged
5 tasks
@evstigneeff
Copy link

evstigneeff commented Sep 18, 2020 •
edited
Loading

@mrmr1993 Is it same issue for my case (I provided crash report above too)? If yes how could we solve it?

@mrmr1993
Copy link
Member

@evstigneeff unfortunately your crash report doesn't contain the log from the snark worker, so I wasn't able to figure out if yours was the same issue or not.

The team is currently working on a bugfix release, including the fix for this issue, there should be an update on discord when it's ready.

@evstigneeff
Copy link

@mrmr1993 Snark worker was crashing yesterday until Coda re-deployed the network from the genesis block (and built new 0.0.16-beta7 docker image) and it stared working fine then. Now it keeps crashing again (see crash report below). Your help would be greatly appreciated.

    2020-09-19 19:15:48 UTC [Info] here are the seeds: [{12D3KooWAFFq2yEQFFzhU5dt64AWqawRuomG9hL8rSmm5vxhAsgr: [/dns4/pickles.o1test.net/tcp/10001]}]
    2020-09-19 19:15:48 UTC [Info] hacking peers: $peers
    peers: "(/dns4/pickles.o1test.net/tcp/10001/p2p/12D3KooWAFFq2yEQFFzhU5dt64AWqawRuomG9hL8rSmm5vxhAsgr)"
    2020-09-19 19:15:48 UTC [Info] starting refreshing cpl 0 with key CIQAAALMM4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA (routing table size was 0)
    2020-09-19 19:15:48 UTC [Info] failed when refreshing routing table
    2020-09-19 19:15:48 UTC [Info] starting refreshing cpl 0 with key CIQAAA57REAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA (routing table size was 0)
    2020-09-19 19:15:48 UTC [Info] failed when refreshing routing table
    2020-09-19 19:15:48 UTC [Info] starting refreshing cpl 0 with key CIQAAALTPEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA (routing table size was 0)
    2020-09-19 19:15:48 UTC [Info] failed when refreshing routing table
    2020-09-19 19:15:48 UTC [Info] starting refreshing cpl 0 with key CIQAAAHYDEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA (routing table size was 0)
    2020-09-19 19:15:48 UTC [Info] failed when refreshing routing table
    2020-09-19 19:15:48 UTC [Info] starting refreshing cpl 0 with key CIQAAAJO6EAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA (routing table size was 0)
    2020-09-19 19:15:48 UTC [Info] starting refreshing cpl 0 with key CIQAAAB7VIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA (routing table size was 0)
    2020-09-19 19:15:48 UTC [Info] failed when refreshing routing table
    2020-09-19 19:15:48 UTC [Info] failed when refreshing routing table
    2020-09-19 19:15:48 UTC [Info] got error on dial: dial tcp4 127.0.0.1:42849: connect: connection refused
    2020-09-19 19:15:48 UTC [Info] got error on dial: dial tcp4 127.0.0.1:10001: connect: connection refused
    2020-09-19 19:15:48 UTC [Info] bootstrapped with 12D3KooWAFFq2yEQFFzhU5dt64AWqawRuomG9hL8rSmm5vxhAsgr
    2020-09-19 19:15:48 UTC [Info] Coda daemon is now connecting
    2020-09-19 19:15:48 UTC [Info] Coda daemon is now listening
    2020-09-19 19:15:48 UTC [Info] Pausing block production while bootstrapping
    2020-09-19 19:15:48 UTC [Info] Created GraphQL server at: http://localhost:3085/graphql
    2020-09-19 19:15:48 UTC [Info] Initializing plugins
    2020-09-19 19:15:48 UTC [Info] Daemon ready. Clients can now connect
    2020-09-19 19:15:48 UTC [Info] got error on dial: dial tcp4 127.0.0.1:10400: connect: connection refused
    2020-09-19 19:15:48 UTC [Info] got error on dial: dial tcp4 127.0.0.1:41833: connect: connection refused
    2020-09-19 19:15:48 UTC [Info] starting refreshing cpl 0 with key CIQAAAG47YAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA (routing table size was 2)
    2020-09-19 19:15:48 UTC [Info] finished refreshing cpl 0, routing table size is now 2
    2020-09-19 19:15:48 UTC [Info] starting refreshing cpl 1 with key CIQAAABQUYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA (routing table size was 2)
    2020-09-19 19:15:48 UTC [Info] finished refreshing cpl 1, routing table size is now 2
    2020-09-19 19:15:53 UTC [Info] DiscoverNAT error:no NAT found
    Transaction_snark.system: 24.935s
    {"timestamp":"2020-09-19 19:16:13.606863Z","level":"Debug","source":{"module":"Snark_worker__Functor","location":"File 
    \"src/lib/snark_worker/functor.ml\", line 163, characters 8-20"},"message":"Snark worker working directory $dir","metadata": 
    {"dir":"/root","pid":145,"process":"Snark Worker"}}
    {"timestamp":"2020-09-19 19:16:13.607347Z","level":"Debug","source":{"module":"Snark_worker__Functor","location":"File 
    \"src/lib/snark_worker/functor.ml\", line 175, characters 6-18"},"message":"Snark worker using daemon $addr","metadata": 
    {"addr":"127.0.0.1:8301","pid":145,"process":"Snark Worker"}}
    {"timestamp":"2020-09-19 19:16:20.926507Z","level":"Debug","source":{"module":"Snark_worker__Functor","location":"File 
    \"src/lib/snark_worker/functor.ml\", line 163, characters 8-20"},"message":"Snark worker working directory $dir","metadata": 
   {"dir":"/root","pid":145,"process":"Snark Worker"}}
   {"timestamp":"2020-09-19 19:16:20.926704Z","level":"Debug","source":{"module":"Snark_worker__Functor","location":"File 
   \"src/lib/snark_worker/functor.ml\", line 175, characters 6-18"},"message":"Snark worker using daemon $addr","metadata": 
   {"addr":"127.0.0.1:8301","pid":145,"process":"Snark Worker"}}
   + echo 'Coda process exited with status code 99'
   + sleep 10
   Coda process exited with status code 99
   2020-09-19 19:16:23 UTC [Info] Daemon child process 119 terminated after receiving signal "sigterm"
   2020-09-19 19:16:23 UTC [Error] Child process of kind "Verifier" with pid 119 has terminated
   + kill 13
   + '[' '!' -f stay_alive ']'
   + exit 0

@garethtdavies garethtdavies mentioned this issue Sep 21, 2020
Closed
@mrmr1993
Copy link
Member

Closing this as fixed, the transaction causing the bug now doesn't crash the snark worker since #6034.

@evstigneeff can you open a new issue for that crash? It seems to be separate from the one originally reported here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mrmr1993 mrmr1993

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mrmr1993 @gretzke @evstigneeff