Some end to end tests are flaky in the CI

[jpraynaud]

Why

Some flakiness occurs often in the CI end to end tests.

What

Fix the CI so that the end to end tests are completing in success most of the time.
We have noticed that the failures are related to different errors since we have upgraded the devnet (#1425).
Also the P2P version of the test seem to fail more often than its non P2P counterpart.

How

• Investigate the source of the problems
• Fix the problems

Flakiness History

New version of the devnet

Timeout exhausted for target epoch to be reached

• https://github.com/input-output-hk/mithril/actions/runs/7730541945/job/21076341583#step:6:345
• https://github.com/input-output-hk/mithril/actions/runs/7730541945/job/21098978755#step:6:409
• https://github.com/input-output-hk/mithril/actions/runs/7729440161/job/21073168550#step:6:230
• https://github.com/input-output-hk/mithril/actions/runs/7729440161/job/21073696381#step:6:231
• https://github.com/input-output-hk/mithril/actions/runs/7725764904/job/21060857068#step:6:330
• https://github.com/input-output-hk/mithril/actions/runs/7725727345/job/21060748226#step:6:309
• https://github.com/input-output-hk/mithril/actions/runs/7725000099/job/21059128351#step:6:408
• https://github.com/input-output-hk/mithril/actions/runs/7724354954/job/21056538735#step:6:230
• https://github.com/input-output-hk/mithril/actions/runs/7724354954/job/21056538921#step:6:230
• https://github.com/input-output-hk/mithril/actions/runs/7724213678/job/21056060534#step:6:329
• https://github.com/input-output-hk/mithril/actions/runs/7714842280/job/21028217003#step:6:345
• https://github.com/input-output-hk/mithril/actions/runs/7740918215/job/21107065721#step:6:330

Minimum expected mithril stake distribution epoch not reached : XX < YY

• https://github.com/input-output-hk/mithril/actions/runs/7730541945/job/21076341825#step:6:448
• https://github.com/input-output-hk/mithril/actions/runs/7730541945/job/21099458455#step:6:448
• https://github.com/input-output-hk/mithril/actions/runs/7729288973/job/21072318517#step:6:449
• https://github.com/input-output-hk/mithril/actions/runs/7725764904/job/21061158992#step:6:448
• https://github.com/input-output-hk/mithril/actions/runs/7725764904/job/21061410161#step:6:449
• https://github.com/input-output-hk/mithril/actions/runs/7725000099/job/21058486155#step:6:448
• https://github.com/input-output-hk/mithril/actions/runs/7725000099/job/21059042341#step:6:448
• https://github.com/input-output-hk/mithril/actions/runs/7724667934/job/21057430894#step:6:449
• https://github.com/input-output-hk/mithril/actions/runs/7716046987/job/21032096255#step:6:448
• https://github.com/input-output-hk/mithril/actions/runs/7740166625/job/21104734361#step:6:385
• https://github.com/input-output-hk/mithril/actions/runs/7740166625/job/21104734697#step:6:448
• https://github.com/input-output-hk/mithril/actions/runs/7740166625/job/21105061902#step:6:449
• https://github.com/input-output-hk/mithril/actions/runs/7740918215/job/21107277439#step:6:449
• https://github.com/input-output-hk/mithril/actions/runs/7740918215/job/21107588562#step:6:485

Timeout exhausted assert_node_producing_mithril_stake_distribution, no response from http://localhost:8080/aggregator/artifact/mithril-stake-distributions

P2P mode only?

• https://github.com/input-output-hk/mithril/actions/runs/7730541945/job/21099178088#step:6:446
• https://github.com/input-output-hk/mithril/actions/runs/7727108290/job/21065019249#step:6:447

Write era marker on chain exited with status code: 1

• https://github.com/input-output-hk/mithril/actions/runs/7730541945/job/21098978457#step:6:114

Run devnet exited with status code: 1

• https://github.com/input-output-hk/mithril/actions/runs/7724778463/job/21058022907#step:6:130

Previous version of the devnet

Genesis creation error

• The list of signers must not be empty to create a signer builder.:

  • https://github.com/input-output-hk/mithril/actions/runs/6862601032/job/18660752453#step:6:177
  • https://github.com/input-output-hk/mithril/actions/runs/6641163836/job/18043327082#step:6:177
  • https://github.com/input-output-hk/mithril/actions/runs/6562715519/job/17826641490#step:6:175
  • https://github.com/input-output-hk/mithril/actions/runs/6562715519/job/17826641490#step:6:175
  • https://github.com/input-output-hk/mithril/actions/runs/6549385737/job/17786376395#step:6:171
  • https://github.com/input-output-hk/mithril/actions/runs/6543549419/job/17768423928#step:6:181
  • https://github.com/input-output-hk/mithril/actions/runs/6535943138/job/17746701721#step:6:174
  • https://github.com/input-output-hk/mithril/actions/runs/6534726773/job/17742869443#step:6:173
  • https://github.com/input-output-hk/mithril/actions/runs/6534521855/job/17742147744#step:6:183
  • https://github.com/input-output-hk/mithril/actions/runs/6532219113/job/17735201690#step:6:180
  • https://github.com/input-output-hk/mithril/actions/runs/6530470341/job/17730245863#step:6:174
  • https://github.com/input-output-hk/mithril/actions/runs/6421452579/job/17436044895#step:6:182
  • https://github.com/input-output-hk/mithril/actions/runs/6014751757/job/16315354780#step:6:181
  • https://github.com/input-output-hk/mithril/actions/runs/5823107791/job/15789449719#step:6:177
  • https://github.com/input-output-hk/mithril/actions/runs/5724576192/job/15511512723#step:6:174
  • https://github.com/input-output-hk/mithril/actions/runs/5693045701/job/15431431094#step:6:178
  • https://github.com/input-output-hk/mithril/actions/runs/6082951484/job/16501969802#step:6:176
  • https://github.com/input-output-hk/mithril/actions/runs/6274778274/job/17041007700#step:6:178

• Missing signers for epoch:

  • https://github.com/input-output-hk/mithril/actions/runs/6589834831/job/17905346903#step:6:179
  • https://github.com/input-output-hk/mithril/actions/runs/6623726420/job/17991492493#step:6:188
  • https://github.com/input-output-hk/mithril/actions/runs/6669176714/job/18126547808#step:6:187
  • https://github.com/input-output-hk/mithril/actions/runs/6822250606/job/18554256758#step:6:186

Unpack error in client

• failed to iterate over archive:
  • https://github.com/input-output-hk/mithril/actions/runs/5795331954/job/15706824941#step:6:426
• failed to unpack file:
  • https://github.com/input-output-hk/mithril/actions/runs/5822636317/job/15787917750#step:6:426
  • https://github.com/input-output-hk/mithril/actions/runs/5691195686/job/15426058439?pr=1105#step:6:423

Not enough signers in certificate

• Certificate is not signed by expected number of signers: 1 < 2:
  • https://github.com/input-output-hk/mithril/actions/runs/6534726773/job/17742869085#step:6:334
  • https://github.com/input-output-hk/mithril/actions/runs/6035642063/job/16376550136#step:6:339
  • https://github.com/input-output-hk/mithril/actions/runs/6085408612/job/16509584771#step:6:328
  • https://github.com/input-output-hk/mithril/actions/runs/6219770444/job/16878651879#step:6:342
  • https://github.com/input-output-hk/mithril/actions/runs/6259198175/job/16994905478#step:6:335
  • https://github.com/input-output-hk/mithril/actions/runs/6259409309/job/16995531238#step:6:335
  • https://github.com/input-output-hk/mithril/actions/runs/6315096866/job/17147100746#step:6:334
  • https://github.com/input-output-hk/mithril/actions/runs/6575288976/job/17862321281#step:6:344

Signing stake distribution error

• Minimum expected mithril stake distribution epoch not reached : 13 < 14:
  • https://github.com/input-output-hk/mithril/actions/runs/6223139723/job/16888665901#step:6:316