tests: Repeat functional tests for 1 minute

[serathius]

Data corruption was only found when functional failure injections was executed multiple times for period of 1 minute. Executing failure once is not enough to confirm if issue exist.

[codecov-commenter]

Codecov Report

Merging #13926 (8aa8519) into main (b018742) will decrease coverage by 0.02%.
The diff coverage is n/a.

❗ Current head 8aa8519 differs from pull request most recent head d8be293. Consider uploading reports for the commit d8be293 to get more accurate results

@@            Coverage Diff             @@
##             main   #13926      +/-   ##
==========================================
- Coverage   72.41%   72.38%   -0.03%     
==========================================
  Files         469      469              
  Lines       38411    38411              
==========================================
- Hits        27814    27804      -10     
- Misses       8798     8816      +18     
+ Partials     1799     1791       -8     

Flag	Coverage Δ
all	72.38% <ø> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/etcdserver/api/rafthttp/peer_status.go	87.87% <0.00%> (-12.13%)	⬇️
server/auth/simple_token.go	80.00% <0.00%> (-8.47%)	⬇️
server/etcdserver/api/rafthttp/peer.go	87.01% <0.00%> (-8.45%)	⬇️
client/pkg/v3/tlsutil/tlsutil.go	83.33% <0.00%> (-8.34%)	⬇️
server/storage/mvcc/watchable_store.go	85.14% <0.00%> (-3.99%)	⬇️
server/etcdserver/cluster_util.go	70.35% <0.00%> (-3.17%)	⬇️
server/proxy/grpcproxy/watch.go	93.64% <0.00%> (-2.90%)	⬇️
server/etcdserver/api/v3election/election.go	66.66% <0.00%> (-2.78%)	⬇️
client/pkg/v3/testutil/leak.go	60.17% <0.00%> (-2.66%)	⬇️
client/v3/op.go	75.09% <0.00%> (-0.77%)	⬇️
... and 13 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b018742...d8be293. Read the comment docs.

[serathius]

cc @ptabor @ahrtr

[ahrtr]

Should we make the duration configurable or add a TODO note here?

[serathius]

Need to rethink if we prefer to have 1 failure injection per test and run multiple tests or have multiple injections per test. Decision should depend on how it's easier to adjust the runs in different scenarios:

1. Presubmits in PR (short length, 1 run is enough per PR),
2. Qualification on main branch (medium length, up to 1h once a day)
3. Qualification of release (long length, run for days, only when releasing).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.