Index service test suite should go through the system

[jasontedor]

I discovered this while working on inlining global checkpoints. The IndexServiceTests#testRescheduleAsyncFsync test failed on my branch. The idea behind this test is that we set the translog durability to async, index a document, and then busily check that the translog does not need to be synced. The expectation is that an async fsync task should fire and at some point fsync the translog. This test was passing on master, failing on my branch. It was passing on master for the wrong reason. The problem is that setting durability to async never stuck but were instead on request durability. Thus, a request fsync had already fired so it did not matter if an async fsync never fired. The details of exactly why are explained in cb46e97. The reason this fails on my branch is because with inlining global checkpoints, the translog does indeed need another fsync after the replication operation completes because we persist the global checkpoint into the translog checkpoint. Since we were stuck on request durability, this fsync never fired and the test failed.

I pushed cb46e97 to fix this test so the test goes through the system and the async durability setting sticks. There are other tests in this test suite doing the same thing. I think that should be updated, but I would rather spend my immediate time on inlining global checkpoints so this issue is a placeholder for addressing these tests in the future.

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

@dnhatn do you mind spending some time triaging this and seeing what needs to be done?

[dnhatn]

@bleskes Looking now.

[dnhatn]

I opened #29682.