[BUG] AdmissionControlMultiNodeIT.testAdmissionControlRejectionOnEnforced is flaky

[peternied]

Describe the bug

Logs https://build.ci.opensearch.org/job/gradle-check/32002/testReport/org.opensearch.ratelimitting.admissioncontrol/AdmissionControlMultiNodeIT/testAdmissionControlRejectionOnEnforced/

Error Message

java.lang.AssertionError: initial cluster state not set yet

Stacktrace

java.lang.AssertionError: initial cluster state not set yet
	at __randomizedtesting.SeedInfo.seed([EECBE83E5D054700]:0)
	at org.opensearch.cluster.service.ClusterApplierService.state(ClusterApplierService.java:215)
	at org.opensearch.cluster.service.ClusterService.state(ClusterService.java:172)
	at org.opensearch.node.ResourceUsageCollectorService.collectLocalNodeResourceUsageStats(ResourceUsageCollectorService.java:127)
	at org.opensearch.node.ResourceUsageCollectorService.lambda$doStart$3(ResourceUsageCollectorService.java:144)
	at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Related component

Build

To Reproduce

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.ratelimitting.admissioncontrol.AdmissionControlMultiNodeIT.testAdmissionControlRejectionOnEnforced" -Dtests.seed=EECBE83E5D054700 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-US -Dtests.timezone=Asia/Magadan -Druntime.java=21
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.ratelimitting.admissioncontrol.AdmissionControlMultiNodeIT.testAdmissionControlRejectionOnEnforced" -Dtests.seed=EECBE83E5D054700 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-US -Dtests.timezone=Asia/Magadan -Druntime.java=21
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.ratelimitting.admissioncontrol.AdmissionControlMultiNodeIT.testAdmissionControlRejectionOnEnforced" -Dtests.seed=EECBE83E5D054700 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-US -Dtests.timezone=Asia/Magadan -Druntime.java=21
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.ratelimitting.admissioncontrol.AdmissionControlMultiNodeIT.testAdmissionControlRejectionOnEnforced" -Dtests.seed=EECBE83E5D054700 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-US -Dtests.timezone=Asia/Magadan -Druntime.java=21
NOTE: test params are: codec=Lucene99, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=es-US, timezone=Asia/Magadan
NOTE: Linux 5.15.0-1048-aws amd64/Eclipse Adoptium 21.0.1 (64-bit)/cpus=32,threads=1,free=563780280,total=805306368
NOTE: All tests run in this JVM: [ConcurrentSearchTasksIT, SplitIndexIT, IndicesStatsBlocksIT, IndexAliasesIT, ShardStateActionIT, DiscoveryNodeRoleIT, UpgradeSettingsIT, ExtensionsManagerIT, ShardIndexingPressureIT, GlobalCheckpointListenersIT, IndicesExistsIT, SegmentReplicationDisruptionIT, ReopenWhileClosingIT, AdmissionControlMultiNodeIT]

Expected behavior

All test should pass

Additional Details

No response

[ajaymovva]

Closing as this test is now removed.

[peternied]

@ajaymovva Could you please reference the pull request where this was done. I am curious - if this test was removed how is the test coverage on the scenario happening?

[ajaymovva]

This was an multi node integration test which is failing and we want to change the flow of testing on more than one node. So we removed the multi node cases and added single node cases for test the code flow for all cases.
We are planning to add multi node tests as well in the next PR's for the whole AdmissionControl Framework.
This PR where we removed the multi node integration tests: https://github.com/opensearch-project/OpenSearch/pull/12702/files

[peternied]

Thanks @ajaymovva my final request :) - where is the issue where these additional tests are planned?

[ajaymovva]

This is the meta issue we are tracking : #9504