[FEATURE] Create Integration Test framework

[owaiskazi19]

Is your feature request related to a problem?

Currently, the project just has unit tests for the features built until now.

• A test framework need to be implemented for developers to write integration tests.
• Run integ test on CI for every PR. [FEATURE] Setup Integration Test CI #56

What solution would you like?

Taking an example from ml-commons. Integtest task in build.gradle like

What alternatives have you considered?

A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

[dbwiddis]

We currently have an IT framework as part of the template.

This test validates that the plugin was installed.
https://github.com/opensearch-project/opensearch-ai-flow-framework/blob/main/src/test/java/org/opensearch/flowframework/FlowFrameworkPluginIT.java

That test runs with every CI.

What is currently missing?

[owaiskazi19]

Ah! I missed the framework completely @dbwiddis. Thanks for the pointer.
Also, IMO we should have a GHA to run integ test separately than our usual build CI.

[dbwiddis]

Sure, we should run them separately.

Also, the IT tests is flaky. It often fails with a 404 error, but works on retry.
Any idea how to fix it?

org.opensearch.client.ResponseException: method [GET], host [http://0.0.0.0:0/], URI [/_cat/plugins], status line [HTTP/1.1 404 Not Found]