[Observability AI Assistant] Add tests for serverless projects

[miltonhultgren]

Summary

With most of our features being available in serverless Observability projects we should have some test automation in place to make sure it doesn't break without us noticing.

Adding tests for serverless is the same as adding tests for stateful but in a different folder xpack/test_serverless/functional/test_suites/observability but these test suites cannot rely on any test tools available in stateful tests.

So we need to decide on either duplicating all of our setup and helper code or extract that code into packages and refactor the existing stateful tests to use those packages instead before adding serverless tests.

Notes

While testing ELSER (TINY_ELSER) in a Serverless project, we need to call await ml.api.assureMlStatsIndexExists(); before importing the model, not after as the other (non-serverless) ML tests show.

[elasticmachine]

Pinging @elastic/obs-knowledge-team (Team:obs-knowledge)

[neptunian]

Tests were duplicated for API integration in #192219. Many were skipped and below are issues still needed to address:

• [Obs AI Assistant] Update knowledge base setup route to accept a model id #192757
• [Obs AI Assistant] fix knowledge base tests failing in serverless #192886
• [Obs AI Assistant] Fix tests using Llm proxy in serverless #192751
• [Obs AI Assistant] Fetch user instructions using id instead of username for knowledge base instructions #192701
• [Obs AI Assistant] unskip summarize tests #192497
• [Obs AI Assistant] Serverless tests should use cookie based auth instead of api keys for internal routes #192711
• [Obs AI Assistant] Deployment agnostic API integration tests #192718
• serverless functional tests
• inquire with ml-ui-team to have the ability to delete system indices which we do after uninstalling tiny elser with .ml indices

[neptunian]

QAF issue to add custom config to the MKI pipeline https://github.com/elastic/qaf-tests/issues/126

[emma-raffenne]

Tests were duplicated for API integration in #192219. Many were skipped and below are issues still needed to address:

• [Obs AI Assistant] Update knowledge base setup route to accept a model id #192757[ ] [Obs AI Assistant] fix knowledge base tests failing in serverless #192886[ ] [Obs AI Assistant] Fix tests using Llm proxy in serverless #192751[ ] [Obs AI Assistant] Fetch user instructions using id instead of username for knowledge base instructions #192701[ ] [Obs AI Assistant] unskip summarize tests #192497[ ] [Obs AI Assistant] Serverless tests should use cookie based auth instead of api keys for internal routes #192711[ ] [Obs AI Assistant] Deployment agnostic API integration tests #192718[ ] serverless functional tests[ ] inquire with ml-ui-team to have the ability to delete system indices which we do after uninstalling tiny elser with .ml indices

Extracting this issue to a meta one with these as sub-issues.
Thanks @neptunian