Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add base FTR test coverage for inference APIs #198000

Merged
merged 36 commits into from
Dec 4, 2024
Merged

Add base FTR test coverage for inference APIs #198000

merged 36 commits into from
Dec 4, 2024
+823 −25

Conversation

pgayvallet
Copy link
Contributor

@pgayvallet pgayvallet commented Oct 28, 2024 •
edited by kibanamachine
Loading

Summary

Part of https://github.com/elastic/kibana-team/issues/1271

This PR introduces the first set of end to end integration test for the inference APIs, and the tooling required to do so (see issue for more context)

  • Add a dedicated pipeline for ai-infra GenAI tests. pipeline is triggered when:
    • genAI stack connectors, or ai-infra owned code is changed
    • when the ci:all-gen-ai-suites label is present on a PR
    • on merge
  • adapt the ftr_configs.sh script to load GenAI connector configuration from vault when a specific var env is set
  • create the @kbn/gen-ai-functional-testing package, which for now only contains utilities to load the GenAI connector configuration in FTR tests
  • Add FTR integration tests for the chatComplete API of the inference plugin

@pgayvallet pgayvallet added release_note:skip Skip the PR/issue when compiling release notes Team:AI Infra AppEx AI Infrastructure Team v8.17.0 backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) labels Oct 28, 2024
pgayvallet
pgayvallet commented Oct 28, 2024
View reviewed changes
Copy link
Contributor Author

@pgayvallet pgayvallet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quick comments for @pheyos

x-pack/test/functional_gen_ai/common/connectors.ts Outdated Show resolved Hide resolved
x-pack/test/functional_gen_ai/inference/tests/index.ts Outdated Show resolved Hide resolved
@pgayvallet
Copy link
Contributor Author

/ci

@pgayvallet
Copy link
Contributor Author

/ci

@pgayvallet
Copy link
Contributor Author

/ci

@pgayvallet
Copy link
Contributor Author

/ci

@pgayvallet
Copy link
Contributor Author

/ci

@pgayvallet pgayvallet added backport:version Backport to applied version labels and removed backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) labels Dec 4, 2024
@pgayvallet pgayvallet enabled auto-merge (squash) December 4, 2024 10:14
@pgayvallet pgayvallet merged commit 14ad13b into elastic:main Dec 4, 2024
8 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12159929564

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/gen-ai-functional-testing - 8 +8
@kbn/sse-utils-server 2 3 +1
total +9
Unknown metric groups

API count

id before after diff
@kbn/gen-ai-functional-testing - 10 +10
@kbn/sse-utils-server 2 4 +2
total +12

ESLint disabled line counts

id before after diff
@kbn/gen-ai-functional-testing - 2 +2
@kbn/test-suites-xpack 727 729 +2
total +4

Total ESLint disabled count

id before after diff
@kbn/gen-ai-functional-testing - 2 +2
@kbn/test-suites-xpack 752 754 +2
total +4

History

@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
❌ 8.x Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 198000

Questions ?

Please refer to the Backport tool documentation

@pgayvallet pgayvallet mentioned this pull request Dec 4, 2024
Merged
@pgayvallet
Copy link
Contributor Author

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

pgayvallet added a commit to pgayvallet/kibana that referenced this pull request Dec 4, 2024
@pgayvallet
Add base FTR test coverage for inference APIs (elastic#198000)
81c15ed 
## Summary

Part of elastic/kibana-team#1271

This PR introduces the first set of end to end integration test for the
inference APIs, and the tooling required to do so (see issue for more
context)

- Add a dedicated pipeline for ai-infra GenAI tests. pipeline is
triggered when:
  - genAI stack connectors, or ai-infra owned code is changed
  - when the `ci:all-gen-ai-suites` label is present on a PR
  - on merge
- adapt the `ftr_configs.sh` script to load GenAI connector
configuration from vault when a specific var env is set
- create the `@kbn/gen-ai-functional-testing` package, which for now
only contains utilities to load the GenAI connector configuration in FTR
tests
- Add FTR integration tests for the `chatComplete` API of the
`inference` plugin

---------

Co-authored-by: kibanamachine <[email protected]>
(cherry picked from commit 14ad13b)

# Conflicts:
#	.buildkite/pipelines/on_merge.yml
#	.github/CODEOWNERS
@kibanamachine kibanamachine mentioned this pull request Dec 4, 2024
Merged
@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Dec 5, 2024
@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

pgayvallet added a commit that referenced this pull request Dec 5, 2024
@pgayvallet @kibanamachine
[8.x] Add base FTR test coverage for inference APIs (#198000) (#202916)
4ffcaed 
# Backport

This will backport the following commits from `main` to `8.x`:
- [Add base FTR test coverage for inference APIs
(#198000)](#198000)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Pierre
Gayvallet","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-04T12:39:45Z","message":"Add
base FTR test coverage for inference APIs (#198000)\n\n##
Summary\r\n\r\nPart of
https://github.com/elastic/kibana-team/issues/1271\r\n\r\nThis PR
introduces the first set of end to end integration test for
the\r\ninference APIs, and the tooling required to do so (see issue for
more\r\ncontext)\r\n\r\n- Add a dedicated pipeline for ai-infra GenAI
tests. pipeline is\r\ntriggered when:\r\n - genAI stack connectors, or
ai-infra owned code is changed\r\n - when the `ci:all-gen-ai-suites`
label is present on a PR\r\n - on merge\r\n- adapt the `ftr_configs.sh`
script to load GenAI connector\r\nconfiguration from vault when a
specific var env is set\r\n- create the `@kbn/gen-ai-functional-testing`
package, which for now\r\nonly contains utilities to load the GenAI
connector configuration in FTR\r\ntests\r\n- Add FTR integration tests
for the `chatComplete` API of the\r\n`inference`
plugin\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"14ad13b6a3d7a87f4dfa9d7c1ed6c055242c9808","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:version","Team:AI
Infra","v8.18.0","ci:all-gen-ai-suites"],"number":198000,"url":"https://github.com/elastic/kibana/pull/198000","mergeCommit":{"message":"Add
base FTR test coverage for inference APIs (#198000)\n\n##
Summary\r\n\r\nPart of
https://github.com/elastic/kibana-team/issues/1271\r\n\r\nThis PR
introduces the first set of end to end integration test for
the\r\ninference APIs, and the tooling required to do so (see issue for
more\r\ncontext)\r\n\r\n- Add a dedicated pipeline for ai-infra GenAI
tests. pipeline is\r\ntriggered when:\r\n - genAI stack connectors, or
ai-infra owned code is changed\r\n - when the `ci:all-gen-ai-suites`
label is present on a PR\r\n - on merge\r\n- adapt the `ftr_configs.sh`
script to load GenAI connector\r\nconfiguration from vault when a
specific var env is set\r\n- create the `@kbn/gen-ai-functional-testing`
package, which for now\r\nonly contains utilities to load the GenAI
connector configuration in FTR\r\ntests\r\n- Add FTR integration tests
for the `chatComplete` API of the\r\n`inference`
plugin\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"14ad13b6a3d7a87f4dfa9d7c1ed6c055242c9808"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/198000","number":198000,"mergeCommit":{"message":"Add
base FTR test coverage for inference APIs (#198000)\n\n##
Summary\r\n\r\nPart of
https://github.com/elastic/kibana-team/issues/1271\r\n\r\nThis PR
introduces the first set of end to end integration test for
the\r\ninference APIs, and the tooling required to do so (see issue for
more\r\ncontext)\r\n\r\n- Add a dedicated pipeline for ai-infra GenAI
tests. pipeline is\r\ntriggered when:\r\n - genAI stack connectors, or
ai-infra owned code is changed\r\n - when the `ci:all-gen-ai-suites`
label is present on a PR\r\n - on merge\r\n- adapt the `ftr_configs.sh`
script to load GenAI connector\r\nconfiguration from vault when a
specific var env is set\r\n- create the `@kbn/gen-ai-functional-testing`
package, which for now\r\nonly contains utilities to load the GenAI
connector configuration in FTR\r\ntests\r\n- Add FTR integration tests
for the `chatComplete` API of the\r\n`inference`
plugin\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"14ad13b6a3d7a87f4dfa9d7c1ed6c055242c9808"}},{"branch":"8.x","label":"v8.18.0","labelRegex":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <[email protected]>
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Dec 5, 2024
hop-dev pushed a commit to hop-dev/kibana that referenced this pull request Dec 5, 2024
@pgayvallet @kibanamachine @hop-dev
Add base FTR test coverage for inference APIs (elastic#198000)
1b55216 
## Summary

Part of elastic/kibana-team#1271

This PR introduces the first set of end to end integration test for the
inference APIs, and the tooling required to do so (see issue for more
context)

- Add a dedicated pipeline for ai-infra GenAI tests. pipeline is
triggered when:
  - genAI stack connectors, or ai-infra owned code is changed
  - when the `ci:all-gen-ai-suites` label is present on a PR
  - on merge
- adapt the `ftr_configs.sh` script to load GenAI connector
configuration from vault when a specific var env is set
- create the `@kbn/gen-ai-functional-testing` package, which for now
only contains utilities to load the GenAI connector configuration in FTR
tests
- Add FTR integration tests for the `chatComplete` API of the
`inference` plugin

---------

Co-authored-by: kibanamachine <[email protected]>
SoniaSanzV pushed a commit to SoniaSanzV/kibana that referenced this pull request Dec 9, 2024
@pgayvallet @kibanamachine @SoniaSanzV
Add base FTR test coverage for inference APIs (elastic#198000)
8bdbfdc 
## Summary

Part of elastic/kibana-team#1271

This PR introduces the first set of end to end integration test for the
inference APIs, and the tooling required to do so (see issue for more
context)

- Add a dedicated pipeline for ai-infra GenAI tests. pipeline is
triggered when:
  - genAI stack connectors, or ai-infra owned code is changed
  - when the `ci:all-gen-ai-suites` label is present on a PR
  - on merge
- adapt the `ftr_configs.sh` script to load GenAI connector
configuration from vault when a specific var env is set
- create the `@kbn/gen-ai-functional-testing` package, which for now
only contains utilities to load the GenAI connector configuration in FTR
tests
- Add FTR integration tests for the `chatComplete` API of the
`inference` plugin

---------

Co-authored-by: kibanamachine <[email protected]>
SoniaSanzV pushed a commit to SoniaSanzV/kibana that referenced this pull request Dec 9, 2024
@pgayvallet @kibanamachine @SoniaSanzV
Add base FTR test coverage for inference APIs (elastic#198000)
cac4e10 
## Summary

Part of elastic/kibana-team#1271

This PR introduces the first set of end to end integration test for the
inference APIs, and the tooling required to do so (see issue for more
context)

- Add a dedicated pipeline for ai-infra GenAI tests. pipeline is
triggered when:
  - genAI stack connectors, or ai-infra owned code is changed
  - when the `ci:all-gen-ai-suites` label is present on a PR
  - on merge
- adapt the `ftr_configs.sh` script to load GenAI connector
configuration from vault when a specific var env is set
- create the `@kbn/gen-ai-functional-testing` package, which for now
only contains utilities to load the GenAI connector configuration in FTR
tests
- Add FTR integration tests for the `chatComplete` API of the
`inference` plugin

---------

Co-authored-by: kibanamachine <[email protected]>
@pgayvallet pgayvallet mentioned this pull request Dec 9, 2024
Merged
pgayvallet added a commit that referenced this pull request Dec 9, 2024
@pgayvallet
GenAI FTR tests: switch to daily job (#203369)
1f2a76a 
## Summary

Part of elastic/kibana-team#1271
Follow-up of #198000

Fix #203205
Fix #203062

- Fix the problem with the Gemini token count events 
- Unskip the test suite
- Remove the suite from the `on-merge` pipeline
- Add a dedicated pipeline to run the tests daily
pgayvallet added a commit to pgayvallet/kibana that referenced this pull request Dec 9, 2024
@pgayvallet
GenAI FTR tests: switch to daily job (elastic#203369)
fa30b5e 
## Summary

Part of elastic/kibana-team#1271
Follow-up of elastic#198000

Fix elastic#203205
Fix elastic#203062

- Fix the problem with the Gemini token count events
- Unskip the test suite
- Remove the suite from the `on-merge` pipeline
- Add a dedicated pipeline to run the tests daily

(cherry picked from commit 1f2a76a)

# Conflicts:
#	.buildkite/pipeline-resource-definitions/locations.yml
#	.buildkite/pipelines/on_merge.yml
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 9, 2024
@pgayvallet @kibanamachine @CAWilson94
Add base FTR test coverage for inference APIs (elastic#198000)
d6da75f 
## Summary

Part of elastic/kibana-team#1271

This PR introduces the first set of end to end integration test for the
inference APIs, and the tooling required to do so (see issue for more
context)

- Add a dedicated pipeline for ai-infra GenAI tests. pipeline is
triggered when:
  - genAI stack connectors, or ai-infra owned code is changed
  - when the `ci:all-gen-ai-suites` label is present on a PR
  - on merge
- adapt the `ftr_configs.sh` script to load GenAI connector
configuration from vault when a specific var env is set
- create the `@kbn/gen-ai-functional-testing` package, which for now
only contains utilities to load the GenAI connector configuration in FTR
tests
- Add FTR integration tests for the `chatComplete` API of the
`inference` plugin

---------

Co-authored-by: kibanamachine <[email protected]>
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 9, 2024
@pgayvallet @CAWilson94
GenAI FTR tests: switch to daily job (elastic#203369)
e9454b4 
## Summary

Part of elastic/kibana-team#1271
Follow-up of elastic#198000

Fix elastic#203205
Fix elastic#203062

- Fix the problem with the Gemini token count events 
- Unskip the test suite
- Remove the suite from the `on-merge` pipeline
- Add a dedicated pipeline to run the tests daily
Samiul-TheSoccerFan pushed a commit to Samiul-TheSoccerFan/kibana that referenced this pull request Dec 10, 2024
@pgayvallet @Samiul-TheSoccerFan
GenAI FTR tests: switch to daily job (elastic#203369)
9da8243 
## Summary

Part of elastic/kibana-team#1271
Follow-up of elastic#198000

Fix elastic#203205
Fix elastic#203062

- Fix the problem with the Gemini token count events 
- Unskip the test suite
- Remove the suite from the `on-merge` pipeline
- Add a dedicated pipeline to run the tests daily
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 12, 2024
@pgayvallet @kibanamachine @CAWilson94
Add base FTR test coverage for inference APIs (elastic#198000)
6dee752 
## Summary

Part of elastic/kibana-team#1271

This PR introduces the first set of end to end integration test for the
inference APIs, and the tooling required to do so (see issue for more
context)

- Add a dedicated pipeline for ai-infra GenAI tests. pipeline is
triggered when:
  - genAI stack connectors, or ai-infra owned code is changed
  - when the `ci:all-gen-ai-suites` label is present on a PR
  - on merge
- adapt the `ftr_configs.sh` script to load GenAI connector
configuration from vault when a specific var env is set
- create the `@kbn/gen-ai-functional-testing` package, which for now
only contains utilities to load the GenAI connector configuration in FTR
tests
- Add FTR integration tests for the `chatComplete` API of the
`inference` plugin

---------

Co-authored-by: kibanamachine <[email protected]>
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 12, 2024
@pgayvallet @CAWilson94
GenAI FTR tests: switch to daily job (elastic#203369)
b9e299c 
## Summary

Part of elastic/kibana-team#1271
Follow-up of elastic#198000

Fix elastic#203205
Fix elastic#203062

- Fix the problem with the Gemini token count events 
- Unskip the test suite
- Remove the suite from the `on-merge` pipeline
- Add a dedicated pipeline to run the tests daily
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@legrego legrego legrego approved these changes

@dmlemeshko dmlemeshko dmlemeshko approved these changes

@delanni delanni delanni approved these changes

@dgieselaar dgieselaar dgieselaar approved these changes

Assignees
No one assigned
Labels
backport:version Backport to applied version labels ci:all-gen-ai-suites release_note:skip Skip the PR/issue when compiling release notes Team:AI Infra AppEx AI Infrastructure Team v8.18.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@pgayvallet @elasticmachine @kibanamachine @dgieselaar @legrego @delanni @dmlemeshko