-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add base FTR test coverage for inference APIs (#198000)
## Summary Part of elastic/kibana-team#1271 This PR introduces the first set of end to end integration test for the inference APIs, and the tooling required to do so (see issue for more context) - Add a dedicated pipeline for ai-infra GenAI tests. pipeline is triggered when: - genAI stack connectors, or ai-infra owned code is changed - when the `ci:all-gen-ai-suites` label is present on a PR - on merge - adapt the `ftr_configs.sh` script to load GenAI connector configuration from vault when a specific var env is set - create the `@kbn/gen-ai-functional-testing` package, which for now only contains utilities to load the GenAI connector configuration in FTR tests - Add FTR integration tests for the `chatComplete` API of the `inference` plugin --------- Co-authored-by: kibanamachine <[email protected]>
- Loading branch information
1 parent
5fa4af9
commit 14ad13b
Showing
31 changed files
with
823 additions
and
25 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| steps: | ||
| - group: AppEx AI-Infra genAI tests | ||
| key: ai-infra-gen-ai | ||
| depends_on: | ||
| - build | ||
| - quick_checks | ||
| - checks | ||
| - linting | ||
| - linting_with_types | ||
| - check_types | ||
| - check_oas_snapshot | ||
| steps: | ||
| - command: .buildkite/scripts/steps/test/ftr_configs.sh | ||
| env: | ||
| FTR_CONFIG: "x-pack/test/functional_gen_ai/inference/config.ts" | ||
| FTR_CONFIG_GROUP_KEY: 'ftr-ai-infra-gen-ai-inference-api' | ||
| FTR_GEN_AI: "1" | ||
| label: AppEx AI-Infra Inference APIs FTR tests | ||
| key: ai-infra-gen-ai-inference-api | ||
| timeout_in_minutes: 50 | ||
| parallelism: 1 | ||
| agents: | ||
| machineType: n2-standard-4 | ||
| preemptible: true | ||
| retry: | ||
| automatic: | ||
| - exit_status: '-1' | ||
| limit: 3 | ||
| - exit_status: '*' | ||
| limit: 1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| ## local version of the connector config | ||
| connector_config.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| # @kbn/gen-ai-functional-testing | ||
|
|
||
| Package exposing various utilities for GenAI/LLM related functional testing. | ||
|
|
||
| ## Features | ||
|
|
||
| ### LLM connectors | ||
|
|
||
| Utilizing LLM connectors on FTR tests can be done via the `getPreconfiguredConnectorConfig` and `getAvailableConnectors` tools. | ||
|
|
||
| `getPreconfiguredConnectorConfig` should be used to define the list of connectors when creating the FTR test's configuration. | ||
|
|
||
| ```ts | ||
| import { getPreconfiguredConnectorConfig } from '@kbn/gen-ai-functional-testing' | ||
|
|
||
| export default async function ({ readConfigFile }: FtrConfigProviderContext) { | ||
| const xpackFunctionalConfig = {...}; | ||
| const preconfiguredConnectors = getPreconfiguredConnectorConfig(); | ||
|
|
||
| return { | ||
| ...xpackFunctionalConfig.getAll(), | ||
| kbnTestServer: { | ||
| ...xpackFunctionalConfig.get('kbnTestServer'), | ||
| serverArgs: [ | ||
| ...xpackFunctionalConfig.get('kbnTestServer.serverArgs'), | ||
| `--xpack.actions.preconfigured=${JSON.stringify(preconfiguredConnectors)}`, | ||
| ], | ||
| }, | ||
| }; | ||
| } | ||
| ``` | ||
|
|
||
| then the `getAvailableConnectors` can be used during the test suite to retrieve the list of LLM connectors. | ||
|
|
||
| For example to run some predefined test suite against all exposed LLM connectors: | ||
|
|
||
| ```ts | ||
| import { getAvailableConnectors } from '@kbn/gen-ai-functional-testing'; | ||
|
|
||
| export default function (providerContext: FtrProviderContext) { | ||
| describe('Some GenAI FTR test suite', async () => { | ||
| getAvailableConnectors().forEach((connector) => { | ||
| describe(`Using connector ${connector.id}`, () => { | ||
| myTestSuite(connector, providerContext); | ||
| }); | ||
| }); | ||
| }); | ||
| } | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the "Elastic License | ||
| * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side | ||
| * Public License v 1"; you may not use this file except in compliance with, at | ||
| * your election, the "Elastic License 2.0", the "GNU Affero General Public | ||
| * License v3.0 only", or the "Server Side Public License, v 1". | ||
| */ | ||
|
|
||
| export { | ||
| AI_CONNECTORS_VAR_ENV, | ||
| getPreconfiguredConnectorConfig, | ||
| getAvailableConnectors, | ||
| type AvailableConnector, | ||
| type AvailableConnectorWithId, | ||
| } from './src/connectors'; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the "Elastic License | ||
| * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side | ||
| * Public License v 1"; you may not use this file except in compliance with, at | ||
| * your election, the "Elastic License 2.0", the "GNU Affero General Public | ||
| * License v3.0 only", or the "Server Side Public License, v 1". | ||
| */ | ||
|
|
||
| module.exports = { | ||
| preset: '@kbn/test/jest_node', | ||
| rootDir: '../..', | ||
| roots: ['<rootDir>/packages/kbn-gen-ai-functional-testing'], | ||
| }; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| { | ||
| "type": "shared-common", | ||
| "id": "@kbn/gen-ai-functional-testing", | ||
| "owner": "@elastic/appex-ai-infra", | ||
| "devOnly": true | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| { | ||
| "name": "@kbn/gen-ai-functional-testing", | ||
| "private": true, | ||
| "version": "1.0.0", | ||
| "license": "Elastic License 2.0 OR AGPL-3.0-only OR SSPL-1.0" | ||
| } |
11 changes: 11 additions & 0 deletions
11
packages/kbn-gen-ai-functional-testing/scripts/format_connector_config.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the "Elastic License | ||
| * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side | ||
| * Public License v 1"; you may not use this file except in compliance with, at | ||
| * your election, the "Elastic License 2.0", the "GNU Affero General Public | ||
| * License v3.0 only", or the "Server Side Public License, v 1". | ||
| */ | ||
|
|
||
| require('@kbn/babel-register').install(); | ||
| require('../src/manage_connector_config').formatCurrentConfig(); |
11 changes: 11 additions & 0 deletions
11
packages/kbn-gen-ai-functional-testing/scripts/retrieve_connector_config.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the "Elastic License | ||
| * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side | ||
| * Public License v 1"; you may not use this file except in compliance with, at | ||
| * your election, the "Elastic License 2.0", the "GNU Affero General Public | ||
| * License v3.0 only", or the "Server Side Public License, v 1". | ||
| */ | ||
|
|
||
| require('@kbn/babel-register').install(); | ||
| require('../src/manage_connector_config').retrieveFromVault(); |
11 changes: 11 additions & 0 deletions
11
packages/kbn-gen-ai-functional-testing/scripts/upload_connector_config.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the "Elastic License | ||
| * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side | ||
| * Public License v 1"; you may not use this file except in compliance with, at | ||
| * your election, the "Elastic License 2.0", the "GNU Affero General Public | ||
| * License v3.0 only", or the "Server Side Public License, v 1". | ||
| */ | ||
|
|
||
| require('@kbn/babel-register').install(); | ||
| require('../src/manage_connector_config').uploadToVault(); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the "Elastic License | ||
| * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side | ||
| * Public License v 1"; you may not use this file except in compliance with, at | ||
| * your election, the "Elastic License 2.0", the "GNU Affero General Public | ||
| * License v3.0 only", or the "Server Side Public License, v 1". | ||
| */ | ||
|
|
||
| import { schema } from '@kbn/config-schema'; | ||
|
|
||
| /** | ||
| * The environment variable that is used by the CI to load the connectors configuration | ||
| */ | ||
| export const AI_CONNECTORS_VAR_ENV = 'KIBANA_TESTING_AI_CONNECTORS'; | ||
|
|
||
| const connectorsSchema = schema.recordOf( | ||
| schema.string(), | ||
| schema.object({ | ||
| name: schema.string(), | ||
| actionTypeId: schema.string(), | ||
| config: schema.recordOf(schema.string(), schema.any()), | ||
| secrets: schema.recordOf(schema.string(), schema.any()), | ||
| }) | ||
| ); | ||
|
|
||
| export interface AvailableConnector { | ||
| name: string; | ||
| actionTypeId: string; | ||
| config: Record<string, unknown>; | ||
| secrets: Record<string, unknown>; | ||
| } | ||
|
|
||
| export interface AvailableConnectorWithId extends AvailableConnector { | ||
| id: string; | ||
| } | ||
|
|
||
| const loadConnectors = (): Record<string, AvailableConnector> => { | ||
| const envValue = process.env[AI_CONNECTORS_VAR_ENV]; | ||
| if (!envValue) { | ||
| return {}; | ||
| } | ||
|
|
||
| let connectors: Record<string, AvailableConnector>; | ||
| try { | ||
| connectors = JSON.parse(Buffer.from(envValue, 'base64').toString('utf-8')); | ||
| } catch (e) { | ||
| throw new Error( | ||
| `Error trying to parse value from KIBANA_AI_CONNECTORS environment variable: ${e.message}` | ||
| ); | ||
| } | ||
| return connectorsSchema.validate(connectors); | ||
| }; | ||
|
|
||
| /** | ||
| * Retrieve the list of preconfigured connectors that should be used when defining the | ||
| * FTR configuration of suites using the connectors. | ||
| * | ||
| * @example | ||
| * ```ts | ||
| * import { getPreconfiguredConnectorConfig } from '@kbn/gen-ai-functional-testing' | ||
| * | ||
| * export default async function ({ readConfigFile }: FtrConfigProviderContext) { | ||
| * const xpackFunctionalConfig = {...}; | ||
| * const preconfiguredConnectors = getPreconfiguredConnectorConfig(); | ||
| * | ||
| * return { | ||
| * ...xpackFunctionalConfig.getAll(), | ||
| * kbnTestServer: { | ||
| * ...xpackFunctionalConfig.get('kbnTestServer'), | ||
| * serverArgs: [ | ||
| * ...xpackFunctionalConfig.get('kbnTestServer.serverArgs'), | ||
| * `--xpack.actions.preconfigured=${JSON.stringify(preconfiguredConnectors)}`, | ||
| * ], | ||
| * }, | ||
| * }; | ||
| * } | ||
| * ``` | ||
| */ | ||
| export const getPreconfiguredConnectorConfig = () => { | ||
| return loadConnectors(); | ||
| }; | ||
|
|
||
| export const getAvailableConnectors = (): AvailableConnectorWithId[] => { | ||
| return Object.entries(loadConnectors()).map(([id, connector]) => { | ||
| return { | ||
| id, | ||
| ...connector, | ||
| }; | ||
| }); | ||
| }; |
Oops, something went wrong.