-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Security Solution] Initial OpenAPI codegen implementation (#163186)
**Resolves: elastic/security-team#7134 ## Summary Implemented request and response schema generation from OpenAPI specifications. The code generator script scans the `x-pack/plugins/security_solution/common/api` directory, locates all `*.schema.yaml` files, and generates a corresponding `*.gen.ts` artifact for each, containing `zod` schema definitions. <hr/> Right now, all generation sources are set to `x-codegen-enabled: false` to prevent the creation of duplicate schemas. Maintaining the old `io-ts` schemas alongside the new `zod` ones could potentially lead to confusion among developers. Thus, the recommended migration strategy is to incrementally replace old schema usages with new ones, subsequently removing outdated ones. I'll be implementing this approach in the upcoming PRs. ### How to use the generator If you need to test the generator locally, enable several sources and run the generator script to see the results. Navigate to `x-pack/plugins/security_solution` and run `yarn openapi:generate` <img width="916" alt="image" src="https://github.com/elastic/kibana/assets/1938181/be1a8a61-b9ed-4359-bc3e-bf393f256859"> Important note: if you want to enable route schemas, ensure you also enable all their dependencies, such as common schemas. Failing to do so will result in the generated code importing non-existent files. ### Example Input file (`x-pack/plugins/security_solution/common/api/detection_engine/model/error_schema.schema.yaml`): ```yaml openapi: 3.0.0 info: title: Error Schema version: 'not applicable' paths: {} components: schemas: ErrorSchema: type: object required: - error properties: id: type: string rule_id: $ref: './rule_schema/common_attributes.schema.yaml#/components/schemas/RuleSignatureId' list_id: type: string minLength: 1 item_id: type: string minLength: 1 error: type: object required: - status_code - message properties: status_code: type: integer minimum: 400 message: type: string ``` Generated output file (`x-pack/plugins/security_solution/common/api/detection_engine/model/error_schema.gen.ts`): ```ts /* * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one * or more contributor license agreements. Licensed under the Elastic License * 2.0; you may not use this file except in compliance with the Elastic License * 2.0. */ import { z } from 'zod'; /* * NOTICE: Do not edit this file manually. * This file is automatically generated by the OpenAPI Generator `yarn openapi:generate`. */ import { RuleSignatureId } from './rule_schema/common_attributes.gen'; export type ErrorSchema = z.infer<typeof ErrorSchema>; export const ErrorSchema = z.object({ id: z.string().optional(), rule_id: RuleSignatureId.optional(), list_id: z.string().min(1).optional(), item_id: z.string().min(1).optional(), error: z.object({ status_code: z.number().min(400), message: z.string(), }), }); ``` --------- Co-authored-by: kibanamachine <[email protected]>
- Loading branch information
1 parent
3d8e425
commit bc37dc2
Showing
27 changed files
with
803 additions
and
73 deletions.
There are no files selected for viewing
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
21 changes: 21 additions & 0 deletions
21
x-pack/plugins/security_solution/common/api/detection_engine/model/warning_schema.gen.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import { z } from 'zod'; | ||
|
|
||
| /* | ||
| * NOTICE: Do not edit this file manually. | ||
| * This file is automatically generated by the OpenAPI Generator `yarn openapi:generate`. | ||
| */ | ||
|
|
||
| export type WarningSchema = z.infer<typeof WarningSchema>; | ||
| export const WarningSchema = z.object({ | ||
| type: z.string(), | ||
| message: z.string(), | ||
| actionPath: z.string(), | ||
| buttonLabel: z.string().optional(), | ||
| }); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
11 changes: 11 additions & 0 deletions
11
x-pack/plugins/security_solution/scripts/openapi/generate.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| require('../../../../../src/setup_node_env'); | ||
| const { generate } = require('./openapi_generator'); | ||
|
|
||
| generate(); |
19 changes: 19 additions & 0 deletions
19
x-pack/plugins/security_solution/scripts/openapi/lib/fix_eslint.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import execa from 'execa'; | ||
| import { resolve } from 'path'; | ||
|
|
||
| const KIBANA_ROOT = resolve(__dirname, '../../../../../../'); | ||
|
|
||
| export async function fixEslint(path: string) { | ||
| await execa('npx', ['eslint', '--fix', path], { | ||
| // Need to run eslint from the Kibana root directory, otherwise it will not | ||
| // be able to pick up the right config | ||
| cwd: KIBANA_ROOT, | ||
| }); | ||
| } |
12 changes: 12 additions & 0 deletions
12
x-pack/plugins/security_solution/scripts/openapi/lib/format_output.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import execa from 'execa'; | ||
|
|
||
| export async function formatOutput(path: string) { | ||
| await execa('npx', ['prettier', '--write', path]); | ||
| } |
21 changes: 21 additions & 0 deletions
21
x-pack/plugins/security_solution/scripts/openapi/lib/remove_gen_artifacts.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import fs from 'fs/promises'; | ||
| import globby from 'globby'; | ||
| import { resolve } from 'path'; | ||
|
|
||
| /** | ||
| * Removes any *.gen.ts files from the target directory | ||
| * | ||
| * @param folderPath target directory | ||
| */ | ||
| export async function removeGenArtifacts(folderPath: string) { | ||
| const artifactsPath = await globby([resolve(folderPath, './**/*.gen.ts')]); | ||
|
|
||
| await Promise.all(artifactsPath.map((artifactPath) => fs.unlink(artifactPath))); | ||
| } |
77 changes: 77 additions & 0 deletions
77
x-pack/plugins/security_solution/scripts/openapi/openapi_generator.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| /* eslint-disable no-console */ | ||
|
|
||
| import SwaggerParser from '@apidevtools/swagger-parser'; | ||
| import chalk from 'chalk'; | ||
| import fs from 'fs/promises'; | ||
| import globby from 'globby'; | ||
| import { resolve } from 'path'; | ||
| import { fixEslint } from './lib/fix_eslint'; | ||
| import { formatOutput } from './lib/format_output'; | ||
| import { removeGenArtifacts } from './lib/remove_gen_artifacts'; | ||
| import { getApiOperationsList } from './parsers/get_api_operations_list'; | ||
| import { getComponents } from './parsers/get_components'; | ||
| import { getImportsMap } from './parsers/get_imports_map'; | ||
| import type { OpenApiDocument } from './parsers/openapi_types'; | ||
| import { initTemplateService } from './template_service/template_service'; | ||
|
|
||
| const ROOT_SECURITY_SOLUTION_FOLDER = resolve(__dirname, '../..'); | ||
| const COMMON_API_FOLDER = resolve(ROOT_SECURITY_SOLUTION_FOLDER, './common/api'); | ||
| const SCHEMA_FILES_GLOB = resolve(ROOT_SECURITY_SOLUTION_FOLDER, './**/*.schema.yaml'); | ||
| const GENERATED_ARTIFACTS_GLOB = resolve(COMMON_API_FOLDER, './**/*.gen.ts'); | ||
|
|
||
| export const generate = async () => { | ||
| console.log(chalk.bold(`Generating API route schemas`)); | ||
| console.log(chalk.bold(`Working directory: ${chalk.underline(COMMON_API_FOLDER)}`)); | ||
|
|
||
| console.log(`👀 Searching for schemas`); | ||
| const schemaPaths = await globby([SCHEMA_FILES_GLOB]); | ||
|
|
||
| console.log(`🕵️♀️ Found ${schemaPaths.length} schemas, parsing`); | ||
| const parsedSchemas = await Promise.all( | ||
| schemaPaths.map(async (schemaPath) => { | ||
| const parsedSchema = (await SwaggerParser.parse(schemaPath)) as OpenApiDocument; | ||
| return { schemaPath, parsedSchema }; | ||
| }) | ||
| ); | ||
|
|
||
| console.log(`🧹 Cleaning up any previously generated artifacts`); | ||
| await removeGenArtifacts(COMMON_API_FOLDER); | ||
|
|
||
| console.log(`🪄 Generating new artifacts`); | ||
| const TemplateService = await initTemplateService(); | ||
| await Promise.all( | ||
| parsedSchemas.map(async ({ schemaPath, parsedSchema }) => { | ||
| const components = getComponents(parsedSchema); | ||
| const apiOperations = getApiOperationsList(parsedSchema); | ||
| const importsMap = getImportsMap(parsedSchema); | ||
|
|
||
| // If there are no operations or components to generate, skip this file | ||
| const shouldGenerate = apiOperations.length > 0 || components !== undefined; | ||
| if (!shouldGenerate) { | ||
| return; | ||
| } | ||
|
|
||
| const result = TemplateService.compileTemplate('schemas', { | ||
| components, | ||
| apiOperations, | ||
| importsMap, | ||
| }); | ||
|
|
||
| // Write the generation result to disk | ||
| await fs.writeFile(schemaPath.replace('.schema.yaml', '.gen.ts'), result); | ||
| }) | ||
| ); | ||
|
|
||
| // Format the output folder using prettier as the generator produces | ||
| // unformatted code and fix any eslint errors | ||
| console.log(`💅 Formatting output`); | ||
| await formatOutput(GENERATED_ARTIFACTS_GLOB); | ||
| await fixEslint(GENERATED_ARTIFACTS_GLOB); | ||
| }; |
Oops, something went wrong.