Automated OpenAPI spec generation

[pgayvallet]

This issue is here to discuss about the feasibility and the possible technical solutions to generate openAPI specs for Kibana endpoints.

Due to the language we are using, this is far from trivial, and way more difficult than it would be done in a compiled language with proper reflection support.

I won't include the global metadata, common types and such in this discussion, as these are more or less constants and do not (I think) represent any technical challenge.

Just for the record, this is what an OpenAPI path spec looks like

{
  "/pets": {
    "get": {
      "description": "Returns pets based on ID",
      "summary": "Find pets by ID",
      "operationId": "getPetsById",
      "responses": {
        "200": {
          "description": "pet response",
          "content": {
            "*/*": {
              "schema": {
                "type": "array",
                "items": {
                  "$ref": "#/components/schemas/Pet"
                }
              }
            }
          }
        },
        "default": {
          "description": "error payload",
          "content": {
            "text/html": {
              "schema": {
                "$ref": "#/components/schemas/ErrorModel"
              }
            }
          }
        }
      }
    },
    "parameters": [
      {
        "name": "id",
        "in": "path",
        "description": "ID of pet to use",
        "required": true,
        "schema": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "style": "simple"
      }
    ]
  }
}

To simplify and focus on the real challenge, we would need, for each defined endpoint, to be able to retrieve, and convert to OpenAPI format:

• The path, method and authentication mode of the endpoint
• The type of the parameters
• The type of the response payload

Possible solutions

Retrieve the information at runtime

One option would be to have a script that starts Kibana, let all plugins registered their handlers, and then work from the routes that have been registered to core's httpService. This would require to start Kibana in a 'sandboxed' mode that would stop after setup, and collect the routes from core, and then work from there.

The path, method and authentication mode of the endpoint

These would be directly accessible from the route's config

The type of the parameters

We would have access to body, params and query's validation schemas. We would 'just' have to be able to interpret and convert them to OA format at runtime. Of course, this is not something that is currently doable, and would requires to expose metadata on all of kbn/config-schema types to be able to perform reflect-ish operations on the schema.

The type of the response payload

I don't think there is any way to retrieve this information using this approach.

Use AST to parse and retrieve the informations

The second option would be to use AST to parse the code, find all calls to Router#get|post|XXX, and work from there (which is already some heavy parsing work tbh). If it's probably more powerful, this also sounds like colossal work to handle.

The path, method and authentication mode of the endpoint

We should be able to parse the route's configuration and retrieve these infos from there. Note that we would need to be able to resolve variables to scenarios such as path: prefix + "/endpoint".

The type of the parameters

We should be able to retrieve the validation schema nodes, and then use typeChecker.getTypeAtLocation to retrieve the schema type. From there, we would need to retrieve the schema's result type, and then convert it to OpenAPI format.

The type of the response payload

We should be able to find calls to res.ok and use typeChecker.getTypeAtLocation to infer the returned type. We would then need to convert the TS type to openAPI format. I see that https://github.com/grantila/ts-to-openapi exists, so it should probably be doable ourselves (or finding a more used/adopted lib)

Let the endpoints owners specify most of the OpenAPI metadata.

This is what is done by https://www.npmjs.com/package/express-openapi for example

module.exports.get.apiDoc = {
  description: 'A description for retrieving a user.',
  tags: ['users'],
  operationId: 'getUser',
  // parameters for this operation
  parameters: [
    {
      in: 'query',
      name: 'firstName',
      type: 'string'
    }
  ],
  responses: {
    default: {
      $ref: '#/definitions/Error'
    }
  }
};

Not sure this has real value compared to just have solutions teams manually maintain their openAPI spec outside of the raw code.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[rudolf]

Just brainstorming here, haven't given this a lot of thought, but could we generate source code from the OpenAPI spec instead of generating OpenAPI spec from the source code?

If you "compile" the spec it would generate a my_openapi_routes.ts file which exposes a method like mountOpenApiRoutes(router: IRouter, routeHandlers: MyRouteHandlers). The MyRouteHandlers type will be defined inside my_openapi_routes.ts and will ensure that a route handler is defined for every route and method defined by the spec. We could also generate types from the json schema so that the route handlers have typed parameters and expected return types for the responses.

[pgayvallet]

Would need to think more about it, but this seems to be a viable option yes. Seems technically way easier (not easy, easier) than the other way around.

[efreeti]

I think in general it is preferable to keep OpenAPI definitions as your source of truth. This makes the solution language agnostic and allows to generate implementations in multiple languages.

There are projects like this though - https://tsoa-community.github.io/docs/getting-started.html#defining-our-first-model
Java has a similar set of libs to annotate your controllers, but Java has more support for reflection to process method parameters, return types, etc.

[jfsiii]

I think deciding the source of truth and direction is as much a team/workflow decision as a technical one. That said, my preference is to use JSON Schema & OpenAPI to spec shape & behavior and then generate TS types, JS clients, and servers from that.

What @rudolf describes is essentially the plan for Ingest Manager but ours is currently more manual/gradual

Ingest Manager OpenAPI overview

• Define endpoints in OpenAPI
  • Live under https://github.com/elastic/kibana/tree/master/x-pack/plugins/ingest_manager/common/openapi
  • Author using multiple files (per route, schema type, etc) for easier maintenance, better code diffs, etc
  • Provide a single bundle w/o $refs for better tooling support
  • Use JSON schema for as many parts as possible (params, responses, etc) to make it easier to generate TS types & share w/others (ours is currently OpenAPI. Will convert later but not sure if big bang or gradual)
• Generate any artifacts (single spec file, TS types, JS clients) needed with script run manually or git hook
  • bundle example: npx swagger-cli bundle -o bundled.json -t json entrypoint.yaml
  • only TS types example: npx openapi-typescript bundled.json --output schema.ts
  • JS client (browser) + TS types: npx swagger-typescript-api -p bundled.json -n client.ts
• Gradually add those artifacts to each route
  • Can come entirely from spec or only partially
  • e.g. Ingest Manager creates its own validators for a few routes using the JSON schema & AJV

    kibana/x-pack/plugins/ingest_manager/server/routes/agent/index.ts

    Lines 168 to 178 in c355dfe

    	// Agent enrollment
    	router.post(
    	{
    	path: AGENT_API_ROUTES.ENROLL_PATTERN,
    	validate: {
    	body: makeValidator(PostAgentEnrollRequestBodyJSONSchema),
    	},
    	options: { tags: [LIMITED_CONCURRENCY_ROUTE_TAG] },
    	},
    	postAgentEnrollHandler
    	);

  • the path and method post are added manually, but it'd be trivial to find that info for a specific schema via the operationId (e.g. post-fleet-agents-enroll) or loop through all entries

There are many options for generating clients. Many (most?) allow you to choose a transport (nodejs http, fetch, axios, etc) but no customization in the client code that's generated. We'll want one which allows us to supply our own templates so we can use core.http or anything else we want to add. OpenAPI Generator is probably the most popular & well-supported. swagger-typescript-api also looks pretty nice (it's templates are at https://github.com/acacode/swagger-typescript-api/tree/master/src/templates but you can provide your own)

[spong]

This is great -- thanks for detailing this out @pgayvallet! Just want to share this issue we created over on the Security Detections side that captures what an initial PoC could look like from our perspective: #81964

[efreeti]

There are many options for generating clients. Many (most?) allow you to choose a transport (nodejs http, fetch, axios, etc) but no customization in the client code that's generated. We'll want one which allows us to supply our own templates so we can use core.http or anything else we want to add. OpenAPI Generator is probably the most popular & well-supported. swagger-typescript-api also looks pretty nice (it's templates are at https://github.com/acacode/swagger-typescript-api/tree/master/src/templates but you can provide your own)

I was mostly working with Java/Maven generator, but in any case supplying your own templates is pretty easy, main concern is who would maintain those templates.

[lcawl]

FYI while generating specifications for a handful of Kibana APIs (e.g. cases, ML), it became clear that there need to be two specs for each Kibana API due to the optional space ID parameter in the path. Per https://swagger.io/docs/specification/describing-parameters/, "In OpenAPI, a path parameter is defined using in: path. ...Also remember to add required: true, because path parameters are always required...". Therefore to be correct, we need to have two identical specifications for each endpoint--one with and one without the optional {space-id}.

For the purposes of the documentation generated from these specifications, however, I'm leaning toward only including the path that contains the space ID. Otherwise, we're ending up with two pages for each API, where the only difference is one path parameter. If you have questions or feedback, let me know!

[rudolf]

@lcawl that sounds like a pragmatic solution and we could always document if that space id is left out Kibana will use the default space.

[kobelb]

Has there been any progress on this? Currently, @lcawl is manually creating open API specs for the response ops team's HTTP APIs, and the response ops team is having to manually maintain these. This manual process is laborious, error prone, and a significant amount of work. Any automation that could be provided would be incredibly beneficial.

[pgayvallet]

Has there been any progress on this?

No, it hasn't be prioritized, and I doubt the team will be able to take the time to find a proper solution anytime soon given our recent change of focus.

[rudolf]

There's been several proposals to try to prioritise this as part of a larger effort like Kibana Persistence Toolkit and Backward Compatible API Architecture but we failed to get buy-in on these.

I would really like to see us invest in this and it can create a lot of value with the versioned API layer required for serverless.

At a high-level it feels like this space is still somewhat immature so I suspect we'd need a guinea pig team to be an early adopter and find all the kinks.

There's a helpful analysis of the landscape here https://blog.simonireilly.com/posts/typescript-openapi

On the two sides of the spectrum:

1. Generate OpenApi from routes/code https://tsoa-community.github.io/docs/introduction.html
2. Generate TS types from OpenAPI https://github.com/metadevpro/openapi3-ts (API itself) https://github.com/ThomasAribart/json-schema-to-ts (validation schemas)

I would lean towards (2) and it feels like something we could incrementally adopt by e.g. starting to just get types for validation schemas as that might cover the biggest surface of where the spec and code starts diverging.

[rudolf]

Linking to @jloleysens space time PoC #156357

[xcrzx]

Another PoC exploring possibilities of generating runtime schemas, API routes implementation, and API client from OpenAPI spec: #157477

[xcrzx]

For reference, you can follow our progress on all tasks related to migrating to the OpenAPI specification on this meta issue: https://github.com/elastic/security-team/issues/6726.

[rayafratkina]

@jloleysens can you please update?

[jloleysens]

Core team recently started supporting generating OAS types from code. See this epic to track progress on implementation.

In collaboration with teams (providing the missing pieces of information) and docs team (@lcawl) we are hoping to provide missing OAS for Kibana's public endpoints.

[rudolf]

With all the work in #180056 completed, this is considered done.