Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REST API: Provide a way to add extra JSON to generated OpenAPI schema endpoints #4035

Closed
1 of 2 tasks
nlykkei opened this issue Mar 28, 2024 · 25 comments · Fixed by #4703
Closed
1 of 2 tasks

REST API: Provide a way to add extra JSON to generated OpenAPI schema endpoints #4035

nlykkei opened this issue Mar 28, 2024 · 25 comments · Fixed by #4703
Assignees
Labels
feature-request feature request help wanted Could use a second pair of eyes/hands openapi-schema triage Pending triage from maintainers

Comments

@nlykkei
Copy link
Contributor

nlykkei commented Mar 28, 2024

Use case

It would be helpful, if you would support adding extra JSON to generated OpenAPI schema endpoints.

We would like to use the generated OpenAPI scheme to provision an AWS API Gateway REST API using a OpenAPI 3.0 spec. For this, we need to add e.g. x-amazon-apigateway-integration objects.

The only way to accomplish this is by mutating the generated API spec using a post-processing?

https://docs.powertools.aws.dev/lambda/python/latest/core/event_handler/api_gateway/#customizing-api-operations

Example:

  /presigned_put_url:
    get:
      tags:
        - client support
      operationId: getPresignedPutUrl
      description: returns a url for uploading files to the bff backend
      parameters:
        - in: query
          name: file_name
          description: Name of the destination file
          example: my_file.txt
          schema:
            type: string
      responses:
        <<: *default-responses
        "200":
          description: 200 New content - returned when new content available
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/PutUrlResponse"
      x-amazon-apigateway-integration:
        <<: *apigw-integration-201
        uri: arn:aws:apigateway:${region}:lambda:path/2015-03-31/functions/${presigned_put_arn}/invocations
      security: *oidc-security

Solution/User Experience

I would like to add extra JSON at the endpoint level, using e.g. extra_json, to embed custom AWS API Gateway objects, where they are used

Alternative solutions

I could manipulate the generated OpenAPI spec myself using post-processing

Acknowledgment

@nlykkei nlykkei added feature-request feature request triage Pending triage from maintainers labels Mar 28, 2024
Copy link

boring-cyborg bot commented Mar 28, 2024

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@leandrodamascena
Copy link
Contributor

Hello @rubenfonseca! I would like to hear your opinion here.
To me, it makes some sense we inject some additional JSON/keys into the OpenAPI specification, but I'm not sure if we are breaking the OpenAPI contract and may have additional problems with this.

@leandrodamascena leandrodamascena added openapi-schema and removed triage Pending triage from maintainers labels Mar 28, 2024
@rubenfonseca
Copy link
Contributor

@leandrodamascena Yes, you can add custom keys prefixed with "x-" in an OpenAPI schema while still maintaining its validity. These keys, also known as "vendor extensions," allow you to include additional metadata or vendor-specific information in the schema without violating the OpenAPI specification.

The OpenAPI specification allows for the use of vendor extensions, which are properties that begin with the prefix "x-". These extensions can be added at various levels of the schema, such as the root level, object definitions, operation objects, and more. So this would break anything.

It's a nice feature that is heavily explorer by @aws/pdk for instance

@rubenfonseca rubenfonseca self-assigned this Apr 8, 2024
@rubenfonseca
Copy link
Contributor

@nlykkei I was thinking adding a new optional parameter to all the HTTP method annotations, for instance:

@app.get("/hello", vendor_extensions={...}), where vendor_extensions is of type Optional[dict[str, Any]]

Question: does this make sense to you?

I'm also considering what levels should this be added to. Documentation says:

Extensions are supported on the root level of the API spec and in the following places:

  • info section
  • paths section, individual paths and operations
  • operation parameters
  • responses
  • tags
  • security schemes

Following the same approach, we could add a vendor_extensions optional field to all those places.

@rubenfonseca rubenfonseca moved this from Ideas to Working on it in Powertools for AWS Lambda (Python) Apr 8, 2024
@nlykkei
Copy link
Contributor Author

nlykkei commented Apr 8, 2024

Thank you, Ruben.

I'm also considering what levels should this be added to. Documentation says:

Which documentation are you referring to?

Yes, it would sense to add vendor_extensions to those places, so users can inject custom JSON into the generated spec, like the x-amazon-apigateway-integrationobject above.

It could also be used to inject other means of authentication, like API key, HTTP Basic, etc. OAuth 2.0 doesn't cover all use cases :)

@rubenfonseca
Copy link
Contributor

Sorry, forgot the link to the documentation!

I would prefer not to abuse this mechanism to just add random things to the schema :P For security specifically, please see the plan I shared in the other thread!

If you want to take 1 (or more) of this tasks, please let me know :D otherwise I'll be happy to help

@rubenfonseca
Copy link
Contributor

@nlykkei I'm planning to work on this tomorrow if that's ok :)

@nlykkei
Copy link
Contributor Author

nlykkei commented Apr 16, 2024

@rubenfonseca Thank you 🎉

Consider Pydantic's Field, it has an attribute called json_schema_extra for providing extra JSON schema properties:

A dict or callable to provide extra JSON schema properties.

https://docs.pydantic.dev/latest/api/fields/

Such customizability is important for injecting e.g. AWS API Gateway specific objects into the generated OpenAPI spec. Furthermore, depending on the securitySchemes and security implementation for OAuth 2.0, this level of customizability will also be required for users to generate other types of OpenAPI security mechanisms, e.g. API key, HTTP Basic, etc.

I would prefer not to abuse this mechanism to just add random things to the schema :P For security specifically, please see the plan I shared in #4036!

I wouldn't think to much about abusing this mechanism, as neither does Pydantic. It's a mean for users to extend Powertools for AWS Lambda OpenAPI generation capabilities.

I'm really sorry that I haven't had more time. I'm an open source developer by heart, thus my current job takes too much of my time

@nlykkei
Copy link
Contributor Author

nlykkei commented Apr 16, 2024

Sorry, forgot the link to the documentation!

It shows an example of embedding API Gateway authorizer extension to have API Gateway create a custom Lambda authorizer using the OpenAPI spec

@rubenfonseca
Copy link
Contributor

I think your point of using json_schema_extra is EXCELLENT for this feature! Thank you so much for pointing this out!

@nlykkei
Copy link
Contributor Author

nlykkei commented Apr 17, 2024

I'm happy you found it useful! :) I think Pydantic serves as a good reference

@EricOddz
Copy link

Hello, is there any news about this issue?

@leandrodamascena
Copy link
Contributor

Hi @EricOddz! Not yet, I'm working on some v3 issues, but I'll try to look into it next Monday, ok?

@EricOddz
Copy link

thank you so much!!🚀

@leandrodamascena
Copy link
Contributor

Hey, I need help here to confirm whether we are going on the right path or not. I read this thread again a few times and it seems that we are assuming that vendor/openapi extensions are only supported in tags, paths (all levels), operations, parameters, responses and security, as this is what is described in the Swagger example.

But I'm looking at the OpenAPI 3.0 and OpenAPI 3.1 schemas definition and it seems that vendor extensions are accepted at practically all levels of the OpenAPI schema, including root. This brings me some concerns:

1 - Currently we are unable to differentiate the Root and Info object, that is, if the customer creates a definition in get_openapi_json_schema function it will assume the same values for Info and Root, I find this confusing, don't you? I think we should assume to add this information in the info section and forget about Root.

app.py

from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from aws_lambda_powertools.utilities.typing import LambdaContext

app = APIGatewayRestResolver(enable_validation=True)


@app.get("/hello")
def get_todo_title() -> str:
    return app.get_openapi_json_schema(vendor_extensions={"x-project":  {"powertools": "x"}})


def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return app.resolve(event, context)

image

2 - We cannot define an extension in the Root of "paths", but I believe it is possible to define it per operation and per operation parameter.

I propose that we adopt vendor extensions in the following sections and inform in our documentation that only these are supported. From the analysis I did, this covers most of the use cases and some tools that create some type of vendor extension:

  • info
  • path operation
  • path parameters
  • response
  • security schemes

I would like your opinion here.

Thanks

@leandrodamascena leandrodamascena added the help wanted Could use a second pair of eyes/hands label May 29, 2024
@leandrodamascena leandrodamascena moved this from Working on it to Pending review in Powertools for AWS Lambda (Python) May 29, 2024
@dywilson-firstam
Copy link

dywilson-firstam commented May 30, 2024

I'd like to chime in with some additional examples we use these for specifically for api gateway:

  • servers
    image
  • root attributes
    image

@leandrodamascena
Copy link
Contributor

Thanks a lot for your contribution @dywilson-firstam, looks like we have more cases to cover. I'm adding the tag researching and I'm going to do some exploratory testing of all the scenarios we need to cover to get this working in a way that helps our customers.

@leandrodamascena
Copy link
Contributor

We plan to include this support in the release scheduled for June 27th.

@leandrodamascena leandrodamascena moved this from Backlog to Working on it in Powertools for AWS Lambda (Python) Jun 12, 2024
@heitorlessa heitorlessa added the triage Pending triage from maintainers label Jun 24, 2024
@heitorlessa
Copy link
Contributor

Adding triage label to decide whether we make it ready for this Thursday's release

@leandrodamascena
Copy link
Contributor

Unfortunately, it is not possible to release this feature tomorrow. I'm experiencing some issues with the Pydantic models for this implementation and need to figure out how to resolve them.

Rescheduling to the next release: 10/07/2024.

@leandrodamascena
Copy link
Contributor

Updating this issue: We have a PR to add support for OpenAPI extensions. I need to improve the code and tests in this PR, but it is currently working and creating the schema as expexed:

from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from aws_lambda_powertools.event_handler.openapi.models import Server, APIKey, APIKeyIn

app = APIGatewayRestResolver(enable_validation=True)

servers = Server(
    url="http://example.com", 
    description="Example server", 
    openapi_extensions={"x-amazon-apigateway-endpoint-configuration": {"vpcEndpoint": "myendpointid"}} # SERVER LEVEL
    )


@app.get("/hello", openapi_extensions={"x-amazon-apigateway-integration": {"type": "aws", "uri": "my_lambda_arn"}}) # OPERATION LEVEL
def hello():
    return app.get_openapi_json_schema(
        servers=[servers], 
        security_schemes={
            "apikey": APIKey(
                    name="X-API-KEY",
                    description="API KeY", 
                    in_=APIKeyIn.header, 
                    openapi_extensions={"x-amazon-apigateway-authorizer": "custom"} # SECURITY SCHEME LEVEL
                )
            },
        openapi_extensions={"x-amazon-apigateway-gateway-responses": {"DEFAULT_4XX"}}) # ROOT LEVEL

def lambda_handler(event, context):
    return app.resolve(event, context)

openapi schema

{
   "openapi":"3.0.3",
   "info":{
      "title":"Powertools API",
      "version":"1.0.0"
   },
   "servers":[
      {
         "url":"http://example.com",
         "description":"Example server",
         "x-amazon-apigateway-endpoint-configuration":{
            "vpcEndpoint":"myendpointid"
         }
      }
   ],
   "paths":{
      "/hello":{
         "get":{
            "summary":"GET /hello",
            "operationId":"hello_hello_get",
            "responses":{
               "200":{
                  "description":"Successful Response",
                  "content":{
                     "application/json":{
                        
                     }
                  }
               },
               "422":{
                  "description":"Validation Error",
                  "content":{
                     "application/json":{
                        "schema":{
                           "$ref":"#/components/schemas/HTTPValidationError"
                        }
                     }
                  }
               }
            },
            "x-amazon-apigateway-integration":{
               "type":"aws",
               "uri":"my_lambda_arn"
            }
         }
      }
   },
   "components":{
      "schemas":{
         "HTTPValidationError":{
            "properties":{
               "detail":{
                  "items":{
                     "$ref":"#/components/schemas/ValidationError"
                  },
                  "type":"array",
                  "title":"Detail"
               }
            },
            "type":"object",
            "title":"HTTPValidationError"
         },
         "ValidationError":{
            "properties":{
               "loc":{
                  "items":{
                     "anyOf":[
                        {
                           "type":"string"
                        },
                        {
                           "type":"integer"
                        }
                     ]
                  },
                  "type":"array",
                  "title":"Location"
               },
               "type":{
                  "type":"string",
                  "title":"Error Type"
               }
            },
            "type":"object",
            "required":[
               "loc",
               "msg",
               "type"
            ],
            "title":"ValidationError"
         }
      },
      "securitySchemes":{
         "apikey":{
            "type":"apiKey",
            "description":"API KeY",
            "in":"header",
            "name":"X-API-KEY",
            "x-amazon-apigateway-authorizer":"custom"
         }
      }
   },
   "x-amazon-apigateway-gateway-responses":[
      "DEFAULT_4XX"
   ]
}

@leandrodamascena
Copy link
Contributor

The pull request to add support for the OpenAPI extension feature is about to be merged, it's missing some last review - #4703.

We've decided not to include it in the next release (11/07). Instead, we'll include it in the Powertools Python Alpha releases starting Friday (12/07) - https://pypi.org/project/aws-lambda-powertools/#history.

The reason we've decided not to add it in this release is that we prefer to allow customers some time to test it in an alpha release before deploying it to production environments. This is a sensitive addition to the OpenAPI feature, and that's why we've decided to introduce it as an alpha release.

Copy link
Contributor

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

@github-actions github-actions bot added the pending-release Fix or implementation already in dev waiting to be released label Jul 15, 2024
@leandrodamascena
Copy link
Contributor

The OpenAPI extensions feature is available in this pre-release: https://pypi.org/project/aws-lambda-powertools/2.41.1a2/

If possible, test and report any type of bug or improvement before we go into production in 2 weeks.

Copy link
Contributor

This is now released under 2.42.0 version!

@github-actions github-actions bot removed the pending-release Fix or implementation already in dev waiting to be released label Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request feature request help wanted Could use a second pair of eyes/hands openapi-schema triage Pending triage from maintainers
Projects
Status: Coming soon
Development

Successfully merging a pull request may close this issue.

6 participants