Add OpenAPI 3 specification and source generation for REST API (#175)

* index file json schema added. Signed-off-by: Michael Valdron <[email protected]> * Swagger 2.0 converted to OpenAPI 3.0 Signed-off-by: Michael Valdron <[email protected]> * oapi-codegen added as dependency. Signed-off-by: Michael Valdron <[email protected]> * Codegen config for server package schema types added. Signed-off-by: Michael Valdron <[email protected]> * V2 index file schema added. Signed-off-by: Michael Valdron <[email protected]> * correct operationIds in index server spec Signed-off-by: Michael Valdron <[email protected]> * endpoint code gen config added. Signed-off-by: Michael Valdron <[email protected]> * index server dependencies modified. Signed-off-by: Michael Valdron <[email protected]> * REST API endpoint definitions and bindings generated. Signed-off-by: Michael Valdron <[email protected]> * Code refactored to use generated source. Signed-off-by: Michael Valdron <[email protected]> * starter project route added with other openapi schema changes. Signed-off-by: Michael Valdron <[email protected]> * rename swagger.yaml to openapi.yaml Signed-off-by: Michael Valdron <[email protected]> * index and download starter project api endpoint signatures refactored. Signed-off-by: Michael Valdron <[email protected]> * Endpoints added to openapi spec. Signed-off-by: Michael Valdron <[email protected]> * template header new year updates. Signed-off-by: Michael Valdron <[email protected]> * source generation config files moved to base directory of index server. Signed-off-by: Michael Valdron <[email protected]> * imports.tmpl file for source generation header added to index server. Signed-off-by: Michael Valdron <[email protected]> * typing changes to openapi spec of index server. Signed-off-by: Michael Valdron <[email protected]> * codegen script added. Signed-off-by: Michael Valdron <[email protected]> * fix deps from rebase Signed-off-by: Michael Valdron <[email protected]> * root, viewer, oci server endpoints added to openapi spec with schema fixups Signed-off-by: Michael Valdron <[email protected]> * new types generated Signed-off-by: Michael Valdron <[email protected]> * endpoint changes. Signed-off-by: Michael Valdron <[email protected]> * Set base directory of build registry script, now can be run from any pwd Signed-off-by: Michael Valdron <[email protected]> * proxy and static endpoints reverted from OpenAPI, OpenAPI validation middleware only applies to OpenAPI routes. Signed-off-by: Michael Valdron <[email protected]> * correct invalid changes introduced in f991981 Signed-off-by: Michael Valdron <[email protected]> * index schemas OpenAPI specs removed for a future issue Signed-off-by: Michael Valdron <[email protected]> * codegen happens on build Signed-off-by: Michael Valdron <[email protected]> * query parameter schema changes. Signed-off-by: Michael Valdron <[email protected]> * update source to updated schemas. Signed-off-by: Michael Valdron <[email protected]> * codegen changes. Signed-off-by: Michael Valdron <[email protected]> * add README documentation on OpenAPI code generation. Signed-off-by: Michael Valdron <[email protected]> --------- Signed-off-by: Michael Valdron <[email protected]>
devfile · Jul 10, 2023 · 605081d · 605081d
1 parent 65033fd
commit 605081d
Show file tree

Hide file tree

Showing 429 changed files with 79,251 additions and 15,827 deletions.
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -1,3 +1,17 @@
+#
+#   Copyright 2020-2023 Red Hat, Inc.
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
 name: CI
 
 on:
@@ -28,6 +42,19 @@ jobs:
       - name: Check if registry-library build is working
         run: cd registry-library && bash ./build.sh
 
+      - name: Check index-server code generation
+        run: |
+          cd index/server
+          export GOPATH=$(go env GOPATH)
+          go install github.com/deepmap/oapi-codegen/cmd/[email protected]
+          bash codegen.sh
+          GEN_DIFFS=$(git diff --name-only --diff-filter=ACMRT | grep .gen.go$ | xargs)
+          if [[ ! -z "${GEN_DIFFS}" ]]
+          then
+            echo "generated source does not match current changes "
+            exit 1
+          fi
+
       - name: Run Gosec Security Scanner
         run: |
           export PATH=$PATH:$(go env GOPATH)/bin
@@ -53,6 +80,11 @@ jobs:
       - name: Check out code
         uses: actions/checkout@v2
 
+      - name: Setup Go environment
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.18
+
       - name: Check license
         run: |
           export PATH=$PATH:$(go env GOPATH)/bin
@@ -67,7 +99,11 @@ jobs:
           fi
 
       - name: Check if index server build is working
-        run: cd index/server && bash ./build.sh
+        run: |
+          cd index/server
+          export GOPATH=$(go env GOPATH)
+          go install github.com/deepmap/oapi-codegen/cmd/[email protected]
+          bash ./build.sh
 
       - name: Test index server
         run: cd index/server && go test ./... -coverprofile cover.out
@@ -91,7 +127,7 @@ jobs:
       - name: Setup Go environment
         uses: actions/setup-go@v2
         with:
-          go-version: 1.17
+          go-version: 1.18
       - name: Setup Minikube
         uses: manusa/[email protected]
         with:
@@ -101,7 +137,10 @@ jobs:
           github token: ${{ secrets.GITHUB_TOKEN }}
           start args: '--addons=ingress'
       - name: Run the devfile registry integration tests
-        run: bash .ci/run_tests_minikube_linux.sh
+        run: | 
+          export GOPATH=$(go env GOPATH)
+          go install github.com/deepmap/oapi-codegen/cmd/[email protected]
+          bash .ci/run_tests_minikube_linux.sh
 
   test_staging:
     name: Test Staging Devfile Registry
@@ -112,7 +151,7 @@ jobs:
       - name: Setup Go environment
         uses: actions/setup-go@v2
         with:
-          go-version: 1.17
+          go-version: 1.18
       - name: Run the devfile registry integration tests
         run: |
           # Run the integration tests

diff --git a/.github/workflows/pushimage-next.yaml b/.github/workflows/pushimage-next.yaml
@@ -1,5 +1,5 @@
 #
-#   Copyright 2020-2022 Red Hat, Inc.
+#   Copyright 2020-2023 Red Hat, Inc.
 #
 #   Licensed under the Apache License, Version 2.0 (the "License");
 #   you may not use this file except in compliance with the License.
@@ -27,14 +27,22 @@ jobs:
     steps:
       - name: Check out registry support source code
         uses: actions/checkout@v2
+      - name: Setup Go environment
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.18
       - name: Login to Quay
         uses: docker/login-action@v1
         with:
           registry: quay.io
           username: ${{ secrets.QUAY_USERNAME }}
           password: ${{ secrets.QUAY_PASSWORD }}
       - name: Build the index server base image
-        run: cd index/server && bash ./build.sh
+        run: | 
+          cd index/server
+          export GOPATH=$(go env GOPATH)
+          go install github.com/deepmap/oapi-codegen/cmd/[email protected]
+          bash ./build.sh
       - name: Push the index server base image
         run: cd index/server && bash ./push.sh quay.io/devfile/devfile-index-base:next
 

diff --git a/README.md b/README.md
@@ -28,6 +28,11 @@ followed by
 docker push <registry>/<username>/devfile-index:latest
 ```
 
+See the following for more on the component specific build process:
+
+- [Building the OCI Registry](oci-registry/README.md#build)
+- [Building the Index Server](index/server/README.md#build)
+
 ## Deploy
 
 ### Via the Devfile Registry Operator

diff --git a/build_registry.sh b/build_registry.sh
@@ -5,11 +5,15 @@
 # and want to test all of the components together
 shopt -s expand_aliases
 set -eux
+
+# Set base registry support directory
+BASE_DIR=$(dirname $0)
+
 #set the docker alias if necessary
-. ./setenv.sh
+. ${BASE_DIR}/setenv.sh
 
 # Build the index server base image
-. ./index/server/build.sh
+. ${BASE_DIR}/index/server/build.sh
 
 # Build the test devfile registry image
-docker build -t devfile-index:latest -f .ci/Dockerfile .
+docker build -t devfile-index:latest -f ${BASE_DIR}/.ci/Dockerfile ${BASE_DIR}
diff --git a/index/server/README.md b/index/server/README.md
@@ -6,12 +6,42 @@ Provides REST API support for devfile registries and serves [devfile registry vi
 
 For more information on REST API docs: [registry-REST-API.adoc](registry-REST-API.adoc)
 
+## Defining Endpoints
+
+Edit the OpenAPI spec `openapi.yaml`, under `paths` you can define your endpoint, e.g. `GET /foo`:
+
+```yaml
+paths:
+    /foo:
+        get:
+            summary: <short summary of what your endpoint does>
+            description: <a long description of what your endpoint does>
+            # 'serveFoo' points to handler function 'ServeFoo'
+            operationId: serveFoo
+            parameters: # the OpenAPI specifications of the endpoint parameters 
+                # spec for passing a bar query parameter /foo?bar=<string>
+                - name: bar
+                  in: query
+                  description: <description for parameter>
+                  required: false
+                  schema:
+                      type: string
+            responses: # the OpenAPI specifications for the endpoint responses
+                default:
+                    description: <description of the response>
+                    content:
+                        # Content type(s)
+                        text/html: {}
+```
+
+See [swagger.io/docs](https://swagger.io/docs/specification/paths-and-operations) for more information.
+
 ## Build
 
 The registry index server is built into a container image, `devfile-index-base:latest`, by running the following script:
 
 ```sh
-bash index/server/build.sh
+bash build.sh
 ```
 
 You retag it with one of the two command:
@@ -31,15 +61,32 @@ podman tag devfile-index-base:latest <new-image-tag>
 Push your image into the an image repository with the following:
 
 ```sh
-bash index/server/push.sh <new-image-tag>
+bash push.sh <new-image-tag>
 ```
 
 For example, if the image repository is quay.io then use the pattern `quay.io/<user>/devfile-index-base`:
 
 ```sh
-bash index/server/push.sh quay.io/someuser/devfile-index-base
+bash push.sh quay.io/someuser/devfile-index-base
 ```
 
+### Source Generation
+
+Index server build uses the CLI tool `oapi-codegen` to generate the schema types `pkg/server/types.gen.go` and endpoint definition `pkg/server/endpoint.gen.go` sources. When changing the OpenAPI specification, such as [defining endpoints](#defining-endpoints), it is required to regenerate these changes into the source. 
+
+The source generation can be done by manually building the index server with: 
+
+```bash
+bash build.sh
+```
+or to just generate the source files by running:
+
+```bash
+bash codegen.sh
+```
+
+**Important**: When committing to this repository, it is *required* to include the up to date source generation in your pull requests. Not including up to date source generation will lead to the PR check to fail.
+
 ## Testing
 
 Endpoint unit testing is defined under `pkg/server/endpoint_test.go` and can be performed by running the following:

diff --git a/index/server/build.sh b/index/server/build.sh
@@ -3,5 +3,8 @@
 # Build the index container for the registry
 buildfolder="$(realpath $(dirname ${BASH_SOURCE[0]}))"
 
+# Generate OpenAPI endpoint and type definitions
+bash ${buildfolder}/codegen.sh
+
 # Build the index server
 docker build -t devfile-index-base:latest $buildfolder
diff --git a/index/server/codegen.sh b/index/server/codegen.sh
@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# Base of the index server directory
+projectfolder="$(realpath $(dirname ${BASH_SOURCE[0]}))"
+
+# Generate source types from OpenAPI spec
+echo "Generating type source.."
+cd ${projectfolder} && ${GOPATH}/bin/oapi-codegen -config config/types.yaml openapi.yaml && cd - > /dev/null
+if [ $? != 0 ]
+then
+    echo "error with generating type source "
+    exit 1
+fi
+
+# Generate endpoint bindings source from OpenAPI spec
+echo "Generating endpoint bindings source.."
+cd ${projectfolder} && ${GOPATH}/bin/oapi-codegen -config config/endpoint.yaml openapi.yaml && cd - > /dev/null
+if [ $? != 0 ]
+then
+    echo "error with generating endpoint bindings source "
+    exit 1
+fi
+
+echo "Done."
diff --git a/index/server/config/endpoint.yaml b/index/server/config/endpoint.yaml
@@ -0,0 +1,25 @@
+package: server
+generate:
+  gin-server: true
+  embedded-spec: true
+output: pkg/server/endpoint.gen.go
+output-options:
+  user-templates:
+    imports.tmpl: |-
+      //
+      // Copyright 2023 Red Hat, Inc.
+      //
+      // Licensed under the Apache License, Version 2.0 (the "License");
+      // you may not use this file except in compliance with the License.
+      // You may obtain a copy of the License at
+      //
+      //     http://www.apache.org/licenses/LICENSE-2.0
+      //
+      // Unless required by applicable law or agreed to in writing, software
+      // distributed under the License is distributed on an "AS IS" BASIS,
+      // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+      // See the License for the specific language governing permissions and
+      // limitations under the License.
+      //
+      // Code generated by {{.ModuleName}} version {{.Version}}. **DO NOT EDIT**
+      package {{.PackageName}}
diff --git a/index/server/config/types.yaml b/index/server/config/types.yaml
@@ -0,0 +1,25 @@
+package: server
+generate:
+  models: true
+output: pkg/server/types.gen.go
+output-options:
+  skip-prune: true
+  user-templates:
+    imports.tmpl: |-
+      //
+      // Copyright 2023 Red Hat, Inc.
+      //
+      // Licensed under the Apache License, Version 2.0 (the "License");
+      // you may not use this file except in compliance with the License.
+      // You may obtain a copy of the License at
+      //
+      //     http://www.apache.org/licenses/LICENSE-2.0
+      //
+      // Unless required by applicable law or agreed to in writing, software
+      // distributed under the License is distributed on an "AS IS" BASIS,
+      // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+      // See the License for the specific language governing permissions and
+      // limitations under the License.
+      //
+      // Code generated by {{.ModuleName}} version {{.Version}}. **DO NOT EDIT**
+      package {{.PackageName}}