Generation of Lattice API Docs

[tfogo]

We want the Lattice API docs to be automatically generated. They cannot be generated currently since they're using the Gin HTTP framework and not Carbon. We have to first investigate possibilities for how this could be done.

Current thoughts:

• Swagger/APIB spec: Could generate the documentation from this but would still have to manually maintain the APIB file... so it's a limited win.
• Light wrapper around Gin: Gin's model binding and validation is imperative, so I don't think we'll be able to inspect the Gin router object to generate a full API spec. But we could create a wrapper around Gin so that each handler has a query params, request, and response object associated with it. Then these are automatically bound by the handler. We can then create a tool which inspects the Gin router and generates documentation - similar to our CLI docgen. Still need to investigate this idea more.

Open issues:

• Look into https://goswagger.io/ which is used by k8
• Discuss docs build script with team (mainly kevin and tim)
• Bug introduced by me i think in swag. RunJobRequest has environment field be of type string instead of map[string]ValueOrSecret
• Control model names and try to remove go package name from it
• try to do custom schema as a temp solution for non-supported types such as maps

[abdulito]

Started looking into this. Found an issue on Gin's with some interesting information

gin-gonic/gin#155

So thought about it and currently leaning towards a light wrapper around Gin. Will discuss with the team when i have something solid.

[tfogo]

Another option here is using annotations. There are several tools that will generate docs from annotations:

https://github.com/yvasiyarov/swagger
https://github.com/swaggo/swag
https://github.com/betacraft/yaag

[abdulito]

https://github.com/swaggo/swag Looks the most promising for me. Looks like the project is more maintained/active where the other two have no releases and latest commit was since Nov 2017

[abdulito]

Ok so i have been playing with https://github.com/swaggo/swag was able to generate swagger docs for endpoints after adding annotations. There is an issue which i am not able to get around which is wiring swag with our data model since swag currently is able to deal with structs that are defined within a relative path of the API definition

swaggo/swag#181

swaggo/swag#178

Trying to find a workaround ...

[abdulito]

Ok i was able to wire up the document generator with the our data model. A couple of issues came up
1- Descriptions for struct fields are allowed, however, it didn't work for our case and i filled a bug for the swag tool guys swaggo/swag#183

2- schema for fields with custom types are not generated with the base type. For instance System struct has ID of type SystemID which is defined as follows:

type (
	SystemID      string
)

The generated swag definition is as follows:

    "definitions": {
        "lattice/pkg/api/v1.System": {
            "type": "object",
            "properties": {
                "definitionUrl": {
                    "type": "string"
                },
                "id": {
                    "type": "SystemID"
                },
                "services": {
                    "type": "object"
                },
                "state": {
                    "type": "SystemState"
                }
            }
        }
    },

There is a swag issue reported for this: swaggo/swag#164

So until these issues are resolved, i am gonna be working on generating APIB docs from the swagger docs and a script to automate docs generation.

[abdulito]

Ok i started using https://github.com/apiaryio/swagger2blueprint to generate APIB docs and it working well. I also added make api-docs task for generating the docs.

Currently trying to workaround the two issues above.

[abdulito]

Managed to workaround generating descriptions for fields + making custom types be as object

[abdulito]

Working on small refactoring for the API to make it cleaner/more organized and started putting all docs in place.

[abdulito]

Finished refactoring the API to make it much more organized. Put all documentation in place but need to fill in the blanks of writing the actual docs.

Still waiting on custom types issue swaggo/swag#164

[abdulito]

I hooked up aglio with the doc generation. Need to review with the team

[abdulito]

Here is what the api docs look like now.
lattice-api.html.zip

[abdulito]

waiting on swaggo/swag#164 and requested that i contribute so it looks like i am gonna do so. Also currently syncing with master and resolving conflicts.

[abdulito]

Here is the document for how docs are generated: https://github.com/mlab-lattice/lattice/blob/api-docs/docs/api/api-doc-generation.md

[abdulito]

Currently waiting on the mock api #185 since this is branched off of it.

Also started looking on swag issue above and i will make a contribution.

[abdulito]

merged mock api so we are good for this. Getting very close to figuring out swag 164 (custom types issue)

Also will be considering using ReDoc for generating docs from swagger directly instead of going through API Blueprints. Need to discuss with the team.

[abdulito]

Tried redoc generated html file

redoc-static.html.zip

[abdulito]

Created a pull request for swag 164 issue (custom types)

[abdulito]

Ok slate looks good. It is definitely much more sophisticated and involved since you will have to fork their repository and start a server.

I used https://github.com/mermade/widdershins to export openapi (swagger) to slate md.
Here is how it looks like. Note that anchor links won't work since they point at the local server but this should give you a general idea.

lattice_slate.zip

[BenElgar]

👍 Big fan of the slate docs.

[abdulito]

swag PR accepted and merged. So now you'll see systemId showing type showing up as string instead of "v1.SystemID"

[abdulito]

Actually this is the legit github pages doc https://abdulito.github.io/slate

[abdulito]

Created slate docs repository (https://github.com/mlab-lattice/lattice-api-docs) and made it a submodule. Also made the doc generation script export docs to static html.

[abdulito]

I am currently working on finalizing doc build script. And will need updates my docs about the doc generation so folks like @tfogo and @DesignerDave can play/test it.

[abdulito]

Updated the HowTo on the api doc generation. https://github.com/mlab-lattice/lattice/blob/api-docs/docs/api/api-doc-generation.md

@tfogo @DesignerDave Can you guys give it a try and see if you have any problems?

Thanks!

[abdulito]

Updated the docs on how to generate the api docs again and thanks to @DesignerDave for helping me on this.

There are issues that i have found with swag that may not be trivial to fix. For example, body schema for creating a new job (POST /systems/{system}/jobs) which is the v1.RunJobRequest definition has the wrong schema. The environment field being translated to a string instead of being a definitionv1.ContainerEnvironment object. The reason is swag is currently unable to handle complex types such as custom maps so i was wondering if there is a way to hardcode the schema in certain cases and i opened swaggo/swag#207.

Also, the fact the package name is included in the schema definition could be annoying and unfortunately there is no way of removing that, so i opened swaggo/swag#205 and unfortunately it looks it won't be controlled.

With all that being said, and given the fact that we will introduce more complicated schemas, i would like to take a step back and look into other options that gives us more control of schemas being generated. I have been doing a lot of research and looking at what k8 is doing to get ideas and found https://goswagger.io/ which appears to be a very powerful framework. It allows generating an api server/client off of swagger definitions, and also allows generating swagger off of code annotations. So i'd like to give it a try and see if i can get past the ContainerEnvironment issue and see how far we can get with it but it looks promising. I talked to kevin and he is onboard with that.

[abdulito]

Talked to kevin about this and made some experiments with goswagger. We should come up with a plan soon.

[abdulito]

Been studying the different options we have. To summarize, it appears that the only option we have at the moment to get our docs working fully is to go with goswagger.io. It will be pretty much putting something similar to swagger yaml as comments in our code and have a command to consolidate all of that and put it in one swagger yaml file and then generate slate docs out of that.

It will be pretty tedious and we will have to maintain our comments per changes in our api (e.g. schema). I am gonna discuss with team to see how we will proceed in this.

[abdulito]

Reimplemented api docs with goswagger.io and its looking good. It dealt with some issues that swag wasn't able to handle such as map. Updated docs on how to generate api docs.

[abdulito]

Here is the latest docs: build.zip