Nightly builds

[subnetmarco]

It would be useful to setup nightly builds of master (and branches?), so that users can easily try bug fixes on their systems without compiling from source every time or waiting for a next release to be available.

It would also be a way to detect build failures caused by errors in the packages, like new dependencies that have not been specified as required in the generated packages.

[yoanisgil]

@thefosk happy to lend a hand if you need help ;). I think an initial step towards this is to include your Dockerfile in the kong repository, right?

[thibaultcha]

Not necessarily? We don't want the nightly builds to only be Docker. In fact the Dockerfile was in this repository but we decided to split it, and consider it as yet another distribution (package builders are separated, AWS AMIs too, Homebrew too etc...). In fact, since we are working towards an official Docker image I think those must be in their own repository anyways (might be wrong on this).

The point is, we just want to consider Docker as a distribution among others, and let people who use Docker discover the Docker way of using Kong. Nightly builds should ideally consists of the packages too.

---

On another note, what should nightly builds be exactly? If was thinking of building the next branch of the upstream repo as well as be able to manually trigger a build for a hotfix (or other) branch, like we did yesterday for testing the URI encoding fix.

[yoanisgil]

@thibaultcha if the Docker file is in another repo then I don't see a clear way of using Docker Hub, unless a hook chain is in place, which honestly does not look right to me. Anyhow, I see your point about considering Docker just another way of distributing Kong and looks to me like you need something like Jenkins/Bamboo/Travis but you're probably using some of those ;)

[thibaultcha]

Oh no but I wouldn't want to use Dockerhub for the nightly builds. Ideally yes, Jenkins/Travis which we already use and the jobs should perform several builds including the Docker one, and those should be available somewhere, like nightly.getkong.org (really just an example, just saying we should host them somewhere).

Or we could simply give a Dockerfile too. After all users can run the docker build .themselves too... I don't know which one is cleaner.

[yoanisgil]

Looks good to me. Happy to help in anyway possible ;)

[subnetmarco]

Currently when building our releases, we do the following things:

• We tag a new release for Kong
• We execute the build-package.sh script at https://github.com/Mashape/kong-distributions telling it which tag/branch to build, and that creates the pkg, rpm and deb packages.
• After we upload the packages into our Releases section, we create a new tag in https://github.com/Mashape/docker-kong updating the configuration files accordingly, create a new tag in DockerHub, and trigger the build.
• Then we manually update the AWS AMIs.
• We also update manually Luarocks, Homebrew, and Bintray packages.

In order to have a fully automated nightly builds, we need to change some things:

• The scripts are https://github.com/Mashape/kong-distributions assume they will be running in a OSX system, which I don't really like. The script should start an OSX Vagrant instance when building the pkg package, and the only requirement on the build machine should be Docker + Vagrant. Or they should only start OSX into Vagrant with built-in Docker support inside, so that the only dependency would be Vagrant.
• The script is very slow, and builds should be parallelized.
• Our Docker repo currently has its own configuration file stored in a folder. We should use sed to change those couple properties that need to be changed for Docker, so that we can run on any configuration file, even if it's being modified across multiple major releases.
• Docker expects the files to be in a GitHub release - we may want to change this because when building packages for a branch/release that doesn't necessarily mean we have a release.

Updating the AWS AMIs, Homebrew and Bintray is a process that we can relegate to the official releases, so we may decide not to automate them in our Nightly builds (although if there was a script to execute them, that would be great).

[thibaultcha]

About Docker and Nightly builds: those should probably not be pushed to Dockerhub, hence I don't think tagging the release is the way to go. We could simply provide the Dockerfile, updated to point to one of Kong's branches or tag. "Edge release" (because it'd be nice to not just do nightly, but also provide testing releases like yesterday) users could simply get the Dockerfile, and build the image.

[p0pr0ck5]

There are nightly builds for Kong these day! :D https://bintray.com/kong/kong-community-edition-nightly