v2.0.0 - first release with the new versioning scheme

AWS_FOR_FLUENT_BIT_VERSION

@@ -1 +1 @@
-1.3.2
+2.0.0

CHANGELOG.md

@@ -1 +1,16 @@
 # Changelog
+
+### 2.0.0
+
+**Note:** This is the first AWS for Fluent Bit release under our new versioning scheme; previously the image was versioned by the Fluent Bit version it contained. Please see the project README for an explanation of how we version this project.
+
+This release includes:
+* An Amazon Linux 2 Base
+* Fluent Bit 1.3.3
+* Amazon CloudWatch Logs for Fluent Bit 1.0.0
+* Amazon Kinesis Streams for Fluent Bit 1.0.0
+* Amazon Kinesis Firehose for Fluent Bit 1.0.0
+
+Compared to `aws-for-fluent-bit:1.3.2` this release adds:
+* Bug - Allow retries for creating log group - [cloudwatch:18](https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/issues/18)
+* Feature - Add Fluent Bit plugin for Amazon Kinesis Data Streams - [kinesis:1](https://github.com/aws/amazon-kinesis-streams-for-fluent-bit/pull/1)

Dockerfile

@@ -3,8 +3,8 @@ FROM amazonlinux:latest as builder
 # Fluent Bit version; update these for each release
 ENV FLB_MAJOR 1
 ENV FLB_MINOR 3
-ENV FLB_PATCH 2
-ENV FLB_VERSION 1.3.2
+ENV FLB_PATCH 3
+ENV FLB_VERSION 1.3.3
 # branch to pull parsers from in github.com/fluent/fluent-bit-docker-image
 ENV FLB_DOCKER_BRANCH 1.3
 
@@ -100,9 +100,13 @@ COPY --from=aws-fluent-bit-plugins:latest /cloudwatch/THIRD-PARTY \
 COPY --from=aws-fluent-bit-plugins:latest /kinesis-streams/THIRD-PARTY \
     /kinesis-streams/LICENSE \
     /fluent-bit/licenses/kinesis/
+COPY AWS_FOR_FLUENT_BIT_VERSION /AWS_FOR_FLUENT_BIT_VERSION
+
+COPY entrypoint.sh /entrypoint.sh
+RUN chmod +x /entrypoint.sh
 
 # Optional Metrics endpoint
 EXPOSE 2020
 
 # Entry point
-CMD ["/fluent-bit/bin/fluent-bit", "-e", "/fluent-bit/firehose.so", "-e", "/fluent-bit/cloudwatch.so", "-e", "/fluent-bit/kinesis.so", "-c", "/fluent-bit/etc/fluent-bit.conf"]
+CMD /entrypoint.sh

README.md

@@ -1,52 +1,120 @@
 ## AWS for Fluent Bit Docker Image
 
+### Versioning FAQ
+
+The version of the AWS for Fluent Bit image is not linked to the version of Fluent Bit which it contains.
+
+**What does the version number signify?**
+
+We use the standard `major.minor.patch` versioning scheme for our image, AKA Semantic Versioning. The initial release with this versioning scheme is `2.0.0`. An update to the patch version indicates backwards compatible bug fixes, a minor version change indicates new backwards compatible functionality, and a major version change indicates backwards incompatible changes.
+
+The AWS for Fluent Bit image includes the following contents:
+* A base linux image (currently Amazon Linux)
+* Fluent Bit
+* Several Fluent Bit Go Plugins
+
+A change in any one of these pieces would lead to a change in our version number.
+
+**What about the 1.x image tags in your repositories?**
+
+The AWS for Fluent Bit image was launched in July 2019. Between July and October of 2019 we simply versioned the image based on the version of Fluent Bit that it contained. During this time we released `1.2.0`, `1.2.2` and `1.3.2`.
+
+The old versioning scheme was simple and it made it clear which version of Fluent Bit our image contained. However, it had a serious problem- how could we signify that we had changed the other parts of the image? If we did not update Fluent Bit, but updated one of the plugins, how would we signify this in a new release? There was no answer- we could only release an update when Fluent Bit released a new version. We ultimately realized this was unacceptable- bug fixes or new features in our plugins should not be tied to the Fluent Bit release cadence.
+
+Thus, we moved to the a new versioning scheme. Because customers already are relying on the `1.x` tags, we have left them in our repositories. The first version with the new scheme is `2.0.0`. From now on we will follow semantic versioning- but the move from `1.3.2` did not follow semantic versioning. There are no backwards incompatible changes between `aws-for-fluent-bit:1.3.2` and `aws-for-fluent-bit:2.0.0`. Our release notes for `2.0.0` clearly explain the change.
+
+**Does this mean you are diverging from fluent/fluent-bit?**
+
+No. We continue to consume Fluent Bit from its main repository. We are not forking Fluent Bit.
+
 ### Public Images
 
-There are image tags for `latest` and the version of Fluent Bit that is built in the image. The first release is Fluent Bit `1.2.0`.
+Each release updates the `latest` tag and adds a tag for the version of the image.
+
+#### Using SSM to find available versions
+
+As of 2.0.0, there are SSM Public Parameters which allow you to see available versions. These parameters are available in every region that the image is available in. Any AWS account can query these parameters.
+
+To see a list of available version tags, run the following command:
+
+```
+aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/ --query 'Parameters[*].Name'
+```
+
+Example output:
+
+```
+[
+    "/aws/service/aws-for-fluent-bit/latest"
+    "/aws/service/aws-for-fluent-bit/2.0.0"
+]
+```
+
+To see the ECR repository ID for a given image tag, run the following:
+
+```
+$ aws ssm get-parameter --name /aws/service/aws-for-fluent-bit/2.0.0
+{
+    "Parameter": {
+        "Name": "/aws/service/aws-for-fluent-bit/2.0.0",
+        "Type": "String",
+        "Value": "906394416424.dkr.ecr.us-east-1.amazonaws.com/aws-for-fluent-bit:2.0.0",
+        "Version": 1,
+        "LastModifiedDate": 1539908129.759,
+        "ARN": "arn:aws:ssm:us-west-2::parameter/aws/service/aws-for-fluent-bit/2.0.0"
+    }
+}
+```
+
+#### Using SSM Parameters in CloudFormation Templates
+
+You can use these SSM Parameters as parameters in your CloudFormation templates.
+
+```
+Parameters:
+  FireLensImage:
+    Description: Fluent Bit image for the FireLens Container
+    Type: AWS::SSM::Parameter::Value<String>
+    Default: /aws/service/aws-for-fluent-bit/latest
+```
+
+#### Using image tags
+
+You should lock your deployments to a specific version tag. We guarantee that these tags will be immutable- once they are released the will not change.
 
 ##### Docker Hub
 
 [amazon/aws-for-fluent-bit](https://hub.docker.com/r/amazon/aws-for-fluent-bit/tags)
 
 ##### Amazon ECR
 
-We also provide images in Amazon ECR in each AWS Region for high availability.
-
-| Region         | Registry ID  | Full Image URI                                                              |
-|----------------|--------------|-----------------------------------------------------------------------------|
-| us-east-1      | 906394416424 | 906394416424.dkr.ecr.us-east-1.amazonaws.com/aws-for-fluent-bit:latest      |
-| eu-west-1      | 906394416424 | 906394416424.dkr.ecr.eu-west-1.amazonaws.com/aws-for-fluent-bit:latest      |
-| us-west-1      | 906394416424 | 906394416424.dkr.ecr.us-west-1.amazonaws.com/aws-for-fluent-bit:latest      |
-| ap-southeast-1 | 906394416424 | 906394416424.dkr.ecr.ap-southeast-1.amazonaws.com/aws-for-fluent-bit:latest |
-| ap-northeast-1 | 906394416424 | 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:latest |
-| us-west-2      | 906394416424 | 906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest      |
-| sa-east-1      | 906394416424 | 906394416424.dkr.ecr.sa-east-1.amazonaws.com/aws-for-fluent-bit:latest      |
-| ap-southeast-2 | 906394416424 | 906394416424.dkr.ecr.ap-southeast-2.amazonaws.com/aws-for-fluent-bit:latest |
-| eu-central-1   | 906394416424 | 906394416424.dkr.ecr.eu-central-1.amazonaws.com/aws-for-fluent-bit:latest   |
-| ap-northeast-2 | 906394416424 | 906394416424.dkr.ecr.ap-northeast-2.amazonaws.com/aws-for-fluent-bit:latest |
-| ap-south-1     | 906394416424 | 906394416424.dkr.ecr.ap-south-1.amazonaws.com/aws-for-fluent-bit:latest     |
-| us-east-2      | 906394416424 | 906394416424.dkr.ecr.us-east-2.amazonaws.com/aws-for-fluent-bit:latest      |
-| ca-central-1   | 906394416424 | 906394416424.dkr.ecr.ca-central-1.amazonaws.com/aws-for-fluent-bit:latest   |
-| eu-west-2      | 906394416424 | 906394416424.dkr.ecr.eu-west-2.amazonaws.com/aws-for-fluent-bit:latest      |
-| eu-west-3      | 906394416424 | 906394416424.dkr.ecr.eu-west-3.amazonaws.com/aws-for-fluent-bit:latest      |
-| ap-northeast-3 | 906394416424 | 906394416424.dkr.ecr.ap-northeast-3.amazonaws.com/aws-for-fluent-bit:latest |
-| eu-north-1     | 906394416424 | 906394416424.dkr.ecr.eu-north-1.amazonaws.com/aws-for-fluent-bit:latest     |
-| ap-east-1      | 449074385750 | 449074385750.dkr.ecr.ap-east-1.amazonaws.com/aws-for-fluent-bit:latest      |
-| cn-north-1     | 128054284489 | 128054284489.dkr.ecr.cn-north-1.amazonaws.com/aws-for-fluent-bit:latest     |
-| cn-northwest-1 | 128054284489 | 128054284489.dkr.ecr.cn-northwest-1.amazonaws.com/aws-for-fluent-bit:latest |
-| us-gov-east-1  | 161423150738 | 161423150738.dkr.ecr.us-gov-east-1.amazonaws.com/aws-for-fluent-bit:latest  |
-| us-gov-west-1  | 161423150738 | 161423150738.dkr.ecr.us-gov-west-1.amazonaws.com/aws-for-fluent-bit:latest  |
+We also provide images in Amazon ECR for high availability. These images are available in almost every AWS region, included AWS Gov Cloud.
+
+The official way to find the ECR image URIs for your region is to use the SSM Parameters. In your region, run the following command:
+
+```
+aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/
+```
+
+### Plugins
+
+We currently bundle the following projects in this image:
+* [amazon-kinesis-firehose-for-fluent-bit](https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit)
+* [amazon-cloudwatch-logs-for-fluent-bit](https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit)
+* [amazon-kinesis-streams-for-fluent-bit](https://github.com/aws/amazon-kinesis-streams-for-fluent-bit)
 
 ### Development
 
 #### Releasing
 
-Use `make release` to build the image. 
-To run the integration tests, run `make integ-dev`. The `make integ-dev` command will run the integration tests for all of our plugins- 
+Use `make release` to build the image.
+To run the integration tests, run `make integ-dev`. The `make integ-dev` command will run the integration tests for all of our plugins-
 kinesis streams, and cloudwatch.
 
 To run integration tests separately, execute `make integ-cloudwatch` or `make integ-kinesis`.
 
+[Documentation on GitHub steps for releases](Release_Process.md).
+
 #### Developing Features in the AWS Plugins
 
 You can build a version of the image with code in your GitHub fork. To do so, you must need to set the following environment variables.
@@ -67,7 +135,7 @@ export KINESIS_PLUGIN_BRANCH="Your branch on your fork"
 ```
 
 Then run `make cloudwatch-dev` or `make kinesis-dev` to build the image with your changes.
- 
+
 To run the integration tests on your code, execute `make integ-cloudwatch-dev` or `make integ-kinesis-dev`.
 
 ## License

Release_Process.md

@@ -0,0 +1,93 @@
+# Releasing AWS for Fluent Bit
+
+Steps which must performed on GitHub before we put out any images.
+
+### How to write Change log entries
+
+The ECS CLI is a good example: https://github.com/aws/amazon-ecs-cli/blob/master/CHANGELOG.md
+
+Note that the PR or issue associated with each change is listed. Use the issue number if an issue is associated with the change; if there is no issue, use the PR.
+
+For the aws-for-fluent-bit image, the issues will likely be in one of the plugin repos; reference them in markdown like this:
+
+```
+* Feature - Allow retries for creating log group - [cloudwatch:18](https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/issues/18)
+```
+
+
+List features as:
+
+* “Enhancement” - a very small change or improvement. For our fluent bit plugins, something like “improve debug log messages” would be an enhancement.
+* “Feature” - most additions will follow under this category. For our Fluent Bit plugins, any time we add a new config option would be a feature.
+* “Bug” - a change which fixes something which was broken previously.
+
+
+Any change log with a “Feature” should lead to a Minor version bump. A Change log with only “Enhancement” entries can have a patch version bump.
+
+For more, learn about semantic versioning: https://semver.org/
+
+### Before releasing the images
+
+#### Cut “releases” of the AWS Fluent Bit Plugins
+
+The first step before a release is freeze development on the Fluent Bit plugins. No new PRs will be merged until after the release CM has completed.
+
+For every Fluent Bit plugin which has changes since the last AWS for Fluent Bit release, we will create a GitHub release, and a new Git Tag (the tag is auto-created with the GitHub release). We will follow SemVer.
+
+We are not releasing binaries for the Fluent Bit plugins. We will have GitHub releases just because it makes it easy to track changes of the plugins over time.
+
+Steps:
+
+1. Submit a PR to update the Change log for the plugin.
+2. Merge the PR (get an approval), and publish a GitHub release with the same contents as the Change log entry
+
+_Example Change Log Entry_
+
+```
+## 1.1.0
+* Add `log_group_key` field to allow the log group name to be dynamically chosen for each log record (#25)
+```
+
+#### Version Bump and Change Log for AWS For Fluent Bit
+
+Submit a PR to update the version file (https://github.com/aws/aws-for-fluent-bit/blob/master/AWS_FOR_FLUENT_BIT_VERSION), and add an entry to the Changelog (https://github.com/aws/aws-for-fluent-bit/blob/master/CHANGELOG.md).
+
+For the AWS for Fluent Bit Change log, you will copy the “Enhancement”, “Feature” and “Bug” items from the releases for each of the plugins. This is a convenience for customers- they only need to go to one place to see what is new with each release.
+
+_Example Change Log Entry_
+```
+## 2.0.1
+
+This release includes:
+* An Amazon Linux 2 Base
+* Fluent Bit 1.3.2
+* Amazon CloudWatch Logs for Fluent Bit 1.1.0
+* Amazon Kinesis Streams for Fluent Bit 1.0.0
+* Amazon Kinesis Firehose for Fluent Bit 1.0.0
+
+Compared to 2.0.0 this release adds:
+* Feature - add `log_group_key` in CloudWatch Plugin
+```
+#### Draft a GitHub Release
+
+Draft (but do not publish) a GitHub release for the image.
+
+*Tag Version:* v + the version. For example v2.0.1
+
+*Release Title:* AWS for Fluent Bit 2.0.1
+
+*Release Notes:* Same as the Change log entry.
+
+#### Check the version
+
+Run `make release` to run the image to check that the version and Fluent Bit version are correctly printed:
+
+```
+$ docker run -it amazon/aws-for-fluent-bit:latest
+AWS for Fluent Bit Container Image Version 2.0.0
+Fluent Bit v1.3.3
+```
+
+### Release
+
+With those steps complete, the images can now be made public. Once the images and SSM Parameters are all public, publish the GitHub release.

entrypoint.sh

@@ -0,0 +1,6 @@
+tput setaf 5;
+tput bold;
+echo -n "AWS for Fluent Bit Container Image Version "
+cat /AWS_FOR_FLUENT_BIT_VERSION
+tput sgr0;
+/fluent-bit/bin/fluent-bit -e /fluent-bit/firehose.so -e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so -c /fluent-bit/etc/fluent-bit.conf