New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Design for generic builder interface #88

Closed

sam-goodwin wants to merge 2 commits into aws:develop from sam-goodwin:develop

sam-goodwin commented Feb 22, 2019 •

edited

Loading

Proposal to extend aws-lambda-builders to become a generic library for building all AWS projects.

Still more details to be worked out, esp. around the UX experience and implementation details. Looking forward to discussion from those with more context.

Supports: aws/aws-cdk#1435

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Sam Goodwin added 2 commits

February 21, 2019 19:55


          Add design for generic builder interface

6cacc82


          Re-word some stuff

74e0f6a

eladb reviewed

View reviewed changes

eladb left a comment

I love the extension to "ECS" but at this point it feels like a very high level requirement. What does that technically mean to build for ECS? Is it just building docker images? Unless we have a very concrete requirement, I would recommend to start with what we know (i.e. merge SAM CLI logic into Lambda Builders and provide a Docker-interface) and only at the second phase extend to ECS once requirements are clear. Alternatively, dive deep into the requirements.

designs/Generic_Builder_Interface.md

+              --------------------
+              Tools like the [Cloud Development Kit (CDK) CLI](https://github.com/awslabs/aws-cdk) and the [Elastic Contianer Service (ECS) CLI](https://github.com/aws/amazon-ecs-cli) require a single library to encapsulate the logic of assembling a project's source code into an artifact deployable to AWS. Since the CDK covers the entire surface area of AWS, not just serverless use-cases like SAM, the scope of `aws-lambda-builders` must be extended to include non-lambda target environments such as Docker images for ECR. This also asks questions of branding, perhaps requiring a new project named `aws-builders`.
+              The logic for mapping a project's structure to a workflow (e.g. looking for `pom.xml` or `build.gradle` to determine the java project type) is currently bundled into SAM CLI instead of `aws-lambda-builders`, burdening and limiting consumers.

eladb Feb 24, 2019

instead of "burdening and limiting" I would say "which makes it hard to reuse" or something in those lines

designs/Generic_Builder_Interface.md

+              ---------------------
+              Extract the automatic project inference from `sam cli` into this library and extend the RPC interface so that projects other than Lambda functions may also be built and packaged.
+              Rename or create a new package, `aws-builders`, to better represent the (now) wider-surface area.

eladb Feb 24, 2019

"aws-build"?

designs/Generic_Builder_Interface.md


		Rename or create a new package, `aws-builders`, to better represent the (now) wider-surface area.

		Publish Docker images bundling the CLI for various target language/platforms so that consumers need not depend on `pip install` to consume the CLI. Instead, the tool can be invoked with [`docker run`](https://docs.docker.com/engine/reference/commandline/run/), requiring only a single dependency on the Docker toolchain.

eladb Feb 24, 2019

and inline with how SAM CLI is actually using lambda-builders today to ensure that native code is compiled to the correct platform

designs/Generic_Builder_Interface.md

+              What is the problem?
+              --------------------
+              Tools like the [Cloud Development Kit (CDK) CLI](https://github.com/awslabs/aws-cdk) and the [Elastic Contianer Service (ECS) CLI](https://github.com/aws/amazon-ecs-cli) require a single library to encapsulate the logic of assembling a project's source code into an artifact deployable to AWS. Since the CDK covers the entire surface area of AWS, not just serverless use-cases like SAM, the scope of `aws-lambda-builders` must be extended to include non-lambda target environments such as Docker images for ECR. This also asks questions of branding, perhaps requiring a new project named `aws-builders`.

eladb Feb 24, 2019

I would change "must be extended" to "could be extended"

eladb commented Feb 24, 2019

Copy @clareliguori @SoManyHs @rix0rrr

rix0rrr reviewed

View reviewed changes

designs/Generic_Builder_Interface.md

+              For the CDK use-case, we wish to instantiate a language-specific function type and only have to point to the project's location:
+              ```ts
+              const myFunction = new lambda.JavaFunction(stack, 'MyJavaFunction', {

rix0rrr Feb 25, 2019

Couldn't we make use of the newly centralized build/language detection logic to get rid of the need of specifying Java here?

rix0rrr Feb 25, 2019

Also, strictly speaking, shouldn't this be called JvmFunction? :)

eladb Feb 25, 2019

There's a different APIs for defining the build project for different languages. Different way to specify dependencies/handlers/runtime versions. I like the strong typing.

rix0rrr reviewed

View reviewed changes

designs/Generic_Builder_Interface.md

+. Do we want to rename this project or create a new package encapsulating the generic functionality?
+. What to do about `pip install`? Is there a better way to expose the CLI for local installation or inclusion in a `Dockerfile`?
+. Which environments/configurations should we publish Docker containers for? E.g. one container for NPM, one for Java, one for Python, etc. etc.
+. Do we want to provide hooks for developers to customize a build when their use-case is not supported?

rix0rrr Feb 25, 2019

Lack of customization hooks seems like a nonstarter. What if a project build requires additional installed tool dependencies, or private package repositories, or ANYTHING, really. Let's give people the ability to build what they need from the start, and add incremental sugar later.

eladb Feb 25, 2019

customization hooks are usually built into the language specific build system. Maven has like a million hooks, npm has hooks for pre-install post-install, etc.

Contributor

sanathkr Feb 25, 2019

@rix0rrr #4 is tracking passing arbitrary options to build tools. We didn't enable this from the beginning because we wanted builds to work out-of-box with majority of cases without additional work.

rix0rrr reviewed

View reviewed changes

designs/Generic_Builder_Interface.md

+              -   \[x\] Send a Pull Request with this design document
+              -   \[ \] Extract project inference from SAM CLI to `aws-lambda-builders`
+              -   \[ \] Update RPC to support the higher-level use cases (project inference)
+              -   \[ \] Publish a Docker image for each language we support (javascript, java, python, go mod, go dep)

rix0rrr Feb 25, 2019

Build a pipeline for all images we vend!

sam-goodwin mentioned this pull request

Lambda "build" assets aws/aws-cdk#1435

Closed

Member

clareliguori commented Feb 25, 2019

Agree with @eladb, I would start without targeting ecs-cli. I'm not actually sure it makes sense for ecs-cli to leverage this library: the ecs-cli is intended to fit into a user's existing docker-compose workflow, so they use docker-compose directly to assemble everything. In which case, the user does something like

docker-compose build
docker-compose push
ecs-cli -f my-compose-file.yml service up

For CDK + ECS, there is already a Docker "asset" type for building and pushing the Docker image.

sam-goodwin mentioned this pull request

[WIP] build a Function's runtime with a Docker container aws/aws-cdk#1884

Closed

11 tasks

Author

sam-goodwin commented Feb 26, 2019

Preliminary work in the CDK can be seen here: aws/aws-cdk#1884

Would love your feedback @sanathkr

jfuss closed this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet