This package contains the EC2 instance configuration and scripts necessary to enable AWS EC2 Instance Connect.
The AuthorizedKeysCommand is split into three parts:
- eic_run_authorized_keys is the main entry point and wraps the rest in a 5 second timeout
- eic_curl_authorized_keys, which is the entry point for sshd on an ssh call
- eic_parse_authorized_keys, which is the main authorized keys command logic
This split is intentional - as parse takes all necessary pieces as command inputs is can be unit tested independently. curl, however, obviously needs to curl EC2 Instance Metadata Service and so cannot be tested without mocking the actual service.
The curl script verifies we are actually running on an EC2 instance and cURLs relevant information from EC2 Instance Metadata Service and send it to parse.
Note that it must make several curl commands to proceed. If it cannot do so it fast-fails to prevent blocking the ssh daemon.
The command also queries several OCSP staples from EC2 Instance Metadata Service. These OCSP staples are provided from the AWS EC2 Instance Connect Service to avoid needing to query each CRL or OCSP authority at ssh calltime as that would have major performance implications. The staples are passed to and used by parse_authorized_keys to check certificate validity without the need for extra external calls.
In addition to the fields required to complete all the below process, a key fingerprint may be provided. If a key fingerprint is specified then only ssh keys found matching that fingerprint will be returned.
The core idea behind AWS EC2 Instance Connect is that all ssh keys vended have been trusted by AWS. This can be verified by checking each key's signature (more on that later) and vetting the signing certificate used.
The signing certificate goes through a deep verification flow that checks:
- The CN matches the service
- The certificate trust chain can be verified up through the Amazon Trust Services Certificate Authority
- The entire certificate chain is valid - ie, none of them have been revoked
The first and second are done via standard openssl checks. The third, however, does not query the relevant CRLs or OCSP authorities at runtime to avoid adding a network call to sshd. Instead, OCSP staples are expected to be provided by the invoker (i.e., eic_curl_authorized_keys). As OCSP staples are cryptographically signed and can be verified against a trusted authority these are considered a sufficient validity check.
The keys are expected to be presented to the script in the format
# Key Metadata # Key Metadata # Key Metadata [ssh key] signature
Currently, the expected metadata is, in order,
- Expiration timestamp
- Instance ID
- IAM Caller
- Request ID
Once this data has been loaded, the following checks are run:
- Has the expiration timestamp passed?
- Is the instance ID correct?
- Does the signature match the provided data?
- If a specific key fingerprint was provided to search for, does this key match that fingerprint?
The signature is specifically expected to be for the entire key blob - all metadata entries plus the ssh key. It should have been generated by the AWS EC2 Instance Connect service's private key, which is verified using the vetted signing certificate.
If all of these checks pass then the key will be presented to the ssh daemon. Otherwise, it will be ignored and the next key will be processed.
Any time a key is provided to the ssh daemon it will be logged to the system authpriv log for auditing purposes.
As parse_authorized_keys requires a valid certificate, CA, and OCSP staples, unit testing is a somewhat involved process.
This has been automated to a convenient entry point: bin/unit_test_suite.sh. This will
- Invoke
bin/unit-test/setup_certificates.shto generate a test CA and trust chain - Invoke
bin/unit-test/generate_ocsp.shto generate OCSP staples for the certificates - Iterate over unit-test/input/direct and unit-test/input/unsigned and test all entries via
bin/unit-test/test_authorized_keys.sh, expecting the matching contents of unit-test/expected-output
unit-test/input/direct's contents are passed directly as-is as they are not expected to contain valid signatures.
unit-test/input/unsigned's contents, however, are expected to get far enough in the process to (potentially) need a valid signature. As such, signatures are generated on-the-fly using the pre-generated certificate's private key.
The structure of unit-test/input/unsigned, rather than files, is directories to test. The directory's contents should be in a numeric order that the test script will iterate. Each file should have one test key blob to sign. The actual test input will be the result of generating signatures for each file and constructing an imitation of the service's key bundle.
Integration testing requires running these scriptlets on an actual EC2 instance. This has been scripted for your convenience.
To run integration tests against a given platform (eg, RHEL or Ubuntu):
- Build a test package (for example,
make deb) - Select an AMI of the platform you desire (for example, the latest Ubuntu AMI)
- Configure your VPC for testing
- Configure an EC2 keypair. Make sure to save the private key!
- Configure your security group to allow SSH from your test-run machine
- Set your subnet to auto-assign public IPs. Testing currently requires SSHing via public IP.
- Determine the appropriate EC2 username for your platform (see below)
- Determine the "config name" for your platform (see below)
- Determine instance types you would like to test (or all supported in your subnet's zone)
- The actual tests are invoked by the following command:
./bin/integration_test_suite.sh -r [region] -a [ami id] -u [ec2 username] -k [ec2 keypair name] -s [subnet id] -g [security group id] -l [config name] -p [/path/to/private/key] -i [/path/to/test/package] [-t [instance,type,list]] [-d [/output/directory]]
This will run all tests under the integration-test/test directory against all requested instance types (or all types in the zone if not specified). Any error output will be written in files pathed /path/instance-type/test-name in the output directory. An output directory will be autogenerated if none is specified.
EC2 usernames and "config names" for each supported platform is as follows:
| Platform | EC2 Username | "Config" Name |
|---|---|---|
| Amazon Linux | ec2-user | amazon-linux |
| Ubuntu | ubuntu | ubuntu |
| RHEL | ec2-user | rhel |
Again, please note that if you do not specify instance types then all supported by the subnet's zone will be run. This can take a very long time. It is therefore recommended you specify at least one Nitro and at least one non-Nitro instance type, such as t2.micro and m5.large.
If desired, this scripting can be added to an EC2 instance for further testing. A convenience pair of scripts - bin/make_rpm.sh and bin/make_deb.sh - have been provided to quickly build test packages.
Each may be invoked via make rpm and make deb respectively.
Debian packaging in particular requires you have a GPG key configured on your system.
Be sure to update VERSION! If you use the same VERSION as the latest public release then test instances will not see it as an update!
You can build .rpm packages using Docker via make docker-build-rpm and .deb packages via make docker-build-deb. You can run both via make docker-build. The built packages will be in the out directory. This currently does not support the Amazon-proprietary build process.
Debian packaging tools are intended for testing purposes only. Please follow standard Debian process to submit patches.
"generic.rpm" is used internally to differentiate the specfile for Fedora/RHEL/CentOS from Amazon Linux. There is a separate Amazon-proprietary rpmspec and build process optimized for Amazon Linux.
This package is intended as a simple reference implementation of the instance-side pieces of the EC2 Instance Connect feature, though as time has gone on it has become much more complex than originally intended. Amazon is considering reimplementation in another language for the future but for the time being we will continue to iterate on the shell implementation.
This library is licensed under the Apache 2.0 License.
