Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using aws-sdk-s3 #855

Merged
merged 2 commits into from
Oct 11, 2018

Conversation

JackDanger
Copy link
Contributor

The full aws-sdk gem includes a massive number of dependency gems now that AWS has broken the aws-sdk into many pieces. Trimming this dependency allows applications that depend on omnibus to avoid downloading dozens or hundreds of small packages when they run 'bundle install'

@JackDanger JackDanger requested a review from a team as a code owner September 26, 2018 01:28
@JackDanger JackDanger force-pushed the jackdanger/using-aws-sdk-s3 branch from 3fae903 to ac069a1 Compare September 26, 2018 01:32
@lamont-granquist
Copy link
Contributor

someone should probably flame test this, since I think this is more or less equivalent to an upgrade to aws-sdk-v3(?), and IDK what the rspec testing looks like.

Copy link
Contributor

@tas50 tas50 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the v3 sdk upgrade we’ve been meaning to do. Thanks for making this happen. This should greatly simplify our deps

@mallyvai
Copy link

@lamont-granquist What does flame test mean here?

@lamont-granquist
Copy link
Contributor

someone internal to manually test + double check

@JackDanger
Copy link
Contributor Author

I figured y'all are busy so I ran some tests on this myself. It seems to have identical functionality to what it had prior to this change:

bin/omnibus cache list
                    [CLI] I | 2018-10-01T23:05:14-07:00 | Using config from 'omnibus.rb'
         [Command::Cache] I | 2018-10-01T23:05:14-07:00 | Using config from 'omnibus.rb'
Cached files (by S3 key):
  * backup1.txt
  * backup2.txt
  * backup.tgz

(These happen to be the 3 objects in the given S3 bucket I used as an example)

@JackDanger
Copy link
Contributor Author

@lamont-granquist Pinging on this – I manually tested this myself because I assume nobody's going to prioritize this internally (y'all have real work to do but this week this is my real work :D )

@lamont-granquist
Copy link
Contributor

looks busted on chef/chef

https://github.com/chef/chef/blob/master/omnibus/Gemfile#L3

changing that line to:

gem "omnibus", git: "https://github.com/JackDanger/omnibus", branch: "jackdanger/using-aws-sdk-s3"

and then doing a "bundle update && bundle exec omnibus build chef" results in:

% bundle exec omnibus build chef
                    [CLI] I | 2018-10-04T22:30:54-07:00 | Using config from 'omnibus.rb'
         [Software: chef] W | 2018-10-04T22:30:55-07:00 | Version local_source for software chef was not parseable. Comparison methods such as #satisfies? will not be available for this version.
         [Software: chef] W | 2018-10-04T22:30:55-07:00 | Version local_source for software chef was not parseable. Comparison methods such as #satisfies? will not be available for this version.
 [Software: config_guess] W | 2018-10-04T22:30:55-07:00 | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
Building chef 14.6.13+20181005053054...
 [Software: config_guess] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for config_guess
 [Software: config_guess] W | 2018-10-04T22:30:55-07:00 | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
 [Software: config_guess] W | 2018-10-04T22:30:55-07:00 | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
  [Software: preparation] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for preparation
[NullFetcher: preparation] I | 2018-10-04T22:30:55-07:00 | Fetching `preparation' (nothing to fetch)
         [Software: zlib] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for zlib
[Software: pkg-config-lite] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for pkg-config-lite
      [Software: libtool] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for libtool
       [Software: xproto] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for xproto
  [Software: util-macros] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for util-macros
   [Software: makedepend] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for makedepend
      [Software: openssl] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for openssl
      [Software: cacerts] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for cacerts
       [Software: libffi] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for libffi
      [Software: libyaml] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for libyaml
         [Software: ruby] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for ruby
     [Software: rubygems] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for rubygems
  [NullFetcher: rubygems] I | 2018-10-04T22:30:55-07:00 | Fetching `rubygems' (nothing to fetch)
      [Software: bundler] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for bundler
   [NullFetcher: bundler] I | 2018-10-04T22:30:55-07:00 | Fetching `bundler' (nothing to fetch)
         [Software: ohai] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for ohai
         [Software: ohai] W | 2018-10-04T22:30:55-07:00 | Version master for software ohai was not parseable. Comparison methods such as #satisfies? will not be available for this version.
         [Software: ohai] W | 2018-10-04T22:30:55-07:00 | Version master for software ohai was not parseable. Comparison methods such as #satisfies? will not be available for this version.
   [Software: appbundler] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for appbundler
   [Software: appbundler] W | 2018-10-04T22:30:55-07:00 | Version master for software appbundler was not parseable. Comparison methods such as #satisfies? will not be available for this version.
   [Software: appbundler] W | 2018-10-04T22:30:55-07:00 | Version master for software appbundler was not parseable. Comparison methods such as #satisfies? will not be available for this version.
      [Software: liblzma] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for liblzma
      [Software: libxml2] I | 2018-10-04T22:30:55-07:00 | Resolving manifest entry for libxml2
#<Thread:0x00007fb69a2a09d8@/Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/thread_pool.rb:57 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
	36: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/thread_pool.rb:61:in `block (2 levels) in initialize'
	35: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/thread_pool.rb:61:in `catch'
	34: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/thread_pool.rb:62:in `block (3 levels) in initialize'
	33: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/thread_pool.rb:62:in `loop'
	32: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/thread_pool.rb:64:in `block (4 levels) in initialize'
	31: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/project.rb:1066:in `block (3 levels) in download'
	30: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/software.rb:888:in `fetch'
	29: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/fetchers/net_fetcher.rb:83:in `fetch'
	28: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/logger.rb:59:in `info'
	27: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/logger.rb:105:in `add'
	26: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/fetchers/net_fetcher.rb:83:in `block in fetch'
	25: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/fetchers/net_fetcher.rb:147:in `download_url'
	24: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/s3_cache.rb:136:in `url_for'
	23: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/s3_helpers.rb:61:in `client'
	22: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/s3_helpers.rb:61:in `new'
	21: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-s3-1.21.0/lib/aws-sdk-s3/resource.rb:14:in `initialize'
	20: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/base.rb:99:in `new'
	19: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-s3-1.21.0/lib/aws-sdk-s3/client.rb:233:in `initialize'
	18: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/base.rb:19:in `initialize'
	17: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/base.rb:62:in `build_config'
	16: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:150:in `build!'
	15: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:177:in `apply_defaults'
	14: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:189:in `resolve'
	13: from /Users/lamont/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/set.rb:338:in `each'
	12: from /Users/lamont/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/set.rb:338:in `each_key'
	11: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:189:in `block in resolve'
	10: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:200:in `value_at'
	 9: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:204:in `resolve_defaults'
	 8: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:57:in `each'
	 7: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:57:in `each'
	 6: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:205:in `block in resolve_defaults'
	 5: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:70:in `call'
	 4: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-s3-1.21.0/lib/aws-sdk-s3/plugins/s3_signer.rb:13:in `block in <class:S3Signer>'
	 3: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-s3-1.21.0/lib/aws-sdk-s3/plugins/s3_signer.rb:189:in `build_v4_signer'
	 2: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-s3-1.21.0/lib/aws-sdk-s3/plugins/s3_signer.rb:189:in `new'
	 1: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sigv4-1.0.3/lib/aws-sigv4/signer.rb:121:in `initialize'
/Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sigv4-1.0.3/lib/aws-sigv4/signer.rb:517:in `extract_credentials_provider': missing credentials, provide credentials with one of the following options: (Aws::Sigv4::Errors::MissingCredentialsError)
  - :access_key_id and :secret_access_key
  - :credentials
  - :credentials_provider
Traceback (most recent call last):
	36: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/thread_pool.rb:61:in `block (2 levels) in initialize'
	35: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/thread_pool.rb:61:in `catch'
	34: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/thread_pool.rb:62:in `block (3 levels) in initialize'
	33: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/thread_pool.rb:62:in `loop'
	32: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/thread_pool.rb:64:in `block (4 levels) in initialize'
	31: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/project.rb:1066:in `block (3 levels) in download'
	30: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/software.rb:888:in `fetch'
	29: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/fetchers/net_fetcher.rb:83:in `fetch'
	28: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/logger.rb:59:in `info'
	27: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/logger.rb:105:in `add'
	26: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/fetchers/net_fetcher.rb:83:in `block in fetch'
	25: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/fetchers/net_fetcher.rb:147:in `download_url'
	24: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/s3_cache.rb:136:in `url_for'
	23: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/s3_helpers.rb:61:in `client'
	22: from /Users/lamont/.rvm/gems/ruby-2.5.1/bundler/gems/omnibus-ac069a1465aa/lib/omnibus/s3_helpers.rb:61:in `new'
	21: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-s3-1.21.0/lib/aws-sdk-s3/resource.rb:14:in `initialize'
	20: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/base.rb:99:in `new'
	19: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-s3-1.21.0/lib/aws-sdk-s3/client.rb:233:in `initialize'
	18: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/base.rb:19:in `initialize'
	17: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/base.rb:62:in `build_config'
	16: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:150:in `build!'
	15: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:177:in `apply_defaults'
	14: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:189:in `resolve'
	13: from /Users/lamont/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/set.rb:338:in `each'
	12: from /Users/lamont/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/set.rb:338:in `each_key'
	11: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:189:in `block in resolve'
	10: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:200:in `value_at'
	 9: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:204:in `resolve_defaults'
	 8: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:57:in `each'
	 7: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:57:in `each'
	 6: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:205:in `block in resolve_defaults'
	 5: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-core-3.30.0/lib/seahorse/client/configuration.rb:70:in `call'
	 4: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-s3-1.21.0/lib/aws-sdk-s3/plugins/s3_signer.rb:13:in `block in <class:S3Signer>'
	 3: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-s3-1.21.0/lib/aws-sdk-s3/plugins/s3_signer.rb:189:in `build_v4_signer'
	 2: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sdk-s3-1.21.0/lib/aws-sdk-s3/plugins/s3_signer.rb:189:in `new'
	 1: from /Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sigv4-1.0.3/lib/aws-sigv4/signer.rb:121:in `initialize'
/Users/lamont/.rvm/gems/ruby-2.5.1/gems/aws-sigv4-1.0.3/lib/aws-sigv4/signer.rb:517:in `extract_credentials_provider': missing credentials, provide credentials with one of the following options: (Aws::Sigv4::Errors::MissingCredentialsError)
  - :access_key_id and :secret_access_key
  - :credentials
  - :credentials_provider

@JackDanger JackDanger force-pushed the jackdanger/using-aws-sdk-s3 branch from 5b2d338 to 519b214 Compare October 5, 2018 05:56
@JackDanger
Copy link
Contributor Author

@lamont-granquist Thanks for sharing a way I can better test this. It looks like the aws-sdk-v3 throws an error when the AWS credentials are blank and no fallback has been specified. I've added 519b214 which fixes this.

The full aws-sdk gem includes a massive number of dependency gems now that AWS has broken the aws-sdk into many pieces. Trimming this dependency allows applications that depend on omnibus to avoid downloading dozens or hundreds of small packages when they run 'bundle install'

Signed-off-by: Jack Danger <[email protected]>
Signed-off-by: Jack Danger <[email protected]>
@JackDanger JackDanger force-pushed the jackdanger/using-aws-sdk-s3 branch from 519b214 to 185f21d Compare October 11, 2018 10:06
@JackDanger
Copy link
Contributor Author

@lamont-granquist I just rebased this and would love a re-review after fixing the nil credentials problem you found.

@lamont-granquist
Copy link
Contributor

yeah that fixed the S3Fetcher.

@lamont-granquist lamont-granquist merged commit 902a205 into chef:master Oct 11, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants