feature: adding openstack and static cloud support

This change sets up basic ironfan features for use with an openstack cloud, as well as adding a fake cloud provider called "static" that lets you specify (normally) cloud-discoverable parameters directly in the cluster definition. Squashed commit of the following: commit a80592a2adf35ca6db3821c2b7897b9e6dad603b Merge: b4149c9 1572b76 Author: Chris Howe <[email protected]> Date: Mon Mar 10 13:12:17 2014 -0500 Merge branch 'master' of github.com:howech/ironfan into 20140310-feature-openstack-support Conflicts: Gemfile Gemfile.lock ironfan.gemspec lib/ironfan/dsl/component.rb commit 1572b76 Merge: 1925e5e 5edcf9a Author: Chris Howe <[email protected]> Date: Tue Feb 25 10:58:47 2014 -0600 Merge pull request #4 from erikmack/master display floating IP's, also bump excon commit 1925e5e Merge: 5a18c58 be86033 Author: Chris Howe <[email protected]> Date: Wed Feb 19 15:06:55 2014 -0600 Merge pull request #3 from howech/static-cloud Static cloud commit be86033 Author: Chris Howe <[email protected]> Date: Wed Feb 19 14:58:54 2014 -0600 minor changes to help the world work better commit 5edcf9a Author: Erik Mackdanz <[email protected]> Date: Wed Feb 19 14:49:15 2014 -0600 Bump excon dep commit fe78a40 Author: Erik Mackdanz <[email protected]> Date: Wed Feb 19 12:56:44 2014 -0600 Display floating IP address in public IP fields commit 5a18c58 Merge: 79f5289 bb88ae4 Author: Chris Howe <[email protected]> Date: Wed Feb 19 12:28:19 2014 -0600 Merge pull request #2 from erikmack/master Factor out cloud-specific security group code commit 06c92dd Author: Chris Howe <[email protected]> Date: Fri Feb 7 11:56:40 2014 -0600 Added a non-cloud cloud provider called 'static' commit 79f5289 Author: Chris Howe <[email protected]> Date: Wed Jan 29 16:05:59 2014 -0600 Added elastic ip support for openstack. Shored up public ip address code. commit bb88ae4 Author: Erik Mackdanz <[email protected]> Date: Tue Jan 28 16:33:16 2014 -0600 Factor out cloud-specific security group code commit b5591dc Author: Chris Howe <[email protected]> Date: Mon Jan 27 13:00:54 2014 -0600 machine state tweaks for openstack commit 353862f Author: Chris Howe <[email protected]> Date: Fri Jan 17 15:39:20 2014 -0600 Fixed an unintended edit :Fixed an unintende commit dbd9f67 Author: Chris Howe <[email protected]> Date: Fri Jan 17 14:37:48 2014 -0600 changed wait for ssh behavior to resect ssh rules commit 0ed441a Author: Chris Howe <[email protected]> Date: Fri Jan 17 11:27:01 2014 -0600 changed cluster and facet groups to use server.security_group instead of cloud.security group commit c72a466 Author: Chris Howe <[email protected]> Date: Thu Jan 16 16:35:13 2014 -0600 Added a security_groups container to dsl/compute, and changed providers to respect both cloud defined and compute defined security groups. commit 610cf62 Merge: 6f99623 1b82f8f Author: Chris Howe <[email protected]> Date: Tue Jan 14 11:22:42 2014 -0800 Merge pull request #1 from erikmack/master Make the DNS search domain configurable at the cloud layer. commit 1b82f8f Author: Erik Mackdanz <[email protected]> Date: Tue Jan 14 12:46:56 2014 -0600 Make the DNS search domain configurable at the cloud layer. The NFS server will perform a reverse DNS query on any clients that try to connect. Only clients whose reverse DNS response matches this domain are allowed to connect. commit 6f99623 Author: Chris Howe <[email protected]> Date: Mon Jan 13 10:15:17 2014 -0600 fixed security groups for openstack commit cd8f930 Author: Chris Howe <[email protected]> Date: Sun Jan 12 10:01:34 2014 -0600 Added some better statuses. Changed the default to actually wait for ssh ports. commit bee2a0d Author: Chris Howe <[email protected]> Date: Fri Jan 10 04:51:34 2014 +0000 fixed a keypair bug commit 8788ccf Author: Chris Howe <[email protected]> Date: Thu Jan 9 19:34:01 2014 +0000 better support for public/private ip address commit e92811a Author: Chris Howe <[email protected]> Date: Thu Jan 9 19:32:57 2014 +0000 better support for public/private ip address commit f756cce Author: Chris Howe <[email protected]> Date: Wed Jan 8 15:45:44 2014 +0000 Added basic support for an openstack cloud provider. Change-Id: If34e82013bdb321d8114d6a507ef84df3567b9ed
infochimps-labs · Mar 14, 2014 · 44d2ba5 · 44d2ba5
1 parent 124342b
commit 44d2ba5
Show file tree

Hide file tree

Showing 27 changed files with 1,646 additions and 106 deletions.
diff --git a/Gemfile b/Gemfile
@@ -1,5 +1,6 @@
 source 'https://rubygems.org'
 
+
 gemspec
 
 group :development do

diff --git a/Gemfile.lock b/Gemfile.lock
@@ -0,0 +1,198 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    archive-tar-minitar (0.5.2)
+    builder (3.2.2)
+    bunny (0.7.9)
+    chef (10.26.0)
+      bunny (>= 0.6.0, < 0.8.0)
+      erubis
+      highline (>= 1.6.9)
+      json (>= 1.4.4, <= 1.7.7)
+      mixlib-authentication (>= 1.3.0)
+      mixlib-cli (>= 1.1.0)
+      mixlib-config (>= 1.1.2)
+      mixlib-log (>= 1.3.0)
+      mixlib-shellout
+      moneta (< 0.7.0)
+      net-ssh (~> 2.6)
+      net-ssh-multi (~> 1.1.0)
+      ohai (>= 0.6.0)
+      rest-client (>= 1.0.4, < 1.7.0)
+      treetop (~> 1.4.9)
+      uuidtools
+      yajl-ruby (~> 1.1)
+    chef-zero (1.5.1)
+      hashie (~> 2.0)
+      json
+      mixlib-log (~> 1.3)
+      moneta (< 0.7.0)
+      puma (~> 1.6)
+    coderay (1.0.9)
+    columnize (0.3.6)
+    configliere (0.4.18)
+      highline (>= 1.5.2)
+      multi_json (>= 1.1)
+    diff-lcs (1.2.5)
+    erubis (2.7.0)
+    excon (0.31.0)
+    ffi (1.9.0)
+    fog (1.19.0)
+      builder
+      excon (~> 0.31.0)
+      formatador (~> 0.2.0)
+      mime-types
+      multi_json (~> 1.0)
+      net-scp (~> 1.1)
+      net-ssh (>= 2.1.3)
+      nokogiri (~> 1.5)
+      ruby-hmac
+    formatador (0.2.4)
+    git (1.2.5)
+    gorillib (0.5.0)
+      configliere (>= 0.4.13)
+      json
+      multi_json (>= 1.1)
+    guard (1.8.1)
+      formatador (>= 0.2.4)
+      listen (>= 1.0.0)
+      lumberjack (>= 1.0.2)
+      pry (>= 0.9.10)
+      thor (>= 0.14.6)
+    guard-rspec (3.0.2)
+      guard (>= 1.8)
+      rspec (~> 2.13)
+    guard-yard (2.1.0)
+      guard (>= 1.1.0)
+      yard (>= 0.7.0)
+    hashie (2.0.5)
+    highline (1.6.19)
+    ipaddress (0.8.0)
+    jeweler (1.8.4)
+      bundler (~> 1.0)
+      git (>= 1.2.5)
+      rake
+      rdoc
+    json (1.5.4)
+    linecache19 (0.5.12)
+      ruby_core_source (>= 0.1.4)
+    listen (1.2.2)
+      rb-fsevent (>= 0.9.3)
+      rb-inotify (>= 0.9)
+      rb-kqueue (>= 0.2)
+    lumberjack (1.0.3)
+    method_source (0.8.1)
+    mime-types (2.0)
+    mini_portile (0.5.2)
+    mixlib-authentication (1.3.0)
+      mixlib-log
+    mixlib-cli (1.3.0)
+    mixlib-config (1.1.2)
+    mixlib-log (1.6.0)
+    mixlib-shellout (1.1.0)
+    moneta (0.6.0)
+    multi_json (1.8.2)
+    net-scp (1.1.2)
+      net-ssh (>= 2.6.5)
+    net-ssh (2.7.0)
+    net-ssh-gateway (1.2.0)
+      net-ssh (>= 2.6.5)
+    net-ssh-multi (1.1)
+      net-ssh (>= 2.1.4)
+      net-ssh-gateway (>= 0.99.0)
+    nokogiri (1.6.1)
+      mini_portile (~> 0.5.0)
+    ohai (6.16.0)
+      ipaddress
+      mixlib-cli
+      mixlib-config
+      mixlib-log
+      mixlib-shellout
+      systemu
+      yajl-ruby
+    oj (2.1.2)
+    polyglot (0.3.3)
+    pry (0.9.12.2)
+      coderay (~> 1.0.5)
+      method_source (~> 0.8)
+      slop (~> 3.4)
+    puma (1.6.3)
+      rack (~> 1.2)
+    rack (1.5.2)
+    rake (10.1.0)
+    rb-fsevent (0.9.3)
+    rb-inotify (0.9.0)
+      ffi (>= 0.5.0)
+    rb-kqueue (0.2.0)
+      ffi (>= 0.5.0)
+    rbvmomi (1.6.0)
+      builder
+      nokogiri (>= 1.4.1)
+      trollop
+    rdoc (4.0.1)
+      json (~> 1.4)
+    redcarpet (2.3.0)
+    rest-client (1.6.7)
+      mime-types (>= 1.16)
+    rspec (2.14.1)
+      rspec-core (~> 2.14.0)
+      rspec-expectations (~> 2.14.0)
+      rspec-mocks (~> 2.14.0)
+    rspec-core (2.14.7)
+    rspec-expectations (2.14.4)
+      diff-lcs (>= 1.1.3, < 2.0)
+    rspec-mocks (2.14.4)
+    ruby-debug-base19 (0.11.25)
+      columnize (>= 0.3.1)
+      linecache19 (>= 0.5.11)
+      ruby_core_source (>= 0.1.4)
+    ruby-debug19 (0.11.6)
+      columnize (>= 0.3.1)
+      linecache19 (>= 0.5.11)
+      ruby-debug-base19 (>= 0.11.19)
+    ruby-hmac (0.4.0)
+    ruby_core_source (0.1.5)
+      archive-tar-minitar (>= 0.5.2)
+    ruby_gntp (0.3.4)
+    simplecov (0.7.1)
+      multi_json (~> 1.0)
+      simplecov-html (~> 0.7.1)
+    simplecov-html (0.7.1)
+    slop (3.4.5)
+    systemu (2.5.2)
+    thor (0.18.1)
+    treetop (1.4.14)
+      polyglot
+      polyglot (>= 0.3.1)
+    trollop (2.0)
+    uuidtools (2.1.4)
+    yajl-ruby (1.1.0)
+    yard (0.8.6.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler (~> 1.0)
+  chef (~> 10.16)
+  chef-zero
+  diff-lcs (~> 1.2.5)
+  excon (~> 0.31.0)
+  fog (~> 1.19)
+  formatador (~> 0.2)
+  gorillib (~> 0.5.0)
+  guard (~> 1)
+  guard-rspec
+  guard-yard
+  jeweler (>= 1.6)
+  json (= 1.5.4)
+  oj (>= 1.2)
+  pry
+  rake
+  rbvmomi
+  redcarpet (>= 2.1)
+  rspec (~> 2.8)
+  ruby-debug19
+  ruby_gntp
+  simplecov (>= 0.5)
+  yard (>= 0.7)
diff --git a/ironfan.gemspec b/ironfan.gemspec
@@ -29,4 +29,5 @@ Gem::Specification.new do |gem|
   gem.add_dependency('json',       '1.5.4')
 
   gem.add_development_dependency('bundler', '~> 1.0')
+
 end
diff --git a/lib/chef/knife/bootstrap/ubuntu12.04-ironfan.erb b/lib/chef/knife/bootstrap/ubuntu12.04-ironfan.erb
@@ -14,6 +14,13 @@ set -e
 
 <%= (@config[:verbosity].to_i > 1 ? 'set -v' : '') %>
 
+echo "deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main" | sudo tee /etc/apt/sources.list.d/opscode.list
+
+# Make sure that opscode chef is on the apt repo list.
+sudo mkdir -p /etc/apt/trusted.gpg.d
+gpg --keyserver keys.gnupg.net --recv-keys 83EF826A
+gpg --export [email protected] | sudo tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null
+
 date > /etc/box_build_time
 
 echo -e "`date` \n\n**** \n**** apt update:\n****\n"

diff --git a/lib/chef/knife/cluster_launch.rb b/lib/chef/knife/cluster_launch.rb
@@ -55,6 +55,12 @@ class ClusterLaunch < Knife
         :boolean     => true,
         :default     => false
 
+      option :wait_ssh,
+        :long        => "--[no-]wait-ssh",
+        :description => "Wait for the target machine to open an ssh port",
+        :boolean     => true,
+        :default     => true
+
       def _run
         load_ironfan
         die(banner) if @name_args.empty?
@@ -110,10 +116,10 @@ def _run
       def perform_after_launch_tasks(computer)
         # Try SSH
         unless config[:dry_run]
-          Ironfan.step(computer.name, 'trying ssh', :white)
-          # FIXME: This is EC2-specific, abstract it
-          address = computer.machine.vpc_id.nil? ? computer.machine.public_hostname : computer.machine.public_ip_address
-          nil until tcp_test_ssh(address){ sleep @initial_sleep_delay ||= 10  }
+          if config[:wait_ssh]
+            Ironfan.step(computer.name, 'trying ssh', :white)
+            nil until wait_for_ssh(computer){ sleep @initial_sleep_delay ||= 10  }
+          end
         end
 
         # Run Bootstrap
@@ -123,31 +129,6 @@ def perform_after_launch_tasks(computer)
         end
       end
 
-      def tcp_test_ssh(target)
-        tcp_socket = TCPSocket.new(target, 22)
-        readable = IO.select([tcp_socket], nil, nil, 5)
-        if readable
-          Chef::Log.debug("sshd accepting connections on #{target}, banner is #{tcp_socket.gets}")
-          yield
-          true
-        else
-          false
-        end
-      rescue Errno::ETIMEDOUT
-	Chef::Log.debug("ssh to #{target} timed out")
-        false
-      rescue Errno::ECONNREFUSED
-	Chef::Log.debug("ssh connection to #{target} refused")
-        sleep 2
-        false
-      rescue Errno::EHOSTUNREACH
-	Chef::Log.debug("ssh host #{target} unreachable")
-        sleep 2
-        false
-      ensure
-        tcp_socket && tcp_socket.close
-      end
-
       def warn_or_die_on_bogus_servers(target)
         ui.info("")
         ui.info "Cluster has servers in a transitional or undefined state (shown as 'bogus'):"

diff --git a/lib/chef/knife/ironfan_knife_common.rb b/lib/chef/knife/ironfan_knife_common.rb
@@ -185,6 +185,41 @@ def run_bootstrap(computer)
       end
     end
 
+    def wait_for_ssh(computer)
+      ssh = Chef::Knife::Ssh.new
+      ssh.ui = ui
+      ssh.name_args = [ computer.name, "ls" ]
+      ssh.config[:ssh_user] = Chef::Config[:knife][:ssh_user] || config[:ssh_user]
+      ssh.config[:ssh_password] = config[:ssh_password]
+      ssh.config[:ssh_port] = Chef::Config[:knife][:ssh_port] || config[:ssh_port]
+      ssh.config[:ssh_gateway] = Chef::Config[:knife][:ssh_gateway] || config[:ssh_gateway]
+      ssh.config[:forward_agent] = Chef::Config[:knife][:forward_agent] || config[:forward_agent]
+      ssh.config[:identity_file] = Chef::Config[:knife][:identity_file] || config[:identity_file]
+      ssh.config[:manual] = true
+      ssh.config[:host_key_verify] = Chef::Config[:knife][:host_key_verify] || config[:host_key_verify]
+      ssh.config[:on_error] = :raise
+      session = ssh.session
+      return true
+    rescue Errno::ETIMEDOUT
+      Chef::Log.debug("ssh to #{computer.name} timed out")
+      return false
+    rescue Errno::ECONNREFUSED
+      Chef::Log.debug("ssh connection to #{computer.name} refused")
+      yield
+      return false
+    rescue Errno::EHOSTUNREACH
+      Chef::Log.debug("ssh host #{computer.name} unreachable")
+      yield
+      return false
+    rescue 
+      Chef::Log.debug("something else went wrong while wating for ssh host #{computer.name}")
+      raise
+      return false
+    else
+      session && session.close
+      session = nil
+    end
+
     #
     # Utilities
     #

diff --git a/lib/ironfan/dsl/cloud.rb b/lib/ironfan/dsl/cloud.rb
@@ -14,6 +14,8 @@ def self.receive(obj, &block)
             when :virtualbox then VirtualBox
             when :vsphere    then Vsphere
             when :rds        then Rds
+            when :openstack  then OpenStack
+            when :static     then Static
             else raise "Unsupported cloud #{obj[:name]}"
             end
         end

diff --git a/lib/ironfan/dsl/component.rb b/lib/ironfan/dsl/component.rb
@@ -110,14 +110,11 @@ def wire_to(compute, full_server_cluster_v, keys)
         discovery = {discovers: keys.reverse.inject(full_server_cluster_v){|hsh,key| {key => hsh}}}
         (compute.facet_role || compute.cluster_role).override_attributes(discovery)
 
-        # FIXME: This is Ec2-specific and probably doesn't belong here.
         client_group_v = client_group(compute)
         server_group_v = security_group(full_server_cluster_v)
 
-        if create_security_groups
-          group_edge(compute.cloud(:ec2), client_group_v, :authorized_by_group, server_group_v)
-          group_edge(compute.cloud(:ec2), client_group_v, :authorize_group,     server_group_v) if bidirectional
-        end
+        group_edge(compute, client_group_v, :authorized_by_group, server_group_v)
+        group_edge(compute, client_group_v, :authorize_group,     server_group_v) if bidirectional
 
         Chef::Log.debug("discovered #{announce_name} for #{cluster_name}: #{discovery}")
       end

diff --git a/lib/ironfan/dsl/compute.rb b/lib/ironfan/dsl/compute.rb
@@ -22,6 +22,7 @@ class Compute < Ironfan::Dsl
       collection :run_list_items, RunListItem,             :resolver => :merge_resolve, :key_method => :name
       collection :clouds,       Ironfan::Dsl::Cloud,       :resolver => :merge_resolve, :key_method => :name
       collection :volumes,      Ironfan::Dsl::Volume,      :resolver => :merge_resolve, :key_method => :name
+      collection :security_groups, Ironfan::Dsl::SecurityGroup, :resolver => :merge_resolve, :key_method => :name
 
       # Resolve these normally (overriding on each layer)
       magic      :environment,  Symbol,                    :default => :_default
Original file line number	Diff line number	Diff line change
Expand Up		@@ -29,4 +29,5 @@ Gem::Specification.new do \|gem\|
		gem.add_dependency('json', '1.5.4')

		gem.add_development_dependency('bundler', '~> 1.0')

		end