Merge pull request #416 from codenrhoden/enhancement/gce_iam_support

Implement GCE application default credentials

.docs/user-guide/config.md

@@ -697,6 +697,7 @@ VirtualBox|Yes
 EBS|Yes
 EFS|No
 RBD|No
+GCEPD|Yes
 
 #### Ignore Used Count
 By default accounting takes place during operations that are performed

.docs/user-guide/storage-providers.md

@@ -708,7 +708,7 @@ libstorage:
   attached to another node. Mounting and writing to such a volume could lead to
   data corruption.
 
-## GCEPD
+## GCE Persistent Disk
 
 The Google Compute Engine Persistent Disk (GCEPD) driver registers a driver
 named `gcepd` with the `libStorage` driver manager and is used to connect and
@@ -718,11 +718,17 @@ mount Google Compute Engine (GCE) persistent disks with GCE machine instances.
 ### Requirements
 
 * GCE account
-* Service account credentials in JSON for GCE project. If not using the Compute
-  Engine default service account, create a new service account with the Service
-  Account Actor role, and create/download a new private key in JSON format. see
+* The libStorage server must be running on a GCE instance created with a Service
+  Account with appropriate permissions, or a Service Account credentials file
+  in JSON format must be supplied. If not using the Compute Engine default
+  Service Account with the Cloud Platform/"all cloud APIs" scope, create a new
+  Service Account via the [IAM Portal](https://console.cloud.google.com/iam-admin/serviceaccounts).
+  This Service Account requires the `Compute Engine/Instance Admin`,
+  `Compute Engine/Storage Admin`, and `Project/Service Account Actor` roles.
+  Then create/download a new private key in JSON format. see
   [creating a service account](https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount)
-  for details.
+  for details. The libStorage service must be restarted in order for permissions
+  changes on a service account to take effect.
 
 ### Configuration
 
@@ -739,36 +745,45 @@ gcepd:
 
 #### Configuration Notes
 
-* The `keyfile` parameter is required. It specifies a path on disk to a file
-  containing the JSON-encoded service account credentials. This file can be
-  downloaded from the GCE web portal.
+* The `keyfile` parameter is optional. It specifies a path on disk to a file
+  containing the JSON-encoded Service Account credentials. This file can be
+  downloaded from the GCE web portal. If `keyfile` is specified, the GCE
+  instance's service account is not considered, and is not necessary. If
+  `keyfile` is *not* specified, the application will try to lookup
+  [application default credentials](https://developers.google.com/identity/protocols/application-default-credentials).
+  This has the effect of looking for credentials in the priority described
+  [here](https://godoc.org/golang.org/x/oauth2/google#FindDefaultCredentials).
 * The `zone` parameter is optional, and configures the driver to *only* allow
   access to the given zone. Creating and listing disks from other zones will be
   denied. If a zone is not specified, the zone from the client Instance ID will
   be used when creating new disks.
-* The `defaultDiskType` parameter is optional, and specified what type of disk
+* The `defaultDiskType` parameter is optional and specifies what type of disk
   to create, either `pd-standard` or `pd-ssd`. When not specified, the default
   is `pd-ssd`.
 * The `tag` parameter is optional, and causes the driver to create or return
-  disks that have a matching tag. The tag is implemented by serializing a JSON
-  structure in to the `Description` field of a GCE disk. Use of this parameter
-  is encouraged, as the driver will only return volumes that have been created
-  by the driver, which is most useful to eliminate listing the boot disks of
-  every GCE disk in your project/zone.
+  disks that have a matching tag. The tag is implemented by using the GCE
+  label functionality available in the beta API. The value of the `tag`
+  parameter is used as the value for a label with the key `libstoragetag`.
+  Use of this parameter is encouraged, as the driver will only return volumes
+  that have been created by the driver, which is most useful to eliminate
+  listing the boot disks of every GCE disk in your project/zone. If you wsih to
+  "expose" previously created disks to the `GCEPD` driver, you can edit the
+  labels on the existing disk to have a key of `libstoragetag` and a value
+  matching that given in `tag`.
 
 ### Runtime behavior
 
 * The GCEPD driver enforces the GCE requirements for disk sizing and naming.
   Disks must be created with a minimum size of 10GB. Disk names must adhere to
-  the regular expression of [a-z]([-a-z0-9]*[a-z0-9])?, which means the first
+  the regular expression of `[a-z]([-a-z0-9]*[a-z0-9])?`, which means the first
   character must be a lowercase letter, and all following characters must be a
   dash, lowercase letter, or digit, except the last character, which cannot be a
   dash.
 * If the `zone` parameter is not specified in the driver configuration, and a
   request is received to list all volumes that does not specify a zone in the
   InstanceID header, volumes from all zones will be returned.
 * By default, all disks will be created with type `pd-ssd`, which creates an SSD
-  based disk. If you wish to created disks that are not SSD-based, change the
+  based disk. If you wish to create disks that are not SSD-based, change the
   default via the driver config, or the type can be changed at creation time by
   using the `Type` field of the create request.
 
@@ -781,7 +796,11 @@ driver name.
 ### Troubleshooting
 
 * Make sure that the JSON credentials file as specified in the `keyfile`
- configuration parameter is present and accessible.
+  configuration parameter is present and accessible, or that you are running in
+  a GCE instance created with a Service Account attached. Whether using a
+  `keyfile` or the Service Account associated with the GCE instance, the Service
+  Account must have the appropriate permissions as described in
+  `Configuration Notes`
 
 ### Examples
 
@@ -795,7 +814,7 @@ libstorage:
         driver: gcepd
         gcepd:
           keyfile: /etc/gcekey.json
-	  tag: rexray
+          tag: rexray
 ```
 
 ### Caveats
@@ -811,3 +830,9 @@ libstorage:
   including the root persistent disk. See
   [GCE Disks](https://cloud.google.com/compute/docs/disks/) docs for more
   details.
+* If running libStorage server in a mode where volume mounts will not be
+  performed on the same host where libStorage server is running, it should be
+  possible to use a Service Account without the `Service Account Actor` role, but
+  this has not been tested. Note that if persistent disk mounts are to be
+  performed on *any* GCE instances that have a Service Account associated with
+  the, the `Service Account Actor` role is required.

drivers/storage/gcepd/gcepd.go

@@ -30,9 +30,10 @@ const (
 )
 
 func init() {
-	r := gofigCore.NewRegistration("GCE")
+	r := gofigCore.NewRegistration("GCEPD")
 	r.Key(gofig.String, "", "",
-		"Required: JSON keyfile for service account", "gcepd.keyfile")
+		"If defined, location of JSON keyfile for service account",
+		"gcepd.keyfile")
 	r.Key(gofig.String, "", "",
 		"If defined, limit GCE access to given zone", "gcepd.zone")
 	r.Key(gofig.String, "", DefaultDiskType, "Default GCE disk type",

drivers/storage/gcepd/storage/gce_storage.go

@@ -8,8 +8,7 @@ import (
 	"fmt"
 	"hash"
 	"io/ioutil"
-	"os"
-	"path/filepath"
+	"net/http"
 	"regexp"
 	"strings"
 	"sync"
@@ -19,13 +18,15 @@ import (
 
 	gofig "github.com/akutz/gofig/types"
 	goof "github.com/akutz/goof"
+	"github.com/akutz/gotil"
 
 	"github.com/codedellemc/libstorage/api/context"
 	"github.com/codedellemc/libstorage/api/registry"
 	"github.com/codedellemc/libstorage/api/types"
 	"github.com/codedellemc/libstorage/drivers/storage/gcepd"
 	"github.com/codedellemc/libstorage/drivers/storage/gcepd/utils"
 
+	"golang.org/x/oauth2"
 	"golang.org/x/oauth2/google"
 	compute "google.golang.org/api/compute/v0.beta"
 	"google.golang.org/api/googleapi"
@@ -51,6 +52,8 @@ type driver struct {
 	zone            string
 	defaultDiskType string
 	tag             string
+	tokenSource     oauth2.TokenSource
+	svcAccount      string
 }
 
 func init() {
@@ -68,29 +71,41 @@ func (d *driver) Name() string {
 // Init initializes the driver.
 func (d *driver) Init(context types.Context, config gofig.Config) error {
 	d.config = config
+
 	d.keyFile = d.config.GetString("gcepd.keyfile")
-	if d.keyFile == "" {
-		return goof.New("GCE service account keyfile is required")
-	}
-	if !filepath.IsAbs(d.keyFile) {
-		cwd, err := os.Getwd()
+	if d.keyFile != "" {
+		if !gotil.FileExists(d.keyFile) {
+			return goof.Newf("keyfile at %s does not exist", d.keyFile)
+		}
+		pID, err := d.extractProjectID()
+		if err != nil || pID == nil || *pID == "" {
+			return goof.New("Unable to set project ID from keyfile")
+		}
+		d.projectID = pID
+		context.Info("Will authenticate using local JSON credentials")
+	} else {
+		// We are using application default credentials
+		defCreds, err := google.FindDefaultCredentials(
+			context, compute.ComputeScope)
 		if err != nil {
-			return goof.New("Unable to determine CWD")
+			return goof.WithError(
+				"Unable to get application default credentials",
+				err)
+		}
+		d.projectID = &defCreds.ProjectID
+		if *d.projectID == "" {
+			return goof.New(
+				"Unable to get project ID from default creds")
 		}
-		d.keyFile = filepath.Join(cwd, d.keyFile)
+		d.tokenSource = defCreds.TokenSource
+		context.Info("Will authenticate using app default credentials")
 	}
-	d.zone = d.config.GetString("gcepd.zone")
 
+	d.zone = d.config.GetString("gcepd.zone")
 	if d.zone != "" {
 		context.Infof("All access is restricted to zone: %s", d.zone)
 	}
 
-	pID, err := d.extractProjectID()
-	if err != nil || pID == nil || *pID == "" {
-		return goof.New("Unable to set project ID from keyfile")
-	}
-	d.projectID = pID
-
 	d.defaultDiskType = config.GetString("gcepd.defaultDiskType")
 
 	switch d.defaultDiskType {
@@ -129,13 +144,17 @@ func (d *driver) Login(ctx types.Context) (interface{}, error) {
 	defer sessionsL.Unlock()
 
 	var (
-		ckey string
-		hkey = md5.New()
+		ckey   string
+		hkey   = md5.New()
+		client *http.Client
 	)
 
 	// Unique connections to google APIs are based on project ID
-	// Project ID is embedded in the service account key JSON
+	// optionally there may be an additional service account
 	writeHkey(hkey, d.projectID)
+	if d.svcAccount != "" {
+		writeHkey(hkey, &d.svcAccount)
+	}
 	ckey = fmt.Sprintf("%x", hkey.Sum(nil))
 
 	// if the session is cached then return it
@@ -150,34 +169,48 @@ func (d *driver) Login(ctx types.Context) (interface{}, error) {
 		"projectID": *d.projectID,
 	}
 
-	serviceAccountJSON, err := d.getKeyFileJSON()
-	if err != nil {
-		ctx.WithFields(fields).Errorf(
-			"Could not read service account credentials file: %s",
-			err)
-		return nil, err
-	}
+	if d.keyFile != "" {
+		serviceAccountJSON, err := d.getKeyFileJSON()
+		if err != nil {
+			ctx.WithFields(fields).Errorf(
+				"Could not read service account credentials file: %s",
+				err)
+			return nil, err
+		}
 
-	config, err := google.JWTConfigFromJSON(
-		serviceAccountJSON,
-		compute.ComputeScope,
-	)
-	if err != nil {
-		ctx.WithFields(fields).Errorf(
-			"Could not create JWT Config From JSON: %s", err)
-		return nil, err
+		config, err := google.JWTConfigFromJSON(
+			serviceAccountJSON,
+			compute.ComputeScope,
+		)
+		if err != nil {
+			ctx.WithFields(fields).Errorf(
+				"Could not create JWT Config From JSON: %s", err)
+			return nil, err
+		}
+		d.svcAccount = config.Email
+		writeHkey(hkey, &config.Email)
+		ckey = fmt.Sprintf("%x", hkey.Sum(nil))
+		client = config.Client(ctx)
+	} else {
+		// Using application default credentials
+		if d.tokenSource == nil {
+			return nil, goof.New("Token Source is nil")
+		}
+
+		client = oauth2.NewClient(ctx, d.tokenSource)
 	}
 
-	client, err := compute.New(config.Client(ctx))
+	svc, err := compute.New(client)
 	if err != nil {
 		ctx.WithFields(fields).Errorf(
 			"Could not create GCE service connection: %s", err)
 		return nil, err
+
 	}
 
-	sessions[ckey] = client
+	sessions[ckey] = svc
 	ctx.Info("GCE service connection created and cached")
-	return client, nil
+	return svc, nil
 }
 
 // NextDeviceInfo returns the information about the driver's next available

drivers/storage/gcepd/tests/README.md

@@ -21,12 +21,15 @@ instance. You will also need to copy the JSON file with your service account
 credentials.
 
 Using an SSH session to connect to the GCE instance, please export the required
-GCE credentials used by the GCE storage driver:
+GCE service account credentials used by the GCE storage driver:
 
 ```bash
-export GCE_KEYFILE=/etc/gcekey.json
+export GCEPD_KEYFILE=/etc/gcekey.json
 ```
 
+If `GCEPD_KEYFILE` is not exported, the default location the test binary looks for
+a credentials file is at `/tmp/gce_key.json`.
+
 The tests may now be executed with the following command:
 
 ```bash