Initial Auditing Configuration and Usage of the kubernetes.io GCP Organization #213

hh · 2019-04-03T08:36:14Z

As an community security auditor
I want to retrieve the iam policies and permissions for the kubernetes.io GCP organisation
In order to  provide transparency into who can access and configure community maintained resources

Given an gcloud account within the k8s-infra-gcp-auditors google group
When ./audit.sh is run and a PR created with the resulting policy changes
Then iam configuration can be peered reviewed / audited

k8s-ci-robot · 2019-04-03T08:36:18Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

hh · 2019-04-03T08:54:06Z

Initial audit of a policy change: https://kubernetes.slack.com/messages/CCK68P2Q2/convo/C09QZ4DQB-1554150053.006600/

thockin

Great starting point

audit/README.md

thockin · 2019-04-03T15:26:09Z

audit/README.md

+  * Audit report
+
+How to automate:
+  * How do we audit for iam changes as they happen, rather than polling


We could (once this is working as we want and once we have our cluster up) run this as a CronJob daily or even hourly, posting any deltas to slack or something

thockin · 2019-04-03T16:43:26Z

audit/audit.sh

+do
+    gcloud organizations get-iam-policy $CNCF_GCP_ORG --format=$format \
+           > cncf-org-policy.$format
+    gcloud projects get-iam-policy kubernetes-public --format=$format \


Rather than "kubernetes-public" shouldn't we be doing a loop over all projects?

e.g.

gcloud projects list \ --filter "parent.id=758905017065" \ --format "value(name, projectNumber)" \ | while read NAME NUM; do \ gcloud projects get-iam-policy $NAME --format=$format > $NAME.policy.$format gcloud iam roles list --project=$NAME --format=$format > $NAME.roles.$format done

Also, I vote for JSON - much easier to visually verify

Currently doing json and yaml, I can remove yaml if you like.

audit/audit.sh

thockin · 2019-04-03T16:48:28Z

audit/audit.sh

+mkdir -p buckets
+for BUCKET in `gsutil ls -p kubernetes-public | awk -F/ '{print $3}'`
+do
+    gsutil ls -r gs://$BUCKET/ > buckets/$BUCKET.txt


Do we need to audit the contents of buckets, or just permissions? This will trigger a lot of updates -- e.g. billing is dumped at least daily.

Just permissions are fine, I think there are object/file/folder specific permissions that can be set.
We may need a way to capture these or set a remove the ability to set per file perms and only have bucket level.
Thoughts?

audit/audit.sh

thockin · 2019-05-10T05:24:32Z

Ping on this. As we start expanding scope, the fully formed version of this would make me feel better...

spiffxp · 2019-05-15T15:48:11Z

@thockin to take a final pass on this

hh · 2019-06-15T16:57:19Z

audit/audit-gcp.sh

+
+# gcloud organizations describe $CNCF_GCP_ORG 2>&1
+# ERROR: (gcloud.organizations.describe)
+# User [[email protected]] does not have permission to access organization []


@thockin Can we add 'gcloud.organizations.describe' to the k8s-infra-gcp-auditors?

That's not the permission name. I added 'Organization Viewer' which seems to have the right permissions. Try it and let me know?

did you try this again?

hh · 2019-06-15T18:31:45Z

audit/k8s-infra-dev-cluster-turnup.services.yaml

@@ -0,0 +1,358 @@
+---


Note that the dev-cluster-turnup has a many more services enabled than our primary one.

hh · 2019-06-16T01:15:12Z

audit/audit-gcp.sh

+    for service in `gcloud services list --filter state:ENABLED --format=json \
+                    | jq -r .[].config.name | sed s:.googleapis.com::`
+    do
+        case $service in


We do a bit of iterating over the services. How far do we want to dig into the configuration of each.

I am less concerned about service configs than IAM and access.

Understood. Dumping the entire config is a bit out of scope, but I added it the documentation/TODO completeness later.

spiffxp · 2019-06-26T15:53:00Z

/cc

thockin

Can we get a form of this committed soon, so we can use it as we turn more things on? Then iterate..

spiffxp · 2019-07-10T16:00:59Z

this has grown quite a lot, do we need to open up a new PR?

spiffxp · 2019-07-10T17:06:14Z

@thockin @hh WDYT about merging this as-is and iterating from there?

hh · 2019-07-10T17:23:49Z

Since adding a few new projects the case statement iterating over services has more unhandled services.

cat log.mkd | grep Unhandled | sort | uniq
# Unhandled Service admin #
# Unhandled Service bigtable #
# Unhandled Service bigtableadmin #
# Unhandled Service cloudapis #
# Unhandled Service cloudbuild #
# Unhandled Service clouddebugger #
# Unhandled Service cloudresourcemanager #
# Unhandled Service cloudscheduler #
# Unhandled Service cloudtrace #
# Unhandled Service container #
# Unhandled Service containeranalysis #
# Unhandled Service containerregistry #
# Unhandled Service containerscanning #
# Unhandled Service datastore #
# Unhandled Service groupssettings #
# Unhandled Service iam #
# Unhandled Service iamcredentials #
# Unhandled Service pubsub #
# Unhandled Service servicemanagement #
# Unhandled Service serviceusage #
# Unhandled Service sql-component #
# Unhandled Service stackdriver #

hh · 2019-07-10T17:25:11Z

These are the new perms and the commands are commented out in audit-gcp under each TODO:

TODO: compute Needs compute.projects.get
TODO: Ensure bucket-policy-only, for simplicity in Auditing
TODO: logging needs serviceusage.services.use
TODO: monitoring needs serviceusage.services.use
TODO: storage-api needs storage.buckets.get for auditors
TODO: Verify how Big Query is configured / audited
TODO: Verify how OS Login is configured / audited

hh · 2019-07-10T17:33:07Z

I'm keen with merging as is and evolving.

spiffxp · 2019-07-10T17:41:51Z

/approve
/lgtm
/hold
I'm going to leave the /hold cancel to @thockin

k8s-ci-robot · 2019-07-10T17:42:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

thockin

Mostly very minor comments - would love to get this in and then do a group review of the data.

thockin · 2019-07-10T23:43:06Z

audit/audit-gcp.sh

@@ -0,0 +1,100 @@
+#!/bin/bash
+# set -x -e
+CNCF_GCP_ORG=758905017065


You could source infra/gcp/lib.sh if there are any more utilities to re-use. This is unlikely to change :)

thockin · 2019-07-10T23:43:29Z

audit/audit-gcp.sh

+
+# gcloud organizations describe $CNCF_GCP_ORG 2>&1
+# ERROR: (gcloud.organizations.describe)
+# User [[email protected]] does not have permission to access organization []


did you try this again?

thockin · 2019-07-10T23:45:10Z

audit/audit-gcp.sh

@@ -0,0 +1,100 @@
+#!/bin/bash


need copyright header

thockin · 2019-07-10T23:45:47Z

audit/audit-gcp.sh

+    | while read PROJECT NUM; do \
+    export CLOUDSDK_CORE_PROJECT=$PROJECT
+    echo "### Auditing Project: ${PROJECT} ###"
+    mkdir -p $PROJECT


Since this assumes CWD is the script's root, maybe we should check that and print a warning if it is not?

thockin · 2019-07-10T23:49:01Z

audit/audit-gcp.sh

+            storage-api.googleapis.com)
+                echo TODO: Add storage.buckets.get for auditors
+                echo ...to kubernetes_public_billing and any newer buckets...
+                echo TODO: Ensure bucket-policy-only, for simplicity in Auditing


We do set bucketpolicyonly now. We do not set logging and we do not set cors. Should we?

thockin · 2019-07-10T23:50:41Z

audit/buckets/kubernetes-public.kubernetes_public_billing.txt

@@ -0,0 +1,156 @@
+gs://kubernetes_public_billing/billing--2019-01-10.csv


We probably should not be dumping bucket contents or VM names or anything that is likely to be constantly changing. Unlikely that any 2 audits could produce the same data for this.

thockin · 2019-07-10T23:51:14Z

audit/audit-gcp.sh

+                    #### gsutil cors get gs://$BUCKET/
+                    #### gsutil logging get gs://$BUCKET/
+                    gsutil iam get gs://$BUCKET/ > $PROJECT/buckets/$BUCKET.iam.json
+                    gsutil ls -r gs://$BUCKET/ > $PROJECT/buckets/$BUCKET.txt


We should not do this - it will be always changing

thockin · 2019-07-10T23:51:45Z

audit/audit-gcp.sh

+            compute)
+                echo TODO: $service Needs compute.projects.get
+                #### gcloud compute project-info describe
+                #### gcloud compute instances list --format=$format > $PROJECT/services/compute.instances.$format


we should not list VMs - they will change as clusters get upgraded

disks are somewhat dubiously useful, too. Clusters seems OK

thockin · 2019-07-10T23:52:43Z

audit/audit-gcp.sh

+                echo Processing: $service
+                mkdir -p dns
+                gcloud dns project-info describe $PROJECT --format=$format > $PROJECT/services/dns.info.$format
+                gcloud dns managed-zones list --format=$format > $PROJECT/services/dns.zones.$format


same as above - audit should catch metadata, not data

thockin · 2019-07-10T23:54:09Z

.../k8s-artifacts-graveyard/buckets/asia.artifacts.k8s-artifacts-graveyard.appspot.com.iam.json

@@ -0,0 +1,33 @@
+{
+  "bindings": [


graveyard is going away, so we can nuke all these files

dims · 2019-07-24T16:02:01Z

/assign @hh

spiffxp · 2019-08-07T15:45:16Z

merge and iterate

thockin · 2019-10-23T17:40:25Z

Ping - I think it's back in your court, @hh

hh · 2019-10-29T16:40:02Z

/hold

hh · 2019-10-29T16:40:55Z

/hold cancel

hh · 2019-10-29T16:42:38Z

/retest

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Apr 3, 2019

k8s-ci-robot requested a review from cblecker April 3, 2019 08:36

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 3, 2019

k8s-ci-robot requested a review from nikhita April 3, 2019 08:36

k8s-ci-robot added the wg/k8s-infra label Apr 3, 2019

thockin reviewed Apr 3, 2019

View reviewed changes

hh force-pushed the cncf-ci-audit branch 2 times, most recently from e25e19f to d8718f6 Compare May 1, 2019 15:27

hh mentioned this pull request May 1, 2019

Setup a job to automatically run and PR the results of audit/audit-gcp.sh #244

Closed

hh force-pushed the cncf-ci-audit branch from d8718f6 to c4eaa12 Compare May 15, 2019 15:43

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 15, 2019

hh added 3 commits May 16, 2019 04:01

initial kubernetes.io org iam policy dump

792d578

updates for 1st of May

82d819b

update for 15th of May

ca16382

hh force-pushed the cncf-ci-audit branch from c4eaa12 to ca16382 Compare May 15, 2019 16:02

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 15, 2019

June 16th 2019 Audit

3f5ee1b

hh commented Jun 15, 2019

View reviewed changes

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 15, 2019

hh commented Jun 15, 2019

View reviewed changes

hh force-pushed the cncf-ci-audit branch from e60bda7 to 3f5ee1b Compare June 15, 2019 18:35

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 15, 2019

hh commented Jun 16, 2019

View reviewed changes

hh mentioned this pull request Jun 16, 2019

Create IAM Dump scripts #207

Closed

k8s-ci-robot requested a review from spiffxp June 26, 2019 15:53

thockin self-assigned this Jun 29, 2019

thockin reviewed Jun 29, 2019

View reviewed changes

Updating for 11th of July

122e4e5

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 10, 2019

k8s-ci-robot assigned spiffxp Jul 10, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 10, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 10, 2019

thockin reviewed Jul 10, 2019

View reviewed changes

k8s-ci-robot assigned hh Jul 24, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 29, 2019

k8s-ci-robot merged commit 9d12330 into kubernetes:master Oct 29, 2019

		@@ -0,0 +1,156 @@
		gs://kubernetes_public_billing/billing--2019-01-10.csv

Initial Auditing Configuration and Usage of the kubernetes.io GCP Organization #213

Initial Auditing Configuration and Usage of the kubernetes.io GCP Organization #213

Conversation

hh commented Apr 3, 2019 • edited by thockin Loading

k8s-ci-robot commented Apr 3, 2019

hh commented Apr 3, 2019

thockin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented May 10, 2019

spiffxp commented May 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spiffxp commented Jun 26, 2019

thockin left a comment

Choose a reason for hiding this comment

spiffxp commented Jul 10, 2019

spiffxp commented Jul 10, 2019

hh commented Jul 10, 2019

hh commented Jul 10, 2019

hh commented Jul 10, 2019

spiffxp commented Jul 10, 2019

k8s-ci-robot commented Jul 10, 2019

thockin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dims commented Jul 24, 2019

spiffxp commented Aug 7, 2019

thockin commented Oct 23, 2019

hh commented Oct 29, 2019

hh commented Oct 29, 2019

hh commented Oct 29, 2019

hh commented Apr 3, 2019 •

edited by thockin

Loading