From 66c13c74c4c2f08615fed3d1eac0339a932eb8d9 Mon Sep 17 00:00:00 2001 From: Greg Cobb Date: Tue, 25 Jun 2024 08:26:14 -0700 Subject: [PATCH] Update workstation scripts to be more broadly useful (#64) * Import print_env_info script from capi-ci - It is no longer needed by capi-ci, since removing Slack integration from the bosh-lite pool. - It will be less lonely here, since this is where the rest of the bosh-lite pool scripts live. * Simplify claim_bosh_lite function - Remove dependency on git-together - Remove dependency on direnv - Remove dependency on lastpass CLI - Remove Pivotal Tracker integration - No longer prompts for Pivotal Tracker story id - Adds Jira integration (not really; just kidding) - No longer changes your working directory - Fix hostname for zsh (no $HOSTNAME environment variable) * Tidy up target_bosh function - Mostly just adding some stdout lines to clarify what the command is doing * Tidy unclaim_bosh_lite function - Remove dependency on git-together - Unset more environment variables - Fix hostname for zsh (no $HOSTNAME environment variable) * Use git commit instead of git ci - cause not everyone uses aliases * Simplify print_env_info - Remove story column - Remove staleness. Bosh lites are eternally fresh! - Gracefully handle case when no environment are claimed. Previously this would would trigger an "ambiguous ref" error from git, due to weirdness when globbing empty directories. * Delete can-i-push - No longer used - Depends the dead canibump app (RIP) * Unclaim also unsets CREDHUB_CA_CERT env var * Don't log in to CredHub - Per CredHub CLI output: Login ignored as the CREDHUB_* environment variables are present. You do NOT need to run this "login" command in order to run other "credhub" commands when you set the CREDHUB_* environment variables. The command only validates your credentials. For backwards compatibility reasons, it (in this particular use case) does not error out. You should not use it with the environment variables set. The "credhub login" command is useful in the case you do not have the environment variables set and you pass your credentials to "credhub login" through command line arguments. - Add some guards for commands that require a targeted bosh. For instance, if they require credhub env vars to be set =) * Remove some dubious commands - Remove the `set_capi_*_pipeline` scripts. They didn't do that much and have some baked-in assumptions (e.g. having a fly target named "capi"). They didn't seem worth parameterizing to make them more portable, but we can do it if people care enough. - Remove `gcs_to_claimed` script. I'm not really sure what this did, but it doesn't seem to be widely used, and it's not very portable. It also duplicates some of the behavior in `claim_bosh_lite`, which I'd rather not have to maintain independently. * Add shebang to bin/fixcommitter * Clean up deploy scripts - Add additional stoutput to help explain what is going on - Add guards to make sure there is an environment targeted - Reincarnate `use-created-capi.yml` ops file, formerly of capi-ci, but now gone. This ops file is used to build and deploy a capi release from your local filesystem, thanks to bosh handling local file urls. - Loosen assumptions about local directory structure. The commands now let you provide environment variables to override the default assumption that you have a relevant repos in a `~/workspace` directory. - Miscellaneous other small style changes, because I can * Additional minor tidying - Soften more directory structure opinions - Change some output formatting to be more consistent with other commands - Remove indenting output by piping to sed, since it doesn't work for non-stdout terminal output (e.g. bosh CLI output is often not pipe-able) * Remove bin/story - Most CAPI work is tracked via GitHub issues now; Tracker is not used by most/all CAPI contributors. - Story is no longer needed for claiming bosh lites (via prior commit) * Tidy seed_users - Take org/space name as arguments, akin to bootstrap_cf - Change default names to "org" and "space", instead of "test", so that they match bootstrap_cf * Remove commit_with_shortlog - Does not work; It attempted to call bin/staged_shortlog, which doesn't exist. - The actual staged_shortlog is in capi-release/scripts * Fix run_bridge_tests - The script assumed that it was inside capi-release's scripts directory. It was not. * Remove watch_cc_units - Consistent with removing other minor concourse-related scripts (see previous commit) - Provides dubious value vs using fly yourself, and it has a hard-coded target name * Remove bin/load-key - Part of larger effort to remove lastpass dependency - Most CAPI contributors don't use lastpass anymore * Remove bin/pivotal_login * Update capidoc with recent changes - Add some missing scripts * Small UI improvements to scripts * Reorganize README - Move heavy-weight installation instructions to the bottom, since they are not in common use - Emphasize the light-weight installation instructions (adding scripts to path and sourcing functions) - Add section espousing the wonders of the bosh lite pool - Remove some "Pivotal" references * Purge lib directory - Remove most opinions about development environment (vim, ruby dev, git, GOPATH, etc) - Limit to dev-machine-agnostic functions for CAPI development --- README.md | 334 +++++++++++++++++++------- assets/capidoc.yml | 26 +- assets/use-created-capi.yml | 7 + bin/can-i-push | 42 ---- bin/cf_admin_password | 13 +- bin/commit_with_shortlog | 4 - bin/create_and_deploy | 20 +- bin/create_and_upload | 27 ++- bin/deploy | 2 +- bin/deploy_only_new_capi | 22 +- bin/fixcommitter | 2 + bin/gcs_to_claimed | 32 --- bin/load-key | 15 -- bin/mysql_bosh_lite | 14 +- bin/pivotal_login | 231 ------------------ bin/print_env_info | 46 +++- bin/psql_bosh_lite | 14 +- bin/quick_deploy | 32 ++- bin/run_bridge_tests | 4 +- bin/seed_users | 24 +- bin/set_capi_backup_metadata_pipeline | 19 -- bin/set_capi_bosh_lite_pipeline | 17 -- bin/set_capi_pipeline | 12 - bin/story | 71 ------ bin/target_cf | 18 +- bin/target_uaa | 22 +- bin/watch_cc_units | 5 - lib/aliases.bash | 22 -- lib/claim-bosh-lite.bash | 78 +----- lib/environment-variables.bash | 24 -- lib/misc.bash | 28 --- lib/target-bosh.bash | 28 ++- lib/unclaim-bosh-lite.bash | 18 +- 33 files changed, 466 insertions(+), 807 deletions(-) create mode 100644 assets/use-created-capi.yml delete mode 100755 bin/can-i-push delete mode 100755 bin/commit_with_shortlog delete mode 100755 bin/gcs_to_claimed delete mode 100755 bin/load-key delete mode 100755 bin/pivotal_login delete mode 100755 bin/set_capi_backup_metadata_pipeline delete mode 100755 bin/set_capi_bosh_lite_pipeline delete mode 100755 bin/set_capi_pipeline delete mode 100755 bin/story delete mode 100755 bin/watch_cc_units delete mode 100644 lib/aliases.bash delete mode 100644 lib/environment-variables.bash diff --git a/README.md b/README.md index 3eb14ce..4c406ba 100644 --- a/README.md +++ b/README.md @@ -1,68 +1,10 @@ -## capi workstation setup 🐋 +## CAPI Workstation Helpers 🐋 -Hello there! This repo is intended to provide light-weight setup to developing on CAPI projects. +Hello there! This repo is intended to help with development work on CAPI +projects. ![capi](https://im-01.gifer.com/9Y0s.gif) -## Dependencies -* Mac OSX Sierra -* pip - -## Installation - -``` - mkdir -p ~/workspace && cd ~/workspace - git clone git@github.com:cloudfoundry/capi-workspace.git && cd capi-workspace -``` - -You can now either install via `./install.sh` or `./install-core.sh`. - -Or curl the bootstrap script, which will create the `workspace` directory and clone this repo for you. -This is primarily intended for quickly setting up virtual workstations. - -``` -bash <(curl -s https://raw.githubusercontent.com/cloudfoundry/capi-workspace/main/bootstrap.sh) -``` -**Make sure you ssh in as the "pivotal" user** - -## Manual steps (to be automated later) -* Open System Preferences / Users & Groups / / Login Items - * Add `flycut` from Applications - * Add `spectacle` from Applications -* Log out/Log in - * This will cause Flycut and Spectacle to run and ask for permissions -* Open up Rubymine manually and select `License Server` and copy in the the rubymine license from the labs license server (http://omaha.pivotallabs.com:8080/licenseServer in the SF office) - - If you are not on a Pivotal Network, you need to VPN into one in order to access this server. -* Install the `mine` cli shortcut (RubyMine -> Tools -> Create Command-line Launcher...) -* If you are using Goland do the same two previous steps for Goland -* If you have access to the private CAPI backlog, add a secure note called `tracker_api_token` to LastPass with your Pivotal Tracker API token. This token will be used in scripts such as `claim_bosh_lite` to pull story titles currently in flight. - -## Contributing to this repo - -* kindly consider when a change is worth making to `install-core.sh` versus `install.sh`. -* [bash-it](https://github.com/Bash-it/bash-it) We use bash-it to organize and streamline our bash settings. This includes stuff like color schemes, aliases, shell settings, and the shell prompt formatting. Adding "plugins" to `custom-bash-it-plugins` will cause them to be installed in every new shell. - -## What this installation does - -`install-core.sh`: intended to be more friendly for developers who don't want this repository to take over their machine. This is intended to be the minimal set of things required for working with a CAPI repository. -* Only installs core utilities, languages, services, and binaries. -* Is NOT responsible for installing & configuring any editors (nvim/vscode/intellij) -* Is NOT responsible for installing tools for improving developer workflows (jq/rg) -* Is NOT responsible for applying any git configuration -* Is NOT responsible for loading in scripts that need to be `source`'d. -* Is NOT responsible for modifying your `$PATH` to include `script` - -`./install.sh`: runs `install-core.sh` and configures the machine with many more packages/preferences - -## install.sh or install-core.sh ? - -`install-core.sh` is a subset of `install.sh` and is intended to be less invasive to the machine, without compromising on provisioning a workstation suitable to working on CAPI projects. - -For fresh workstations && full-time CAPI developers, install.sh might make more sense. - -For folks working on multiple projects, `install-core.sh` might make more sense. - -For folks with existing workstations & configuration, `install-core.sh` might make more sense. ## Directory Overview @@ -71,38 +13,32 @@ Description of the folders in this project: Folder | Description ---------------- | ----------- assets | random static files -bash-it | all the bash-it stuff, only bash-it stuff +bash-it | all the [bash-it](https://github.com/Bash-it/bash-it) stuff, only bash-it stuff helpers | random helper files lib | scripts that could be sourced into your shell bin | scripts that can be added to your `$PATH` install-scripts | executable scripts that install a thing or two -Why is `lib` separate from `bash-it`? Not everyone uses bash-it, so `lib` is similar to `scripts`, giving people a way to manually load the things they care about in. For example, one may manually (and minimally) load in capi-workspace content by sourcing a file like this in their `.bash_profile`: +Why is `lib` separate from `bash-it`? Not everyone uses bash-it, so `lib` is +similar to `scripts`, giving people a way to manually load the things they care +about in. + +## Minimal Install + +One may manually (and minimally) load in capi-workspace content by including +the following in their `~/.zshrc` (or equivalent): ```bash -❯ bat bash_sources/load_capi_workspace.zsh -───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── - │ File: bash_sources/load_capi_workspace.zsh -───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── - 1 │ - 2 │ source ~/workspace/capi-workspace/lib/pullify.bash >/dev/null - 3 │ source ~/workspace/capi-workspace/lib/target-bosh.bash >/dev/null - 4 │ source ~/workspace/capi-workspace/lib/claim-bosh-lite.bash >/dev/null - 5 │ source ~/workspace/capi-workspace/lib/unclaim-bosh-lite.bash >/dev/null - 6 │ source ~/workspace/capi-workspace/lib/deploy_only_new_capi.bash >/dev/null - 7 │ - 8 │ export PATH="$PATH:$HOME/workspace/capi-workspace/bin" -───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── -``` - -## CAPI Commands +source <(cat $HOME/workspace/capi-workspace/lib/*) > /dev/null +path+=("$HOME/workspace/capi-workspace/bin") +``` + +## CAPI Helper Commands This table is generated from `assets/capidoc.yml`. It is dynamically available in your terminal using the `capi` command. ``` -auth - load-key Loads the an ssh key in the currently logged in lastpass called "github-private-key". Run eval $(ssh-agent) if it fails backup-restore compare_db_rows Compares counts of two db's rows to validate backup & restore bosh-deploy @@ -117,7 +53,6 @@ bosh-lite-pool cf_admin_password Get the CF admin password for a current bosh target (requires credhub) cfu Alias for seed_users claim_bosh_lite Claim an available bosh lite CF - gcs_to_claimed Convert bosh-lite environment files stored in gcs to a claimed bosh-lite mysql_bosh_lite Connect to current bosh target's MySQL DB print_env_info List all the claimed bosh-lites in the pool psql_bosh_lite Connect to current bosh target's MySQL DB @@ -132,18 +67,15 @@ capi-release-author sync_package_specs Add go submodule dependencies to bosh package specs sync_submodule_config Sync git submodules with go dependencies using gosub unused_blobs Detects unused blobs from the blobs.yml file -ci - set_capi_pipeline Set the capi pipeline to your local config file (fly set-pipeline) - watch_cc_units Watch CAPI CI unit tests. For a specific run, -b build_number (fly watch) file-system v Open matching location in vim (fasd -e vim) z Change current directory to directory matching provided pattern (ex: z ng) git - commit_with_shortlog Git commit with a staged_shortlog fixcommitter Cleans up committer/author after a rebase g git status gd git diff gdc git diff --cached + git-open Open a git repo in your browser, courtesy of Paul Irish. pullify Pull down all PRs for current git repo as branches staged_shortlog Get commit shortlogs and authors from submodule changes update Update all the git submodules @@ -160,9 +92,239 @@ testing cats_cleanup Deletes builpacks, orgs, quotes, and service brokers in a targeted CF check_certificate_expirations Reads YAML files, checks for expired (or premature) certs. Useful for pipeline troubleshooting delete_orgs Delete all orgs for current cf target + generate_integration_config Create an integration_config.json for running CATS against current bosh target. run_bridge_tests Run Bridge unit tests sits Runs sits against a bosh lite workspaces capi-workspaces List and connect to existing K8s CAPI workspaces create-capi-workspace Create a CAPI workspace in K8s cluster ``` + +## Using the Bosh Lite Pool + +As a perk, [CAPI +approvers](https://github.com/cloudfoundry/community/blob/main/toc/working-groups/app-runtime-interfaces.md) +get access to a pool of cf-deployment environments, deployed on bosh lites in a +CFF-managed IaaS account. These environments are disposable, and are only +intended for development use. + +Though it is technically possible to use the pool without them, there are a +number of convenience scripts provided by capi-workspace that make the pool +usable by humans. + +### Prerequisites + +- git +- Clone capi-workspace (this repo) +- Add capi-workspace scripts/functions to your shell (see installation instructions above (or below)) +- Clone https://github.com/cloudfoundry/capi-env-pool + +### Example Pool Dev Workflow + +Claim an environment from the pool: +``` +❯ claim_bosh_lite +Claiming 'distaffs'... +Writing out .envrc... +Pushing reservation to capi-env-pool... +Use 'target_bosh distaffs' to set bosh environment variables. +``` + +Observe that you successfully claimed the environment: +``` +❯ print_env_info +Rounding up claimed environments... + +* ENV * * CLAIMED BY * * CLAIMED ON * * CLAIMED SINCE * +apologia Tim Downey tdowney.hostname 3 weeks ago +distaffs Greg Cobb gcobb.hostname 2 minutes ago +riskier Greg Cobb gcobb.hostname 13 days ago +zoned Seth Boyles capi-ws-cc-203 3 weeks ago +``` + +Set bosh environment variables in your shell to "target" that environment: +``` +❯ target_bosh distaffs + +Refreshing bosh lite pool state... +Already up to date. + +Sourcing /.../capi-env-pool/bosh-lites/claimed/distaffs to set bosh environment variables... + +Writing a capi-specific integration_config.json... +Generated /.../capi-env-pool/distaffs/integration_config.json. Enjoy! + +Writing a bosh gateway ssh key... +Generated /.../capi-env-pool/distaffs/bosh.pem. Enjoy! + +Setting BOSH_GW_PRIVATE_KEY, BOSH_ALL_PROXY, and CONFIG environment variables... +Success! +``` + +Log in to the environment with the `cf` CLI: +``` +❯ target_cf +Getting CF admin password from credhub... +Setting API endpoint to https://api.distaffs.app-runtime-interfaces.ci.cloudfoundry.org... +OK + +API endpoint: https://api.distaffs.app-runtime-interfaces.ci.cloudfoundry.org +API version: 3.166.0 + +Not logged in. Use 'cf login' or 'cf login --sso' to log in. +API endpoint: https://api.distaffs.app-runtime-interfaces.ci.cloudfoundry.org + +Authenticating... +OK + +Use 'cf target' to view or set your target org and space. +``` + +Create a default Organization and Space: +``` +❯ bootstrap_cf +Getting CF admin password from credhub... +Setting API endpoint to https://api.distaffs.app-runtime-interfaces.ci.cloudfoundry.org... +OK + +API endpoint: https://api.distaffs.app-runtime-interfaces.ci.cloudfoundry.org +API version: 3.166.0 + +Not logged in. Use 'cf login' or 'cf login --sso' to log in. +API endpoint: https://api.distaffs.app-runtime-interfaces.ci.cloudfoundry.org + +Authenticating... +OK + +Use 'cf target' to view or set your target org and space. +Creating org org as admin... +OK + +TIP: Use 'cf target -o "org"' to target new org +API endpoint: https://api.distaffs.app-runtime-interfaces.ci.cloudfoundry.org +API version: 3.166.0 +user: admin +org: org +No space targeted, use 'cf target -s SPACE' +Creating space space in org org as admin... +OK + +Assigning role SpaceManager to user admin in org org / space space as admin... +OK + +Assigning role SpaceDeveloper to user admin in org org / space space as admin... +OK + +TIP: Use 'cf target -o "org" -s "space"' to target new space +API endpoint: https://api.distaffs.app-runtime-interfaces.ci.cloudfoundry.org +API version: 3.166.0 +user: admin +org: org +space: space +``` + +Deploy the cf-deployment and capi-release versions from your dev machine: +``` +❯ deploy +Uploading new release to 35.185.252.196. + +Syncing bosh blobs... +bosh sync-blobs +... + +Creating bosh release... +create-release --force --name capi +... + +Uploading release to bosh director... +bosh upload-release --rebase +... + +Deploying uploaded release... +bosh deploy cf-deployment.yml -o... +... +Succeeded +``` + +Make local changes and deploy a new CAPI version: +``` +❯ deploy_only_new_capi +Building a new CAPI from local filesystem and deploying to 35.185.252.196. +bosh deploy <(bosh manifest) -o ...use-created-capi.yml ... +... +Succeeded +``` + +Release the environment for deletion, once you are done with it: +``` +❯ unclaim_bosh_lite riskier +Refreshing bosh lite pool state... +Hit enter to release 'riskier' +rm 'riskier/.envrc' +Pushing the release commit to capi-env-pool... +Done! +``` + +## Automated Workstation Setup + +If you want, there are also some opinionated scripts you can run to set up a +new computer for CAPI development. These may be useful if you are dynamically +provisioning remote workstations, or something like that. + +### Dependencies +* MacOS +* pip + +### Running the Installer + +``` + mkdir -p ~/workspace && cd ~/workspace + git clone git@github.com:cloudfoundry/capi-workspace.git && cd capi-workspace +``` + +You can now either install via `./install.sh` or `./install-core.sh`. + +Or curl the bootstrap script, which will create the `workspace` directory and clone this repo for you. +This is primarily intended for quickly setting up virtual workstations. + +``` +bash <(curl -s https://raw.githubusercontent.com/cloudfoundry/capi-workspace/main/bootstrap.sh) +``` + +## Manual steps (to be automated later) +* Open System Preferences / Users & Groups / / Login Items + * Add `flycut` from Applications + * Add `spectacle` from Applications +* Log out/Log in + * This will cause Flycut and Spectacle to run and ask for permissions +* Open up Rubymine manually and enter a `License Server` +* Install the `mine` cli shortcut (RubyMine -> Tools -> Create Command-line Launcher...) +* If you are using Goland do the same two previous steps for Goland +* If you have access to the private CAPI backlog, add a secure note called `tracker_api_token` to LastPass with your Pivotal Tracker API token. This token will be used in scripts such as `claim_bosh_lite` to pull story titles currently in flight. + +## Contributing to this repo + +* kindly consider when a change is worth making to `install-core.sh` versus `install.sh`. +* [bash-it](https://github.com/Bash-it/bash-it) We use bash-it to organize and streamline our bash settings. This includes stuff like color schemes, aliases, shell settings, and the shell prompt formatting. Adding "plugins" to `custom-bash-it-plugins` will cause them to be installed in every new shell. + +## What this installation does + +`install-core.sh`: intended to be more friendly for developers who don't want this repository to take over their machine. This is intended to be the minimal set of things required for working with a CAPI repository. +* Only installs core utilities, languages, services, and binaries. +* Is NOT responsible for installing & configuring any editors (nvim/vscode/intellij) +* Is NOT responsible for installing tools for improving developer workflows (jq/rg) +* Is NOT responsible for applying any git configuration +* Is NOT responsible for loading in scripts that need to be `source`'d. +* Is NOT responsible for modifying your `$PATH` to include `script` + +`./install.sh`: runs `install-core.sh` and configures the machine with many more packages/preferences + +## install.sh or install-core.sh ? + +`install-core.sh` is a subset of `install.sh` and is intended to be less invasive to the machine, without compromising on provisioning a workstation suitable to working on CAPI projects. + +For fresh workstations && full-time CAPI developers, install.sh might make more sense. + +For folks working on multiple projects, `install-core.sh` might make more sense. + +For folks with existing workstations & configuration, `install-core.sh` might make more sense. diff --git a/assets/capidoc.yml b/assets/capidoc.yml index 6ede5f9..68b85a5 100644 --- a/assets/capidoc.yml +++ b/assets/capidoc.yml @@ -26,10 +26,6 @@ check_certificate_expirations: description: Reads YAML files, checks for expired (or premature) certs. Useful for pipeline troubleshooting location: capi-workspace/bin group: testing -commit_with_shortlog: - description: Git commit with a staged_shortlog - location: capi-workspace/bin - group: git compare_db_rows: description: Compares counts of two db's rows to validate backup & restore location: capi-workspace/bin @@ -50,10 +46,14 @@ fixcommitter: description: Cleans up committer/author after a rebase location: capi-workspace/bin group: git -gcs_to_claimed: - description: Convert bosh-lite environment files stored in gcs to a claimed bosh-lite +generate_integration_config: + description: Create an integration_config.json for running CATS against current bosh target. location: capi-workspace/bin - group: bosh-lite-pool + group: testing +git-open: + description: Open a git repo in your browser, courtesy of Paul Irish. + location: capi-workspace/bin + group: git mysql_bosh_lite: description: Connect to current bosh target's MySQL DB location: capi-workspace/bin @@ -147,14 +147,6 @@ gi: description: gem install location: capi-workspace/custom-bash-it-plugins group: ruby-dev -set_capi_pipeline: - description: Set the capi pipeline to your local config file (fly set-pipeline) - location: capi-workspace/bin - group: ci -watch_cc_units: - description: Watch CAPI CI unit tests. For a specific run, -b build_number (fly watch) - location: capi-workspace/bin - group: ci cfu: description: Alias for seed_users location: capi-workspace/bin @@ -200,10 +192,6 @@ attach_debugger: description: Attaches the rubymine debugger to your bosh lite location: capi-release/src/cloud_controller_ng/scripts/short-circuit-cc group: bosh-lite-pool -load-key: - description: Loads the an ssh key in the currently logged in lastpass called "github-private-key". Run eval $(ssh-agent) if it fails - location: capi-workspace/bin - group: auth capi-workspaces: description: List and connect to existing K8s CAPI workspaces location: capi-workspace/bin diff --git a/assets/use-created-capi.yml b/assets/use-created-capi.yml new file mode 100644 index 0000000..530edff --- /dev/null +++ b/assets/use-created-capi.yml @@ -0,0 +1,7 @@ +--- +- type: replace + path: /releases/name=capi + value: + name: capi + version: create + url: file://((capi_release_dir)) \ No newline at end of file diff --git a/bin/can-i-push b/bin/can-i-push deleted file mode 100755 index 98a451b..0000000 --- a/bin/can-i-push +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env bash - -ARG=$(echo $1 | awk '{print tolower($1)}') -REASON=${2} - -if [ "$ARG" = "no" ] -then - if [ -z "${REASON}" ] - then - printf "\e[1;31mReason must be provided when setting \"can-i-push\" to no\e[0m\n" - echo "i.e. can-i-push no \"because i said so\"" - exit 1 - fi - printf "Setting can-i-push to \e[1;31mNO\e[0m...\n" - curl -X PUT "canibump.cfapps.io/$ARG" -d "token=maybe" -d "buildnumber=0" -d "reason=${REASON}" - if [ $? -ne 0 ] - then - printf "\e[1;31mFAILED\e[0m\n" - exit 1 - fi - printf "can-i-push is now \e[1;31mNO\e[0m - see https://canibump.cfapps.io for details\n" -elif [ "$ARG" = "yes" ] -then - printf "Setting can-i-push to \e[1;32mYES\e[0m\n" - curl -X PUT "canibump.cfapps.io/$ARG" -d "token=maybe" -d "buildnumber=0" - if [ $? -ne 0 ] - then - printf "\e[1;31mFAILED\e[0m\n" - exit 1 - fi - printf "can-i-push is now \e[1;32mYES\e[0m\n" -else - YES=$(curl canibump.cfapps.io -s -H "Accept: application/json" | grep true) - if [ "$YES" ] - then - printf "\e[1;32m%s\e[0m\n" "YES" - else - printf "\e[1;31m%s\e[0m - see https://canibump.cfapps.io for details\n" "NO" - exit 1 - fi -fi -tput sgr0 diff --git a/bin/cf_admin_password b/bin/cf_admin_password index 986e41b..2c6b57f 100755 --- a/bin/cf_admin_password +++ b/bin/cf_admin_password @@ -4,24 +4,13 @@ set -e get_cf_admin_password() { echo "Getting CF admin password from credhub... " - - # Newer bosh-lites use $CREDHUB_SECRET instead of $CREDHUB_PASSWORD - # The credhub cli is aware of $CREDHUB_SECRET so it does not need it passed in - # If there are no bosh-lites around anymore that were created before 01/25/2018 - # you can delete the else condition - set +u # $CREDHUB_PASSWORD may be unbound - if [[ -z $CREDHUB_PASSWORD ]]; then - credhub login --skip-tls-validation - else - credhub login -s "$CREDHUB_SERVER" -u "$CREDHUB_USERNAME" -p "$CREDHUB_PASSWORD" --skip-tls-validation - fi set -u cf_admin_password=$(credhub get --name '/bosh-lite/cf/cf_admin_password' --output-json | jq -r '.value') } main() { if [ -z "$BOSH_ENVIRONMENT" ]; then - echo "No bosh targeted. Use \"target_bosh\" before getting credhub password" + echo "No bosh targeted. Use \"target_bosh\" before getting admin password." else get_cf_admin_password echo "CF Admin Password: ${cf_admin_password}" diff --git a/bin/commit_with_shortlog b/bin/commit_with_shortlog deleted file mode 100755 index 2962564..0000000 --- a/bin/commit_with_shortlog +++ /dev/null @@ -1,4 +0,0 @@ -#!/bin/bash -script_dir=$(dirname $0) - -$script_dir/staged_shortlog | git ci -F - diff --git a/bin/create_and_deploy b/bin/create_and_deploy index 182ac74..895a749 100755 --- a/bin/create_and_deploy +++ b/bin/create_and_deploy @@ -1,12 +1,18 @@ #!/bin/bash -set -exu +set -eu + +cfd_dir="${CF_DEPLOYMENT_DIR:-$HOME/workspace/cf-deployment}" +capi_ci_dir="${CAPI_CI_DIR:-$HOME/workspace/capi-ci}" create_and_upload -# deploy to current bosh env -bosh deploy ~/workspace/cf-deployment/cf-deployment.yml \ +printf "\nDeploying uploaded release...\n" +echo "bosh deploy cf-deployment.yml -o..." + +bosh deploy "${cfd_dir}/cf-deployment.yml" \ -v system_domain=$BOSH_LITE_DOMAIN \ - -o ~/workspace/capi-ci/cf-deployment-operations/skip-cert-verify.yml \ - -o ~/workspace/cf-deployment/operations/bosh-lite.yml \ - -o ~/workspace/cf-deployment/operations/use-compiled-releases.yml \ - -o ~/workspace/capi-ci/cf-deployment-operations/use-latest-capi.yml $@ + -o "${capi_ci_dir}/cf-deployment-operations/skip-cert-verify.yml" \ + -o "${cfd_dir}/operations/bosh-lite.yml" \ + -o "${cfd_dir}/operations/use-compiled-releases.yml" \ + -o "${capi_ci_dir}/cf-deployment-operations/use-latest-capi.yml" \ + $@ diff --git a/bin/create_and_upload b/bin/create_and_upload index b2734ee..3f8345d 100755 --- a/bin/create_and_upload +++ b/bin/create_and_upload @@ -1,9 +1,28 @@ #!/bin/bash -set -eu +set -e -# create and upload a release -pushd ~/workspace/capi-release +if [ -z "$BOSH_ENVIRONMENT" ]; then + echo "No bosh targeted. Use \"target_bosh\" before creating and uploading a release." + exit 1 +fi +echo "Uploading new release to $BOSH_ENVIRONMENT." + +set -u + +capi_release_dir="${CAPI_RELEASE_DIR:-$HOME/workspace/capi-release}" + +pushd "${capi_release_dir}" > /dev/null + + printf "\nSyncing bosh blobs...\n" + echo "bosh sync-blobs" bosh sync-blobs + + printf "\nCreating bosh release...\n" + echo "create-release --force --name capi" bosh create-release --force --name capi + + printf "\nUploading release to bosh director...\n" + echo "bosh upload-release --rebase" bosh upload-release --rebase -popd + +popd > /dev/null diff --git a/bin/deploy b/bin/deploy index 6358e12..33c547a 100755 --- a/bin/deploy +++ b/bin/deploy @@ -1,4 +1,4 @@ #!/bin/bash -set -exu +set -e create_and_deploy $@ -n diff --git a/bin/deploy_only_new_capi b/bin/deploy_only_new_capi index cf55d4f..55c4e73 100755 --- a/bin/deploy_only_new_capi +++ b/bin/deploy_only_new_capi @@ -1,13 +1,19 @@ #!/usr/bin/env bash -# you provide your own capi-release directory via the first positional argument -if [[ -n "$1" ]] ; then - CAPI_RELEASE_DIR="$1" ; shift -else - CAPI_RELEASE_DIR="$HOME/workspace/capi-release" +set -e + +capi_release_dir="${CAPI_RELEASE_DIR:-$HOME/workspace/capi-release}" +__dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +if [ -z "$BOSH_ENVIRONMENT" ]; then + echo "No bosh targeted. Use \"target_bosh\" before deploying new CAPI." + exit 1 fi +echo "Building a new CAPI from local filesystem and deploying to $BOSH_ENVIRONMENT." + +echo "bosh deploy <(bosh manifest) -o ...use-created-capi.yml ..." bosh -n deploy <(bosh manifest) \ - -o ~/workspace/capi-ci/cf-deployment-operations/use-created-capi.yml \ - -v capi_release_dir="${CAPI_RELEASE_DIR}" \ - "$@" + -o "${__dir}/../assets/use-created-capi.yml" \ + -v capi_release_dir="${capi_release_dir}" \ + $@ diff --git a/bin/fixcommitter b/bin/fixcommitter index f1347f6..d3d3ecc 100755 --- a/bin/fixcommitter +++ b/bin/fixcommitter @@ -1,3 +1,5 @@ +#!/usr/bin/env bash + do_fixcommitter () { ( committer=$(git show HEAD -s --pretty="tformat:%b" | grep Signed-off | cut -d: -f2); diff --git a/bin/gcs_to_claimed b/bin/gcs_to_claimed deleted file mode 100755 index 7d7deb6..0000000 --- a/bin/gcs_to_claimed +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env bash - -creds_file=$1 -director_file=$2 - -bosh_ca_cert=$(bosh int ${creds_file} --path=/director_ssl/ca) -bosh_client_secret=$(bosh int ${creds_file} --path=/admin_password) -bosh_gw_private_key_contents=$(bosh int ${creds_file} --path=/jumpbox_ssh/private_key) -credhub_secret=$(bosh int ${creds_file} --path=/credhub_admin_client_secret) - -bosh_environment=$(bosh int ${director_file} --path=/instance_groups/name=bosh/networks/name=public/static_ips/0) -bosh_lite_name=$(bosh int ${director_file} --path=/networks/name=default/subnets/0/cloud_properties/network_name) - -cat > "${HOME}/workspace/capi-env-pool/bosh-lites/claimed/${bosh_lite_name}" < $KEY -chmod 600 $KEY -if [ -z $HOURS ]; then - HOURS=12 -fi -ssh-add -D -ssh-add -t ${HOURS}H $KEY -echo "You are logged in now!" diff --git a/bin/mysql_bosh_lite b/bin/mysql_bosh_lite index 10531bb..18596d8 100755 --- a/bin/mysql_bosh_lite +++ b/bin/mysql_bosh_lite @@ -5,13 +5,6 @@ set -e set -o pipefail get_database_password(){ - set +u # $CREDHUB_PASSWORD may be unbound - if [[ -z $CREDHUB_PASSWORD ]]; then - credhub login --skip-tls-validation - else - credhub login -s "$CREDHUB_SERVER" -u "$CREDHUB_USERNAME" -p "$CREDHUB_PASSWORD" --skip-tls-validation - fi - set -u cc_database_password=$(credhub get -n '/bosh-lite/cf/cc_database_password' --output-json | jq -r '.value') } @@ -45,6 +38,13 @@ kill_tree() { kill $pid } +set +u +if [ -z "$BOSH_ENVIRONMENT" ]; then + echo "No bosh targeted. Use \"target_bosh\" before connecting to MySQL." + exit 1 +fi +set -u + { get_database_password && ssh_tunnel && diff --git a/bin/pivotal_login b/bin/pivotal_login deleted file mode 100755 index 00a6e8d..0000000 --- a/bin/pivotal_login +++ /dev/null @@ -1,231 +0,0 @@ -#!/bin/bash - -set -e - -readonly LASTPASS_KEY_LABEL='ssh_private_key' - -function usage { - echo 'Usage: ' - echo 'pivotal_login [options...]' - echo - echo "Beginner's guide" - echo 'Your first time, run:' - echo 'pivotal_login donald.knuth@pivotal.io -t 8 --import-private-key /mnt/path/to/private_key --save-alias dk' - echo 'The next times, run:' - echo 'pivotal_login dk [-t ]' - echo - echo 'List of options:' - echo 'To specify a number of hours for which you want to stay logged in:' - echo '--time-in-hours (-t for short)' - echo "To not be prompted to login to lastpass if you're already logged in:" - echo '-n' - echo 'To import your private key and write it to lastpass, use the option:' - echo '--import-private-key (-k for short)' - echo 'To save your username and settings with a short alias, use:' - echo '--save-alias ' - echo 'Then, you can run "pivotal_login "' - echo - echo 'To logout, just run ' - echo 'pivotal_login --logout (-o for short)' -} - -POSITIONAL=("$@") -HOURS=1 -IMPORT_KEY='false' -LOGIN_TO_LASTPASS='true' - -USERNAME="$1" - -if [[ -z "$USERNAME" ]]; then - echo 'You need to specify your lastpass username or an alias as the first argument.' - echo - usage - exit 2 -fi - -case $USERNAME in - -h|--help) - usage - exit 0 - ;; -esac - -shift # skip username - -while [[ $# -gt 0 ]] -do - key="$1" - - case $key in - -t|--time-in-hours) - HOURS="$2" - shift # past argument - shift # past value - ;; - -k|--import-private-key) - IMPORT_KEY='true' - KEY_FILE_PATH="$2" - shift # past argument - shift # past value - ;; - -n) - LOGIN_TO_LASTPASS='false' - shift - ;; - --save-alias) - SAVE_ALIAS="$2" - shift - shift - ;; - -h|--help) - usage - exit 0 - ;; - *) # unknown option - echo "unrecognized option \"$1\"" - exit 6 - ;; - esac -done -set -- "${POSITIONAL[@]}" # restore positional parameters - -function fetch_private_key { - set +x - KEY=`lpass show "$LASTPASS_KEY_LABEL" --notes 2> /dev/null` - return $? -} - -function import_private_key { - LPASS_USERNAME="$1" - KEY_FILE_PATH="$2" - - if [[ ! -f "$KEY_FILE_PATH" ]]; then - echo "The file \"$KEY_FILE_PATH\" doesn't exist." - exit 3 - fi - - if fetch_private_key ; then - echo "There already is a note called \"$LASTPASS_KEY_LABEL\" in your lastpass vault." - echo 'Aborting' - exit 4 - fi - - cat "$KEY_FILE_PATH" | lpass add --non-interactive --notes "$LASTPASS_KEY_LABEL" - if [[ $? ]]; then - echo 'Your private key was successfully saved in your lastpass vault!' - else - echo "Your private key couldn't be saved to lastpass. Aborting." - exit 5 - fi -} - -readonly STORE="${HOME}/.pivotal_login" - -function save_alias { - ALIAS=$1 - LP_USER=$2 - if [[ ! -f "$STORE" ]]; then - touch $STORE - fi - if grep "^${ALIAS}#" "$STORE"; then - echo "The alias ${ALIAS} is already taken. Pick another one or delete it from ${STORE}" - exit 7 - fi - if echo "$ALIAS" | grep -e '#' -e '@'; then - echo 'The alias cannot contain the characters # or @.' - exit 8 - fi - echo "${ALIAS}#${LP_USER} -t ${HOURS}" >> "${STORE}" -} - -function find_alias { - ALIAS=$1 - - set +e - - [[ -f ${STORE} ]] || touch ${STORE} - - STORE_LINE="$(grep "^${ALIAS}#" "${STORE}")" - if [[ "$?" = 0 ]]; then - ALIAS_OPTIONS=$(echo "$STORE_LINE" | cut -d# -f2) - else - echo "The alias \"$ALIAS\" cannot be found." - exit 9 - fi - - set -e -} - -function options_from_username { - USERNAME=$1 - - if echo "$USERNAME" | grep '@' > /dev/null; then - LPASS_USERNAME=$1 - else - find_alias $USERNAME - OPTIONS="$ALIAS_OPTIONS" - fi -} - -function logout { - set -e - /usr/bin/ssh-add -D - lpass logout -f - echo 'You sucessfully logged out!' - exit 0 -} - -function validate_ssh_agent_running { - set +e - \ssh-add -L > /dev/null 2>&1 - - if [[ $? == 2 ]]; then - echo "We could't connect to a running ssh-agent, we recommend adding:" - echo - echo 'eval $(ssh-agent)' - echo - echo 'to your .profile script so that it is run on login' - exit 10 - fi -} - -validate_ssh_agent_running - -if [[ "$USERNAME" = '-o' || "$USERNAME" = "--logout" ]]; then - logout -fi - -options_from_username "$USERNAME" - -if [[ -n "$OPTIONS" ]]; then - shift # get rid of alias - options=($OPTIONS $@) - $0 ${options[@]} - exit $? -fi - -if [[ ${LOGIN_TO_LASTPASS} = 'true' ]]; then - export LPASS_AGENT_TIMEOUT=$((HOURS * 60 * 60)) - lpass login "${LPASS_USERNAME}" - unset LPASS_AGENT_TIMEOUT -fi - -if [[ "$IMPORT_KEY" = 'true' ]]; then - import_private_key "$LPASS_USERNAME" "$KEY_FILE_PATH" -fi - -if ! fetch_private_key ; then - echo "Your private key couldn't be found in lastpass." - echo "Set it to be able to login." - exit 1 -fi - -if [[ -n ${SAVE_ALIAS} ]]; then - save_alias "$SAVE_ALIAS" "$LPASS_USERNAME" -fi - -/usr/bin/ssh-add -D -/usr/bin/ssh-add -t "${HOURS}H" - <<< "${KEY}" -echo "Identity added for ${HOURS} hours" - -unset KEY diff --git a/bin/print_env_info b/bin/print_env_info index 42e518a..bf9256e 100755 --- a/bin/print_env_info +++ b/bin/print_env_info @@ -1,7 +1,45 @@ #!/bin/bash set -e -pushd ~/workspace/capi-env-pool > /dev/null - git pull -n > /dev/null - ~/workspace/capi-ci/ci/bosh-lite/src/print_env_info -popd > /dev/null +env_pool="${CAPI_ENV_POOL_DIR:-$HOME/workspace/capi-env-pool}" +pool_dir="${env_pool}/bosh-lites" + +function print_env_info { + local blue + blue='\033[0;34m' + nc='\033[0m' + + echo -e "${blue}Rounding up claimed environments...${nc}" + + pushd "${pool_dir}/claimed" > /dev/null + git pull -n > /dev/null + if [ -z "$(ls .)" ]; then + echo "No claimed environments." + exit 0 + fi + + claimed_files="$(git log --reverse --name-only --pretty=format: -- * | sort | uniq | xargs)" + + output="${blue}\n* ENV *\t* CLAIMED BY *\t* CLAIMED ON *\t* CLAIMED SINCE *\n${nc}" + + for file in ${claimed_files}; do + file="$( basename "${file}" )" + author="$(git log --max-count=1 --pretty='format:%an' "${file}")" + committer="$(git log --max-count=1 --pretty='format:%cn' "${file}")" + claimed_since="$(git log --max-count=1 --pretty='format:%ar' "${file}")" + workstation="$(git log --max-count=1 --pretty='format:%s' "${file}" | sed -E "s/^manually claim [^[:space:]]+ on ([^[:space:]]+).*$/\1/")" + + if [ "${author}" != "${committer}" ]; then + claimed_by="${author}+${committer}" + else + claimed_by="${author}" + fi + + output="${output}${file}\t${claimed_by}\t${workstation}\t${claimed_since}\n" + done + + popd > /dev/null + echo -e "$output" | column -t -s $'\t' +} + +print_env_info diff --git a/bin/psql_bosh_lite b/bin/psql_bosh_lite index c90e99e..c4597b4 100755 --- a/bin/psql_bosh_lite +++ b/bin/psql_bosh_lite @@ -5,13 +5,6 @@ set -e set -o pipefail get_database_password(){ - set +u # $CREDHUB_PASSWORD may be unbound - if [[ -z $CREDHUB_PASSWORD ]]; then - credhub login --skip-tls-validation - else - credhub login -s "$CREDHUB_SERVER" -u "$CREDHUB_USERNAME" -p "$CREDHUB_PASSWORD" --skip-tls-validation - fi - set -u cc_database_password=$(credhub get -n '/bosh-lite/cf/cc_database_password' --output-json | jq -r '.value') } @@ -41,6 +34,13 @@ kill_tree() { kill $pid } +set +u +if [ -z "$BOSH_ENVIRONMENT" ]; then + echo "No bosh targeted. Use \"target_bosh\" before connecting to Postgres." + exit 1 +fi +set -u + { get_database_password && ssh_tunnel && diff --git a/bin/quick_deploy b/bin/quick_deploy index 25b7308..411fa52 100755 --- a/bin/quick_deploy +++ b/bin/quick_deploy @@ -1,17 +1,29 @@ #!/bin/sh -set -exu +set -e TARFILE=$(mktemp -t qd) TARBASE=$(basename $TARFILE) SLEEP=15 QUICK_DEPLOY=$HOME/.cf/last_quick_deploy +ccng_dir="${CLOUD_CONTROLLER_DIR:-$HOME/workspace/capi-release/src/cloud_controller_ng}" -pushd ~/workspace/capi-release/src/cloud_controller_ng +if [ -z "$BOSH_ENVIRONMENT" ]; then + echo "No bosh targeted. Use \"target_bosh\" before deploying, quickly." + exit 1 +fi +echo "Deploying to $BOSH_ENVIRONMENT, posthaste!" + +set -u + +printf "\nGathering modified Cloud Controller files...\n" +pushd $ccng_dir > /dev/null deletedFiles=$(git status -s | awk '$1 == "D" { print $2 }') if [ -f $QUICK_DEPLOY ] ; then + echo "Detected prior quick deploy in $QUICK_DEPLOY." changedFiles=$(find . -type f -newer $QUICK_DEPLOY \! -name '*.log' \! -name '*.txt' \! -name workspace.xml | grep -v -e spec/ -e docs/ ) || true else + echo "No prior quick deploy detected." changedFiles=$(git status -s | awk '$1 != "D" { print $2 }') fi if [ -z "$changedFiles" -a -z "$deletedFiles" ] ; then @@ -19,13 +31,15 @@ pushd ~/workspace/capi-release/src/cloud_controller_ng exit 0 fi tar cf $TARFILE $changedFiles -popd +popd > /dev/null SDIR=/var/vcap/packages/cloud_controller_ng/cloud_controller_ng +printf "\nbosh SCPing modified files to CAPI bosh instances...\n" targets=$(bosh is --json | jq -r '.Tables[0].Rows[] | select(.process_state="running") | select(.instance | test("\\A(?:api|cc-worker|scheduler)/")) | .instance') for target in $targets do + printf "\nCopying modified files to ${target}\n" bosh scp $TARFILE "${target}:/tmp/${TARBASE}" for x in $deletedFiles ; do bosh ssh $target "sudo rm -f $SDIR/$x" @@ -34,17 +48,21 @@ do done MONIT=/var/vcap/bosh/bin/monit +printf "\nMonit restarting api Cloud Controller jobs...\n" bosh ssh api "sudo $MONIT restart cloud_controller_ng; sudo $MONIT restart cloud_controller_worker_local_1 ; sudo $MONIT restart cloud_controller_worker_local_2" +printf "\nMonit restarting cc-worker Cloud Controller jobs...\n" bosh ssh cc-worker "sudo $MONIT restart cloud_controller_worker_1" -bosh ssh scheduler "sudo $MONIT restart cloud_controller_clock" +printf "\nMonit restarting scheduler Cloud Controller jobs...\n" +bosh ssh scheduler "sudo $MONIT restart cloud_controller_clock; sudo $MONIT restart cc_deployment_updater" -echo "sleeping for $SLEEP seconds before checking status..." +printf "\nSleeping for $SLEEP seconds before checking status...\n" sleep $SLEEP +printf "\nChecking status of bosh jobs...\n" for target in $targets do - echo $target - bosh ssh $target "sudo $MONIT summary" + echo "CAPI jobs for ${target}:" + bosh ssh $target "sudo $MONIT summary | grep -E 'cloud_controller|cc'" done touch $QUICK_DEPLOY diff --git a/bin/run_bridge_tests b/bin/run_bridge_tests index 4432946..2cc04f3 100755 --- a/bin/run_bridge_tests +++ b/bin/run_bridge_tests @@ -2,8 +2,8 @@ set -e -scripts_path="./$(dirname $0)" -bridge_path="$scripts_path/../src/code.cloudfoundry.org" +readonly CAPI_RELEASE_DIR="${CAPI_RELEASE_DIR:-$HOME/workspace/capi-release}" +bridge_path="${CAPI_RELEASE_DIR}/src/code.cloudfoundry.org" pushd $bridge_path > /dev/null ginkgo -r -keepGoing -p -trace -randomizeAllSpecs -progress --race \ diff --git a/bin/seed_users b/bin/seed_users index 464f08a..9b02148 100755 --- a/bin/seed_users +++ b/bin/seed_users @@ -2,19 +2,31 @@ set -ue +orgname=${1:-org} +spacename=${2:-space} + main() { + echo "Creating users in ${orgname}/${spacename}..." cf create-user space_developer password - cf set-space-role space_developer test test SpaceDeveloper + cf set-space-role space_developer $orgname $spacename SpaceDeveloper + cf create-user space_manager password - cf set-space-role space_manager test test SpaceManager + cf set-space-role space_manager $orgname $spacename SpaceManager + cf create-user space_auditor password - cf set-space-role space_auditor test test SpaceAuditor + cf set-space-role space_auditor $orgname $spacename SpaceAuditor + + cf create-user space_supporter password + cf set-space-role space_supporter $orgname $spacename SpaceSupporter + cf create-user billing_manager password - cf set-org-role billing_manager test BillingManager + cf set-org-role billing_manager $orgname BillingManager + cf create-user org_manager password - cf set-org-role org_manager test OrgManager + cf set-org-role org_manager $orgname OrgManager + cf create-user org_auditor password - cf set-org-role org_auditor test OrgAuditor + cf set-org-role org_auditor $orgname OrgAuditor } main diff --git a/bin/set_capi_backup_metadata_pipeline b/bin/set_capi_backup_metadata_pipeline deleted file mode 100755 index 6763aaf..0000000 --- a/bin/set_capi_backup_metadata_pipeline +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/bash - -set -e - -echo "Pulling 'capi-ci-private'..." -pushd ~/workspace/capi-ci-private > /dev/null - git pull -r -popd > /dev/null - -if ! fly -t capi status ; then - fly -t capi login -b -fi - -# Using lpass with Backup-Restore credentials -# until we figure out how to integrate with shared resources, (e.g clusters, buckets etc) -fly -t capi set-pipeline \ - --check-creds \ - --pipeline backup-metadata \ - --config "${HOME}/workspace/capi-ci/ci/pipeline-backup-metadata.yml" diff --git a/bin/set_capi_bosh_lite_pipeline b/bin/set_capi_bosh_lite_pipeline deleted file mode 100755 index 461f3f9..0000000 --- a/bin/set_capi_bosh_lite_pipeline +++ /dev/null @@ -1,17 +0,0 @@ -#!/bin/bash - -set -e - -echo "Pulling 'capi-ci-private'..." -pushd ~/workspace/capi-ci-private > /dev/null - git pull -r -popd > /dev/null - -if ! fly -t capi status ; then - fly -t capi login -b -fi - -fly -t capi \ - set-pipeline \ - -p bosh-lite \ - -c ~/workspace/capi-ci/ci/pipeline-bosh-lite.yml diff --git a/bin/set_capi_pipeline b/bin/set_capi_pipeline deleted file mode 100755 index ba360a2..0000000 --- a/bin/set_capi_pipeline +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/bash - -set -e - -if ! fly -t capi status ; then - fly -t capi login -b -fi - -fly -t capi \ - set-pipeline \ - -p capi \ - -c ~/workspace/capi-ci/ci/pipeline.yml diff --git a/bin/story b/bin/story deleted file mode 100755 index a200b0d..0000000 --- a/bin/story +++ /dev/null @@ -1,71 +0,0 @@ -#!/usr/bin/env bash - - -get_stories() { - story_json="$(curl -s -H "Content-Type: application/json" \ - "https://www.pivotaltracker.com/services/v5/projects/$CAPI_WORKSPACE_TRACKER_ID/stories?with_state=started" | tr '**' ' ')" - echo $story_json | tr '\n' ' ' | jq '.[] | "#\(.id) \(.name)"' - echo "No story" -} - -story_selector() { - stories="$(get_stories | sed 's/#//g')" - - PS3="Select a story or 'q' to quit: " - OLD_IFS=$IFS - IFS=$'\n' - select story in $stories; do - echo "$story" - break - done - IFS=$OLD_IFS -} - -backlog_selector() { - backlogs=("CAPI 966314" "VAT 2196383" "CAKE 2468497") - - PS3="Select a backlog or 'q' to quit: " - OLD_IFS=$IFS - IFS=$'\n' - select backlog in "${backlogs[@]}"; do - echo "$backlog" - break - done - IFS=$OLD_IFS -} - -story() { - if [ -z "$CAPI_WORKSPACE_TRACKER_ID" ]; then - backlog_selection="$(backlog_selector)" - CAPI_WORKSPACE_TRACKER_ID="$(echo "$backlog_selection" | awk '{print $2;}')" - fi - - if [ -z "$CAPI_WORKSPACE_TRACKER_ID" ]; then - echo "No backlog selected. Exiting..." - return - fi - - story_selection="$(story_selector)" - story_id="$(echo "$story_selection" | awk '{print $1;}' | sed s/\"//)" - - if [[ "$story_selection" == "No story" ]]; then - echo "Unsetting current story." - export STORY_NUM='' - return - fi - - if [ -z "$story_selection" ]; then - echo "No story selected. Exiting..." - return - fi - - STORY_TITLE="$(curl -s "https://www.pivotaltracker.com/services/v5/projects/${CAPI_WORKSPACE_TRACKER_ID}/stories/${story_id}" | jq -r .name)"; - STORY_URL="$(curl -s "https://www.pivotaltracker.com/services/v5/projects/${CAPI_WORKSPACE_TRACKER_ID}/stories/${story_id}" | jq -r .url)"; - - export STORY_NUM="#${story_id} [${STORY_TITLE}](${STORY_URL})" - - # reset commit message now to include story - git author $(git config git-together.active | sed -e 's/+/ /g') -} - -story diff --git a/bin/target_cf b/bin/target_cf index f0391b9..6edbad3 100755 --- a/bin/target_cf +++ b/bin/target_cf @@ -4,18 +4,6 @@ set -ue get_cf_admin_password() { echo "Getting CF admin password from credhub... " - - # Newer bosh-lites use $CREDHUB_SECRET instead of $CREDHUB_PASSWORD - # The credhub cli is aware of $CREDHUB_SECRET so it does not need it passed in - # If there are no bosh-lites around anymore that were created before 01/25/2018 - # you can delete the else condition - set +u # $CREDHUB_PASSWORD may be unbound - if [[ -z $CREDHUB_PASSWORD ]]; then - credhub login --skip-tls-validation - else - credhub login -s "$CREDHUB_SERVER" -u "$CREDHUB_USERNAME" -p "$CREDHUB_PASSWORD" --skip-tls-validation - fi - set -u cf_admin_password=$(credhub get --name '/bosh-lite/cf/cf_admin_password' --output-json | jq -r '.value') } @@ -25,6 +13,12 @@ login() { } main() { + set +u + if [ -z "$BOSH_ENVIRONMENT" ]; then + echo "No bosh targeted. Use \"target_bosh\" before targeting Cloud Foundry." + exit 1 + fi + set -u get_cf_admin_password login } diff --git a/bin/target_uaa b/bin/target_uaa index dc5ee81..ab66a21 100755 --- a/bin/target_uaa +++ b/bin/target_uaa @@ -1,28 +1,28 @@ #!/usr/bin/env bash -set -ue +set -e get_uaa_admin_password() { - echo "Getting UAA admin password from credhub... " - set +u # $CREDHUB_PASSWORD may be unbound - if [[ -z $CREDHUB_PASSWORD ]]; then - credhub login --skip-tls-validation - else - credhub login -s "$CREDHUB_SERVER" -u "$CREDHUB_USERNAME" -p "$CREDHUB_PASSWORD" --skip-tls-validation - fi + echo "Getting UAA admin password from credhub..." set -u uaa_admin_password=$(credhub get --name '/bosh-lite/cf/uaa_admin_client_secret' --output-json | jq -r '.value') + echo "UAA admin password: ${uaa_admin_password}" } login() { + printf "\nTargeting UAA with uaac...\n" uaac target uaa.${BOSH_LITE_DOMAIN} --skip-ssl-validation + printf "\nGetting access token for admin client...\n" uaac token client get admin -s $uaa_admin_password - echo "UAA password = ${uaa_admin_password}" } main() { - get_uaa_admin_password - login + if [ -z "$BOSH_ENVIRONMENT" ]; then + echo "No bosh targeted. Use \"target_bosh\" before targeting UAA." + else + get_uaa_admin_password + login + fi } main diff --git a/bin/watch_cc_units b/bin/watch_cc_units deleted file mode 100755 index 50d30a8..0000000 --- a/bin/watch_cc_units +++ /dev/null @@ -1,5 +0,0 @@ -#!/usr/bin/env bash - -set -e - -fly -t capi watch -j capi/cc-unit-tests $* diff --git a/lib/aliases.bash b/lib/aliases.bash deleted file mode 100644 index 2bf4fe5..0000000 --- a/lib/aliases.bash +++ /dev/null @@ -1,22 +0,0 @@ -#!/usr/bin/env bash - -alias vim=nvim - -# fasd -alias v='fasd -e vim' -[ -f /usr/local/etc/profile.d/autojump.sh ] && . /usr/local/etc/profile.d/autojump.sh - -# ruby -alias b='bundle exec' -alias bake='echo "bundling..." && bundle install --quiet && echo "done bundling" && bundle exec rake' -alias pgbake='echo "bundling..." && bundle install --quiet && echo "done bundling" && DB=postgres bundle exec rake' -alias mybake='echo "bundling..." && bundle install --quiet && echo "done bundling" && DB=mysql bundle exec rake' - -# k8s -alias k=kubectl - -# Git aliases -alias gd='git diff' -alias gdc='git diff --cached' -alias gst='git status' - diff --git a/lib/claim-bosh-lite.bash b/lib/claim-bosh-lite.bash index ac0be7f..2acf911 100755 --- a/lib/claim-bosh-lite.bash +++ b/lib/claim-bosh-lite.bash @@ -1,71 +1,12 @@ env_pool="$HOME/workspace/capi-env-pool" -readonly LASTPASS_API_KEY_LABEL='tracker_api_token' - -# TODO: have this script choose Tracker ID like our "story" script -project_id=1945579; - -function fetch_tracker_api_token() { - set +x - API_TOKEN=$(lpass show "$LASTPASS_API_KEY_LABEL" --notes 2> /dev/null) - return $? -} - -function get_stories() { - if [ -z "$API_TOKEN" ]; then - story_json="$(curl -s -H "Content-Type: application/json" \ - "https://www.pivotaltracker.com/services/v5/projects/$project_id/stories?with_state=started" | tr '**' ' ')" - else - story_json="$(curl -s -H "Content-Type: application/json" -H "X-TrackerToken: $API_TOKEN" \ - "https://www.pivotaltracker.com/services/v5/projects/$project_id/stories?with_state=started" | tr '**' ' ')" - fi - echo "$story_json" | tr '\n' ' ' | jq -r '.[] | "#\(.id) \(.name)"' - echo "No story" -} - -function story_selector() { - stories="$(get_stories | sed 's/#//g')" - - PS3="Select a story or 'q' to quit: " - OLD_IFS=$IFS - IFS=$'\n' - select story in $stories; do - echo $story - break - done - IFS=$OLD_IFS -} function claim_bosh_lite() { - git_authors=$(git config --get git-together.active) - if [ -z "$git_authors" ]; then - echo "please set your git authors before running this!" - echo "maybe run 'story' while you're at it." - return - fi - - fetch_tracker_api_token - if [ -z "$API_TOKEN" ]; then - echo "Could not find tracker API token in your lastpass vault as a note under \"$LASTPASS_API_KEY_LABEL\"" - echo -e 'Attempting to pull stories without token \n' - fi - - if [ -n "$1" ] ; then - STORY=$1 - shift - else - echo "Please select a story for your bosh lite" - - STORY=$(story_selector) - fi - - if [ -z "${STORY}" ]; then - echo "Canceling bosh lite claim. Goodbye." + git_user=$(git config --get user.name) + if [ -z "$git_user" ]; then + echo "Please set your git user before running this!" return fi - STORY=$(echo $STORY | cut -c 1-20)... - echo Selected story \"${STORY}\" - env_dir=$( set -e @@ -83,7 +24,7 @@ function claim_bosh_lite() { for f in $(ls -tr ./bosh-lites/unclaimed/*); do test -f "$f" || continue - msg "Claiming $( basename $f )..." + msg "Claiming '$( basename $f )'..." claim_specific_environment $(basename $f) return $? done @@ -177,10 +118,7 @@ echo -e "${green}\n## Target UAA API, login as uaa admin ##${nc}" echo "target_uaa" echo -e "${green}\n## Retrieve CF admin password ##${nc}" -echo 'credhub login --skip-tls-validation' -echo "credhub get --name '/bosh-lite/cf/cf_admin_password' --output-json | jq -r '.value'" -#TODO: Get this variant working: -#echo 'CF_PASSWORD=$(credhub get --name "/bosh-lite/cf/cf_admin_password" --output-json | jq -r ".value" | tee /dev/tty)' +echo "cf_admin_password" echo -e "${green}\n## Unclaim this environment ##${nc}" echo "unclaim_bosh_lite ${env_name}" @@ -197,7 +135,7 @@ EOF } function commit_and_push() { - git ci --quiet --message "manually claim ${env} on ${HOSTNAME} [$STORY]" --no-verify + git commit --quiet --message "manually claim ${env} on $( hostname )" --no-verify msg "Pushing reservation to $( basename $PWD )..." git push --quiet } @@ -213,9 +151,7 @@ EOF ) if [[ "$?" == 0 ]]; then - direnv allow "${env_dir}" - echo "Changing directory to '${env_dir}'..." - cd "${env_dir}" + echo "Use 'target_bosh $( basename $env_dir )' to set bosh environment variables." fi } diff --git a/lib/environment-variables.bash b/lib/environment-variables.bash deleted file mode 100644 index 9b3ca40..0000000 --- a/lib/environment-variables.bash +++ /dev/null @@ -1,24 +0,0 @@ -export BOSH_SHA2=true - -# Set nvim as default editor -export GIT_EDITOR=nvim -export EDITOR=nvim - -# Setup GOPATH -export GOPATH=$HOME/go -export PATH=$PATH:$GOPATH/bin - -# Add capi-workspace/scripts to PATH -export PATH="$PATH:$HOME/workspace/capi-workspace/bin" - - -# Add CLI out directory to PATH -export PATH="$PATH:$HOME/go/src/code.cloudfoundry.org/cli/out" -# Add brew-installed python to the path. -# Must go at the beginning so that it's prioritized over the system installation. -export PATH="/usr/local/opt/python/libexec/bin:$PATH" - -# specify locale so vim will stop shouting -export LC_ALL=en_US.UTF-8 - -launchctl setenv PATH $PATH \ No newline at end of file diff --git a/lib/misc.bash b/lib/misc.bash index 1d3f457..6fbeb1c 100644 --- a/lib/misc.bash +++ b/lib/misc.bash @@ -5,17 +5,6 @@ # Luan's vim config, where they can put their custom bits # ------------------------------------------------------------- -# capi likes `alias g="git status"`, other teams like `alias g=git`, so do both: -alias g >/dev/null 2>&1 && unalias g - -function g() { - case $# in - 0) git status ;; - *) git "$@" ;; - esac -} - -# Misc aliases alias cfu="seed_users" alias roundup_bosh_lites="print_env_info" alias bosh_lites="print_env_info" @@ -33,22 +22,5 @@ function cf_auth_config() { function int() { export CF_INT_API=https://api.$BOSH_LITE_DOMAIN - - credhub login --skip-tls-validation export CF_INT_PASSWORD=$(credhub get --name '/bosh-lite/cf/cf_admin_password' --output-json | jq -r '.value') } - -function let_me_pull() { - local ssh_url="$(git remote get-url --push origin)" - local https_url="$(echo $ssh_url | awk '{gsub(/git@github.com:/,"https://github.com/")}1')" - if ! echo ${ssh_url} | grep 'git@' > /dev/null; then - echo 'push url doesnt seem to be ssh... exiting without changing anything' - return - fi - - echo 'setting fetch to https and push to ssh' - git remote set-url origin "${https_url}" - git remote set-url origin --push "${ssh_url}" - git remote -v - echo 'success!' -} diff --git a/lib/target-bosh.bash b/lib/target-bosh.bash index 5aece9d..c2b78cc 100755 --- a/lib/target-bosh.bash +++ b/lib/target-bosh.bash @@ -1,13 +1,10 @@ #!/usr/bin/env bash target_bosh() { - # tput setaf 1 = red - # tput setaf 2 = green - # tput setaf 9 = reset color - - local env_pool="$HOME/workspace/capi-env-pool" + local env_pool="${CAPI_ENV_POOL_DIR:-$HOME/workspace/capi-env-pool}" local claimed_dir="$env_pool/bosh-lites/claimed" + printf "\nRefreshing bosh lite pool state...\n" pushd ${claimed_dir} >/dev/null git pull popd >/dev/null @@ -19,25 +16,30 @@ target_bosh() { env_file=${claimed_dir}/${1} if [ -f "$env_file" ]; then + echo "\nSourcing ${env_file} to set bosh environment variables...\n" source "$env_file" - env_ssh_key_path="${env_pool}/${1}/bosh.pem" + env_integration_config_path="${env_pool}/${1}/integration_config.json" + if [ ! -f "${env_integration_config_path}" ]; then + echo "Writing a capi-specific integration_config.json..." + generate_integration_config > "${env_integration_config_path}" + echo -e "Generated ${env_integration_config_path}. Enjoy!\n" + fi + env_ssh_key_path="${env_pool}/${1}/bosh.pem" if [ ! -f "${env_ssh_key_path}" ]; then + echo "Writing a bosh gateway ssh key..." echo "${BOSH_GW_PRIVATE_KEY_CONTENTS}" > "${env_ssh_key_path}" chmod 0600 "${env_ssh_key_path}" + echo -e "Generated ${env_ssh_key_path}. Enjoy!\n" fi - if [ ! -f "${env_integration_config_path}" ]; then - echo "writing ${1} a capi-specific integration_config.json..." - generate_integration_config > "${env_integration_config_path}" - fi - - echo "$(tput setaf 2)Success!$(tput setaf 9)" - + printf "Setting BOSH_GW_PRIVATE_KEY, BOSH_ALL_PROXY, and CONFIG environment variables...\n" export BOSH_GW_PRIVATE_KEY="${env_ssh_key_path}" export BOSH_ALL_PROXY=ssh+socks5://${BOSH_GW_USER}@${BOSH_ENVIRONMENT}:22?private-key=${BOSH_GW_PRIVATE_KEY} export CONFIG="${env_integration_config_path}" + + echo "$(tput setaf 2)Success!$(tput setaf 9)" else echo "$(tput setaf 1)Environment '${1}' does not exist. Valid environments are:$(tput setaf 9)" ls ${claimed_dir} diff --git a/lib/unclaim-bosh-lite.bash b/lib/unclaim-bosh-lite.bash index ae4ba20..2eaebc5 100755 --- a/lib/unclaim-bosh-lite.bash +++ b/lib/unclaim-bosh-lite.bash @@ -1,7 +1,7 @@ function unclaim_bosh_lite() { - git_authors=$(git config --get git-together.active) - if [ -z "$git_authors" ]; then - echo "please set your git authors before running this!" + git_user=$(git config --get user.name) + if [ -z "$git_user" ]; then + echo "Please set your git user before running this!" return fi @@ -26,6 +26,7 @@ function unclaim_bosh_lite() { return 1 fi + echo "Refreshing bosh lite pool state..." git pull -n -r --quiet --no-verify function mark_broken { @@ -37,7 +38,7 @@ function unclaim_bosh_lite() { return 1 fi - printf "Hit enter to release ${env} " + printf "Hit enter to release '${env}' " read -r git mv "${file}" "${broken_pool}/unclaimed/" @@ -45,7 +46,7 @@ function unclaim_bosh_lite() { git rm -rf "${env}" && \rm -rf "${env}" fi - git ci --quiet -m"releasing $env on ${HOSTNAME} [nostory]" --no-verify + git commit --quiet -m"releasing $env on $( hostname )" --no-verify echo "Pushing the release commit to $( basename "$PWD" )..." git push --quiet } @@ -56,10 +57,11 @@ function unclaim_bosh_lite() { ) unset BOSH_CA_CERT BOSH_CLIENT BOSH_CLIENT_SECRET BOSH_ENVIRONMENT \ - BOSH_GW_USER BOSH_GW_HOST BOSH_LITE_DOMAIN BOSH_GW_PRIVATE_KEY_CONTENTS \ - BOSH_GW_PRIVATE_KEY + BOSH_DEPLOYMENT BOSH_GW_USER BOSH_GW_HOST BOSH_LITE_DOMAIN \ + BOSH_GW_PRIVATE_KEY_CONTENTS BOSH_GW_PRIVATE_KEY CONFIG BOSH_ALL_PROXY \ + CREDHUB_SERVER CREDHUB_CLIENT CREDHUB_SECRET CREDHUB_CA_CERT - echo "Done" + echo "Done!" } export -f unclaim_bosh_lite