Skip to content

Commit

Permalink
Merge pull request #753 from AlexsLemonade/kurtwheeler/improve-readme
Browse files Browse the repository at this point in the history
[WIP] Improve readme and many scripts.
  • Loading branch information
kurtwheeler authored Oct 26, 2018
2 parents a9b9391 + 6754925 commit 3374c06
Show file tree
Hide file tree
Showing 6 changed files with 307 additions and 158 deletions.
2 changes: 1 addition & 1 deletion .circleci/run_terraform.sh
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ else
fi

# New deployment
TF_VAR_user=circleci TF_VAR_stage=$ENVIRONMENT ./deploy.sh -e $ENVIRONMENT -v $CIRCLE_TAG
./deploy.sh -e $ENVIRONMENT -v $CIRCLE_TAG -u circleci
exit_code=$?

exit $exit_code
247 changes: 124 additions & 123 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,13 @@ Refine.bio currently has four sub-projects contained within this repo:
- [Virtual Environment](#virtual-environment)
- [Common Dependecies](#common-dependecies)
- [Services](#services)
- [Nomad](#nomad)
- [Postgres](#postgres)
- [Nomad](#nomad)
- [Testing](#testing)
- [Development Helpers](#development-helpers)
- [API](#api)
- [Common](#common)
- [Foreman](#foreman)
- [Workers](#workers)
- [Style](#style)
- [Gotchas](#gotchas)
- [Running Locally](#running-locally)
Expand All @@ -41,19 +44,16 @@ Refine.bio currently has four sub-projects contained within this repo:
- [Processor Jobs](#processor-jobs)
- [Creating Quantile Normalization Reference Targets](#creating-quantile-normalization-reference-targets)
- [Checking on Local Jobs](#checking-on-local-jobs)
- [Testing](#testing-1)
- [API](#api)
- [Common](#common)
- [Foreman](#foreman)
- [Workers](#workers)
- [Development Helpers](#development-helpers-1)
- [Style](#style-1)
- [Production Deployment](#production-deployment)
- [Development Helpers](#development-helpers)
- [Cloud Deployment](#cloud-deployment)
- [Terraform](#terraform)
- [Docker Images](#docker-images)
- [Autoscaling and Setting Spot Prices](#autoscaling-and-setting-spot-prices)
- [Running Jobs](#running-jobs)
- [Log Consumption](#log-consumption)
- [Tearing Down](#tearing-down)
- [Support](#support)
- [Meta-README](#meta-readme)
- [License](#license)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->
Expand Down Expand Up @@ -81,7 +81,7 @@ unsupported by this project.
#### Linux

The following services will need to be installed:
- [Python3 and Pip]: `sudo apt-get -y install python3-pip`
- Python3 and Pip: `sudo apt-get -y install python3-pip`
- [Docker](https://www.docker.com/community-edition): Be sure to follow the
[post installation steps]
(https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user)
Expand All @@ -92,7 +92,7 @@ so Docker does not need sudo permissions.
- [jq](https://stedolan.github.io/jq/)
- [iproute2](https://wiki.linuxfoundation.org/networking/iproute2)

Instructions for installing Docker and Nomad can be found by
Instructions for installing Docker, Terraform, and Nomad can be found by
following the link for each service. git-crypt, jq, and iproute2 can be installed via
`sudo apt-get install git-crypt jq iproute2`.

Expand All @@ -112,11 +112,11 @@ This sets pip to install all packages in your user directory so sudo is not requ
The following services will need to be installed:
- [Docker for Mac](https://www.docker.com/docker-mac)
- [Terraform](https://www.terraform.io/)
- [Nomad](https://www.nomadproject.io/docs/install/index.html#precompiled-binaries)
- [Homebrew](https://brew.sh/)
- [git-crypt](https://www.agwa.name/projects/git-crypt/)
- [iproute2mac](https://github.com/brona/iproute2mac)
- [jq](https://stedolan.github.io/jq/)
- [Nomad](https://www.nomadproject.io/docs/install/index.html#precompiled-binaries) This may not work for Mac. [Citation Needed]

Instructions for installing Docker, Nomad, and Homebrew can be found by
following the link for those services. The others on that list can
Expand Down Expand Up @@ -152,25 +152,6 @@ command:
run in a local Docker container, but Nomad must be run on your
development machine.

##### Nomad

Similarly, you will need to run a local
[Nomad](https://www.nomadproject.io/) service in development
mode. Assuming you have followed the [installation instructions](#installation), you
can do so with:

```bash
sudo -E ./run_nomad.sh
```

(_Note:_ This step may take some time because it downloads lots of files.)

Nomad is an orchestration tool which Refine.bio uses to run
`Downloader` and `Processor` jobs. Jobs are queued by sending a message to
the Nomad agent, which will then launch a Docker container which runs
the job. If address conflicts emerge, old Docker containers can be purged
with `docker container prune -f`.

##### Postgres

To start a local Postgres server in a Docker container, use:
Expand Down Expand Up @@ -201,6 +182,32 @@ If you need to access a `psql` shell for inspecting the database, you can use:
./run_psql_shell.sh
```

or if you have `psql` installed this command will give you a better shell experience:

```
source common.sh && PGPASSWORD=mysecretpassword psql -h $(get_docker_db_ip_address) -U postgres -d data_refinery
```

##### Nomad

Similarly, you will need to run a local [Nomad](https://www.nomadproject.io/) service in development mode.
Unfortunately it seems that Nomad is not currently supporting using both Docker and Mac [Citation Needed].
However if you run Linux and you have followed the [installation instructions](#installation), you
can run Nomad with:

```bash
sudo -E ./run_nomad.sh
```

(_Note:_ This step may take some time because it downloads lots of files.)

Nomad is an orchestration tool which Refine.bio uses to run
`Surveyor`, `Downloader`, `Processor` jobs. Jobs are queued by sending a message to
the Nomad agent, which will then launch a Docker container which runs
the job. If address conflicts emerge, old Docker containers can be purged
with `docker container prune -f`.


### Testing

The end to end tests require a separate Nomad client to be running so
Expand All @@ -217,20 +224,58 @@ To run the entire test suite:
./run_all_tests.sh
```

(_Note:_ Running all the tests can take some time because it downloads a lot of files)
(_Note:_ Running all the tests can take some time, especially the first time because it downloads a lot of files.)

These tests will also be run continuosly for each commit via CircleCI.

### Development Helpers
For more granular testing, you can just run the tests for specific parts of the system.

It can be useful to have an interactive Python interpreter running within the
context of the Docker container. The `run_shell.sh` script has been provided
for this purpose. It is in the top level directory so that if you wish to
reference it in any integrations its location will be constant. However, it
is configured by default for the Foreman project. The interpreter will
have all the environment variables, dependencies, and Django configurations
for the Foreman project. There are instructions within the script describing
how to change this to another project.
#### API
To just run the API tests:

```bash
./api/run_tests.sh
```

#### Common
To just run the common tests:

```bash
./common/run_tests.sh
```

#### Foreman
To just run the foreman tests:

```bash
./foreman/run_tests.sh
```

#### Workers
To just run the workers tests:

```bash
./workers/run_tests.sh
```

If you only want to run tests with a specific tag, you can do that too. For
example, to run just the salmon tests:

```bash
./workers/run_tests.sh -t salmon
```

All of our worker tests are tagged, generally based on the Docker image required to run them.
Possible values for worker test tags are:
- affymetrix
- agilent
- downloaders
- illumina
- no_op
- qn (short for quantile normalization)
- salmon
- smasher
- transcriptome

### Style

Expand All @@ -251,13 +296,13 @@ linter.
During development, you make encounter some occasional strangeness. Here's
some things to watch out for:

- If builds are failing, increase the size of Docker's memory allocation.
- Since we use multiple Docker instances, don't forget to `./update_models`
- If builds are failing, increase the size of Docker's memory allocation. (Mac only.)
- If Docker images are failing mysteriously during creation, it may
be the result of Docker's `Docker.qcow2` or `Docker.raw` file filling. You
can prune old images with `docker system prune -a`.
- If it's killed abruptly, the containerized Postgres images can be
left in an unrecoverable state. Annoying.
- Since we use multiple Docker instances, don't forget to `./update_models`

## Running Locally

Expand Down Expand Up @@ -462,53 +507,6 @@ nomad logs -verbose b30e4edd
This command will output both the stderr and stdout logs from the container
which ran that allocation. The allocation is really a Refine.bio job.

### Testing

To run the entire test suite:

```bash
./run_all_tests.sh
```

These tests will also be run continuosly for each commit via CircleCI.

For more granular testing, you can just run the tests for specific parts of the system.

#### API
To just run the API tests:

```bash
./api/run_tests.sh
```

#### Common
To just run the common tests:

```bash
./common/run_tests.sh
```

#### Foreman
To just run the foreman tests:

```bash
./foreman/run_tests.sh
```

#### Workers
To just run the workers tests:

```bash
./workers/run_tests.sh
```

If you only want to run tests with a specific tag, you can do that too. For
example, to run just the salmon tests:

```bash
./workers/run_tests.sh -t salmon
```


### Development Helpers

Expand All @@ -521,39 +519,25 @@ have all the environment variables, dependencies, and Django configurations
for the Foreman project. There are instructions within the script describing
how to change this to another project.

### Style

R files in this repo follow
[Google's R Style Guide](https://google.github.io/styleguide/Rguide.xml).
Python Files in this repo follow
[PEP 8](https://www.python.org/dev/peps/pep-0008/). All files (including
Python and R) have a line length limit of 100 characters.

A `setup.cfg` file has been included in the root of this repo which specifies
the line length limit for the autopep8 and flake8 linters. If you run either
linter within the project's directory tree, it will enforce a line length limit
of 100 instead of 80. This will also be true for editors which rely on either
linter.

## Production Deployment
## Cloud Deployment

Refine.bio requires an active, credentialed AWS account with appropriate permissions to create network infrastructure, users, compute instances and databases.

### Terraform

Once you have Terraform installed and your AWS account credentials installed, you can plan your terraform deployment like so (from the `infrastructure` directory):
Once you have Terraform installed and your AWS account credentials installed, you're ready to deploy. The correct way to deploy to the cloud is by running the `deploy.sh` script. This script will perform additional
configuration steps, such as setting environment variables, setting up Nomad job specifications, and performing database migrations. It can be used from the `infrastructure` directory like so:

```bash
TF_VAR_user=myusername TF_VAR_stage=dev TF_VAR_region=us-east-1 terraform plan
./deploy.sh -u myusername -e dev -r us-east-1 -v v1.0.0 -d my-dockerhub-repo
```

If that worked fine, then to deploy:
This will spin up the whole system. It will usually take about 15 minutes, most of which is spent waiting for the Postgres instance to start.
The command above would spin up a development stack in the `us-east-1` region where all the resources' names would end with `-myusername-dev`.
All of the images used in that stack would come from `my-dockerhub-repo` and would be tagged with `v1.0.0`.

```bash
TF_VAR_user=myusername TF_VAR_stage=dev TF_VAR_region=us-east-1 ./deploy.sh
```
The `-e` specifies the environment you would like to spin up. You may specify, `dev`, `staging`, or `prod`. `dev` is meant for individuals to test infrastructure changes or to run large tests. `staging` is to test the overall system before re-deploying to `prod`.

This will spin up the whole system. It will usually take about 15 minutes, most of which is spent waiting for the Postgres instance to start.

To see what's been created at any time, you can:
```
Expand All @@ -563,21 +547,25 @@ terraform state list
If you want to change a single entity in the state, you can use

```
terraform taint <your-entity-from-state-list>; tf plan; tf apply;
terraform taint <your-entity-from-state-list>
```

To tear down the entire system:
And then rerun `deploy.sh` with the same parameters you originally ran it with.

```
terraform destroy
```

For convenience, a `deploy.sh` script is also provided, which will perform additional
steps to configure (such as setting up Nomad job specifications and performing database migrations) and prepare the entire system. It can be used simply (from the `infrastructure` directory), like so:
### Docker Images

Refine.bio uses a number of different Docker images to run different pieces of the system.
By default, refine.bio will pull images from the Dockerhub repo `ccdlstaging`.
If you would like to use images you have built yourself you can set the `TF_VAR_dockerhub_repo` variable to the name of the Dockerhub repo where you have pushed your images.

To make building and pushing your own images easier, the `update_my_docker_images.sh` has been provided.
The `-d` option will allow you to specify which repo you'd like to push to.
If the Dockerhub repo requires you to be logged in, you should do so before running the script using `docker login`.
The -v option allows you to specify the version, which will both end up on the Docker images you're building as the SYSTEM_VERSION environment variable and also will be the docker tag for the image.

`update_my_docker_images.sh` will not build the dr_affymetrix image, because this image requires a lot of resources and time to build.

```
./deploy.sh -e dev
```

### Autoscaling and Setting Spot Prices

Expand Down Expand Up @@ -660,13 +648,26 @@ awslogs get data-refinery-log-group-myusername-dev log-stream-api-nginx-access-*

will show all of the API access logs made by Nginx.


### Tearing Down

A stack that has been spun up via `deploy.sh -u myusername -e dev` can be taken down with `destroy_terraform.sh -u myusername -e dev`.
The same username and environment must be passed into `destroy_terraform.sh` as were used to run `deploy.sh` either via the -e and -u options or by specifying `TF_VAR_stage` or `TF_VAR_user` so that the script knows which to take down.
Note that this will prompt you for confirmation before actually destroying all of your cloud resources.

## Support

Refine.bio is supported by
[Alex's Lemonade Stand Foundation](https://www.alexslemonade.org/),
with some initial development supported by the Gordon and Betty Moore
Foundation via GBMF 4552 to Casey Greene.

## Meta-README

The table of contents for this README is generated using `doctoc`.
`doctoc` can be installed with: `sudo npm install -g doctoc`
Once `doctoc` is installed the table of contents can be re-generated with: `doctoc README.md`

## License

BSD 3-Clause License.
Loading

0 comments on commit 3374c06

Please sign in to comment.