Some extra developer conveniences for Docker image. (envoyproxy#437)

* Some extra developer conveniences for Docker image.

* Refactor do_ci.sh to split the cmake and environment setup from the
  build and test. This makes it convenient for devs to invoke arbitrary
  make targets in the Docker image, e.g.

  docker run -t -i -v $PWD:/source lyft/envoy-build:latest /bin/bash -c \
    "cd /source && . ./ci/build_setup.sh && make fix_format"

* Add EXTRA_TEST_ARGS env var to pass addition flags to envoy-test. This
  is useful, for example, when applying a --gtest_filter to restrict the
  test run to only failing tests.

* Add a do_ci.sh fix_format target.

* Add gdb, strace to the image and a RUN_TEST_UNDER environment variable
  that can be used to have envoy-test run under these programs.

* UNIT_TEST_ONLY env var that will skip the non-envoy-test aspects of
  the test process.

Example command for debugging only tests matching *Dns*:

docker run -t -i -v $PWD:/source lyft/envoy-build:latest /bin/bash -c \
  "cd /source && UNIT_TEST_ONLY=1 RUN_TEST_UNDER='gdb --args' ./ci/do_ci.sh debug \
  --gtest_filter='*Dns*'

ci/README.md

@@ -0,0 +1,44 @@
+# Developer use of CI Docker image
+
+The Docker image at `lyft/envoy-build:<hash>` is used for Travis CI checks, where `<hash>` is specified in
+[.travis.yml](https://github.com/lyft/envoy/blob/master/.travis.yml). Developers
+may work with `lyft/envoy-build:latest` to provide a self-contained environment for
+building Envoy binaries and running tests that reflects the latest built image.
+
+An example basic invocation to build a debug image and run all tests is:
+
+  `docker run -t -i -u $(id -u):$(id -g) -v <SOURCE_DIR>:/source lyft/envoy-build:latest /bin/bash -c "cd /source && ci/do_ci.sh debug"`
+
+This bind mounts `<SOURCE_DIR>`, which allows for changes on the local
+filesystem to be consumed and outputs build artifacts in `<SOURCE_DIR>/build`.
+The static Envoy binary can be found in `<SOURCE_DIR>/build/source/exe/envoy`.
+
+The `do_ci.sh` targets are:
+
+* `asan` &mdash; build and run tests with [AddressSanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizer).
+* `coverage` &mdash; build and run tests, generating coverage information in `<SOURCE_DIR>/build/coverage.html`.
+* `debug` &mdash; build debug binary and run tests.
+* `docs` &mdash; build documentation, generated docs are found in `<SOURCE_DIR>/generated`.
+* `fix_format`&mdash; run `clang-format` 3.6 on entire source tree.
+* `normal` &mdash; build unstripped optimized binary and run tests .
+* `server_only` &mdash; build stripped optimized binary only.
+
+A convenient shell function to define is:
+
+  `run_envoy_docker() { docker run -t -i -u $(id -u):$(id -g) -v $PWD:/source lyft/envoy-build:latest /bin/bash -c "cd /source && $*";}`
+
+This then allows for a simple invocation of `run_envoy_docker './ci/do_ci.sh debug'` from the
+Envoy source tree.
+
+## Advanced developer features
+
+* Any parameters following the `do_ci.sh` target are passed in as command-line
+  arguments to the `envoy-test` binary during unit test execution. This allows
+  for [GTest](https://github.com/google/googletest) flags to be passed, e.g.
+  `run_envoy_docker './ci/do_ci.sh debug --gtest_filter="*Dns*"'`.
+
+* A `UNIT_TEST_ONLY` environment variable is available to control test execution to limit testing to
+  just unit tests, e.g. `run_envoy_docker 'UNIT_TEST_ONLY=1 ./ci/do_ci.sh debug --gtest_filter="*Dns*"'`.
+
+* A `RUN_TEST_UNDER` environment variable is available to specify an executable to run the tests
+  under. For example, to run a subset of tests under `gdb`: `run_envoy_docker 'RUN_TEST_UNDER="gdb --args" UNIT_TEST_ONLY=1 ./ci/do_ci.sh debug --gtest_filter="*Dns*"'`.

ci/build_container/build_container.sh

@@ -6,6 +6,8 @@ set -e
 apt-get update
 apt-get install -y wget software-properties-common make cmake git python python-pip clang-format-3.6 bc
 apt-get install -y golang
+# For debugging.
+apt-get install -y gdb strace
 add-apt-repository -y ppa:ubuntu-toolchain-r/test
 apt-get update
 apt-get install -y g++-4.9

ci/build_setup.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+
+# Configure environment variables, generate makefiles and switch to build
+# directory in preparation for an invocation of the generated build makefiles.
+
+set -e
+
+export CC=gcc-4.9
+export CXX=g++-4.9
+export HEAPCHECK=normal
+export PPROF_PATH=/thirdparty_build/bin/pprof
+
+NUM_CPUS=`grep -c ^processor /proc/cpuinfo`
+
+if [[ "$1" == "coverage" ]]; then
+  EXTRA_CMAKE_FLAGS="-DENVOY_CODE_COVERAGE:BOOL=ON"
+elif [[ "$1" == "asan" ]]; then
+  EXTRA_CMAKE_FLAGS="-DENVOY_SANITIZE:BOOL=ON -DENVOY_DEBUG:BOOL=OFF"
+elif [[ "$1" == "debug" ]]; then
+  EXTRA_CMAKE_FLAGS="-DENVOY_DEBUG:BOOL=ON"
+elif [[ "$1" == "server_only" ]]; then
+  EXTRA_CMAKE_FLAGS="-DENVOY_DEBUG:BOOL=OFF -DENVOY_STRIP:BOOL=ON"
+else
+  EXTRA_CMAKE_FLAGS="-DENVOY_DEBUG:BOOL=OFF"
+fi
+
+mkdir -p build
+cd build
+
+cmake \
+$EXTRA_CMAKE_FLAGS \
+-DENVOY_COTIRE_MODULE_DIR:FILEPATH=/thirdparty/cotire-cotire-1.7.8/CMake \
+-DENVOY_GMOCK_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
+-DENVOY_GPERFTOOLS_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
+-DENVOY_GTEST_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
+-DENVOY_HTTP_PARSER_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
+-DENVOY_LIBEVENT_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
+-DENVOY_NGHTTP2_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
+-DENVOY_SPDLOG_INCLUDE_DIR:FILEPATH=/thirdparty/spdlog-0.11.0/include \
+-DENVOY_TCLAP_INCLUDE_DIR:FILEPATH=/thirdparty/tclap-1.2.1/include \
+-DENVOY_OPENSSL_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
+-DENVOY_LIGHTSTEP_TRACER_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
+-DENVOY_PROTOBUF_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
+-DENVOY_PROTOBUF_PROTOC:FILEPATH=/thirdparty_build/bin/protoc \
+-DENVOY_GCOVR:FILEPATH=/thirdparty/gcovr-3.3/scripts/gcovr \
+-DENVOY_RAPIDJSON_INCLUDE_DIR:FILEPATH=/thirdparty/rapidjson-1.1.0/include \
+-DENVOY_GCOVR_EXTRA_ARGS:STRING="-e test/* -e build/*" \
+-DENVOY_EXE_EXTRA_LINKER_FLAGS:STRING=-L/thirdparty_build/lib \
+-DENVOY_TEST_EXTRA_LINKER_FLAGS:STRING=-L/thirdparty_build/lib \
+..
+
+cmake -L || true

ci/do_ci.sh

@@ -1,67 +1,42 @@
 #!/bin/bash
 
-set -e
+# Run a CI build/test target, e.g. docs, asan.
 
-export CC=gcc-4.9
-export CXX=g++-4.9
-export HEAPCHECK=normal
-export PPROF_PATH=/thirdparty_build/bin/pprof
+set -e
 
-NUM_CPUS=`grep -c ^processor /proc/cpuinfo`
 echo "building using $NUM_CPUS CPUs"
 
 if [[ "$1" == "docs" ]]; then
   echo "docs build..."
   make docs
   exit 0
+fi
+
+. "$(dirname "$0")"/build_setup.sh
+
+if [[ "$1" == "fix_format" ]]; then
+  echo "fix_format..."
+  make fix_format
+  exit 0
 elif [[ "$1" == "coverage" ]]; then
   echo "coverage build with tests..."
-  EXTRA_CMAKE_FLAGS="-DENVOY_CODE_COVERAGE:BOOL=ON"
   TEST_TARGET="envoy.check-coverage"
 elif [[ "$1" == "asan" ]]; then
   echo "asan build with tests..."
-  EXTRA_CMAKE_FLAGS="-DENVOY_SANITIZE:BOOL=ON -DENVOY_DEBUG:BOOL=OFF"
   TEST_TARGET="envoy.check"
 elif [[ "$1" == "debug" ]]; then
   echo "debug build with tests..."
-  EXTRA_CMAKE_FLAGS="-DENVOY_DEBUG:BOOL=ON"
   TEST_TARGET="envoy.check"
 elif [[ "$1" == "server_only" ]]; then
   echo "normal build server only..."
-  EXTRA_CMAKE_FLAGS="-DENVOY_DEBUG:BOOL=OFF -DENVOY_STRIP:BOOL=ON"
   TEST_TARGET="envoy"
 else
   echo "normal build with tests..."
-  EXTRA_CMAKE_FLAGS="-DENVOY_DEBUG:BOOL=OFF"
   TEST_TARGET="envoy.check"
 fi
 
-mkdir -p build
-cd build
-
-cmake \
-$EXTRA_CMAKE_FLAGS \
--DENVOY_COTIRE_MODULE_DIR:FILEPATH=/thirdparty/cotire-cotire-1.7.8/CMake \
--DENVOY_GMOCK_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
--DENVOY_GPERFTOOLS_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
--DENVOY_GTEST_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
--DENVOY_HTTP_PARSER_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
--DENVOY_LIBEVENT_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
--DENVOY_NGHTTP2_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
--DENVOY_SPDLOG_INCLUDE_DIR:FILEPATH=/thirdparty/spdlog-0.11.0/include \
--DENVOY_TCLAP_INCLUDE_DIR:FILEPATH=/thirdparty/tclap-1.2.1/include \
--DENVOY_OPENSSL_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
--DENVOY_LIGHTSTEP_TRACER_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
--DENVOY_PROTOBUF_INCLUDE_DIR:FILEPATH=/thirdparty_build/include \
--DENVOY_PROTOBUF_PROTOC:FILEPATH=/thirdparty_build/bin/protoc \
--DENVOY_GCOVR:FILEPATH=/thirdparty/gcovr-3.3/scripts/gcovr \
--DENVOY_RAPIDJSON_INCLUDE_DIR:FILEPATH=/thirdparty/rapidjson-1.1.0/include \
--DENVOY_GCOVR_EXTRA_ARGS:STRING="-e test/* -e build/*" \
--DENVOY_EXE_EXTRA_LINKER_FLAGS:STRING=-L/thirdparty_build/lib \
--DENVOY_TEST_EXTRA_LINKER_FLAGS:STRING=-L/thirdparty_build/lib \
-..
-
-cmake -L || true
+shift
+export EXTRA_TEST_ARGS="$@"
 
 make check_format
 make -j$NUM_CPUS $TEST_TARGET

docs/install/building.rst

@@ -3,13 +3,7 @@ Building
 
 The Envoy build system uses cmake. In order to ease initial building and for a quick start, we
 provide an Ubuntu 14 based docker container that has everything needed inside of it to build
-and *statically link* envoy. The following command will build the server.
-
-.. code-block:: console
-
-  docker run -t -i -v <SOURCE_DIR>:/source lyft/envoy-build:latest /bin/bash -c "cd /source && ci/do_ci.sh server_only"
-
-See :repo:`ci/do_ci.sh` for other possible targets (to run tests, etc.).
+and *statically link* envoy, see :repo:`ci/README.md`.
 
 In order to build manually, cmake is used like so:
 

test/run_envoy_tests.sh

@@ -41,7 +41,12 @@ openssl x509 -req -days -365 -in $TEST_CERT_DIR/unittestcert_expired.csr -sha256
 
 # First run the normal unit test suite
 cd $SOURCE_DIR
-$BINARY_DIR/test/envoy-test
+$RUN_TEST_UNDER $BINARY_DIR/test/envoy-test $EXTRA_TEST_ARGS
+
+if [ "$UNIT_TEST_ONLY" = "1" ]
+then
+  exit 0
+fi
 
 # Now start the real server, hot restart it twice, and shut it all down as a basic hot restart
 # sanity test.