Skip to content

Latest commit

 

History

History
180 lines (121 loc) · 5.65 KB

developing.md

File metadata and controls

180 lines (121 loc) · 5.65 KB
Raw

Developing Genny

This page describes the details of developing in Genny: How to run tests, debug, sanitizers, etc. For information on how to write actors and workloads, see Using Genny.

Commit Queue

Genny uses the Evergreen Commit-Queue. When you have received approval for your PR, simply comment evergreen merge and your PR will automatically be tested and merged.

Running Genny Self-Tests

These self-tests are exercised in CI. For the exact invocations that the CI tests use, see evergreen.yml. All genny commands now use ./run-genny. For a full list run ./run-genny --help.

Linters

Lint Python:

./run-genny lint-python # add --fix to fix any errors.

Lint Workload and other YAML:

./run-genny lint-yaml

C++ Unit-Tests

./run-genny cmake-test

Python Unit-Tests

./run-genny self-test

For more fine-tuned testing (eg. running a single test or excluding some) you can manually invoke the test binaries:

./build/src/gennylib/gennylib_test "My testcase"

Read more about what parameters you can pass here.

Benchmark Tests

The above self-test line also runs so-called "benchmark" tests. They can take a while to run and may fail occasionally on local developer machines, especially if you have an IDE or web browser open while the test runs.

If you want to run all the tests except perf tests you can manually invoke the test binaries and exclude perf tests:

# Build Genny first: `./scripts/lamp [...]`
./build/src/gennylib/gennylib_test '~[benchmark]'

Actor Integration Tests

The Actor tests use resmoke to set up a real MongoDB cluster and execute the test binary. The resmoke yaml config files that define the different cluster configurations are defined in src/resmokeconfig.

# Example:
./run-genny resmoke-test --suites src/resmokeconfig/genny_standalone.yml

Each yaml configuration file will only run tests that are associated with their specific tags. (Eg. genny_standalone.yml will only run tests that have been tagged with the [standalone] tag.)

When creating a new Actor, ./run-genny create-new-actor will generate a new test case template to ensure the new Actor can run against different MongoDB topologies, please update the template as needed so it uses the newly-created Actor.

Patch-Testing and Evergreen

When restarting any of Genny's Evergreen self-tests, make sure you restart all the tasks not just failed tasks. This is because Genny's tasks rely on being run in dependency-order on the same machine. Rescheduled tasks don't re-run dependent tasks.

See this page's section on Patch-Testing Genny Changes with Sys-Perf / DSI if you are writing a workload or making changes to more than just this repo.

Debugging

IDEs can debug Genny if it is built with the Debug build type:

./run-genny install -- -DCMAKE_BUILD_TYPE=Debug

Genny has a verbose output mode:

./run-genny workload -v debug

Actor information can be printed for additional help, eg: when an actor is about to begin or when it has ended

Code Style and Limitations

Don't get cute.

Please see CONTRIBUTING.md for code-style etc. Note that we're pretty vanilla.

Generating Doxygen Documentation

We use Doxygen with a configuration that relies on llvm for its parsing and graphviz for generating diagrams. As such you must compile Doxygen with the appropriate support for these:

brew install graphviz
brew install doxygen --with-llvm --with-graphviz

Then generate and open Doxygen docs with the following:

doxygen .doxygen
open build/docs/html/index.html

Sanitizers

Genny is periodically manually tested to be free of unknown sanitizer errors. These are not currently run in a CI job. If you are adding complicated code and are afraid of undefined behavior or data-races etc, you can run the clang sanitizers yourself easily.

Run ./run-genny install --help for information on what sanitizers there are.

To run with ASAN:

./run-genny install --build-system make --sanitizer asan
./run-genny self-test
# Pick a workload YAML that uses your Actor below
ASAN_OPTIONS="detect_container_overflow=0" ./run-genny workload -- run ./src/workloads/docs/HelloWorld.yml

The toolchain isn't instrumented with sanitizers, so you may get false-positives for Boost, hence the ASAN_OPTIONS flag.

Updating canned artifacts for re-smoke tests

The canned artifacts used for re-smoke tests can get outdated over time and will be required to update periodically. Here is how to update the canned artifacts.

  1. Download the latest artifacts and upload them to the S3 bucket.
    • Download the tgz package for a specific platform
    • The downloaded package should be uploaded to S3 bucket - s3://dsi-donot-remove/compile_artifacts/. Make sure to make the uploaded file public.
  2. Update run_tests.py with new S3 URLs
    • CANNED_ARTIFACTS needs to be updated with the new S3 URL.