Skip to content

Commit

Permalink
Merge pull request #1781 from newrelic/super_easy_magic_test_commands
Browse files Browse the repository at this point in the history
Add test running script and update test readme
  • Loading branch information
tannalynn authored Feb 7, 2023
2 parents dbaea42 + 968a543 commit a5ff73e
Show file tree
Hide file tree
Showing 2 changed files with 270 additions and 6 deletions.
89 changes: 83 additions & 6 deletions test/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,38 @@ We use Minitest for the Ruby agent. The following command runs the unit tests w

bundle exec rake test

## Running Specific Tests

You can also run a single unit test file like this:
## Super Easy Testing Setup

bundle exec ruby test/new_relic/agent_test.rb
There is a script in test/script/run_tests that makes it easy to run the unit, env, and multiverse tests.

And to run a single test within that file (note that when using the -n argument, you can either supply the entire test name as a string or a partial name as a regex):
Examples

./test/script/run_tests -u # unit tests
./test/script/run_tests -e 61 # env tests running rails61
./test/script/run_tests -m rake # multiverse tests running rake in all envs and prepend/chain
./test/script/run_tests -q rake # multiverse tests running rake only env 0 and prepend

If you set up shell alias for these, it will make it super convenient to run

add these aliases in ~/.bash_profile (or ~/.zshenv, depending on what shell you use)

alias bert="./test/script/run_tests -u"
alias bere="./test/script/run_tests -e"
alias berm="./test/script/run_tests -m"
alias bermq="./test/script/run_tests -q"
alias ber="./test/script/run_tests"

Also, if you set it to the full path instead of relative, then you can run it from anywhere and it will work in wherever you are (for example, if you've `cd`-ed into infinite_tracing and run `bert` it will run the infinite tracing tests if you're using the full path, but error if its relative)

Then you'll be able to run the tests super easy like

bert # run all unit tests
bere 61 # run env tests for rails 6.1
berm rake # run all rake multiverse suites
bermq rake # run multiverse rake env=0 method=prepend
ber -h # explains all the args for each option

bundle exec ruby test/new_relic/agent_test.rb -n /test_shutdown/

## Running Tests in a Rails Environment

Expand All @@ -26,7 +49,61 @@ In CI, these unit tests are run against all supported major.minor versions of Ra

bundle exec rake test:env

### Running Specific Tests


## Running Specific Tests

These env variables work for both the unit tests and env tests.

Running a specific test file

TEST="path/to/test_file_you_want_to_run.rb" bundle exec rake test


Running a specific test by name

TESTOPTS="--name=test_name_of_test_to_run" bundle exec rake test


You can also specify both

TEST="path/to/test_file_you_want_to_run.rb" TESTOPTS="--name=test_name_of_test_to_run" bundle exec rake test


## Specify a seed

If you're running into intermittent failures that seem to be related to the order tests are run in, you can specify a seed to the randomization

TESTOPTS="--seed=12345" bundle exec rake test


## Multiverse

To run a multiverse suite

bundle exec rake test:multiverse[suite_name]

More detailed multiverse information available in the [multiverse readme](./multiverse/README.md)





## Other Ways To Do Things

### Unit Tests Only - Specify File and Test Name
This doesn't work for the env tests, but this is also an option when running just the unit tests.

You can also run a single unit test file like this:

bundle exec ruby test/new_relic/agent_test.rb

And to run a single test within that file (note that when using the -n argument, you can either supply the entire test name as a string or a partial name as a regex):

bundle exec ruby test/new_relic/agent_test.rb -n /test_shutdown/


### Env Tests Only - Specify File

The file environment variable can be added to the test:env invocation to run a specific unit file. It can be exact file name, or a wildcard pattern. Multiple file patterns can be specified by separating with a comma with no spaces surrounding:

Expand Down
187 changes: 187 additions & 0 deletions test/script/run_tests
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
#!/bin/bash

help() {
echo "Runs the unit, env, or multiverse tests"
echo
echo "Syntax: run_tests [-u|e|m|q] arg1 arg2 arg3"
echo " options:"
echo " -u Runs unit tests"
echo " optional arg 1: file_path OR test_name"
echo " optional arg 2: test_name (if also passing in file_path in arg 1)"
echo " Examples: "
echo " ./test/script/run_tests -u"
echo " ./test/script/run_tests -u test_name"
echo " ./test/script/run_tests -u path/to/test_file.rb"
echo " ./test/script/run_tests -u path/to/test_file.rb test_name"
echo " "
echo " -e Runs env tests"
echo " arg 1: rails version to test. The leading 'rails' can be left off. 61,70 works like rails61,rails70"
echo " optional arg 2: file_path OR test_name"
echo " optional arg 3: test_name (if also passing in file_path in arg 2)"
echo " Examples: "
echo " ./test/script/run_tests -e 61"
echo " ./test/script/run_tests -e 61,71 test_name"
echo " ./test/script/run_tests -e rails61 path/to/test_file.rb"
echo " ./test/script/run_tests -e rails61,rails71 path/to/test_file.rb test_name"
echo " "
echo " -m Runs multiverse tests"
echo " arg 1: multiverse suite to run (or any other args you wanna pass in to multiverse)"
echo " the 'debug' multiverse arg is automatically included"
echo " optional arg 2: file_path OR test_name"
echo " optional arg 3: test_name (if also passing in file_path in arg 2)"
echo " Examples: "
echo " ./test/script/run_tests -m rake"
echo " ./test/script/run_tests -m rake,env=2"
echo " ./test/script/run_tests -m rake test_name"
echo " ./test/script/run_tests -m rake path/to/test_file.rb"
echo " ./test/script/run_tests -m rake path/to/test_file.rb test_name"
echo " "
echo " -q Runs quick multiverse tests. Prepend method only and env 0, unless env number is passed in"
echo " arg 1: multiverse suite to run (or any other args you wanna pass in to multiverse)"
echo " the 'debug' multiverse arg is automatically included"
echo " optional arg 2: env# OR file_path OR test_name"
echo " optional arg 3: file_path OR test_name (if also passing in env # in arg 2)"
echo " optional arg 4: test_name (if also passing in file_path in arg 3)"
echo " Examples: "
echo " ./test/script/run_tests -m rake"
echo " ./test/script/run_tests -m rake 2"
echo " ./test/script/run_tests -m rake test_name"
echo " ./test/script/run_tests -m rake path/to/test_file.rb"
echo " ./test/script/run_tests -m rake path/to/test_file.rb test_name"
echo " "
}


unit_command() {
# echo "ENV TEST="$TEST" TESTOPTS="$TESTOPTS""
# echo "bundle exec rake test -q;"
bundle exec rake test -q;
}

env_test_command() {
# echo "ENV TEST="$TEST" TESTOPTS="$TESTOPTS""
# echo "bundle exec rake test:env["$@"];"
bundle exec rake test:env["$@"];
}

multiverse_command() {
# echo "ENV TEST="$TEST" TESTOPTS="$TESTOPTS""
# echo "bundle exec rake test:multiverse["$@",debug];"
bundle exec rake test:multiverse["$@",debug];
}

# doesn't overwrite testopts that already exist,
# that way you can still pass in a seed if you want
set_test_opts() {
test_opts_result=""$TESTOPTS" --name="$1""
}

# organizes the args for mutiverse and calls the command
run_multiverse() {
if [[ -n "$3" ]]; then
# echo "running file and name"
multiverse_command "$1",file="$2",name="$3";
elif [[ -n "$2" && "$2" =~ ^test_ ]]; then
# echo "running name"
multiverse_command "$1",name="$2";
elif [[ -n "$2" ]]; then
# echo "running file"
multiverse_command "$1",file="$2";
else
# echo "running suite $1"
multiverse_command "$1";
fi
}

# calls multiverse but only env 0 or specified and method prepend
run_multiverse_quick() {
if [[ "$2" =~ [0-9]+ && ! "$2" =~ ^test ]]; then
run_multiverse "$1",env="$2",method=prepend "$3" "$4"
else
run_multiverse "$1",env=0,method=prepend "$2" "$3"
fi
}

# organizes the args for env tests and calls the command
run_env_tests() {
# this will add "rails" before any number so you can pass in just "61" or "61,70"
ENVARGS=$(echo "$1" | sed -E -e 's/(\,)|(\,rails)/,rails/g' | sed '/^rails/!s/^/rails/');

if [[ -n "$3" ]]; then
# echo "running file and name"
set_test_opts $3
TEST="../../../$2" TESTOPTS="$test_opts_result" env_test_command "$ENVARGS";
elif [[ -n "$2" && "$2" =~ ^test_ ]]; then
# echo "running name"
set_test_opts $2
TESTOPTS="$test_opts_result" env_test_command "$ENVARGS";
elif [[ -n "$2" ]]; then
# echo "running file"
TEST="../../../$2" env_test_command "$ENVARGS";
else
# echo "running env tests"
env_test_command "$ENVARGS";
fi
}

# organizes the args for the unit tests and calls the command
run_unit_tests() {
if [[ -n "$2" ]]; then
# echo "running file and name"
set_test_opts $2
TEST="$1" TESTOPTS="$test_opts_result" unit_command;
elif [[ -n "$1" && "$1" =~ ^test_ ]]; then
# echo "running name"
set_test_opts $1
TESTOPTS="$test_opts_result" unit_command;
elif [[ -n "$1" ]]; then
# echo "running file"
TEST="$1" unit_command;
else
# echo "running unit tests"
unit_command;
fi
}

# helper method to output the help info and error and then exit the script
output_arg_error(){
help
echo $1
exit
}

while getopts "huemq" option; do
shift 1 # removes the option (ex: -u) from the args so we can pass all the args with $@
case $option in
h)
help
exit;;

u)
run_unit_tests $@
exit;;

e)
if ! [[ -n "$1" ]]; then
output_arg_error "ERROR: Must specify at least one rails version"
fi
run_env_tests $@
exit;;

m)
if ! [[ -n "$1" ]]; then
output_arg_error "ERROR: Must specify multiverse suite"
fi
run_multiverse $@
exit;;

q)
if ! [[ -n "$1" ]]; then
output_arg_error "ERROR: Must specify multiverse suite"
fi
run_multiverse_quick $@
exit;;
esac
done

help

0 comments on commit a5ff73e

Please sign in to comment.