New test level for metadata deploy - RunTestSuite

[alan-morey]

What are you trying to do
It would be nice to be able to specify a test suite name as part of deployment command that would be used to specify which tests should be executed as part of the deployment, as an alternative to the RunSpecifiedTests option.

For example, if I have apackage with manifest as follows:

package.xml

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
  <types>
    <members>Foo</members>
    <members>FooTest</members>
    <members>Bar</members>
    <members>BarTest</members>
    <name>ApexClass</name>
  </types>

  <types>
    <members>MyTestSuite</members>
    <name>ApexTestSuite</name>
  </types>
</Package>

And the test suite specifies the tests I want executed during deployment:

MyTestsuite.testSuite

<?xml version="1.0" encoding="UTF-8"?>
<ApexTestSuite xmlns="http://soap.sforce.com/2006/04/metadata">
    <testClassName>FooTest</testClassName>
    <testClassName>BarTest</testClassName>
</ApexTestSuite>

Describe the solution you'd like
I would like to use a test level named RunTestSuite for example, that would require a test suite name be specified, possibily by reusing --runtests or with another flag. With this test level I don't have to explicitly list all my test classes I want to have run during deployment. Instead the tool would extract the list of tests from the test suite and run those.

Example:

$ sfdx force:mdapi:deploy \
  --testlevel RunTestSuite \
  --runtests MyTestSuite \
  --deploydir mypackage \
  --wait -1 \
  -u alan

# Maybe also supports multiple test suites:
$ sfdx force:mdapi:deploy \
  --testlevel RunTestSuite \
  --runtests MyTestSuite1,MyTestSuite2,MyTestSuiteN \
  --deploydir mypackage \
  --wait -1 \
  -u alan

[dcarroll]

Assuming --runtests doesn't handle a test suite, I think this is a great feature. Have you verified that --runtest cannot handle a suite name?

[alan-morey]

@dcarroll You can not use --runtests currently with a test suite name.

Here is a bash script I wrote which kind of implements the feature, it requires xmlstartlet

#!/usr/bin/env bash
XPATH_PACKAGE_TEST_SUITES='/_:Package/_:types/_:name[.="ApexTestSuite"]/../_:members'
XPATH_TEST_SUITE_CLASSES='/_:ApexTestSuite/_:testClassName'
MANIFEST=package.xml

TEST_SUITES=$(xmlstarlet select --template --value-of $XPATH_PACKAGE_TEST_SUITES $MANIFEST)
TEST_SUITES=$(echo $TEST_SUITES | tr ' ' '\n' | sort | uniq)
if [ -z "$TEST_SUITES" ]; then
    echo "*WARNING* Skipping test run, $MANIFEST does not specify any test suite(s)"
    exit 1
fi

echo -e "Apex Test Suites in $MANIFEST:\n - $(echo $TEST_SUITES | sed 's/ /\n - /g')"
echo

TEST_CLASSES=''
for TEST_SUITE in $TEST_SUITES; do
    TEST_SUITE_FILE="src/testSuites/$TEST_SUITE.testSuite"
    TEST_CLASSES+="\n"$(xmlstarlet select --template --value-of $XPATH_TEST_SUITE_CLASSES $TEST_SUITE_FILE)
done

TEST_CLASSES=$(echo -e $TEST_CLASSES | tr ' ' '\n' | sort | uniq)
if [ -z "$TEST_CLASSES" ]; then
    echo "*WARNING* Skipping test run, no test classes in parsed test suite(s)"
    exit 1
fi

echo -e "Apex Test Classes:\n - $(echo $TEST_CLASSES | sed 's/ /\n - /g')"
echo

RUN_TESTS=$(echo $TEST_CLASSES | tr ' ' ,)

sfdx force:mdapi:deploy --verbose --checkonly --deploydir build/deploy --wait -1 --testlevel RunSpecifiedTests --runtests $RUN_TESTS -$@

[dcarroll]

@alan-morey Ok, cool. I'll let the team know and will see if we can get this on the roadmap.

EDIT: Looks like you can specify a test suite in the test:run command for force:apex. Might this be a temporary work around? Reason I ask is that this request requires a modification/enhancement of the Metadata API.

sfdx force:apex:test:run -s MySuite,MyOtherSuite -c --json

FYI @clairebianchi

[alan-morey]

@dcarroll I'm aware of the option to run test suites with force:apex:test:run, I use that alot during development and on CI jobs. The feature I'm looking for is specifically for deployments, so that I can meet code coverage requirements.

I realize that the Metadata API would probably have to be the first candidate for implentation of this feature, on the plus side, it would be available for other deployment tools such as Changesets.

[dcarroll]

@alan-morey Yup. Makes sense and is something that the Metadata API team has on the backlog somewhere. Hopefully, you post will help move it up the priority list.

@kfidelak FYI

[akalatksy]

Very disappointed that it wasn't implemented initially.
A really helpful feature for CI deployments.

[shantanu0805]

Checking on this, is there an update on this feature? It would be of immense help to run test suites along with deployments without having to create them manually in the target org.

[clairebianchi]

Sadly the mdapi commands are not owned the Salesforce CLI team. I have passed on this feedback to their the MDAPI team's PM, but will be closing this issue here.

[maelmonnier]

In case it can help, there is an helper in the SFDX plugin of Accenture to extract all test classes from a test suite : https://github.com/Accenture/sfpowerkit#sfpowerkitsourceapextestsuiteconvert (sfpowerkit:source:apextestsuite:convert)

[jointovinay]

What are you trying to do It would be nice to be able to specify a test suite name as part of deployment command that would be used to specify which tests should be executed as part of the deployment, as an alternative to the RunSpecifiedTests option.

For example, if I have apackage with manifest as follows:

package.xml

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
  <types>
    <members>Foo</members>
    <members>FooTest</members>
    <members>Bar</members>
    <members>BarTest</members>
    <name>ApexClass</name>
  </types>

  <types>
    <members>MyTestSuite</members>
    <name>ApexTestSuite</name>
  </types>
</Package>

And the test suite specifies the tests I want executed during deployment:

MyTestsuite.testSuite

<?xml version="1.0" encoding="UTF-8"?>
<ApexTestSuite xmlns="http://soap.sforce.com/2006/04/metadata">
    <testClassName>FooTest</testClassName>
    <testClassName>BarTest</testClassName>
</ApexTestSuite>

Describe the solution you'd like I would like to use a test level named RunTestSuite for example, that would require a test suite name be specified, possibily by reusing --runtests or with another flag. With this test level I don't have to explicitly list all my test classes I want to have run during deployment. Instead the tool would extract the list of tests from the test suite and run those.

Example:

$ sfdx force:mdapi:deploy \
  --testlevel RunTestSuite \
  --runtests MyTestSuite \
  --deploydir mypackage \
  --wait -1 \
  -u alan

# Maybe also supports multiple test suites:
$ sfdx force:mdapi:deploy \
  --testlevel RunTestSuite \
  --runtests MyTestSuite1,MyTestSuite2,MyTestSuiteN \
  --deploydir mypackage \
  --wait -1 \
  -u alan

Hi,

I've created my test suite in same way. i'm getting error while running below command,

sfdx force:apex:test:run -s "testSuites/TestSuite.testSuite"
ERROR running force:apex:test:run: Invalid ID or name: testSuites/TestSuite.testSuite
r2ufav@osx-56zmlvcg-u sfdx-gitlab-coe-project %