Add goTestUnit and goTestIntegration to magefile #7766

andrewkroh · 2018-07-26T17:28:31Z

Consider these targets as incubating. They are not used by any of the Makefiles yet.
I added it to the Windows CI only to start testing it out and to resolve an issue where
compilation could fail and success would be reported by the powershell script.

$ mage -h goTestUnit
mage gotestunit:

GoTestUnit executes the Go unit tests.
Use TEST_COVERAGE=true to enable code coverage profiling.
Use RACE_DETECTOR=true to enable the race detector.

$ TEST_COVERAGE=true RACE_DETECTOR=true mage goTestUnit
>> go test: Unit Testing
SUMMARY:
  Fail:     0
  Skip:     2
  Pass:     807
  Packages: 70
  Duration: 21.459313277s
  Coverage Report: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.html
  JUnit Report:    /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.xml
  Output File:     /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.out
>> go test: Unit Test Passed

$ TEST_COVERAGE=true RACE_DETECTOR=true mage goTestUnit
>> go test: Unit Testing
FAILURES:
Package: github.com/elastic/beats/libbeat/processors
Test:    TestDemo
processor_test.go:36: Only failing tests are logged. But you can use 'mage -v goTestUnit'
	to see all of the go test output or just view the output file list in the summary.
----
SUMMARY:
  Fail:     1
  Skip:     2
  Pass:     807
  Packages: 70
  Duration: 21.53730358s
  Coverage Report: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.html
  JUnit Report:    /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.xml
  Output File:     /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.out
>> go test: Unit Test Failed
Error: go test failed: 1 test failures
$ echo $?
1

Consider these targets as incubating. They are not used by any of the Makefiles yet. I added it to the Windows CI only to start testing it out and to resolve an issue where compilation could fail and success would be reported by the powershell script. ``` $ mage -h goTestUnit mage gotestunit: GoTestUnit executes the Go unit tests. Use TEST_COVERAGE=true to enable code coverage profiling. Use RACE_DETECTOR=true to enable the race detector. ``` ``` $ TEST_COVERAGE=true RACE_DETECTOR=true mage goTestUnit >> go test: Unit Testing SUMMARY: Fail: 0 Skip: 2 Pass: 807 Packages: 70 Duration: 21.459313277s Coverage Report: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.html JUnit Report: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.xml Output File: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.out >> go test: Unit Test Passed ``` ``` $ TEST_COVERAGE=true RACE_DETECTOR=true mage goTestUnit >> go test: Unit Testing FAILURES: Package: github.com/elastic/beats/libbeat/processors Test: TestDemo processor_test.go:36: Only failing tests are logged. But you can use 'mage -v goTestUnit' to see all of the go test output or just view the output file list in the summary. ---- SUMMARY: Fail: 1 Skip: 2 Pass: 807 Packages: 70 Duration: 21.53730358s Coverage Report: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.html JUnit Report: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.xml Output File: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.out >> go test: Unit Test Failed Error: go test failed: 1 test failures $ echo $? 1 ```

ruflin · 2018-07-27T07:16:10Z

Reminds me a bit of #4960 Perhaps we could take this opportunity to change the ENV variable names and also change a bit which commands run which tests.

ruflin

++ on this change.

ruflin · 2018-07-27T07:16:52Z

auditbeat/magefile.go

@@ -96,6 +97,20 @@ func Fields() error {
 	return mage.GenerateFieldsYAML("module")
 }

+// GoTestUnit executes the Go unit tests.
+// Use TEST_COVERAGE=true to enable code coverage profiling.


Could we call this just COVERAGE and RACE?

I'm thinking that they should both be prefixed in some way to avoid collisions and be self-describing. How prefixing any variables that control test parameters with TEST_? Like...

TEST_RACE

TEST_COVERAGE

ruflin · 2018-07-27T07:18:46Z

dev-tools/mage/gotest.go

+	}
+	if goTestErr != nil && len(report.Packages) == 0 {
+		// No packages were tested. Probably the code didn't compile.
+		fmt.Println(bytes.NewBuffer(bufferOutput.Bytes()).String())


I wonder if in here we could also use the log package?

In mage the stdlib log package is tied to the -v parameter so it gets used for adding debug information to build.

stdout is the main UI for the build tool. So fmt.Print is used for messages that always need to be presented to the user. The goal is to keep verbosity down so that real warnings and problems are easily observed.

I think switching to a leveled logger would be best similar to how Gradle handles it.

And I try to only fmt.Print when necessary such that the build only gets verbose when there is a failure.

ruflin · 2018-07-27T07:21:26Z

filebeat/magefile.go

+// GoTestUnit executes the Go unit tests.
+// Use TEST_COVERAGE=true to enable code coverage profiling.
+// Use RACE_DETECTOR=true to enable the race detector.
+func GoTestUnit(ctx context.Context) error {


Could we call this just UnitTest? Would be great if the names of the commands would be as intuitive and rememberable as possible (I know, very subjective topic).

I think we should have targets for unitTest and integrationTest. The unitTest target would then have dependency on goUnitTest and pythonUnitTest?

Then do the same thing for with integrationTest by making it depend on goIntegrationTest and pythonIntegrationTest.

Then if needed we also add benchTest and loadTest.

SGTM. Very similar to #4960

For bench and load I wonder if they should be their own target or a ENV variable. We can discuss that in a follow up.

ruflin · 2018-07-27T07:22:40Z

heartbeat/magefile.go

+// GoTestUnit executes the Go unit tests.
+// Use TEST_COVERAGE=true to enable code coverage profiling.
+// Use RACE_DETECTOR=true to enable the race detector.
+func GoTestUnit(ctx context.Context) error {


It's a bit a pity that each Beat needs to have all commands inside. Is there a way that mage could "inherit" commands?

Hopefully in the future this is supported by mage: magefile/mage#108

ruflin · 2018-08-02T08:33:28Z

@elastic/apm-server This might be interesting for the team.

…efile (#8294) * Add goTestUnit and goTestIntegration to magefile Consider these targets as incubating. They are not used by any of the Makefiles yet. I added it to the Windows CI only to start testing it out and to resolve an issue where compilation could fail and success would be reported by the powershell script. ``` $ mage -h goTestUnit mage gotestunit: GoTestUnit executes the Go unit tests. Use TEST_COVERAGE=true to enable code coverage profiling. Use RACE_DETECTOR=true to enable the race detector. ``` ``` $ TEST_COVERAGE=true RACE_DETECTOR=true mage goTestUnit >> go test: Unit Testing SUMMARY: Fail: 0 Skip: 2 Pass: 807 Packages: 70 Duration: 21.459313277s Coverage Report: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.html JUnit Report: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.xml Output File: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.out >> go test: Unit Test Passed ``` ``` $ TEST_COVERAGE=true RACE_DETECTOR=true mage goTestUnit >> go test: Unit Testing FAILURES: Package: github.com/elastic/beats/libbeat/processors Test: TestDemo processor_test.go:36: Only failing tests are logged. But you can use 'mage -v goTestUnit' to see all of the go test output or just view the output file list in the summary. ---- SUMMARY: Fail: 1 Skip: 2 Pass: 807 Packages: 70 Duration: 21.53730358s Coverage Report: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.html JUnit Report: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.xml Output File: /Users/akroh/go/src/github.com/elastic/beats/libbeat/build/TEST-go-unit.out >> go test: Unit Test Failed Error: go test failed: 1 test failures $ echo $? 1 ``` (cherry picked from commit f52f069) * Add github.com/jstemmer/go-junit-report to vendor (cherry picked from commit 5f05ac7)

andrewkroh added review :Infra labels Jul 26, 2018

andrewkroh added 2 commits July 26, 2018 13:43

Add github.com/jstemmer/go-junit-report to vendor

6830908

andrewkroh force-pushed the feature/mage-go-test branch from 1aea460 to 6830908 Compare July 26, 2018 17:43

ruflin reviewed Jul 27, 2018

View reviewed changes

ruflin merged commit 5f05ac7 into elastic:master Aug 2, 2018

graphaelli mentioned this pull request Aug 20, 2018

use mage in windows test runner elastic/apm-server#1301

Merged

graphaelli mentioned this pull request Sep 11, 2018

Cherry-pick #7766 to 6.x: Add goTestUnit and goTestIntegration to magefile #8287

Closed

graphaelli added the v6.5.0 label Sep 11, 2018

graphaelli mentioned this pull request Sep 12, 2018

Cherry-pick #7766 to 6.x: Add goTestUnit and goTestIntegration to magefile #8294

Merged

andrewkroh mentioned this pull request Oct 31, 2018

MS SQL DB metricset #8250

Closed

andrewkroh deleted the feature/mage-go-test branch November 8, 2018 20:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add goTestUnit and goTestIntegration to magefile #7766

Add goTestUnit and goTestIntegration to magefile #7766

andrewkroh commented Jul 26, 2018

ruflin commented Jul 27, 2018

ruflin left a comment

ruflin Jul 27, 2018

andrewkroh Jul 30, 2018

ruflin Aug 2, 2018

ruflin Jul 27, 2018

andrewkroh Jul 30, 2018

ruflin Jul 27, 2018

andrewkroh Jul 30, 2018

ruflin Aug 2, 2018

ruflin Jul 27, 2018

andrewkroh Jul 30, 2018

ruflin commented Aug 2, 2018

Add goTestUnit and goTestIntegration to magefile #7766

Add goTestUnit and goTestIntegration to magefile #7766

Conversation

andrewkroh commented Jul 26, 2018

ruflin commented Jul 27, 2018

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Aug 2, 2018