Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add golangci-lint #2602

Closed
wants to merge 14 commits into from
Closed

Add golangci-lint #2602

wants to merge 14 commits into from

Conversation

daxmc99
Copy link

@daxmc99 daxmc99 commented Oct 29, 2020

Which problem is this PR solving?

Short description of the changes

  • Enable a few linters on golangci-lint to start

Signed-off-by: Dax McDonald [email protected]

Happy Hacktoberfest 🎃

@yurishkuro
Copy link
Member

yurishkuro commented Oct 29, 2020

I would prefer if there were a make target that would allow running all the lint checks locally, not only in GH actions.

@jpkrohling
Copy link
Contributor

I would prefer if there were a make target that would allow running all the lint checks locally, not only in GH actions.

If we have a config in the repo like the .golangci.yml from this PR, it would be a matter of only running golangci-lint in the base directory, and we could still have the GitHub action for it. The advantage of using an action is that it caches the tool, which would save us a minute or two for each build.

That said, I'd rather have this PR use the official action instead of what we have here. Reference: https://github.com/observatorium/opentelemetry-collector-builder/blob/c725aab7de2c3f782f8273ae4cc96faf21747290/.github/workflows/go.yaml#L27-L39

@codecov
Copy link

codecov bot commented Dec 1, 2020

Codecov Report

Merging #2602 (d7b9e41) into master (76ce108) will increase coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2602      +/-   ##
==========================================
+ Coverage   95.55%   95.57%   +0.01%     
==========================================
  Files         214      215       +1     
  Lines        9546     9579      +33     
==========================================
+ Hits         9122     9155      +33     
- Misses        345      346       +1     
+ Partials       79       78       -1     
Impacted Files Coverage Δ
cmd/collector/app/span_processor.go 100.00% <100.00%> (ø)
model/converter/thrift/jaeger/from_domain.go 100.00% <100.00%> (ø)
cmd/collector/app/server/zipkin.go 73.07% <0.00%> (-3.85%) ⬇️
plugin/storage/integration/integration.go 77.34% <0.00%> (-0.56%) ⬇️
cmd/status/command.go 100.00% <0.00%> (ø)
cmd/query/app/static_handler.go 84.82% <0.00%> (+1.78%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b43839...d7b9e41. Read the comment docs.

.github/workflows/ci-golang-lint.yml Outdated Show resolved Hide resolved
.github/workflows/ci-golang-lint.yml Outdated Show resolved Hide resolved
.github/workflows/ci-golang-lint.yml Outdated Show resolved Hide resolved
@daxmc99
Copy link
Author

daxmc99 commented Dec 1, 2020

@yurishkuro thanks for the review!

Do you think this should be a separate yml file? I was planning to nest it under ci-unit-test.yml

@yurishkuro
Copy link
Member

for now let's keep it in ci-unit-tests.yml

Can you actually enable any linters? The job is not checking anything right now.

@jpkrohling
Copy link
Contributor

Can you actually enable any linters?

How did you assess that? The configuration looks sane to me and I would guess that the following linters were used, but I wonder if you are looking at some place with a positive confirmation about the linters that were used for the specific run.

https://github.com/jaegertracing/jaeger/pull/2602/files#diff-6179837f7df53a6f05c522b6b7bb566d484d5465d9894fb04910dd08bb40dcc9R10-R37

@yurishkuro
Copy link
Member

configuration says disable-all: true, and the job completed too quickly

@jpkrohling
Copy link
Contributor

It does say disable-all, but then it enables linters individually. As opposed to enable-all and disable individually. It took about 10s for the linting phase alone, which is what I would expect the job to take.

Ran golangci-lint in 9282ms

https://github.com/jaegertracing/jaeger/pull/2602/checks?check_run_id=1478020447

@yurishkuro
Copy link
Member

gosec alone takes 2min:

$ make lint-gosec
time gosec -quiet -exclude=G104,G107 -exclude-dir=cmd/opentelemetry ./...
      120.03 real       236.28 user       195.93 sys

@daxmc99
Copy link
Author

daxmc99 commented Dec 2, 2020

With the config file, you can use golangci-lint -v run to see what the linter picks up and how long it takes.
There are a few open lint issues. We can address those here or in a separate PR.

Output
golangci-lint -v run
INFO [config_reader] Config search paths: [./ /Users/dax/projects/jaeger /Users/dax/projects /Users/dax /Users /]
INFO [config_reader] Used config file .golangci.yml
INFO [lintersdb] Active 10 linters: [deadcode gofmt goimports golint gosec govet staticcheck structcheck typecheck unused]
INFO [loader] Go packages loading at mode 575 (exports_file|files|imports|name|types_sizes|compiled_files|deps) took 1.012619813s
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 35.883375ms
INFO [linters context/goanalysis] analyzers took 83.376645ms with top 10 stages: buildir: 17.607961ms, goimports: 11.697858ms, gosec: 8.840719ms, gofmt: 7.785945ms, golint: 5.979534ms, SA1012: 1.844425ms, SA1013: 1.646506ms, printf: 1.482018ms, deadcode: 1.420496ms, ctrlflow: 1.231694ms
INFO [linters context/goanalysis] analyzers took 22.52245ms with top 10 stages: buildir: 17.317955ms, U1000: 5.204495ms
INFO [runner/skip dirs] Skipped 1 issues from dir examples/hotrod/services/frontend by pattern (^|/)examples($|/)
INFO [runner/skip dirs] Skipped 1 issues from dir examples/hotrod/services/customer by pattern (^|/)examples($|/)
INFO [runner/skip dirs] Skipped 69 issues from dir examples/hotrod/services/driver by pattern (^|/)examples($|/)
INFO [runner/skip dirs] Skipped 1 issues from dir examples/hotrod/services/route by pattern (^|/)examples($|/)
INFO [runner/max_same_issues] 9/12 issues with text "G404: Use of weak random number generator (math/rand instead of crypto/rand)" were hidden, use --max-same-issues
INFO [runner/max_same_issues] 4/7 issues with text "G306: Expect WriteFile permissions to be 0600 or less" were hidden, use --max-same-issues
INFO [runner/max_same_issues] 4/7 issues with text "File is not `goimports`-ed" were hidden, use --max-same-issues
INFO [runner] Issues before processing: 3983, after processing: 21
INFO [runner] Processors filtering stat (out/in): max_same_issues: 21/38, max_from_linter: 21/21, severity-rules: 21/21, cgo: 3983/3983, filename_unadjuster: 3983/3983, skip_files: 3983/3983, exclude-rules: 39/339, uniq_by_line: 38/39, path_prettifier: 3983/3983, nolint: 39/39, source_code: 21/21, path_shortener: 21/21, sort_results: 21/21, skip_dirs: 3911/3983, autogenerated_exclude: 339/3911, diff: 38/38, max_per_file_from_linter: 38/38, path_prefixer: 21/21, identifier_marker: 339/339, exclude: 339/339
INFO [runner] processing took 38.919486ms with stages: nolint: 16.716437ms, path_prettifier: 7.979696ms, autogenerated_exclude: 6.675571ms, identifier_marker: 3.172654ms, exclude-rules: 1.69026ms, skip_dirs: 1.570182ms, source_code: 442.928µs, cgo: 358.105µs, filename_unadjuster: 255.942µs, max_same_issues: 26.884µs, uniq_by_line: 11.137µs, max_per_file_from_linter: 9.045µs, path_shortener: 4.857µs, max_from_linter: 3.756µs, skip_files: 621ns, exclude: 392ns, diff: 384ns, severity-rules: 264ns, sort_results: 233ns, path_prefixer: 138ns
INFO [runner] linters took 369.097097ms with stages: goanalysis_metalinter: 224.257539ms, unused: 105.765528ms
model/converter/json/from_domain_test.go:142:10: G306: Expect WriteFile permissions to be 0600 or less (gosec)
		err := ioutil.WriteFile(outFile+"-actual.json", buf.Bytes(), 0644)
		       ^
cmd/collector/app/sanitizer/cache/auto_refresh_cache.go:136:40: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
	return (interval / 2) + time.Duration(rand.Int63n(int64(interval/2)))
	                                      ^
cmd/collector/app/server/grpc_test.go:80:19: G102: Binds to all network interfaces (gosec)
	listener, err := net.Listen("tcp", ":0")
	                 ^
pkg/discovery/grpcresolver/grpc_resolver.go:68:12: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
	random := rand.New(rand.NewSource(seed))
	          ^
pkg/cassandra/mocks/Iterator.go:18: File is not `goimports`-ed (goimports)
import cassandra "github.com/jaegertracing/jaeger/pkg/cassandra"
import mock "github.com/stretchr/testify/mock"
pkg/cassandra/mocks/Query.go:18: File is not `goimports`-ed (goimports)
import cassandra "github.com/jaegertracing/jaeger/pkg/cassandra"
import mock "github.com/stretchr/testify/mock"
pkg/cassandra/mocks/Session.go:18: File is not `goimports`-ed (goimports)
import cassandra "github.com/jaegertracing/jaeger/pkg/cassandra"
import mock "github.com/stretchr/testify/mock"
pkg/es/wrapper/wrapper_nolint.go:23:30: method Id should be ID (golint)
func (i IndexServiceWrapper) Id(id string) es.IndexService {
                             ^
pkg/es/wrapper/wrapper_nolint.go:28:30: method BodyJson should be BodyJSON (golint)
func (i IndexServiceWrapper) BodyJson(body interface{}) es.IndexService {
                             ^
cmd/ingester/app/processor/decorator/retry.go:54:18: G404: Use of weak random number generator (math/rand instead of crypto/rand) (gosec)
	rand:           rand.New(rand.NewSource(time.Now().UnixNano())),
	                ^
pkg/queue/bounded_queue_test.go:57:3: SA2001: empty critical section (staticcheck)
		startLock.Unlock()
		^
plugin/sampling/strategystore/static/strategy_store_test.go:346:21: G306: Expect WriteFile permissions to be 0600 or less (gosec)
	require.NoError(t, ioutil.WriteFile(dstFile, srcBytes, 0644))
	                   ^
plugin/sampling/strategystore/static/strategy_store_test.go:367:21: G306: Expect WriteFile permissions to be 0600 or less (gosec)
	require.NoError(t, ioutil.WriteFile(dstFile, []byte(newStr), 0644))
	                   ^
storage/spanstore/token_propagation_test.go:25:2: G101: Potential hardcoded credentials (gosec)
	token := "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhZG1pbiIsIm5hbWUiOiJKb2huIERvZSIsImlhdCI"
	^
cmd/anonymizer/main.go:72:22: G601: Implicit memory aliasing in for loop. (gosec)
				writer.WriteSpan(&span)
				                 ^
model/converter/thrift/jaeger/from_domain.go:106:37: G601: Implicit memory aliasing in for loop. (gosec)
		jaegerTags[idx] = d.keyValueToTag(&kv)
		                                  ^
plugin/storage/badger/spanstore/read_write_test.go:142:4: var `lowId` should be `lowID` (golint)
			lowId := rand.Uint64()
			^
cmd/agent/app/reporter/grpc/reporter_test.go:60:2: SA5001: should check returned error before deferring conn.Close() (staticcheck)
	defer conn.Close()
	^
cmd/agent/app/reporter/grpc/reporter_test.go:98:2: SA5001: should check returned error before deferring conn.Close() (staticcheck)
	defer conn.Close()
	^
cmd/query/app/grpc_handler_test.go:138:12: G102: Binds to all network interfaces (gosec)
	lis, _ := net.Listen("tcp", ":0")
	          ^
cmd/query/app/token_propagation_hander_test.go:31:2: G101: Potential hardcoded credentials (gosec)
	bearerToken := "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhZG1pbiIsIm5hbWUiOiJKb2huIERvZSIsImlhdCI"
	^
INFO File cache stats: 18 entries of total size 105.4KiB
INFO Memory: 16 samples, avg is 90.0MB, max is 141.1MB
INFO Execution took 1.446295162s

I am specifying --only-new-issues in the Github Action config though. I don't believe this will throw any lint issues on the introductory PR. It is the same as running golangci-lint run --new that only outputs linter failures for "new" lint issues found.

Can you actually enable any linters? The job is not checking anything right now.

This is a fairly common style for .golangci-lint ex1 , ex2
Happy to make changes though as needed

gosec alone takes 2min

This seems like a great reason to use golangci-lint, the go sec errors I am seeing look the same, and golangci-lint runs much faster for me (1.446295162 seconds)

@jpkrohling
Copy link
Contributor

I am specifying --only-new-issues in the Github Action config though.

I think this is the key :-)

@yurishkuro
Copy link
Member

I am not sure what only-new-issues means: you still have to scan everything to discover IF there are new issues.

Why can't we run with -v in CI as well? That would at least provide some proof that the linters are running.

.golangci.yml Outdated Show resolved Hide resolved
@daxmc99 daxmc99 marked this pull request as ready for review December 4, 2020 20:56
@daxmc99 daxmc99 requested a review from a team as a code owner December 4, 2020 20:56
uses: golangci/golangci-lint-action@v2
with:
version: v1.33
only-new-issues: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please explain what only new issues means? Does it mean it only scans changed files (if so how does it determine that, against master?), or that it reports errors previously unseen (if so where does it store the state?) ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been dropped since all linters are currently passing.
Per https://golangci-lint.run/usage/configuration/

 # Show only new issues: if there are unstaged changes or untracked files,
  # only those changes are analyzed, else only changes in HEAD~ are analyzed.
  # It's a super-useful option for integration of golangci-lint into existing
  # large codebase. It's not practical to fix all existing issues at the moment
  # of integration: much better don't allow issues in new code.
  # Default is false.

This is meant to allow easier integration into a large codebase by only reporting errors if the added code contains them. It does not report previously seen linter errors (those that are already in master).
This has been dropped though because all outstanding linter issues have been fixed.

@yurishkuro
Copy link
Member

@jpkrohling my main concern with this is that by adding a whole set of new linters with new changes only mode, we're suppressing a lot of existing issues that will make life painful for future contributors, because their PRs may touch files where these issues exist and start breaking, forcing people to fix previously issues and distracting from the main goal of the PRs.

I think I would prefer instead to add linters one by one or in batches along with the corresponding fixes of ALL issues that a given linter flags. That way each PR could be reasonably sized and not push the burden of the actual fixes on the next random person who happens to touch the files.

For example, with the config in this PR:

$ golangci-lint -v --max-issues-per-linter 500 --max-same-issues 500 run | grep Error | wc -l
     189

@jpkrohling
Copy link
Contributor

because their PRs may touch files where these issues exist and start breaking, forcing people to fix previously issues and distracting from the main goal of the PRs

Good point and I agree with your proposal.

@daxmc99
Copy link
Author

daxmc99 commented Dec 8, 2020

I'm happy to either reduce the number of enabled linters or fix the found issues.
To get this merged it might make more sense to enable less linters for now

@yurishkuro
Copy link
Member

I'd recommend enabling only those linters that currently pass on the whole repository.

Signed-off-by: Dax McDonald <[email protected]>
Signed-off-by: Dax McDonald <[email protected]>
Signed-off-by: Dax McDonald <[email protected]>
Signed-off-by: Dax McDonald <[email protected]>
Signed-off-by: Dax McDonald <[email protected]>
crossdock/main.go:112:23: response body must be closed (bodyclose)
Close the response body when the err == nil to avoid leaking resources

Signed-off-by: Dax McDonald <[email protected]>
plugin/storage/badger/spanstore/read_write_test.go:142:4: var `lowId` should be `lowID` (golint)
                        lowId := rand.Uint64()

Signed-off-by: Dax McDonald <[email protected]>
File is not query/app/ui/placeholder.go:20: File is not `goimports`-ed

Signed-off-by: Dax McDonald <[email protected]>
@daxmc99
Copy link
Author

daxmc99 commented Dec 8, 2020

After removing a few linters, there were still some small issues remaining. These all looked like legitimate issues, so I proceeded to fix them

Linter errors

INFO [runner] linters took 142.248311ms with stages: goanalysis_metalinter: 112.589465ms, unused: 7.165119ms
storage/mocks/ArchiveFactory.go:17: File is not goimports-ed (goimports)
import mock "github.com/stretchr/testify/mock"
import spanstore "github.com/jaegertracing/jaeger/storage/spanstore"
import storage "github.com/jaegertracing/jaeger/storage"
storage/mocks/Factory.go:17: File is not goimports-ed (goimports)
import dependencystore "github.com/jaegertracing/jaeger/storage/dependencystore"
import metrics "github.com/uber/jaeger-lib/metrics"
import mock "github.com/stretchr/testify/mock"
import spanstore "github.com/jaegertracing/jaeger/storage/spanstore"
import storage "github.com/jaegertracing/jaeger/storage"
import zap "go.uber.org/zap"
cmd/query/app/ui/placeholder.go:20: File is not goimports-ed (goimports)
"github.com/jaegertracing/jaeger/cmd/query/app/ui/placeholder"
plugin/storage/badger/spanstore/read_write_test.go:142:4: var lowId should be lowID (golint)
lowId := rand.Uint64()
^
model/converter/thrift/jaeger/from_domain.go:106:37: G601: Implicit memory aliasing in for loop. (gosec)
jaegerTags[idx] = d.keyValueToTag(&kv)
^
cmd/anonymizer/main.go:72:22: G601: Implicit memory aliasing in for loop. (gosec)
writer.WriteSpan(&span)
^
crossdock/main.go:112:23: response body must be closed (bodyclose)
res, err := http.Get(healthURL)
^
INFO File cache stats: 7 entries of total size 35.1KiB

Happy to revert any commits as needed

@yurishkuro
Copy link
Member

The run in the CI succeeds, but when I run it locally with the current config file I get a few errors. Where is the discrepancy coming from?

$ golangci-lint -v --max-issues-per-linter 500 --max-same-issues 500 run

@daxmc99
Copy link
Author

daxmc99 commented Dec 8, 2020 via email

Copy link
Member

@yurishkuro yurishkuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right, I just ran it on master so it might have tripped on the issues that you fixed.

Comment on lines 71 to 73
for _, span := range spans {
span := span
writer.WriteSpan(&span)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is weird. I think I know what the linter is complaining about, and a better solution is

Suggested change
for _, span := range spans {
span := span
writer.WriteSpan(&span)
for i := range spans {
writer.WriteSpan(&spans[i])

Copy link
Author

@daxmc99 daxmc99 Dec 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not familiar with the intention of the code but

for _, span := range spans {
	// something with &span

looks like the developer wants to use copies of the original item, not explicitly manipulating the items in the array.
If the intention is to use the items directly then I believe this change makes sense.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

writer treats the spans as immutable, so the copying is not necessary, only extra memory allocation

@@ -111,6 +111,7 @@ func httpHealthCheck(logger *zap.Logger, service, healthURL string) {
for i := 0; i < 240; i++ {
res, err := http.Get(healthURL)
if err == nil && is2xxStatusCode(res.StatusCode) {
res.Body.Close()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fine here because it's a test code, but strictly speaking we must close when err==nil regardless of status code

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I wanted to minimize the diff here

model/converter/thrift/jaeger/from_domain.go Outdated Show resolved Hide resolved
- govet
- gosec
- gofmt
- golint
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find it very suspicious that you're not getting errors from golint given the current exclusions below. Usually thrift-gen produces a lot of errors for golint, which is why we excluded it in the makefile

@yurishkuro
Copy link
Member

Also, what makes it exclude examples?

INFO [runner/skip dirs] Skipped 69 issues from dir examples/hotrod/services/driver by pattern (^|/)examples($|/)
INFO [runner/skip dirs] Skipped 1 issues from dir examples/hotrod/services/customer by pattern (^|/)examples($|/)
INFO [runner/skip dirs] Skipped 1 issues from dir examples/hotrod/services/route by pattern (^|/)examples($|/)
INFO [runner/skip dirs] Skipped 1 issues from dir examples/hotrod/services/frontend by pattern (^|/)examples($|/)

I think examples code is a fair game for linter (unlike generated code).

@jpkrohling
Copy link
Contributor

Closing due to inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migrate lint tools to golangci-lint
3 participants