Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: slow with many methods on parameterized type #51957

Open
seborama opened this issue Mar 26, 2022 · 5 comments
Open

cmd/compile: slow with many methods on parameterized type #51957

seborama opened this issue Mar 26, 2022 · 5 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@seborama
Copy link

seborama commented Mar 26, 2022

What version of Go are you using (go version)?

$ go version
go version go1.18 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOVERSION="go1.18"

What did you do?

git clone [email protected]:seborama/fuego.git -b v11_type_parameters && cd fuego
Edit file comparable_test.go and add a comment anywhere in the file then save. This step is to ensure the build is executed without caching.
go test -timeout 10s -v ./... -count 1 -run TestComparableStream_Max
The execution of the test takes over 1 minute on my computer, rather than < 3s normally.

This problem may have started after I added mapto.go (generated via make generate).

The reason for generating so many parameterised methods comes from issue 49085.

What did you expect to see?

A fast test execution, at most 3 seconds, likely under 1s.

What did you see instead?

The outcome is correct, but the test build takes a drastically long time.

Right now, I only have one computer on which to test this issue.
This may conceivably be related to an unknown particular of my environment set-up, although I cannot think of any.

@randall77
Copy link
Contributor

Can you build and run the test separately? Build with go test -c and then run the resulting binary.
That will let us know for sure whether it is compile time or run time.

@seborama
Copy link
Author

seborama commented Mar 26, 2022

Sure:

# edited comparable_test.go and added `// hi` at the top, then saved, then:
$> time go test -c
go test -c  59.77s user 4.67s system 247% cpu 26.078 total

It is the build of the test binary. Execution is slow compared to Go 1.17 but not as much as the build:

$> time go test -timeout 10s ./... -count 1 -run "TestComparableStream_.*"
ok  	github.com/seborama/fuego/v11	0.344s
?   	github.com/seborama/fuego/v11/generate	[no test files]
go test -timeout 10s ./... -count 1 -run "TestComparableStream_.*"  2.43s user 0.74s system 134% cpu 2.352 total

@seankhliao seankhliao changed the title affected/package: go test exhibits degraded performance cmd/compile: slow with many methods on parameterized type Mar 26, 2022
@seankhliao seankhliao added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 26, 2022
seborama added a commit to seborama/fuego that referenced this issue Mar 26, 2022
@seborama
Copy link
Author

For comparison, I've created a new branch v11_type_parameters_51957 that does away from the parameterised MapTo* methods in mapto.go.
Build performance is back to normal. Execution is also faster.

time go test -c 
go test -c  0.92s user 0.56s system 222% cpu 0.665 total
time go test -timeout 10s ./... -count 1 -run "TestComparableStream_.*" 
ok      github.com/seborama/fuego/v11   0.138s
?       github.com/seborama/fuego/v11/generate  [no test files]
go test -timeout 10s ./... -count 1 -run "TestComparableStream_.*"  0.77s user 0.61s system 188% cpu 0.732 total

@randall77
Copy link
Contributor

There appears to be just a lot of code generated by this package. Like, 74302 functions.

For instance, there are 24299 functions that all came from orderlyConcurrentDoStream. Looks like there's some cross product of two different shapes, times a bunch of local functions in each.

I'm not entirely sure that all of these instantiations are necessary, but at first look there's nothing obviously duplicated.

seborama added a commit to seborama/fuego that referenced this issue Mar 30, 2022
* v11: transition to go 1.18 type parameters (WIP)

* add interface implementation checks

* Checkpoint

* Initial ground zero

* Some updates to README (WIP)

* Makefile

* added common type aliases and improved tests

* more tests, tweaks

* re-org'ed files

* FlatMap

also fix go.mod versioning

* add Predicate

* added: predicate, stream.Filter

* added TestStream_ForEach

* doc updates

* added Stream.MapTo* generated code

* added: LeftReduce, Reduce and BiFunctions: Max, Min, Sum

* README, renamed a param

* Licence file

* README

* refined generated code (adding FlatMapTo* as well)

* added tests

* add ComparableStream with Max and Min

* add ComparableStream.Sum

* test branch for Issue 51957

See golang/go#51957

* Attempt to move forward with Go 1.18

* Add Average, replace Addable with Mathable

Small corrections

* added Stream.GroupBy

* added collectors

* added missing test

* README updates

* add optional + more collectors, etc
@seborama
Copy link
Author

seborama commented Apr 13, 2022

I'd argue that my attempt to use methods with a generic receiver (and then back-fill with functions) is probably inadequate in my use-case.

This article refers:
https://go.dev/blog/when-generics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Status: Triage Backlog
Development

No branches or pull requests

4 participants