Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Out-of-memory error: if I run remove-empty-columns and clean-whitespace before summary verb #1090

Closed
aborruso opened this issue Sep 4, 2022 · 11 comments
Closed

Out-of-memory error: if I run remove-empty-columns and clean-whitespace before summary verb #1090

aborruso opened this issue Sep 4, 2022 · 11 comments
Assignees
@johnkerl
Labels
pending feedback to close

Comments

@aborruso
Copy link
Contributor

aborruso commented Sep 4, 2022

Hi,
I have this large csv file.

If I run on it

mlr --csv remove-empty-columns then clean-whitespace then summary  extras.csv

I have the error below.
If I remove remove-empty-columns or clean-whitespace I have no error.

Thank you

fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x85ac7b, 0x400000})
        runtime/panic.go:1198 +0x71
runtime.sysMap(0xc237400000, 0x1, 0xc000082680)
        runtime/mem_linux.go:169 +0x96
runtime.(*mheap).grow(0x237f8a0, 0x787249)
        runtime/mheap.go:1393 +0x225
runtime.(*mheap).allocSpan(0x237f8a0, 0x1, 0x0, 0x12)
        runtime/mheap.go:1179 +0x165
runtime.(*mheap).alloc.func1()
        runtime/mheap.go:913 +0x69
runtime.systemstack()
        runtime/asm_amd64.s:383 +0x49

goroutine 35 [running]:
runtime.systemstack_switch()
        runtime/asm_amd64.s:350 fp=0xc000116a58 sp=0xc000116a50 pc=0x465ac0
runtime.(*mheap).alloc(0x7f7b1703ff00, 0x400, 0xff, 0x1)
        runtime/mheap.go:907 +0x73 fp=0xc000116aa8 sp=0xc000116a58 pc=0x425753
runtime.(*mcentral).grow(0x2000)
        runtime/mcentral.go:241 +0x65 fp=0xc000116af0 sp=0xc000116aa8 pc=0x4171c5
runtime.(*mcentral).cacheSpan(0x2390868)
        runtime/mcentral.go:161 +0x69e fp=0xc000116b68 sp=0xc000116af0 pc=0x416ffe
runtime.(*mcache).refill(0x7f7b5ea865b8, 0x12)
        runtime/mcache.go:162 +0xaf fp=0xc000116bb8 sp=0xc000116b68 pc=0x41622f
runtime.(*mcache).nextFree(0x7f7b5ea865b8, 0x12)
        runtime/malloc.go:886 +0x85 fp=0xc000116c00 sp=0xc000116bb8 pc=0x40c2c5
runtime.mallocgc(0x68, 0x83c200, 0x1)
        runtime/malloc.go:1077 +0x4e8 fp=0xc000116c80 sp=0xc000116c00 pc=0x40c948
runtime.newobject(0x8)
        runtime/malloc.go:1234 +0x27 fp=0xc000116ca8 sp=0xc000116c80 pc=0x40ce67
github.com/johnkerl/miller/internal/pkg/mlrval.(*Mlrval).Copy(0xc01de70150)
        github.com/johnkerl/miller/internal/pkg/mlrval/mlrval_copy.go:5 +0x29 fp=0xc000116cd8 sp=0xc000116ca8 pc=0x6778a9
github.com/johnkerl/miller/internal/pkg/transformers/utils.(*PercentileKeeper).Ingest(0xc21e9a0180, 0xc21e96bd10)
        github.com/johnkerl/miller/internal/pkg/transformers/utils/percentile_keeper.go:53 +0xd6 fp=0xc000116d30 sp=0xc000116cd8 pc=0x72dfb6
github.com/johnkerl/miller/internal/pkg/transformers.(*TransformerSummary).ingest(0xc000126390, 0x0)
        github.com/johnkerl/miller/internal/pkg/transformers/summary.go:344 +0x21d fp=0xc000116e18 sp=0xc000116d30 pc=0x78733d
github.com/johnkerl/miller/internal/pkg/transformers.(*TransformerSummary).Transform(0xc000126390, 0xc0f933ae10, 0xc0496ccdb0, 0xc00009c0e0, 0xc00009c070)
        github.com/johnkerl/miller/internal/pkg/transformers/summary.go:298 +0x66 fp=0xc000116e48 sp=0xc000116e18 pc=0x7870c6
github.com/johnkerl/miller/internal/pkg/transformers.runSingleTransformerBatch(0xc09b406e70, {0x919040, 0xc000126390}, 0x0, 0x0, 0x0, 0x0, 0xc000103860)
        github.com/johnkerl/miller/internal/pkg/transformers/aaa_chain_transformer.go:269 +0x2ce fp=0xc000116f08 sp=0xc000116e48 pc=0x73654e
github.com/johnkerl/miller/internal/pkg/transformers.runSingleTransformer({0x919040, 0xc000126390}, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        github.com/johnkerl/miller/internal/pkg/transformers/aaa_chain_transformer.go:212 +0x92 fp=0xc000116f90 sp=0xc000116f08 pc=0x7361d2
github.com/johnkerl/miller/internal/pkg/transformers.ChainTransformer·dwrap·1()
        github.com/johnkerl/miller/internal/pkg/transformers/aaa_chain_transformer.go:187 +0x45 fp=0xc000116fe0 sp=0xc000116f90 pc=0x736105
runtime.goexit()
        runtime/asm_amd64.s:1581 +0x1 fp=0xc000116fe8 sp=0xc000116fe0 pc=0x467ba1
created by github.com/johnkerl/miller/internal/pkg/transformers.ChainTransformer
        github.com/johnkerl/miller/internal/pkg/transformers/aaa_chain_transformer.go:187 +0x2ef

goroutine 1 [select, 1 minutes]:
github.com/johnkerl/miller/internal/pkg/stream.Stream({0xc0001191a0, 0x1, 0x1}, 0xc000103860, {0xc000132380, 0x4, 0x4}, {0x91e3b8, 0xc00012e008}, 0x1)
        github.com/johnkerl/miller/internal/pkg/stream/stream.go:92 +0x617
github.com/johnkerl/miller/internal/pkg/entrypoint.processToStdout(...)
        github.com/johnkerl/miller/internal/pkg/entrypoint/entrypoint.go:82
github.com/johnkerl/miller/internal/pkg/entrypoint.Main()
        github.com/johnkerl/miller/internal/pkg/entrypoint/entrypoint.go:61 +0x1eb
main.main()
        github.com/johnkerl/miller/cmd/mlr/main.go:78 +0x685

goroutine 20 [chan receive, 1 minutes]:
github.com/johnkerl/miller/internal/pkg/output.ChannelWriter(0x0, {0x9188a0, 0xc00011bbf0}, 0x0, 0x0, 0x0, 0x0)
        github.com/johnkerl/miller/internal/pkg/output/channel_writer.go:21 +0x5a
created by github.com/johnkerl/miller/internal/pkg/stream.Stream
        github.com/johnkerl/miller/internal/pkg/stream/stream.go:87 +0x5ab

goroutine 36 [chan receive, 1 minutes]:
github.com/johnkerl/miller/internal/pkg/transformers.runSingleTransformer({0x918b80, 0xc00011bbc0}, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        github.com/johnkerl/miller/internal/pkg/transformers/aaa_chain_transformer.go:211 +0x5f
created by github.com/johnkerl/miller/internal/pkg/transformers.ChainTransformer
        github.com/johnkerl/miller/internal/pkg/transformers/aaa_chain_transformer.go:187 +0x2ef
@aborruso
Copy link
Contributor Author

aborruso commented Sep 4, 2022

My way to have the result is use Miller 5 and 6:

mlr5 --csv remove-empty-columns then clean-whitespace extras.csv | mlr6 --csv summary

@johnkerl
Copy link
Owner

Related to #1119

@johnkerl johnkerl changed the title out of memory error: if I run remove-empty-columns and clean-whitespace before summary verb Out-of-memory error: if I run remove-empty-columns and clean-whitespace before summary verb Nov 26, 2022
@johnkerl
Copy link
Owner

@aborruso PRs #1130 #1131 #1132 #1133 merged to head today should be of help -- I was able to run

mlr --csv remove-empty-columns then clean-whitespace then summary  extras.csv

without issue -- can you confirm?

@johnkerl johnkerl self-assigned this Nov 27, 2022
@aborruso
Copy link
Contributor Author

Hi @johnkerl I have compiling problems :(

go build github.com/johnkerl/miller/cmd/mlr
go: downloading github.com/pkg/profile v1.7.0
go: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
go: downloading github.com/mattn/go-isatty v0.0.16
go: downloading github.com/lestrrat-go/strftime v1.0.6
go: downloading github.com/facette/natsort v0.0.0-20181210072756-2cd4dd1e2dcb
go: downloading github.com/johnkerl/lumin v1.0.0
go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
go: downloading github.com/felixge/fgprof v0.9.3
go: downloading github.com/google/pprof v0.0.0-20211214055906-6f57359322fd
# runtime/internal/atomic
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:18:6: Load redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:16:24
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:24:6: Loadp redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:22:32
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:30:6: Load64 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:28:26
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:36:6: LoadAcq redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:34:27
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:41:6: Xadd redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:39:37
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:44:6: Xadd64 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:42:39
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:47:6: Xadduintptr redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:45:47
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:50:6: Xchg redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:48:36
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:53:6: Xchg64 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:51:38
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:56:6: Xchguintptr redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:54:45
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:56:6: too many errors
make: *** [Makefile:10: build] Error 2

@johnkerl
Copy link
Owner

@aborruso I'll make a release

@johnkerl
Copy link
Owner

Meanwhile please try go clean and then make

@aborruso
Copy link
Contributor Author

Meanwhile please try go clean and then make

Now I have

go build github.com/johnkerl/miller/cmd/mlr
# runtime/internal/atomic
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:18:6: Load redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:16:24
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:24:6: Loadp redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:22:32
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:30:6: Load64 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:28:26
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:36:6: LoadAcq redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:34:27
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:41:6: Xadd redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:39:37
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:44:6: Xadd64 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:42:39
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:47:6: Xadduintptr redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:45:47
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:50:6: Xchg redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:48:36
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:53:6: Xchg64 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:51:38
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:56:6: Xchguintptr redeclared in this block
        previous declaration at /usr/local/go/src/runtime/internal/atomic/atomic_amd64.go:54:45
/usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:56:6: too many errors
make: *** [Makefile:10: build] Error 2

@johnkerl
Copy link
Owner

johnkerl commented Nov 27, 2022 •
edited
Loading

@aborruso I don't know ... Go builds generally "just work" (except when, as here, they don't ...) ... what's your go version ?

@aborruso
Copy link
Contributor Author

@aborruso I don't know ... Go builds generally "just work" (except when, as here, they don't ...) ... what's your go version ?

go version go1.15.8 linux/amd64

Thank you

@johnkerl
Copy link
Owner

There has since been a release -- closing. Please let me know if we need to revisit. Thanks!

@aborruso
Copy link
Contributor Author

There has since been a release -- closing. Please let me know if we need to revisit. Thanks!

it works thank you John

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@johnkerl johnkerl

Labels
pending feedback to close
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aborruso @johnkerl