Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

race condition #559

Open
hangxie opened this issue Sep 17, 2023 · 1 comment
Open

race condition #559

hangxie opened this issue Sep 17, 2023 · 1 comment

Comments

@hangxie
Copy link
Contributor

hangxie commented Sep 17, 2023

Got race condition in an older version so I upgraded to latest master branch but still get:`

WARNING: DATA RACE
Read at 0x00c000a5dc50 by goroutine 714:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:341 +0x520
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).Flush()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:370 +0x34
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).WriteStop()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:149 +0x60
  github.com/hangxie/parquet-tools/cmd.ImportCmd.importJSON()
      /Users/xiehang/dev/parquet/parquet-tools/cmd/import.go:116 +0x238
  github.com/hangxie/parquet-tools/cmd.Test_ImportCmd_importJSON_schema_mismatch()
      /Users/xiehang/dev/parquet/parquet-tools/cmd/import_test.go:247 +0xfc
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1595 +0x194
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1648 +0x40

Previous write at 0x00c000a5dc50 by goroutine 723:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1.1()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:297 +0x1c8
  runtime.gopanic()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/runtime/panic.go:920 +0x268
  github.com/xitongsys/parquet-go/encoding.WritePlainINT32()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/encoding/encodingwrite.go:75 +0x3bc
  github.com/xitongsys/parquet-go/encoding.WritePlain()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/encoding/encodingwrite.go:43 +0x408
  github.com/xitongsys/parquet-go/layout.(*Page).EncodingValues()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/layout/page.go:189 +0x140
  github.com/xitongsys/parquet-go/layout.(*Page).DataPageCompress()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/layout/page.go:216 +0x360
  github.com/xitongsys/parquet-go/layout.TableToDataPages()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/layout/page.go:129 +0xb58
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:328 +0x3cc
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func2()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:336 +0x60

Goroutine 714 (running) created at:
  testing.(*T).Run()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1648 +0x5d8
  testing.runTests.func1()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:2054 +0x80
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1595 +0x194
  testing.runTests()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:2052 +0x6d8
  testing.(*M).Run()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1925 +0x908
  main.main()
      _testmain.go:273 +0x2b4

Goroutine 723 (finished) created at:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:289 +0x294
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).Flush()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:370 +0x34
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).WriteStop()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:149 +0x60
  github.com/hangxie/parquet-tools/cmd.ImportCmd.importJSON()
      /Users/xiehang/dev/parquet/parquet-tools/cmd/import.go:116 +0x238
  github.com/hangxie/parquet-tools/cmd.Test_ImportCmd_importJSON_schema_mismatch()
      /Users/xiehang/dev/parquet/parquet-tools/cmd/import_test.go:247 +0xfc
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1595 +0x194
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1648 +0x40
@hangxie hangxie mentioned this issue Sep 17, 2023
Open
@hangxie
Copy link
Contributor Author

hangxie commented Sep 17, 2023 •
edited
Loading

Added a unit test to reproduce this problem in #560

=== RUN   TestWriteStopRaceConditionOnError
==================
WARNING: DATA RACE
Read at 0x00c0004b3fc0 by goroutine 40:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs()
      github.com/xitongsys/parquet-go/writer/writer.go:341 +0x628
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).Flush()
      github.com/xitongsys/parquet-go/writer/writer.go:370 +0x78
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).WriteStop()
      github.com/xitongsys/parquet-go/writer/writer.go:149 +0xb4
  github.com/xitongsys/parquet-go/writer.TestWriteStopRaceConditionOnError()
      github.com/xitongsys/parquet-go/writer/writer_test.go:251 +0x19c
  testing.tRunner()
      testing/testing.go:1595 +0x194
  testing.(*T).Run.func1()
      testing/testing.go:1648 +0x40

Previous write at 0x00c0004b3fc0 by goroutine 41:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1.1()
      github.com/xitongsys/parquet-go/writer/writer.go:297 +0x210
  runtime.gopanic()
      runtime/panic.go:920 +0x268
  github.com/xitongsys/parquet-go/encoding.WritePlainINT64()
      github.com/xitongsys/parquet-go/encoding/encodingwrite.go:81 +0x308
  github.com/xitongsys/parquet-go/encoding.WritePlain()
      github.com/xitongsys/parquet-go/encoding/encodingwrite.go:45 +0x354
  github.com/xitongsys/parquet-go/layout.(*Page).EncodingValues()
      github.com/xitongsys/parquet-go/layout/page.go:189 +0x140
  github.com/xitongsys/parquet-go/layout.(*Page).DataPageCompress()
      github.com/xitongsys/parquet-go/layout/page.go:216 +0x360
  github.com/xitongsys/parquet-go/layout.TableToDataPages()
      github.com/xitongsys/parquet-go/layout/page.go:129 +0xb58
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1()
      github.com/xitongsys/parquet-go/writer/writer.go:328 +0x468
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func2()
      github.com/xitongsys/parquet-go/writer/writer.go:336 +0x60

Goroutine 40 (running) created at:
  testing.(*T).Run()
      testing/testing.go:1648 +0x5d8
  testing.runTests.func1()
      testing/testing.go:2054 +0x80
  testing.tRunner()
      testing/testing.go:1595 +0x194
  testing.runTests()
      testing/testing.go:2052 +0x6d8
  testing.(*M).Run()
      testing/testing.go:1925 +0x908
  main.main()
      _testmain.go:103 +0x2b4

Goroutine 41 (finished) created at:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs()
      github.com/xitongsys/parquet-go/writer/writer.go:289 +0x328
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).Flush()
      github.com/xitongsys/parquet-go/writer/writer.go:370 +0x78
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).WriteStop()
      github.com/xitongsys/parquet-go/writer/writer.go:149 +0xb4
  github.com/xitongsys/parquet-go/writer.TestWriteStopRaceConditionOnError()
      github.com/xitongsys/parquet-go/writer/writer_test.go:251 +0x19c
  testing.tRunner()
      testing/testing.go:1595 +0x194
  testing.(*T).Run.func1()
      testing/testing.go:1648 +0x40
==================
    testing.go:1465: race detected during execution of test
--- FAIL: TestWriteStopRaceConditionOnError (0.00s)
=== NAME
    testing.go:1465: race detected during execution of test
FAIL
coverage: 77.8% of statements
FAIL	github.com/xitongsys/parquet-go/writer	0.272s
FAIL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@hangxie