Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update apache/arrow library to latest to fix slice bounds out of range #37588

Merged
merged 5 commits into from
Jan 10, 2024
Merged

update apache/arrow library to latest to fix slice bounds out of range #37588

merged 5 commits into from
Jan 10, 2024

Conversation

kaiyan-sheng
Copy link
Contributor

Proposed commit message

When running test decoding of a parquet file and compare the number of events along with the content test in decoding_test.go locally, it fails with a run time error slice bounds out of range:

panic: runtime error: slice bounds out of range [:8] with capacity 0

goroutine 212 [running]:
github.com/apache/arrow/go/v12/parquet/internal/encoding.(*PlainByteArrayDecoder).DecodeSpaced(0x14000c86300?, {0x0?, 0x1f?, 0x1?}, 0x1f?, {0x14000c86500?, 0x102d90460?, 0x14000cb21c0?}, 0x14000c86301?)
	/Users/kaiyansheng/go/pkg/mod/github.com/apache/arrow/go/[email protected]/parquet/internal/encoding/byte_array_decoder.go:83 +0x130
github.com/apache/arrow/go/v12/parquet/file.(*byteArrayRecordReader).ReadValuesSpaced(0x1400077c080, 0x0, 0x20?)
	/Users/kaiyansheng/go/pkg/mod/github.com/apache/arrow/go/[email protected]/parquet/file/record_reader.go:841 +0x134
github.com/apache/arrow/go/v12/parquet/file.(*recordReader).ReadRecordData(0x140000b8660, 0x1)
	/Users/kaiyansheng/go/pkg/mod/github.com/apache/arrow/go/[email protected]/parquet/file/record_reader.go:548 +0x288
github.com/apache/arrow/go/v12/parquet/file.(*recordReader).ReadRecords(0x140000b8660, 0x1)
	/Users/kaiyansheng/go/pkg/mod/github.com/apache/arrow/go/[email protected]/parquet/file/record_reader.go:632 +0x330
github.com/apache/arrow/go/v12/parquet/pqarrow.(*leafReader).LoadBatch(0x140000b86c0, 0x1)
	/Users/kaiyansheng/go/pkg/mod/github.com/apache/arrow/go/[email protected]/parquet/pqarrow/column_readers.go:109 +0xe0
github.com/apache/arrow/go/v12/parquet/pqarrow.(*structReader).LoadBatch.func1()
	/Users/kaiyansheng/go/pkg/mod/github.com/apache/arrow/go/[email protected]/parquet/pqarrow/column_readers.go:236 +0x30
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/Users/kaiyansheng/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 150
	/Users/kaiyansheng/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x98

The same test has been failing in build as well. I believe this is a bug in Apache Arrow and has been fixed in apache/arrow#37712. With the new version of Apache Arrow, this test passed locally.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@kaiyan-sheng kaiyan-sheng requested review from a team as code owners January 10, 2024 03:35
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 10, 2024
@kaiyan-sheng kaiyan-sheng self-assigned this Jan 10, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 10, 2024

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kaiyan-sheng? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 10, 2024
@elasticmachine
Copy link
Collaborator

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Start Time: 2024-01-10T03:35:41.231+0000

  • Duration: 60 min 4 sec

Test stats 🧪

Test Results
Failed 0
Passed 27604
Skipped 1931
Total 29535

Steps errors 1

Expand to view the steps failures

Error signal
  • Took 0 min 0 sec . View more details here
  • Description: Error 'org.jenkinsci.plugins.workflow.steps.FlowInterruptedException'

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@elasticmachine
Copy link
Collaborator

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2024-01-10T04:28:48.521+0000

  • Duration: 130 min 54 sec

Test stats 🧪

Test Results
Failed 4
Passed 28738
Skipped 2015
Total 30757

Test errors 4

Expand to view the tests failures

Build&Test / x-pack/libbeat-unitTest / TestParquetWithRandomData/Test_parquet_files_with_rows=20,_and_columns=10 – github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestParquetWithRandomData/Test_parquet_files_with_rows=20,_and_columns=10
--- FAIL: TestParquetWithRandomData/Test_parquet_files_with_rows=20,_and_columns=10 (0.00s)

Build&Test / x-pack/libbeat-unitTest / TestParquetWithRandomData – github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestParquetWithRandomData
--- FAIL: TestParquetWithRandomData (0.00s)
panic: arrow: field with nil DataType [recovered]
	panic: arrow: field with nil DataType

goroutine 7 [running]:
testing.tRunner.func1.2({0x131c500, 0x166b990})
	/var/lib/jenkins/workspace/PR-37588-2-22e6715f-38e6-49c1-9b29-90c4ae5a5fc7/.gvm/versions/go1.21.5.linux.amd64/src/testing/testing.go:1545 +0x238
testing.tRunner.func1()
	/var/lib/jenkins/workspace/PR-37588-2-22e6715f-38e6-49c1-9b29-90c4ae5a5fc7/.gvm/versions/go1.21.5.linux.amd64/src/testing/testing.go:1548 +0x397
panic({0x131c500?, 0x166b990?})
	/var/lib/jenkins/workspace/PR-37588-2-22e6715f-38e6-49c1-9b29-90c4ae5a5fc7/.gvm/versions/go1.21.5.linux.amd64/src/runtime/panic.go:914 +0x21f
github.com/apache/arrow/go/v14/arrow.NewSchemaWithEndian({0xc0003fce00, 0x14, 0xc00008edd8?}, 0x0, 0x0)
	/var/lib/jenkins/workspace/PR-37588-2-22e6715f-38e6-49c1-9b29-90c4ae5a5fc7/pkg/mod/github.com/apache/arrow/go/[email protected]/arrow/schema.go:182 +0x4b4
github.com/apache/arrow/go/v14/arrow.NewSchema(...)
	/var/lib/jenkins/workspace/PR-37588-2-22e6715f-38e6-49c1-9b29-90c4ae5a5fc7/pkg/mod/github.com/apache/arrow/go/[email protected]/arrow/schema.go:168
github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet.createRandomParquet({0x169b1b8, 0xc00031d380}, {0xc00004c690, 0x6a}, 0xa, 0x14)
	/var/lib/jenkins/workspace/PR-37588-2-22e6715f-38e6-49c1-9b29-90c4ae5a5fc7/src/github.com/elastic/beats/x-pack/libbeat/reader/parquet/parquet_test.go:118 +0x277
github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet.TestParquetWithRandomData.func1(0xc00031d380)
	/var/lib/jenkins/workspace/PR-37588-2-22e6715f-38e6-49c1-9b29-90c4ae5a5fc7/src/github.com/elastic/beats/x-pack/libbeat/reader/parquet/parquet_test.go:63 +0x118
testing.tRunner(0xc00031d380, 0xc0003a8b70)
	/var/lib/jenkins/workspace/PR-37588-2-22e6715f-38e6-49c1-9b29-90c4ae5a5fc7/.gvm/versions/go1.21.5.linux.amd64/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 6
	/var/lib/jenkins/workspace/PR-37588-2-22e6715f-38e6-49c1-9b29-90c4ae5a5fc7/.gvm/versions/go1.21.5.linux.amd64/src/testing/testing.go:1648 +0x3ad

Build&Test / x-pack/libbeat-goIntegTest / TestParquetWithRandomData/Test_parquet_files_with_rows=20,_and_columns=10 – github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestParquetWithRandomData/Test_parquet_files_with_rows=20,_and_columns=10
--- FAIL: TestParquetWithRandomData/Test_parquet_files_with_rows=20,_and_columns=10 (0.00s)

Build&Test / x-pack/libbeat-goIntegTest / TestParquetWithRandomData – github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet
    Expand to view the error details

     Failed

    Expand to view the stacktrace

     === RUN   TestParquetWithRandomData
--- FAIL: TestParquetWithRandomData (0.00s)
panic: arrow: field with nil DataType [recovered]
	panic: arrow: field with nil DataType

goroutine 20 [running]:
testing.tRunner.func1.2({0x131c500, 0x166b990})
	/var/lib/jenkins/workspace/PR-37588-2-0f4b8443-e790-40a7-a059-72d9fc65ddbe/.gvm/versions/go1.21.5.linux.amd64/src/testing/testing.go:1545 +0x238
testing.tRunner.func1()
	/var/lib/jenkins/workspace/PR-37588-2-0f4b8443-e790-40a7-a059-72d9fc65ddbe/.gvm/versions/go1.21.5.linux.amd64/src/testing/testing.go:1548 +0x397
panic({0x131c500?, 0x166b990?})
	/var/lib/jenkins/workspace/PR-37588-2-0f4b8443-e790-40a7-a059-72d9fc65ddbe/.gvm/versions/go1.21.5.linux.amd64/src/runtime/panic.go:914 +0x21f
github.com/apache/arrow/go/v14/arrow.NewSchemaWithEndian({0xc000454000, 0x14, 0xc000089dd8?}, 0x0, 0x0)
	/var/lib/jenkins/workspace/PR-37588-2-0f4b8443-e790-40a7-a059-72d9fc65ddbe/pkg/mod/github.com/apache/arrow/go/[email protected]/arrow/schema.go:182 +0x4b4
github.com/apache/arrow/go/v14/arrow.NewSchema(...)
	/var/lib/jenkins/workspace/PR-37588-2-0f4b8443-e790-40a7-a059-72d9fc65ddbe/pkg/mod/github.com/apache/arrow/go/[email protected]/arrow/schema.go:168
github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet.createRandomParquet({0x169b1b8, 0xc000105d40}, {0xc000332540, 0x6a}, 0xa, 0x14)
	/var/lib/jenkins/workspace/PR-37588-2-0f4b8443-e790-40a7-a059-72d9fc65ddbe/src/github.com/elastic/beats/x-pack/libbeat/reader/parquet/parquet_test.go:118 +0x277
github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet.TestParquetWithRandomData.func1(0xc000105d40)
	/var/lib/jenkins/workspace/PR-37588-2-0f4b8443-e790-40a7-a059-72d9fc65ddbe/src/github.com/elastic/beats/x-pack/libbeat/reader/parquet/parquet_test.go:63 +0x118
testing.tRunner(0xc000105d40, 0xc0003bab58)
	/var/lib/jenkins/workspace/PR-37588-2-0f4b8443-e790-40a7-a059-72d9fc65ddbe/.gvm/versions/go1.21.5.linux.amd64/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 19
	/var/lib/jenkins/workspace/PR-37588-2-0f4b8443-e790-40a7-a059-72d9fc65ddbe/.gvm/versions/go1.21.5.linux.amd64/src/testing/testing.go:1648 +0x3ad

Steps errors 9

Expand to view the steps failures

x-pack/filebeat-goIntegTest - mage goIntegTest
  • Took 6 min 54 sec . View more details here
  • Description: mage goIntegTest
x-pack/filebeat-windows-2022-windows-2022 - mage build unitTest
  • Took 15 min 55 sec . View more details here
  • Description: mage build unitTest
x-pack/libbeat-unitTest - mage build unitTest
  • Took 5 min 39 sec . View more details here
  • Description: mage build unitTest
x-pack/libbeat-unitTest - mage build unitTest
  • Took 1 min 41 sec . View more details here
  • Description: mage build unitTest
x-pack/libbeat-unitTest - mage build unitTest
  • Took 1 min 41 sec . View more details here
  • Description: mage build unitTest
x-pack/libbeat-goIntegTest - mage goIntegTest
  • Took 5 min 15 sec . View more details here
  • Description: mage goIntegTest
x-pack/libbeat-goIntegTest - mage goIntegTest
  • Took 1 min 21 sec . View more details here
  • Description: mage goIntegTest
x-pack/libbeat-goIntegTest - mage goIntegTest
  • Took 1 min 17 sec . View more details here
  • Description: mage goIntegTest
Error signal
  • Took 0 min 0 sec . View more details here
  • Description: Error 'hudson.AbortException: script returned exit code 1'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors 4

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: Build&Test / x-pack/libbeat-unitTest / TestParquetWithRandomData/Test_parquet_files_with_rows=20,_and_columns=10 – github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet
  • Name: Build&Test / x-pack/libbeat-unitTest / TestParquetWithRandomData – github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet
  • Name: Build&Test / x-pack/libbeat-goIntegTest / TestParquetWithRandomData/Test_parquet_files_with_rows=20,_and_columns=10 – github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet
  • Name: Build&Test / x-pack/libbeat-goIntegTest / TestParquetWithRandomData – github.com/elastic/beats/v7/x-pack/libbeat/reader/parquet

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@elasticmachine
Copy link
Collaborator

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Start Time: 2024-01-10T16:39:16.017+0000

  • Duration: 9 min 21 sec

Test stats 🧪

Test Results
Failed 0
Passed 3
Skipped 0
Total 3

Steps errors 2

Expand to view the steps failures

Cleanup
  • Took 0 min 0 sec . View more details here
  • Description: rm source.tgz
Error signal
  • Took 0 min 0 sec . View more details here
  • Description: untar: step failed with error null

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2024-01-10T16:43:53.323+0000

  • Duration: 164 min 1 sec

Test stats 🧪

Test Results
Failed 0
Passed 28762
Skipped 2015
Total 30777

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2024-01-10T20:50:05.604+0000

  • Duration: 165 min 46 sec

Test stats 🧪

Test Results
Failed 0
Passed 28762
Skipped 2015
Total 30777

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@kaiyan-sheng kaiyan-sheng merged commit fffe228 into elastic:main Jan 10, 2024
@kaiyan-sheng kaiyan-sheng deleted the fix_filebeat_build branch January 10, 2024 23:43
Scholar-Li pushed a commit to Scholar-Li/beats that referenced this pull request Feb 5, 2024
@kaiyan-sheng @Scholar-Li
update apache/arrow library to latest to fix slice bounds out of range (
4fbd601 
elastic#37588)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cmacknz cmacknz cmacknz approved these changes

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@kaiyan-sheng @elasticmachine @cmacknz @andrewkroh