Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-43837: [Go][IPC] Consolidate StreamWriter and FileWriter, ensuring that EOS indicator is written in file #43890

Merged
merged 1 commit into from
Aug 30, 2024

Conversation

joellubi
Copy link
Member

@joellubi joellubi commented Aug 29, 2024

Rationale for this change

Fixes: #43837

Much of the logic between the ipc stream writer and the file writer was split. This PR changes the file writer so that it uses a stream writer internally, ensuring that a valid stream is embedded within the file.

TODO

What changes are included in this PR?

  • Refactor fileWriter to embed streamWriter and defer relevant methods
  • Add test

Are these changes tested?

Yes

Are there any user-facing changes?

Go-generated IPC files will contain the EOS indicator

Copy link

⚠️ GitHub issue #43837 has been automatically assigned in GitHub to PR creator.

@joellubi
Copy link
Member Author

All "Go producing, ..." scenarios now passing in integration testing workflow

@joellubi joellubi marked this pull request as ready for review August 29, 2024 20:16
@joellubi joellubi requested a review from zeroshade as a code owner August 29, 2024 20:16
@github-actions github-actions bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Aug 29, 2024
@zeroshade
Copy link
Member

The change seems reasonable to me, but we need to figure out what's wrong with the integration tests here before we can merge this

@joellubi
Copy link
Member Author

The change seems reasonable to me, but we need to figure out what's wrong with the integration tests here before we can merge this

The integration tests were failing because I included commits from #43834 which breaks the tests for a number of implementations. I confirmed that the Go scenarios in the workflow were passing on this PR, even though other languages are still failing with the changes to the tests.

I removed the commits now and the integration testing workflow succeeded.

Copy link
Member

@zeroshade zeroshade left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@joellubi joellubi merged commit 63b34c9 into apache:main Aug 30, 2024
26 checks passed
@joellubi joellubi removed the awaiting merge Awaiting merge label Aug 30, 2024
Copy link

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 63b34c9.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

mapleFU pushed a commit to mapleFU/arrow that referenced this pull request Sep 3, 2024
…suring that EOS indicator is written in file (apache#43890)

### Rationale for this change

Fixes: apache#43837

Much of the logic between the ipc stream writer and the file writer was split. This PR changes the file writer so that it uses a stream writer internally, ensuring that a valid stream is embedded within the file.

**TODO**
- [x] Remove @ bkietz's commits

### What changes are included in this PR?

- Refactor `fileWriter` to embed `streamWriter` and defer relevant methods
- Add test

### Are these changes tested?

Yes

### Are there any user-facing changes?

Go-generated IPC files will contain the EOS indicator

* GitHub Issue: apache#43837

Authored-by: Joel Lubinitsky <[email protected]>
Signed-off-by: Joel Lubinitsky <[email protected]>
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this pull request Sep 6, 2024
…suring that EOS indicator is written in file (apache#43890)

### Rationale for this change

Fixes: apache#43837

Much of the logic between the ipc stream writer and the file writer was split. This PR changes the file writer so that it uses a stream writer internally, ensuring that a valid stream is embedded within the file.

**TODO**
- [x] Remove @ bkietz's commits

### What changes are included in this PR?

- Refactor `fileWriter` to embed `streamWriter` and defer relevant methods
- Add test

### Are these changes tested?

Yes

### Are there any user-facing changes?

Go-generated IPC files will contain the EOS indicator

* GitHub Issue: apache#43837

Authored-by: Joel Lubinitsky <[email protected]>
Signed-off-by: Joel Lubinitsky <[email protected]>
khwilson pushed a commit to khwilson/arrow that referenced this pull request Sep 14, 2024
…suring that EOS indicator is written in file (apache#43890)

### Rationale for this change

Fixes: apache#43837

Much of the logic between the ipc stream writer and the file writer was split. This PR changes the file writer so that it uses a stream writer internally, ensuring that a valid stream is embedded within the file.

**TODO**
- [x] Remove @ bkietz's commits

### What changes are included in this PR?

- Refactor `fileWriter` to embed `streamWriter` and defer relevant methods
- Add test

### Are these changes tested?

Yes

### Are there any user-facing changes?

Go-generated IPC files will contain the EOS indicator

* GitHub Issue: apache#43837

Authored-by: Joel Lubinitsky <[email protected]>
Signed-off-by: Joel Lubinitsky <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Integration][Go] IPC files do not embed a valid IPC stream
3 participants