[BUG] ORC writer writes the 3 byte "ORC" header even when the write operation fails #15387
Comments
3 tasks
rapids-bot bot
pushed a commit
that referenced
this issue
Apr 2, 2024
…ritten (#15393) Closes #15386, #15387 The fixes for the two issues overlap, so I included both in a single PR. Expanded the `_closed` flag to an enum that tracks if the operations in `close()` should be performed (one or more tables were written to the sink). This way, we don't perform the steps in close when there is no valid file to write the footer for. This includes: - No `write` calls; - All `write` calls failed; The new enum replaces `skip_close()` that used to fix this issue for a smaller subset of cases. Additionally, writing of the ORC header has been moved after the encode and uses the new state to only write the header in the first `write` call. This way we don't write anything to the sink if there were no `write` calls with the writer, and if the encode failed in the `write`s. Authors: - Vukasin Milovanovic (https://github.com/vuule) - Nghia Truong (https://github.com/ttnghia) Approvers: - Nghia Truong (https://github.com/ttnghia) - David Wendt (https://github.com/davidwendt) URL: #15393
|
closed via #15393 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We should not write any data to the sink unless the entire write is successful.
Proposal:
Move the writing of the header from
init_statetowrite_orc_data_to_sink. This way, any encode/compression errors will result in no changes to the sink.The text was updated successfully, but these errors were encountered: