Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use Hadoop FileSystem.openFile() to open files #3078

Closed
steveloughran opened this issue Nov 25, 2024 · 0 comments · Fixed by #3079
Closed

Use Hadoop FileSystem.openFile() to open files #3078

steveloughran opened this issue Nov 25, 2024 · 0 comments · Fixed by #3079
Labels
Type: enhancement

Comments

@steveloughran
Copy link
Contributor

steveloughran commented Nov 25, 2024 •
edited
Loading

Describe the enhancement requested

Use the hadoop openFile() builder API to open a file, passing down

  • the existing FileStatus (saves a HEAD on cloud stores)
  • a read policy appropriate for parquet files. This ensures that the object stores know what kind of GET/prefect/cache to do.

Component(s)

Core
@steveloughran steveloughran changed the title Hadoop FileSystem.openFile() to open a file, passing down length and read policy Use Hadoop FileSystem.openFile() to open a file Nov 26, 2024
@steveloughran steveloughran changed the title Use Hadoop FileSystem.openFile() to open a file Use Hadoop FileSystem.openFile() to open files Nov 26, 2024
steveloughran added a commit to steveloughran/parquet-mr that referenced this issue Nov 26, 2024
@steveloughran
Use Hadoop FileSystem.openFile() to open files
550bf8f 
* Open files with FileSystem.openFile(), passing in file status
* And read policy of "parquet, vector, random, adaptive"

Addresses apacheGH-3078
steveloughran added a commit to steveloughran/parquet-mr that referenced this issue Nov 26, 2024
@steveloughran
apacheGH-3078: Use Hadoop FileSystem.openFile() to open files
b1b490b 
* Open files with FileSystem.openFile(), passing in file status
* And read policy of "parquet, vector, random, adaptive"
@steveloughran steveloughran mentioned this issue Nov 26, 2024
Merged
steveloughran added a commit to steveloughran/parquet-mr that referenced this issue Nov 27, 2024
@steveloughran
apacheGH-3078: Use Hadoop FileSystem.openFile() to open files
ab31f8f 
* Open files with FileSystem.openFile(), passing in file status
* And read policy of "parquet, vector, random, adaptive"
steveloughran added a commit to steveloughran/parquet-mr that referenced this issue Dec 3, 2024
@steveloughran
apacheGH-3078. openFile() invocation/failure tests.
79a4c04 
The builder API is a real PITA to mock, even though it
works great in application code.

Uses the o.a.h.fs.impl implementation, and I've created
HADOOP-19355. Mark FutureDataInputStreamBuilderImpl as VisibleForTesting
to make it clear that this is allowed.
gszadovszky pushed a commit that referenced this issue Dec 4, 2024
@steveloughran
GH-3078: Use Hadoop FileSystem.openFile() to open files (#3079)
f4a3e8b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

GH-3078: Use Hadoop FileSystem.openFile() to open files steveloughran/parquet-mr
1 participant
@steveloughran