Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docs changes for Hadoop FS default #23366

Merged
merged 1 commit into from
Sep 13, 2024
Merged

Docs changes for Hadoop FS default #23366

merged 1 commit into from
Sep 13, 2024

Conversation

Joelg96
Copy link
Contributor

@Joelg96 Joelg96 commented Sep 11, 2024 •
edited by mosabua
Loading

Description

  • Hadoop FS will now default to false.
  • Removed "and deactivating all legacy support".
  • Added a requirement to each object storage connector to state that selecting a filesystem is a requirement, and included example property.

Additional context and related issues

#23343

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

@cla-bot cla-bot bot added the cla-signed label Sep 11, 2024
@Joelg96 Joelg96 requested a review from jhlodin September 11, 2024 17:18
@github-actions github-actions bot added the docs label Sep 11, 2024
@Joelg96
Copy link
Contributor Author

Joelg96 commented Sep 11, 2024 •
edited
Loading

Just saw the conversation in Slack... Here is the release note Anu proposed:

  • Disable legacy hadoop filesystem fs.hadoop.enabled config by default. Users will have to now explicitly enable the filesystem implementation to use.

@Joelg96 Joelg96 closed this Sep 11, 2024
@Joelg96 Joelg96 reopened this Sep 11, 2024
jhlodin
jhlodin reviewed Sep 11, 2024
View reviewed changes
Copy link
Contributor

@jhlodin jhlodin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left my comments, but since time is of the essence I think we should open this up to engineering review and not leave in draft

docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/hive.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/hive.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/iceberg.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/iceberg.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/iceberg.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage/file-system-hdfs.md Show resolved Hide resolved
@Joelg96 Joelg96 force-pushed the jg/hadoop-fs-default-changes branch from a1c39c3 to 6fd1c19 Compare September 11, 2024 19:45
@Joelg96 Joelg96 requested a review from mosabua September 11, 2024 19:46
@Joelg96 Joelg96 marked this pull request as ready for review September 11, 2024 19:47
@Joelg96 Joelg96 requested a review from anusudarsan September 11, 2024 19:48
mosabua
mosabua requested changes Sep 11, 2024
View reviewed changes
Copy link
Member

@mosabua mosabua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the Delta lake based on the comments and then apply similar changes to all the others

docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/delta-lake.md Show resolved Hide resolved
docs/src/main/sphinx/connector/hive.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage/file-system-hdfs.md Show resolved Hide resolved
@Joelg96 Joelg96 force-pushed the jg/hadoop-fs-default-changes branch from 6fd1c19 to b7715a7 Compare September 11, 2024 20:59
@Joelg96 Joelg96 requested a review from mosabua September 11, 2024 21:00
mosabua
mosabua requested changes Sep 11, 2024
View reviewed changes
Copy link
Member

@mosabua mosabua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we still also have to update the legacy pages some more - please have a look

docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/hive.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/hive.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/hudi.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/hudi.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage/file-system-hdfs.md Show resolved Hide resolved
docs/src/main/sphinx/object-storage/file-system-hdfs.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage/file-system-hdfs.md Outdated Show resolved Hide resolved
@mosabua
Copy link
Member

mosabua commented Sep 11, 2024

I just checked the legacy pages we definitely have to add details about the property now being required to be sent to true for these legacy systems to work.

In fact each one of the legacy painted should have a warning to say that user who why are you in here cause its should use the new one

@mosabua mosabua mentioned this pull request Sep 11, 2024
Merged
@Joelg96 Joelg96 force-pushed the jg/hadoop-fs-default-changes branch 3 times, most recently from a96872f to c30edcb Compare September 12, 2024 16:54
@Joelg96 Joelg96 requested a review from mosabua September 12, 2024 16:54
@Joelg96 Joelg96 force-pushed the jg/hadoop-fs-default-changes branch 2 times, most recently from 42e4642 to 739edf0 Compare September 12, 2024 19:37
mosabua
mosabua requested changes Sep 12, 2024
View reviewed changes
docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/hive.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/hive.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/hudi.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage/file-system-gcs.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage/file-system-hdfs.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage/legacy-azure.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage/legacy-azure.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage/legacy-azure.md Outdated Show resolved Hide resolved
@Joelg96 Joelg96 force-pushed the jg/hadoop-fs-default-changes branch from 739edf0 to 72cc731 Compare September 12, 2024 21:16
@Joelg96 Joelg96 requested a review from mosabua September 12, 2024 21:17
@mosabua
Copy link
Member

mosabua commented Sep 12, 2024

i just noticed we aloso need to update the ibm cos page

@Joelg96 Joelg96 force-pushed the jg/hadoop-fs-default-changes branch from 72cc731 to 1de14d1 Compare September 13, 2024 14:32
@Joelg96
Copy link
Contributor Author

Joelg96 commented Sep 13, 2024

@mosabua, I updated the IBM COS page, let me know if you think that link should direct users elsewhere.

mosabua
mosabua requested changes Sep 13, 2024
View reviewed changes
Copy link
Member

@mosabua mosabua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple things still need adjustment

docs/src/main/sphinx/object-storage.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/object-storage.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved
docs/src/main/sphinx/connector/delta-lake.md Show resolved Hide resolved
docs/src/main/sphinx/object-storage/legacy-azure.md Outdated Show resolved Hide resolved
@Joelg96 Joelg96 force-pushed the jg/hadoop-fs-default-changes branch from 1de14d1 to 33685ad Compare September 13, 2024 19:58
mosabua
mosabua approved these changes Sep 13, 2024
View reviewed changes
Copy link
Member

@mosabua mosabua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now. Thank you.

@mosabua mosabua merged commit cf1d19c into trinodb:master Sep 13, 2024
4 of 8 checks passed
@github-actions github-actions bot added this to the 458 milestone Sep 13, 2024
@mosabua mosabua mentioned this pull request Sep 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jhlodin jhlodin jhlodin left review comments

@mosabua mosabua mosabua approved these changes

@anusudarsan anusudarsan Awaiting requested review from anusudarsan

Assignees
No one assigned
Labels
cla-signed docs
Milestone
458
Development

Successfully merging this pull request may close these issues.
3 participants
@Joelg96 @mosabua @jhlodin