[#5472] improvement(docs): Add example to use cloud storage fileset and polish hadoop-catalog document. #6059

yuqi1129 · 2025-01-02T04:56:46Z

What changes were proposed in this pull request?

Add full example about how to use cloud storage fileset like S3, GCS, OSS and ADLS
Polish how-to-use-gvfs.md and hadoop-catalog-md.
Add document how fileset using credential.

Why are the changes needed?

For better user experience.

Fix: #5472

Does this PR introduce any user-facing change?

N/A.

How was this patch tested?

N/A

docs/cloud-storage-fileset-example.md

docs/hadoop-catalog-with-gcs.md

FANNG1 · 2025-01-06T14:58:25Z

docs/hadoop-catalog-with-s3.md

+## Prerequisites
+
+In order to create a Hadoop catalog with S3, you need to place [`gravitino-aws-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws-bundle) in Gravitino Hadoop classpath located 
+at `${HADOOP_HOME}/share/hadoop/common/lib/`. After that, start Gravitino server with the following command:


in Gravitino Hadoop classpath located at ${HADOOP_HOME}/share/hadoop/common/lib/ , use hadoop catalog classpath not hadoop classpath?

The user should place the jar in {GRAVITINO_HOME}/catalogs/hadoop/libs not ${HADOOP_HOME}/share/hadoop/common/lib/, YES?

seems not fixed

I have confirmed that this has been fixed, can you please reresh the web and see if it has been resolved

docs/hadoop-catalog-with-s3.md

docs/hadoop-catalog.md

docs/how-to-use-gvfs.md

FANNG1 · 2025-01-13T12:14:31Z

IMO, credential vending is the critical part for the fileset use example for cloud storage, but now it lays in some inconspicuous places :(

Do you have any suggestions? The credential part is already with Header 2. do you mean I should move forward this part?

Instead of placing it in a separate part at the last which seems optional and not important, place it in the catalog properties part like ### Configurations for S3 Hadoop Catalog, for most examples use credential vending by default.

yuqi1129 · 2025-01-13T12:21:27Z

IMO, credential vending is the critical part for the fileset use example for cloud storage, but now it lays in some inconspicuous places :(

Do you have any suggestions? The credential part is already with Header 2. do you mean I should move forward this part?

Instead of placing it in a separate part at the last which seems optional and not important, place it in the catalog properties part like ### Configurations for S3 Hadoop Catalog, for most examples use credential vending by default.

I don't believe putting them last means they are not important.
The configuration items for Hadoop catalog in hadoop-catalog.md and for GVFS in credential-vending.md, Do I need make a copy here?

yuqi1129 · 2025-01-13T12:33:39Z

IMO, credential vending is the critical part for the fileset use example for cloud storage, but now it lays in some inconspicuous places :(

Do you have any suggestions? The credential part is already with Header 2. do you mean I should move forward this part?

Instead of placing it in a separate part at the last which seems optional and not important, place it in the catalog properties part like ### Configurations for S3 Hadoop Catalog, for most examples use credential vending by default.

I don't believe putting them last means they are not important.

The configuration items for Hadoop catalog in hadoop-catalog.md and for GVFS in credential-vending.md, Do I need make a copy here?

Other points are

Credential vending may be quite an advanced feature and should be cautious in its usage. I'm afraid this will make primary users difficult to use the example.
I'm not very sure if we need to put it in the basic example just like I have put S3 example in the hadoop-catalog.md which is intentional for HDFS.

If you have any further thoughts on it, please let me know your thoughts and ideas.

mchades · 2025-01-13T14:04:19Z

Credential vending may be quite an advanced feature

Strongly agree that Credential vending is an advanced feature. We can distinguish it from simple examples, using different sections or even separate documents; otherwise, we will make simple examples not simple.

docs/hadoop-catalog-with-adls.md

docs/hadoop-catalog-with-gcs.md

FANNG1 · 2025-01-14T06:22:32Z

docs/hadoop-catalog-with-adls.md

+license: "This software is licensed under the Apache License version 2."
+---
+
+This document describes how to configure a Hadoop catalog with ADLS (Azure Blob Storage).


ADLS is based on Azure Blob Storage, but not Azure Blob Storage

Here, we support Azure Blob Storage, ADLS, and ADLS (v2), I used Azure Blob Storage for all, but there are no abbreviations for "Azure Blob Storage", so I used ADLS to stands for storage service provided by Azure.

Is there a good term to describe all?

seems ADLS is enough

I use this sentences to replace it ADLS (aka. Azure Blob Storage (ABS), or Azure Data Lake Storage (v2))

ADLS couldn't represent ABS, IMO, azure hadoop connector just supports ADLS

FANNG1 · 2025-01-14T06:47:48Z

LGTM except minor comments

…e and account key in python client accordingly.

FANNG1 · 2025-01-14T08:27:54Z

@jerryshao @mchades any other comments?

jerryshao · 2025-01-14T08:35:19Z

I don't have further comment, @mchades can also take a review.

mchades · 2025-01-14T08:46:47Z

docs/hadoop-catalog-index.md

+
+- [Hadoop catalog overview and features](./hadoop-catalog.md): This chapter provides an overview of the Hadoop catalog, its features, capabilities and related configurations.
+- [Manage Hadoop catalog with Gravitino API](./manage-fileset-metadata-using-gravitino.md): This chapter explains how to manage fileset metadata using Gravitino API and provides detailed examples.
+- [Using Hadoop catalog with Gravitino virtual System](how-to-use-gvfs.md): This chapter explains how to use Hadoop catalog with Gravitino virtual System and provides detailed examples.


Gravitino virtual System -> Gravitino virtual file system

…nd polish hadoop-catalog document. (#6059) ### What changes were proposed in this pull request? 1. Add full example about how to use cloud storage fileset like S3, GCS, OSS and ADLS 2. Polish how-to-use-gvfs.md and hadoop-catalog-md. 3. Add document how fileset using credential. ### Why are the changes needed? For better user experience. Fix: #5472 ### Does this PR introduce _any_ user-facing change? N/A. ### How was this patch tested? N/A

yuqi1129 added 2 commits January 2, 2025 12:54

fix

c4fb29a

Merge branch 'main' of github.com:datastrato/graviton into 5472

4791a64

yuqi1129 self-assigned this Jan 2, 2025

yuqi1129 requested a review from jerryshao January 2, 2025 11:42

tengqm reviewed Jan 3, 2025

View reviewed changes

yuqi1129 added 2 commits January 3, 2025 10:19

fix

baf42e1

Merge branch 'main' of github.com:datastrato/graviton into 5472

d86610b

jerqi changed the title ~~[#5472] improvement(docs): Add example to use cloud stroage fileset and polish hadoop-catalog document.~~ [#5472] improvement(docs): Add example to use cloud storage fileset and polish hadoop-catalog document. Jan 3, 2025

yuqi1129 added 2 commits January 3, 2025 19:35

fix

b7eb621

update the docs

1ecc378

tengqm reviewed Jan 4, 2025

View reviewed changes

yuqi1129 added 2 commits January 6, 2025 08:48

polish document again.

d232e92

Again

fbd57ba

yuqi1129 requested review from tengqm and FANNG1 and removed request for tengqm January 6, 2025 03:02

tengqm reviewed Jan 6, 2025

View reviewed changes

docs/hadoop-catalog-with-gcs.md Outdated Show resolved Hide resolved

docs/hadoop-catalog-with-gcs.md Outdated Show resolved Hide resolved

docs/hadoop-catalog-with-gcs.md Outdated Show resolved Hide resolved

yuqi1129 added 3 commits January 6, 2025 19:57

fix

4fb6e79

fix

e481c8d

fix

0a97fc7

FANNG1 reviewed Jan 6, 2025

View reviewed changes

yuqi1129 added 3 commits January 7, 2025 09:58

fix

a6fbe7b

fix

7b47a9b

Polish the doc

ab07455

mchades reviewed Jan 7, 2025

View reviewed changes

docs/hadoop-catalog.md Show resolved Hide resolved

mchades reviewed Jan 7, 2025

View reviewed changes

docs/how-to-use-gvfs.md Outdated Show resolved Hide resolved

yuqi1129 added 2 commits January 7, 2025 12:09

Optimize the docs

6c1aac3

format code.

44014d9

mchades reviewed Jan 7, 2025

View reviewed changes

docs/how-to-use-gvfs.md Outdated Show resolved Hide resolved

docs/how-to-use-gvfs.md Outdated Show resolved Hide resolved

Merge branch 'main' of github.com:datastrato/graviton into 5472

f4968bd

fix

c9eca73

fix

aacd58f

fix

d3a8986

FANNG1 reviewed Jan 14, 2025

View reviewed changes

docs/hadoop-catalog-with-adls.md Outdated Show resolved Hide resolved

fix

54536d9

jerryshao reviewed Jan 14, 2025

View reviewed changes

docs/hadoop-catalog-with-adls.md Outdated Show resolved Hide resolved

jerryshao reviewed Jan 14, 2025

View reviewed changes

docs/hadoop-catalog-with-gcs.md Outdated Show resolved Hide resolved

Merge branch 'main' of github.com:datastrato/graviton into 5472

b2e357f

FANNG1 reviewed Jan 14, 2025

View reviewed changes

yuqi1129 added 2 commits January 14, 2025 14:31

Resolve python code indent and fix table format problem.

1971ba1

Fix incompleted description about endpoint for S3

30f4271

yuqi1129 added 2 commits January 14, 2025 14:52

Optimize ADLS descriptions

65c171c

Fix the problem in apache#5737 that does not change azure account-nam…

1e155d4

…e and account key in python client accordingly.

FANNG1 previously approved these changes Jan 14, 2025

View reviewed changes

mchades reviewed Jan 14, 2025

View reviewed changes

fix

d2d2de3

yuqi1129 dismissed FANNG1’s stale review via d2d2de3 January 14, 2025 09:06

FANNG1 previously approved these changes Jan 14, 2025

View reviewed changes

fix again

e01b201

yuqi1129 dismissed FANNG1’s stale review via e01b201 January 14, 2025 09:29

mchades approved these changes Jan 14, 2025

View reviewed changes

FANNG1 merged commit 5caa9de into apache:main Jan 14, 2025
25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#5472] improvement(docs): Add example to use cloud storage fileset and polish hadoop-catalog document. #6059

[#5472] improvement(docs): Add example to use cloud storage fileset and polish hadoop-catalog document. #6059

yuqi1129 commented Jan 2, 2025 •

edited

Loading

FANNG1 Jan 6, 2025

yuqi1129 Jan 7, 2025

FANNG1 Jan 7, 2025

yuqi1129 Jan 10, 2025

FANNG1 Jan 13, 2025

yuqi1129 Jan 13, 2025

FANNG1 commented Jan 13, 2025

yuqi1129 commented Jan 13, 2025

yuqi1129 commented Jan 13, 2025

mchades commented Jan 13, 2025

FANNG1 Jan 14, 2025

yuqi1129 Jan 14, 2025

FANNG1 Jan 14, 2025

yuqi1129 Jan 14, 2025

FANNG1 Jan 14, 2025

FANNG1 commented Jan 14, 2025

FANNG1 commented Jan 14, 2025

jerryshao commented Jan 14, 2025

mchades Jan 14, 2025

[#5472] improvement(docs): Add example to use cloud storage fileset and polish hadoop-catalog document. #6059

[#5472] improvement(docs): Add example to use cloud storage fileset and polish hadoop-catalog document. #6059

Conversation

yuqi1129 commented Jan 2, 2025 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FANNG1 commented Jan 13, 2025

yuqi1129 commented Jan 13, 2025

yuqi1129 commented Jan 13, 2025

mchades commented Jan 13, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FANNG1 commented Jan 14, 2025

FANNG1 commented Jan 14, 2025

jerryshao commented Jan 14, 2025

Choose a reason for hiding this comment

yuqi1129 commented Jan 2, 2025 •

edited

Loading