Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[storage-file-datalake] Allow DataLakePathClient.move() use sas to authenticate source and destination #12769

Merged
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
fc4f1a3
Make storage-file-datalake move() support adding both sasToken to x-m…
JasonYeMSFT Dec 3, 2020
200041b
Format
JasonYeMSFT Dec 3, 2020
6220e09
Add comment to util function
JasonYeMSFT Dec 3, 2020
4812421
Fallback to use the source's SAS if both
JasonYeMSFT Dec 4, 2020
c9c8977
Reformat
JasonYeMSFT Dec 4, 2020
645f60d
Do not encode sas query parameter values in x-ms-rename-source header
JasonYeMSFT Dec 4, 2020
e8b5d30
Don't set query parameter for snapshot or versionid at the destinatio…
JasonYeMSFT Dec 7, 2020
77b734d
Reformat
JasonYeMSFT Dec 7, 2020
665baf9
Merge branch 'master' of https://github.com/Azure/azure-sdk-for-js in…
JasonYeMSFT Dec 9, 2020
f5ec03d
Resolve merge conflict
JasonYeMSFT Dec 9, 2020
7a779f6
Merge branch 'master' of https://github.com/Azure/azure-sdk-for-js in…
JasonYeMSFT Dec 11, 2020
9f9f29c
Remove fallback logic
JasonYeMSFT Dec 11, 2020
957145d
Encode source path segment by segment to avoid encoding "/" characters
JasonYeMSFT Dec 11, 2020
b7df555
Merge branch 'master' of https://github.com/Azure/azure-sdk-for-js in…
JasonYeMSFT Dec 14, 2020
b3a3b92
Merge branch 'master' of https://github.com/Azure/azure-sdk-for-js in…
JasonYeMSFT Dec 21, 2020
203e012
Add unit tests for move operation with SAS credentials
JasonYeMSFT Dec 21, 2020
067ee46
Merge branch 'master' of https://github.com/Azure/azure-sdk-for-js in…
JasonYeMSFT Dec 23, 2020
f9bc999
Add move related test recordings
JasonYeMSFT Dec 23, 2020
8e3da4a
Merge branch 'master' of https://github.com/Azure/azure-sdk-for-js in…
JasonYeMSFT Jan 6, 2021
055b3c9
Add a bullet for the fix in the change log
JasonYeMSFT Jan 6, 2021
0f964d5
Take PR suggestion
JasonYeMSFT Jan 6, 2021
4ec4e4c
Merge branch 'master' into dev/chuye/fix-file-datalake-move-source-sas
ljian3377 Jan 6, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 31 additions & 5 deletions sdk/storage/storage-file-datalake/src/clients.ts
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,13 @@ import {
} from "./utils/constants";
import { DataLakeAclChangeFailedError } from "./utils/DataLakeAclChangeFailedError";
import { createSpan } from "./utils/tracing";
import { appendToURLPath, setURLPath } from "./utils/utils.common";
import {
appendToURLPath,
encodeURLPathQueries,
getURLQueryString,
setURLPath,
setURLQueries
} from "./utils/utils.common";
import { fsCreateReadStream, fsStat } from "./utils/utils.node";

/**
Expand Down Expand Up @@ -877,7 +883,8 @@ export class DataLakePathClient extends StorageClient {
*
* @see https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/create
*
* @param {string} destinationPath Destination directory path like "directory" or file path "directory/file"
* @param {string} destinationPath Destination directory path like "directory" or file path "directory/file".
* If the destinationPath is authenticated with SAS, add the SAS to the destination path like "directory/file?sasToken".
* @param {PathMoveOptions} [options] Optional. Options when moving directory or file.
* @returns {Promise<PathMoveResponse>}
* @memberof DataLakePathClient
Expand All @@ -891,6 +898,7 @@ export class DataLakePathClient extends StorageClient {
*
* @param {string} destinationFileSystem Destination file system like "filesystem".
* @param {string} destinationPath Destination directory path like "directory" or file path "directory/file"
* If the destinationPath is authenticated with SAS, add the SAS to the destination path like "directory/file?sasToken".
* @param {PathMoveOptions} [options] Optional. Options when moving directory or file.
* @returns {Promise<PathMoveResponse>}
* @memberof DataLakePathClient
Expand Down Expand Up @@ -924,10 +932,28 @@ export class DataLakePathClient extends StorageClient {

// Be aware that decodeURIComponent("%27") = "'"; but encodeURIComponent("'") = "'".
// But since both ' and %27 work with the service here so we omit replace(/'/g, "%27").
const renameSource = `/${this.fileSystemName}/${encodeURIComponent(this.name)}`;
const renameDestination = `/${destinationFileSystem}/${destinationPath}`;
const sourceSas = getURLQueryString(this.dfsEndpointUrl);
JasonYeMSFT marked this conversation as resolved.
Show resolved Hide resolved
const renameSource = !!sourceSas
? encodeURLPathQueries(
`/${this.fileSystemName}/${encodeURIComponent(this.name)}?${sourceSas}`
)
: `/${this.fileSystemName}/${encodeURIComponent(this.name)}`;

const split: string[] = destinationPath.split("?");
let destinationUrl: string;
if (split.length === 2) {
JasonYeMSFT marked this conversation as resolved.
Show resolved Hide resolved
const renameDestination = `/${destinationFileSystem}/${split[0]}`;
destinationUrl = setURLPath(this.dfsEndpointUrl, renameDestination);
destinationUrl = setURLQueries(destinationUrl, split[1]);
} else {
const renameDestination = `/${destinationFileSystem}/${destinationPath}`;
destinationUrl = setURLPath(this.dfsEndpointUrl, renameDestination);
// If the source is authenticating using SAS while the destination doesn't have a SAS, fallback to use the source's SAS on the destination
if (!!sourceSas) {
destinationUrl = setURLQueries(destinationUrl, sourceSas);
}
}

const destinationUrl = setURLPath(this.dfsEndpointUrl, renameDestination);
const destPathClient = new DataLakePathClient(destinationUrl, this.pipeline);

try {
Expand Down
56 changes: 56 additions & 0 deletions sdk/storage/storage-file-datalake/src/utils/utils.common.ts
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,62 @@ export function getURLQueries(url: string): { [key: string]: string } {
return queries;
}

/**
* Get URL query string.
*
* @param {string} url
*/
export function getURLQueryString(url: string): string | undefined {
const urlParsed = URLBuilder.parse(url);
return urlParsed.getQuery();
}

/**
* Set URL query string.
*
* @param {string} url
* @param {string} queryString
*/
export function setURLQueries(url: string, queryString: string): string {
const urlParsed = URLBuilder.parse(url);
urlParsed.setQuery(queryString);
return urlParsed.toString();
}

/**
* Encode URL path query values from an URL string.
*
* @example {filesystem}/{file}?{sasToken}
*
* @export
* @param {string} urlPath
* @returns {string}
*/
export function encodeURLPathQueries(urlPath: string): string {
const split = urlPath.split("?");
if (split.length === 2) {
let queryString = split[1];
queryString = queryString.trim();

let querySubStrings: string[] = queryString.split("&");
querySubStrings = querySubStrings.filter((value: string) => {
const indexOfEqual = value.indexOf("=");
const lastIndexOfEqual = value.lastIndexOf("=");
return (
indexOfEqual > 0 && indexOfEqual === lastIndexOfEqual && lastIndexOfEqual < value.length - 1
);
});
querySubStrings = querySubStrings.map((value: string) => {
const [queryKey, queryValue] = value.split("=");
return `${queryKey}=${encodeURIComponent(queryValue)}`;
});
const encodedQueryString = querySubStrings.join("&");
return `${split[0]}?${encodedQueryString}`;
} else {
return urlPath;
}
}

/**
* Rounds a date off to seconds.
*
Expand Down