Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial checkin for Storage file #4414

Merged
merged 33 commits into from
Jul 24, 2019
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
e43a8b2
Finished the restructure, refactor builder. Added sleep in record fea…
sima-zhu Jul 12, 2019
b7c446b
Fixed the macOS
sima-zhu Jul 12, 2019
889ec07
merge from mainlnie
sima-zhu Jul 12, 2019
8334cb1
Added commit for macOS and linux for debuging the filepath
sima-zhu Jul 12, 2019
3a6470b
Fixed filePath
sima-zhu Jul 12, 2019
0ae0d8a
Remove comments
sima-zhu Jul 12, 2019
bddf4da
Keep the download file.
sima-zhu Jul 13, 2019
ca12657
Added the empty download file.
sima-zhu Jul 13, 2019
cdfeb09
Fixed the failure test in linux and mac
Jul 14, 2019
740e7e1
Fixed the upload download tests.
sima-zhu Jul 16, 2019
456deb3
Fixed the record and playback tests
sima-zhu Jul 16, 2019
b1ea3be
Fix checkstyle
sima-zhu Jul 16, 2019
7c0f87d
URL and snapshot refactor. FileService JavaDoc code snippet done.
sima-zhu Jul 18, 2019
f98107e
Fixed the failed JavaDoc
sima-zhu Jul 18, 2019
6f729e0
Share client java doc code snippet.
sima-zhu Jul 19, 2019
93a5fce
Directory client java doc code snippet.
sima-zhu Jul 19, 2019
e5f1ec7
file
sima-zhu Jul 19, 2019
efb304b
file client java doc code snippet
sima-zhu Jul 20, 2019
f6853df
Merge branch 'storage-post-review1-dev' into storage-file
sima-zhu Jul 22, 2019
67410a8
CLean up the javadoc snippet
sima-zhu Jul 22, 2019
a65ed6d
Merge branch 'storage-file' of https://github.com/sima-zhu/azure-sdk-…
sima-zhu Jul 22, 2019
67aee2a
Addressed comments
sima-zhu Jul 22, 2019
6047a3f
Addressed the comments from PR
sima-zhu Jul 22, 2019
9ceb72b
Fixed README
sima-zhu Jul 22, 2019
ddb43c1
Addressed the comments from PR and meeting,
sima-zhu Jul 23, 2019
bd80931
Addressed the comments from PR and meeting.
sima-zhu Jul 23, 2019
b615c23
Add file version.
sima-zhu Jul 23, 2019
1f83cb2
Fixed the playback test
sima-zhu Jul 23, 2019
738677a
Update storage/client/file/src/main/java/com/azure/storage/file/Direc…
sima-zhu Jul 23, 2019
d673190
Make changes to javadoc and readme
sima-zhu Jul 23, 2019
ca039a0
Merge branch 'storage-file' of https://github.com/sima-zhu/azure-sdk-…
sima-zhu Jul 23, 2019
434123e
Changed the log level to warning and make it non-static
sima-zhu Jul 23, 2019
67e7fcb
Remove the duplicate exclude rule in spotbugs
sima-zhu Jul 23, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,11 @@
<Bug pattern="NP_LOAD_OF_KNOWN_NULL_VALUE"/>
</Match>

<Match>
<Class name="~com\.azure\.(.+)Impl"/>
<Bug pattern="NP_LOAD_OF_KNOWN_NULL_VALUE"/>
</Match>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't duplicate the pattern - put it in the Or clause above instead.


<!-- Public field already exists in the public API surface area even though no writes were seen to this public field.
https://github.com/Azure/azure-sdk-for-java/blob/master/eventhubs/data-plane/azure-eventhubs/src/main/java/com/microsoft/azure/eventhubs/BatchOptions.java#L50
https://github.com/Azure/azure-sdk-for-java/blob/master/eventhubs/data-plane/azure-eventhubs/src/main/java/com/microsoft/azure/eventhubs/BatchOptions.java#L57 -->
Expand Down Expand Up @@ -368,6 +373,7 @@
<Match>
<Or>
<Package name="com.microsoft.azure.storage.blob"/>
<Package name="com.azure.storage.file.models"/>
<Class name="com.azure.storage.blob.HTTPGetterInfo"/>
</Or>
<Bug pattern="NM_CONFUSING"/>
Expand Down
5 changes: 5 additions & 0 deletions pom.client.xml
Original file line number Diff line number Diff line change
Expand Up @@ -393,6 +393,10 @@
<title>Azure Storage - Blobs</title>
<packages>com.azure.storage.blob*</packages>
</group>
<group>
<title>Azure Storage - Files</title>
<packages>com.azure.storage.file*</packages>
</group>
</groups>
<links>
<link>https://docs.oracle.com/javase/8/docs/api/</link>
Expand Down Expand Up @@ -748,5 +752,6 @@
<!-- <module>./sdk/tracing</module>-->
<module>./sdk/identity/azure-identity</module>
<module>./storage/client/blob</module>
<module>./storage/client/file</module>
</modules>
</project>
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,3 @@ private static synchronized void loadProperties() {
}



5 changes: 2 additions & 3 deletions storage/client/blob/src/samples/java/FileTransferExample.java
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.net.URL;
Expand Down Expand Up @@ -139,7 +138,7 @@ private static File createTempEmptyFile(String fileName) throws IOException {
}
}

private static void createTempFileWithFileSize(File f, long size) throws FileNotFoundException, IOException {
private static void createTempFileWithFileSize(File f, long size) throws IOException {
RandomAccessFile raf = new RandomAccessFile(f, "rw");
raf.setLength(size);
raf.close();
Expand All @@ -148,7 +147,7 @@ private static void createTempFileWithFileSize(File f, long size) throws FileNot
private static void checkTwoFilesAreTheSame(File f1, File f2) throws IOException, NoSuchAlgorithmException {
String checksumUpload = getFileChecksum(f1);
String checksumDownload = getFileChecksum(f2);
if (checksumUpload.equals(checksumDownload)) {
if (!checksumUpload.equals(checksumDownload)) {
throw new RuntimeException("The file upload does not match the file download.");
}
}
Expand Down
104 changes: 104 additions & 0 deletions storage/client/file/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<groupId>com.azure</groupId>
<artifactId>azure-client-sdk-parent</artifactId>
<version>1.1.0</version>
<relativePath>../../../pom.client.xml</relativePath>
</parent>

<modelVersion>4.0.0</modelVersion>

<groupId>com.azure</groupId>
<artifactId>azure-storage-file</artifactId>
<version>12.0.0-preview.1</version>

<name>azure-storage-file</name>
<url>https://github.com/Azure/azure-sdk-for-java</url>

<distributionManagement>
<site>
<id>azure-java-build-docs</id>
<url>${site.url}/site/${project.artifactId}</url>
</site>
</distributionManagement>

<scm>
<url>scm:git:https://github.com/Azure/azure-sdk-for-java</url>
<connection>scm:git:[email protected]:Azure/azure-sdk-for-java.git</connection>
<tag>HEAD</tag>
</scm>

<dependencies>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core</artifactId>
<version>1.0.0-preview.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

slf4j-api should come in transitively and not need to be included here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.


<!-- Added this dependency to include necessary annotations used by reactor core.
Without this dependency, javadoc throws a warning as it cannot find enum When.MAYBE
which is used in @Nullable annotation in reactor core classes -->
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
<version>3.0.2</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core-test</artifactId>
<version>1.0.0-preview.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
<version>1.0.0-preview.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>adal4j</artifactId>
<scope>test</scope>
</dependency>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as in queues - is slf4j-simple, reactor-test, and adal4j required here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reactor-test is necessary for async tests.

Removed adal4j , slf4j-simple

<dependency>
<groupId>org.spockframework</groupId>
<artifactId>spock-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>cglib</groupId>
<artifactId>cglib-nodep</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>uk.org.lidalia</groupId>
<artifactId>slf4j-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

</project>
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ public final class SharedKeyCredential {
private static final String AUTHORIZATION_HEADER_FORMAT = "SharedKey %s:%s";

// Pieces of the connection string that are needed.
private static final String ACCOUNT_NAME = "AccountName".toLowerCase();
private static final String ACCOUNT_KEY = "AccountKey".toLowerCase();
private static final String ACCOUNT_NAME = "accountname";
private static final String ACCOUNT_KEY = "accountkey";

private final String accountName;
private final byte[] accountKey;
Expand Down Expand Up @@ -59,7 +59,7 @@ public static SharedKeyCredential fromConnectionString(String connectionString)
HashMap<String, String> connectionStringPieces = new HashMap<>();
for (String connectionStringPiece : connectionString.split(";")) {
String[] kvp = connectionStringPiece.split("=", 2);
connectionStringPieces.put(kvp[0].toLowerCase(), kvp[1]);
connectionStringPieces.put(kvp[0].toLowerCase(Locale.ROOT), kvp[1]);
}

String accountName = connectionStringPieces.get(ACCOUNT_NAME);
Expand Down Expand Up @@ -98,9 +98,13 @@ public String generateAuthorizationHeader(URL requestURL, String httpMethod, Map
*
* @param stringToSign The UTF-8-encoded string to sign.
* @return A {@code String} that contains the HMAC-SHA256-encoded signature.
* @throws InvalidKeyException If the accountKey is not a valid Base64-encoded string.
* @throws RuntimeException for one of the following cases:
* <ul>
* <li> If the HMAC-SHA256 signature for {@code sharedKeyCredentials} fails to generate. </li>
* <li> If the an invalid key has been given to the client. </li>
* </ul>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move to the left.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

*/
public String computeHmac256(final String stringToSign) throws InvalidKeyException {
public String computeHmac256(final String stringToSign) {
try {
/*
We must get a new instance of the Mac calculator for each signature calculated because the instances are
Expand All @@ -111,8 +115,10 @@ public String computeHmac256(final String stringToSign) throws InvalidKeyExcepti
hmacSha256.init(new SecretKeySpec(this.accountKey, "HmacSHA256"));
byte[] utf8Bytes = stringToSign.getBytes(StandardCharsets.UTF_8);
return Base64.getEncoder().encodeToString(hmacSha256.doFinal(utf8Bytes));
} catch (final NoSuchAlgorithmException e) {
throw new Error(e);
} catch (final NoSuchAlgorithmException e) {
throw new RuntimeException("There is no such algorithm. Error Details: " + e.getMessage());
} catch (InvalidKeyException e) {
throw new RuntimeException("Please double check the account key. Error details: " + e.getMessage());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use clientLogger logAndThrow API

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The log and throw API is not available to preview 2.
Issue opened. #4551

}
}

Expand All @@ -121,20 +127,20 @@ private String buildStringToSign(URL requestURL, String httpMethod, Map<String,
contentLength = contentLength.equals("0") ? "" : contentLength;

// If the x-ms-header exists ignore the Date header
String dateHeader = (headers.containsKey("x-ms-date")) ? "" : getStandardHeaderValue(headers,"Date");
String dateHeader = (headers.containsKey("x-ms-date")) ? "" : getStandardHeaderValue(headers, "Date");

return String.join("\n",
httpMethod,
getStandardHeaderValue(headers,"Content-Encoding"),
getStandardHeaderValue(headers,"Content-Language"),
getStandardHeaderValue(headers, "Content-Encoding"),
getStandardHeaderValue(headers, "Content-Language"),
contentLength,
getStandardHeaderValue(headers,"Content-MD5"),
getStandardHeaderValue(headers,"Content-Type"),
getStandardHeaderValue(headers, "Content-MD5"),
getStandardHeaderValue(headers, "Content-Type"),
dateHeader,
getStandardHeaderValue(headers,"If-Modified-Since"),
getStandardHeaderValue(headers,"If-Match"),
getStandardHeaderValue(headers, "If-Modified-Since"),
getStandardHeaderValue(headers, "If-Match"),
getStandardHeaderValue(headers, "If-None-Match"),
getStandardHeaderValue(headers,"If-Unmodified-Since"),
getStandardHeaderValue(headers, "If-Unmodified-Since"),
getStandardHeaderValue(headers, "Range"),
getAdditionalXmsHeaders(headers),
getCanonicalizedResource(requestURL));
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

package com.azure.storage.common.exception;

import com.azure.core.exception.HttpResponseException;
import com.azure.storage.file.models.StorageErrorCode;
import com.azure.storage.file.models.StorageErrorException;

/**
* A {@code StorageException} is thrown whenever Azure Storage successfully returns an error code that is not 200-level.
* Users can inspect the status code and error code to determine the cause of the error response. The exception message
* may also contain more detailed information depending on the type of error. The user may also inspect the raw HTTP
* response or call toString to get the full payload of the error response if present.
* Note that even some expected "errors" will be thrown as a {@code StorageException}. For example, some users may
* perform a getProperties request on an entity to determine whether it exists or not. If it does not exists, an
* exception will be thrown even though this may be considered an expected indication of absence in this case.
*
* <p><strong>Sample Code</strong></p>
* <p>For more samples, please see the <a href="https://github.com/Azure/azure-storage-java/blob/master/src/test/java/com/microsoft/azure/storage/Samples.java">sample file</a></p>
*/
public final class StorageException extends HttpResponseException {
private static final String ERROR_CODE = "x-ms-error-code";
private final String message;

StorageException(StorageErrorException e, String responseBody) {
super(e.getMessage(), e.response(), e);
this.message = responseBody;
}

/**
* @return The error code returned by the service.
*/
public StorageErrorCode errorCode() {
return StorageErrorCode.fromString(super.response().headers().value(ERROR_CODE));
}

/**
* @return The message returned by the service.
*/
public String message() {
return this.message;
}

/**
* @return The status code on the response.
*/
public int statusCode() {
return super.response().statusCode();
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class should not have fluent API - it should be get*.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will start work on common module after the file get merged in.

Create an issue for this.
#4539

Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ public RequestRetryOptions() {
* <p><strong>Sample Code</strong></p>
*
* <p>For more samples, please see the <a href="https://github.com/Azure/azure-storage-java/blob/master/src/test/java/com/microsoft/azure/storage/Samples.java">samples file</a></p>
* @throws IllegalArgumentException If one of the following case exists:
* <ul>
* <li> There is only one null value for retryDelay and maxRetryDelay.<li/>
* <li> Unrecognized retry policy type.<li/>
* </ul>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move left.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

*/
public RequestRetryOptions(RetryPolicyType retryPolicyType, Integer maxTries, Integer tryTimeout,
Long retryDelayInMs, Long maxRetryDelayInMs, String secondaryHost) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@
public final class RequestRetryPolicy implements HttpPipelinePolicy {
private final RequestRetryOptions requestRetryOptions;

/**
* Constructs the policy using the retry options.
*
* @param requestRetryOptions Retry options for the policy.
*/
public RequestRetryPolicy(RequestRetryOptions requestRetryOptions) {
this.requestRetryOptions = requestRetryOptions;
}
Expand Down
Loading