CloudStorageFileSystem: java.net.URISyntaxException: Illegal character in hostname

[kshakir]

Environment details

1. Specify the API at the beginning of the title: CloudStorageFileSystem
2. OS type and version: macOS 13.4.1 (a)
3. Java version: OpenJDK 64-Bit Server VM Temurin-11.0.18+10 (build 11.0.18+10, mixed mode)
4. version(s): 0.126.20-SNAPSHOT

Steps to reproduce

1. Create a bucket in GCS with a name containing an underscore
2. Call Paths.get(URI.create("<path_in_your_bucket")) with the GCS path in your bucket

Code example

Paths.get(URI.create("gs://bucket_with_authority/path"))

Stack trace

java.lang.IllegalArgumentException: Expected scheme-specific part at index 3: gs:

	at com.google.cloud.storage.contrib.nio.CloudStorageUtil.stripPathFromUri(CloudStorageUtil.java:65)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.getPath(CloudStorageFileSystemProvider.java:282)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.getPath(CloudStorageFileSystemProvider.java:97)
	at java.base/java.nio.file.Path.of(Path.java:208)
	at java.base/java.nio.file.Paths.get(Paths.java:97)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProviderTest.testBucketWithAuthority(CloudStorageFileSystemProviderTest.java:836)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at com.google.cloud.testing.junit4.MultipleAttemptsRule$1.evaluate(MultipleAttemptsRule.java:94)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)

External references such as API reference guides

• https://docs.oracle.com/javase/8/docs/api/java/net/URI.html
• https://www.ietf.org/rfc/rfc2396.txt
• https://cloud.google.com/storage/docs/buckets

Any additional information below

According to RCF 2396, a URI may contain an “Authority Component” (section 3.2). That authority may be “Server-based” (section 3.2.2) or “Registry-based” (section 3.2.1).

All Google bucket names are not server-based authorities. The Google bucket naming documentation mentions “You can use a bucket name in a DNS record as part of a CNAME or A redirect.” but it does not say “You must”.

Since the bucket names are not hostnames then under RFC 2396 the URI may still be constructed using the “Registry-based” form. URI.create("gs://bucket_with_authority/path") produces a valid java.net.URI, it’s just not “Server-based” according to the RFC.

The stack trace appears to be caused by the CloudStorageFileSystem internal use of the java.net.URI API. In a number of places the CloudStorageFileSystem attempts to construct URIs with hostnames, or retrieve the hostname from “Server-based” URIs instead of “Registry-based” URIs.

The fix is to internally use the java.net.URI APIs that support all RFC 2396 section 3.2.1 and section 3.2.2 authorities, not the APIs that only support the section 3.2.2 "Server-based" authorities.

Thanks!

[kshakir]

Support case filed: https://console.cloud.google.com/support/cases/detail/v2/45924691

[kshakir]

FYI via the Google support ticket [1], there is also the suggestion that

you can try the workaround [2]

[1] https://console.cloud.google.com/support/cases/detail/v2/45924691
[2] https://stackoverflow.com/questions/28568188/java-net-uri-get-host-with-underscores#:~:text=A%20possible%20workaround%3A

The workaround involves java-storage-nio using reflection to force the private host field public, then overwriting the value. I'm hopeful that #1219 using getAuthority() won't need reflection, and seems like a better fix upon first glance.

Still, I'm just passing along the above option forwarded by Google's support team just in case.