Support WASB scheme in ADLSFileIO

[mrcnc]

This is my third attempt to resolve #10127 (...third time's the charm 😉)

We needed to revert #11504 since it introduced a breaking change for ADLSFileIO clients that were using SAS tokens. This PR fixes the issue by introducing a new host() method to ADLSLocation and uses it in ADLSFileIO when calling applyClientConfiguration.

[mrcnc]

I would like to verify that updating this default behavior of ResolvingFileIO is not considered a breaking change, since it could potentially force clients using the WASB scheme with HadoopFileIO to update their configuration. For example, if clients are currently using SAS tokens in an core-site.xml file, then it seems they need to either configure these in the AzureProperties for ADLSFileIO or explicitly configure HadoopFileIO as the implementation to get the same behavior.

[amogh-jahagirdar]

Not completely sure I follow the concern, if a client is using WASB + HadoopFileIO this means they explicitly configured it or are using a catalog which defaults to using HadoopFileIO (e.g. the HadoopCatalog).

Adding this scheme mapping to ResolvingFileIO just makes it so that when ResolvingFileIO is configured or defaulted to, if a path with "wasb/wasbs" is encountered, we initialize the ADLS file IO. It shouldn't impact a user who has already either explicitly configured HadoopFileIO or is defaulting to that. Unless I'm missing something cc @danielcweeks @bryanck

[mrcnc]

I see the JdbcCatalog uses HadoopFileIO by default so it wouldn't affect these users, but it would affect the REST Catalog which defaults to ResolvingFileIO. Maybe this is ok but needs to be added to release notes for 1.8.0? If not, we can remove it for now or activate this behavior explicitly with another setting?

[amogh-jahagirdar]

I think it's actually important to update the mapping (so imo what's here is correct). I think ResolvingFileIO should always only fallback to HadoopFileIO as a last resort, and in this case we do want wasb/wasbs to resolve to ADLS so that any custom integrations that happens in Iceberg in the ADLSFileIO implementation can be taken advantage of. At the same time we should document it.

In general, I feel like we shouldn't design for the case where a user relies on ResolvingFileIO's fallback HadoopFileIO

[jbonofre]

The mapping should be updated imho. This change makes sense to me here.

[amogh-jahagirdar]

Not completely sure I follow the concern, if a client is using WASB + HadoopFileIO this means they explicitly configured it or are using a catalog which defaults to using HadoopFileIO (e.g. the HadoopCatalog).

Adding this scheme mapping to ResolvingFileIO just makes it so that when ResolvingFileIO is configured or defaulted to, if a path with "wasb/wasbs" is encountered, we initialize the ADLS file IO. It shouldn't impact a user who has already either explicitly configured HadoopFileIO or is defaulting to that. Unless I'm missing something cc @danielcweeks @bryanck

[ajreid21]

From a breaking change perspective, this looks okay to me now.

Is there future intent to use the storageAccount/storageAccountName? Otherwise, it's not being used currently but we're still parsing and storing it.

[mrcnc]

Is there future intent to use the storageAccount/storageAccountName? Otherwise, it's not being used currently but we're still parsing and storing it.

I don't have any future plans currently but think it's probably ok to leave it. One potential use case is if we wanted to override toString() to return the fully qualified URI

[amogh-jahagirdar]

LGTM, @mrcnc thanks for this! I'll wait a bit in case @jbonofre wanted to review.

[jbonofre]

The mapping should be updated imho. This change makes sense to me here.

[amogh-jahagirdar]

Thanks for the change @mrcnc , and thank you for reviewing @jbonofre @ajreid21 !