Self-resolving errors in reading from ADLS Gen2 on 0.5.8

[malmgrek]

Environment

Fedora 35 (Linux)
Python 3.10.4

Delta-rs version:
0.5.8

---

Bug

What happened:
Reading a delta table from remote ADLS Gen2 service, I get two errors of type

[2022-08-23T13:44:34Z ERROR azure_core::policies::retry_policies::retry_policy] server returned error status which will not be retried: 404 Not Found

NOTE: Finally the correct data is returned succesfully, however.

I also tried running the same script on version 0.5.7 and no errors logged in the std output.

What you expected to happen:
Similar behavior as with version 0.5.7.

How to reproduce it:

My script was more or less like so:

import logging
import os

from deltalake import DeltaTable

os.environ["AZURE_STORAGE_ACCOUNT_NAME"] = "storagename"
os.environ["AZURE_STORAGE_ACCOUNT_KEY"] = "accountkey"

dt = DeltaTable(
    "adls2://storagename/path/to/delta-table",
)

• Run with delta-rs version 0.5.8 to reproduce the erratic behavior
• Run with delta-rs version 0.5.7 to reproduce the correct behavior

[roeap]

@malmgrek, the azure integration has gone back and forth on how its working :D.

Luckily we are resoling all of this right now (#761) which includes extensive integration testing ... There are braking changes in how paths are cosntructed for azure, but nothing too bad ..

We also aim to release 0.6 (#762 ) shortly after that is done

[malmgrek]

@roeap Thanks for the quick comments, good to know that a new version is coming up. I will settle for now with 0.5.7 and bump to 0.6.0 afterwards. Thanks for your hard work!

PS: I have some quite high level questions related to Azure integration:

• Do you know if SAS tokens will be supported as an alternative to the envvar AZURE_STORAGE_ACCOUNT_KEY?
• What would be an appropriate forum to ask such and other questions related to e.g. the high level Python API?

EDIT: Include PS questions.

[roeap]

The best way to ask more general questions is the slack channel, there should be a way to join in the README. As for SAS tokens, The currently is a PR open in the object_Store crate that we are migrating to that will include suppoort for SAS tokens in general... There is a little work that needs to be done on this end. but we may just get to include that - could you open an issue for SAS support?

[malmgrek]

Alrighty then, I have now submitted two feature requests:

• Azure SAS support  #766
• Programmatic configuration of Azure storage options in deltalake.DeltaTable #767

[roeap]

@malmgrek - could you check if this is resolved for you now?

[malmgrek]

@roeap - now it is resolved, thanks. 🎉