Add GCS Fs implementation

[VOvchinnikov]

An implementation of GCS Fs, based on #161

Thanks @Zatte for the inital implementation!

Changes vs the original PR:

• Based on the latest master
• Uses client as its base abstraction, not bucket => bucket is thus a part of path, the first component in it (!)
• Has kinks and quirks of the GCS API "ironed out"
• Has its own test suite now, based off the TarFS one with extra parts to test the writing/folders making scenarios
• The suite allows for testing both with mocks and with the real bucket (see commented out block in the tests setup)
• File modes are exactly those that are provided during files/directories opening. If not provided, defaults to 0755

Bonus:

• Supports gs://<bucket>/<path> URLs too, even though it's explicitly excluded from tests, since afero in general does not work with URLs.

The original PR notes, which still apply to this version of the implementation + extra for this version:

Limitations:

• No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version.
• No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version.
• NOTE: Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent.

Performance implications

• Sequential reads are performant
• Sequential writes are performant.
• Seek + Read or ReadAt is performant after the initial seek. (produces a warning)
• Alternating reads/writes to the same file handler are highly inefficient. To get consistent FS behavior using an API that separates readers and writers we close any open readers before an write as well close open writers before a read (ensure the data is committed).
• Seek + Write such as WriteAt, Truncate, Seek+Write will work as expected but with significant overhead. Doing a seek + write will in effect download the old file/object, overlay it with the new writes and save it back. This is done in a streaming fashion so large files will not clog the memory but will trigger a full download and upload of the file/object.

[CLAassistant]

All committers have signed the CLA.

[VOvchinnikov]

@Kargakis Would you maybe interested to take a look into it?

[VOvchinnikov]

A notification bump, since it would actually be nice to have it at least looked at 😢

[UndeadBaneGitHub]

Any news on this one?

[VOvchinnikov]

Bump.

[0xmichalis]

I would love to get this merged but I am hesitant to do without e2e tests so I'll defer to @spf13 on the potential to get infra for tests.

[VOvchinnikov]

@Kargakis Hey, nice to see some activity here :)

What tests are you exactly talking about? New code is actually covered by comprehensive mocked e2e tests, with an ability to switch over to a real bucket for testing, as per this comment.

if you have some bucket to use permanently for the sake of testing (with proper timeouts configured, since tests bombard a bucket), then I’d gladly make a real bucket the main testing target.

Meanwhile the mocks have been adjusted to mimick the real thing as close as it’s necessary for the sake of testing.

[0xmichalis]

What tests are you exactly talking about?

I meant that the existing tests should be using real GCS buckets and not mocks as part of CI.

[VOvchinnikov]

Got it - there's going to be a bit of an issue with that, since some slowing down would have to be introduced to not trigger GCS timeouts with the tests doing so much in so little time.

That said, I'm happy to switch to using the real buckets as soon as there is one - I cannot organize it for your project, unfortunately.

[0xmichalis]

If @spf13 thinks we can merge w/o using GCS I am fine with merging as is, but I would prefer him to sign this off.

[spf13]

I think we should merge it now, but flag it as experimental until we can establish a more complete end2end test.

[0xmichalis]

@spf13 thanks for the response! @VOvchinnikov can you please update the README here to document how to configure the use of a GCS bucket? Thanks!

[0xmichalis]

@spf13 can you be a bit more specific on how to mark this as experimental? Would creating an "experimental" subpackage where we can have gcs suffice?

[spf13]

No, I'd just add it to the documentation. It's not really usable without reading the documentation anyway as it requires unique configuration.

…

On Mon, Nov 8, 2021 at 9:41 AM Michalis Kargakis ***@***.***> wrote: @spf13 <https://github.com/spf13> can you be a bit more specific on how to mark this as experimental? Would creating an "experimental" subpackage where we can have gcs suffice? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#296 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABKKZCLB76SYRZ7XWSNHCDUK7OSJANCNFSM42ERP5UA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[0xmichalis]

Fixed the merge conflicts and added a README section in #331

[0xmichalis]

https://github.com/spf13/afero/releases/tag/v1.7.0