Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some adjustments for Storage Driver and SourceManager to run on the cloud #519

Merged
merged 1 commit into from
Jun 19, 2024

Conversation

andresgutgon
Copy link
Contributor

@andresgutgon andresgutgon commented Jun 19, 2024

Describe your changes

We want to use some methods from these 2 packages in our Latitude cloud and we have to make it work.

TODO

  • Do specs for resolve secrets function

Copy link

changeset-bot bot commented Jun 19, 2024

🦋 Changeset detected

Latest commit: a1bdca2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages
Name Type
@latitude-data/source-manager Patch
@latitude-data/storage-driver Patch
@latitude-data/server Patch
@latitude-data/cli Patch
@latitude-data/display_table Patch
@latitude-data/athena-connector Patch
@latitude-data/bigquery-connector Patch
@latitude-data/clickhouse-connector Patch
@latitude-data/databricks-connector Patch
@latitude-data/duckdb-connector Patch
@latitude-data/mssql-connector Patch
@latitude-data/mysql-connector Patch
@latitude-data/postgresql-connector Patch
@latitude-data/snowflake-connector Patch
@latitude-data/sqlite-connector Patch
@latitude-data/test-connector Patch
@latitude-data/trino-connector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@andresgutgon andresgutgon force-pushed the fix/adjustments-for-storage-driver-and-manager branch 2 times, most recently from 8393d36 to 4c129f2 Compare June 19, 2024 15:40
@andresgutgon andresgutgon removed the WIP label Jun 19, 2024
server
We want to use some methods from these 2 packages in our Latitude cloud
and we have to make it work
@andresgutgon andresgutgon force-pushed the fix/adjustments-for-storage-driver-and-manager branch from 4c129f2 to a1bdca2 Compare June 19, 2024 15:44
const stream = await this.materializedStorage.createWriteStream(filename)
const stream = (await this.materializedStorage.createWriteStream(
filename,
)) as Writable
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why materializedStorage is typed a any. We must be doing something wrong.

Comment on lines +9 to +12
get isS3Driver(): boolean {
return this.type === StorageType.s3
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this for? Can't you just compare the type?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can, but this is nicer

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is for the server to check if the driver is S3

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the an abstract parent class like this should be agnostic to its children subclasses existence.
This is not as big of a deal as if it was on the BaseConnector though, here's only two subclasses (for now). Just OCD.

Copy link
Contributor Author

@andresgutgon andresgutgon Jun 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep pure ocd in relation to the size of the driver code xD

Copy link
Contributor

@csansoon csansoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bit of OCD. Everything else is nice 👌

@andresgutgon andresgutgon merged commit 4f1d88d into canary Jun 19, 2024
4 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Jun 19, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants