Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide the project columns to ConnectorSplitManager.getSplits() #2943

Closed
davidrabinowitz opened this issue Feb 25, 2020 · 4 comments
Closed

Comments

@davidrabinowitz
Copy link
Contributor

At the moment, the projected columns are provided just in the API of ConnectorPageSourceProvider.createPageSource(). For the BigQuery connector, the columns needs to be provided at the ConnectorSplitManager as at this point the ReadSession that contains the columns is created.
This issues blocks Issue #2532

This is per @findepi suggestion at #2532 (comment)

@electrum
Copy link
Member

This should be possible today with applyProjection(). I ran TpchQueryRunner with a breakpoint in the default implementation in ConnectorMetadata, then ran SELECT count(*) FROM tpch.tiny.region and it's called as expected with an empty projection list.

@martint
Copy link
Member

martint commented Feb 25, 2020

I don't think this is the direction we want to go. As David said above, the necessary information should be communicated to the connector via the applyProjection API. If that's not happening, we should figure out why.

@davidrabinowitz
Copy link
Contributor Author

Ok, I'll check it out. Thanks

@davidrabinowitz
Copy link
Contributor Author

Thanks for the tip, indeed I have managed with applyProjection().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants