Add config to reject Iceberg queries without partition filter

[rice668]

Description

For iceberg partitioned tables, we should reject the table scan produced by the planner when the query does not have partition field.

Additional context and related issues

Fixes #17239

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Iceberg
* Add an option to require all scans on partitioned tables to have a filter on a partitioned column. This can be enabled by setting the 
  ``iceberg.query-partition-filter-required`` catalog config property or the ``query_partition_filter_required`` catalog session property to ``true``.

[rice668]

Thank you for @Praveen2112 @marton-bod review, I have made some changes to the code, please help review again when you have time.

[rice668]

@marton-bod Thanks! I have updated the code, Could you please take a look again ?

[marton-bod]

@zhangminglei Can you explain why this
table.getEnforcedPredicate().equals(TupleDomain.all()) is needed? Doesn't this mean that we only enter the block if the table has no enforced predicates?

[rice668]

That's right! @marton-bod , If a table has no enforced predicates, like select * from T where udf(partition_field) = someValue then enter the block.

[alexjo2144]

The reason for the difference, Marton, is that some filters show up as TupleDomains and some show up as Predicates depending on how they can be represented.

WHERE x = 42 for example, is a tuple domain and will show up in the enforced constraint
WHERE x % 2 = 0 will be a Predicate because it cannot be represented as a discrete set or range

but if x is a partition column, either should be fine for this feature

[marton-bod]

Thanks for the explanation! Just curious to hear what would happen in the following example:

CREATE TABLE tbl (a int, b int) partitioning = ARRAY['a'] AS SELECT * FROM ...;
SELECT * FROM tbl WHERE b % 2 = 0;

Would we fail the read query because it has no predicate defined for the partition column a? Or would we skip the check becase there is an enforced predicate for b

[marton-bod]

@findepi @alexjo2144 @findinpath Can you please take a look at this PR so it can make some progress? It generally looks good to me, but there are some places where your expertise and iceberg connector knowledge would be needed. Thanks a lot!

[alexjo2144]

I'd leave one typical test case here in the smoke test and move the rest to BaseIcebergConnectorTest

[rice668]

Oh. Yes, smoke tests are designed to be quick and simple

[alexjo2144]

I saw in the Hive connector version of this we have a set of schemas where the filters are required. Any thoughts on if we should have that configurable here as well?

[marton-bod]

Thanks so much for your review @alexjo2144 !

Any thoughts on if we should have that configurable here as well?

I also suggested to implement that flag as well. Whether we do it in this PR or a follow-up I'm OK with both. cc @zhangminglei

[marton-bod]

@zhangminglei just a friendly bump to see if you could take a look at @alexjo2144's comments. Our team would love to move this feature forward to completion

[marton-bod]

Hi @ebyhr , I think all the PR comments have been addressed now - can you please check if anything else is required, and if not, merge this in? Thanks a lot!

[marton-bod]

@ebyhr I've rebased this PR to the latest master, so it's again ready for review. Thanks!

[marton-bod]

@ebyhr Can you please take a final look? I appreciate that all maintainers are probably busy, but it's been waiting for review/merge for a while now. Thanks a lot!

[rice668]

@ebyhr Could you please take a look again ? Thank you very much.

[marton-bod]

Hi @raunaqmorarka! I think @ebyhr is busy, so could I ask you please to help us with this? We have rebased multiple times and responded to all the review comments so I don't think there is anything outstanding here. Thank you so much!

[raunaqmorarka]

I think this should include the columns from enforced and unenforced tuple domains. See the implementation in DeltaLakeMetadata#applyFilter

[raunaqmorarka]

On the other hand constraintColumns in HiveTableHandle includes only constraint predicate columns. Either way should be okay as long as we clearly document in IcebergTableHandle what is included in constraintColumns

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Marton Bod.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[marton-bod]

@raunaqmorarka I highly appreciate your review. I have addressed your comments. To make it easier to check, I kept the new changes in a separate commit, but once you approve the PR I will squash the commits

[raunaqmorarka]

minor comments
please squash the commits

[marton-bod]

@raunaqmorarka Thanks a lot again for you review, I truly appreciate it!

[findinpath]

Only nits