S3 hooks filter options #19018

sunank200 · 2021-10-15T23:11:17Z

Add more filter options to list_keys of S3Hook

This commit adds following filters to list the keys in list_keys of S3Hook:
- start_after_key filters the any keys after the specified key. start_after_key can be any key in the bucket.
- start_after_datetime filters all the keys with last modified date-time greater than or equal to the start_after_datetime.
- to_datetime filters all the keys with last modified date-time less than or equal to the to_datetime

Implemented as per changes discussed in the previous PR.

closes: #16627

This change wouldn't affect dependencies for other operators like S3DeleteObjectsOperator, S3ListOperator, S3Hook methods:get_wildcard_key, delete_bucket and S3KeysUnchangedSensor.

Corresponding unittest has been added to test_s3.py.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

This commit adds following filters to list the keys in list_keys of S3Hook: - start_after_key filters the any keys after the specified key. start_after_key can be any key in the bucket. - start_after_datetime filters all the keys with last modified datetime greater than or equal to the start_after_datetime. - to_datetime filters all the keys with last modified datetime less than or equal to the to_datetime

Tests to filter keys after specified key and to filter keys based on last modified datetime.

… object_filter callable. Response filter to filter objects based on operations defined by user. It is written in a generic way such that it can be extended in future for different operations.

Fixes for Flake8 error and indentation bug fix to return result when object_filter is None

This reverts commit f2f4cbc.

sunank200 · 2021-10-15T23:16:08Z

@potiuk @dstandish @eladkal could you trigger the CI?

dstandish

i'm not sure we need to add the ResponseFilter abstraction.

What do you think about allowing the user to provide an arbitrary callable that does the filtering that the user needs?

For example, your docstring shows this example:

object_filter={"LastModified__lt": datetime.now()},

But with arbitrary callable, this can be implemented about as simply this way:

object_filter=lambda x: x['LastModified'] < datetime.now(),

This is roughly as compact and roughly as simple, but the benefit is there's no need to understand the options available within the class and how it works, and if user wants to do arbitrary filtering they can.

github-actions · 2021-12-12T00:07:31Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

…time, object_filter callable Implemented as discussed in [closed PR](apache#19018). Add more filter options to list_keys of S3Hook - `start_after_key`: should return only keys greater than this key - `from_datetime`: should return only keys with LastModified attr greater than this equal `from_datetime`. - `to_datetime`: should return only keys with LastModified attr less than this `to_datetime`. - `object_filter`: Function callable that receives the list of the S3 objects, `from_datetime` and `to_datetime` and returns the List of the matched key. Add test for the added argument to `list_keys`. closes: apache#16627

…time, object_filter callable (#22231) Implemented as discussed in [closed PR](#19018). Add more filter options to list_keys of S3Hook - `start_after_key`: should return only keys greater than this key - `from_datetime`: should return only keys with LastModified attr greater than this equal `from_datetime`. - `to_datetime`: should return only keys with LastModified attr less than this `to_datetime`. - `object_filter`: Function callable that receives the list of the S3 objects, `from_datetime` and `to_datetime` and returns the List of the matched key. Add test for the added argument to `list_keys`. closes: #16627

…time, object_filter callable (#22231) Implemented as discussed in [closed PR](apache/airflow#19018). Add more filter options to list_keys of S3Hook - `start_after_key`: should return only keys greater than this key - `from_datetime`: should return only keys with LastModified attr greater than this equal `from_datetime`. - `to_datetime`: should return only keys with LastModified attr less than this `to_datetime`. - `object_filter`: Function callable that receives the list of the S3 objects, `from_datetime` and `to_datetime` and returns the List of the matched key. Add test for the added argument to `list_keys`. closes: #16627 GitOrigin-RevId: 926f6d1894ce9d097ef2256d14a99968638da9c0

sunank200 added 10 commits October 16, 2021 04:52

Unittest for more filter options to list_keys of S3Hook

ac37a41

Tests to filter keys after specified key and to filter keys based on last modified datetime.

Add ResponseFilter class to parse and filter response object based on…

ade6aec

… object_filter callable. Response filter to filter objects based on operations defined by user. It is written in a generic way such that it can be extended in future for different operations.

Update unittest for object_filter callable for s3.

e6691c9

Add ResponseFilter class to parse and filter response object based on…

6c70ab0

… object_filter callable. Response filter to filter objects based on operations defined by user. It is written in a generic way such that it can be extended in future for different operations.

Update unittest for object_filter callable for s3.

0860a7e

Flake8 fixes and indentation fix

6fb7ac4

Fixes for Flake8 error and indentation bug fix to return result when object_filter is None

fix isort error

ca88ed2

fix isort error

91532b4

Revert "Flake8 fixes and indentation fix"

7bad414

This reverts commit f2f4cbc.

boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Oct 15, 2021

sunank200 added 2 commits October 16, 2021 05:25

fix comments as per static checks.

cd90bff

fix pre-commit for test_s3

ceeeccc

eladkal requested a review from dstandish October 26, 2021 20:26

dstandish reviewed Oct 27, 2021

View reviewed changes

github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Dec 12, 2021

github-actions bot closed this Dec 18, 2021

sunank200 mentioned this pull request Mar 13, 2022

Add more filter to s3 hook list_key #22231

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 hooks filter options #19018

S3 hooks filter options #19018

sunank200 commented Oct 15, 2021 •

edited

Loading

sunank200 commented Oct 15, 2021 •

edited

Loading

dstandish left a comment •

edited

Loading

github-actions bot commented Dec 12, 2021

S3 hooks filter options #19018

S3 hooks filter options #19018

Conversation

sunank200 commented Oct 15, 2021 • edited Loading

sunank200 commented Oct 15, 2021 • edited Loading

dstandish left a comment • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Dec 12, 2021

sunank200 commented Oct 15, 2021 •

edited

Loading

sunank200 commented Oct 15, 2021 •

edited

Loading

dstandish left a comment •

edited

Loading