Add more filter to s3 hook list_key #22231
Conversation
boring-cyborg
bot
added
area:providers
provider:amazon
sunank200
force-pushed
the
s3-list-key-filter
branch
from
5bbb8a6 to
cd8ccaa
Compare
| from_datetime: Optional[DateTime] = None, | ||
| to_datetime: Optional[DateTime] = None, |
There was a problem hiding this comment.
I don’t think this needs to take pendulum.DateTime. Normal datetime.datetime works equally well (and is compatible with pendulum.DateTime).
There was a problem hiding this comment.
LastModified in the key returned by boto3 is timezone aware and comparison with DateTime specified by the user would create TypeError: can't compare offset-naive and offset-aware datetimes.
There was a problem hiding this comment.
Native datetime can be timezone-aware as well; on the other hand, using pendulum.DateTime still does not guarantee the instance is timezone-aware.
There was a problem hiding this comment.
Sure. Changed the type to datetime.datetime.
|
Re-triggering the build to rerun doc build |
|
Seems like this is a wider outage of some inventories :( |
|
I think you will need to rebase that one @sunank200 |
sunank200
force-pushed
the
s3-list-key-filter
branch
from
9196f49 to
594f7bd
Compare
sunank200
requested review from
potiuk,
ashb,
jedcunningham,
turbaszek,
mik-laj and
kaxil
as code owners
kaxil
force-pushed
the
s3-list-key-filter
branch
from
594f7bd to
6bfdf44
Compare
…time, object_filter callable Implemented as discussed in [closed PR](apache#19018). Add more filter options to list_keys of S3Hook - `start_after_key`: should return only keys greater than this key - `from_datetime`: should return only keys with LastModified attr greater than this equal `from_datetime`. - `to_datetime`: should return only keys with LastModified attr less than this `to_datetime`. - `object_filter`: Function callable that receives the list of the S3 objects, `from_datetime` and `to_datetime` and returns the List of the matched key. Add test for the added argument to `list_keys`. closes: apache#16627
sunank200
force-pushed
the
s3-list-key-filter
branch
from
6bfdf44 to
9707a1a
Compare
Implemented as discussed in closed PR.
Add more filter options to list_keys of S3Hook
start_after_key: should return only keys greater than this keyfrom_datetime: should return only keys with LastModified attr greater than this equalfrom_datetime.to_datetime: should return only keys with LastModified attr less than thisto_datetime.object_filter: Function callable that receives the list of the S3 objects,from_datetimeandto_datetimeand returns the List of the matched key.Add test for the added argument to
list_keys.closes: #16627
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.