replace `is_datetime_or_timedelta_dtype` #316

rawwar · 2023-10-13T05:31:31Z

Description

removes deprecated is_datetime_or_timedelta_dtype usage

Issues Resolved

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: kalyan <[email protected]>

dhrubo-os · 2023-10-19T23:10:55Z

Could you please check if this change works with panda version 2.0 & 2.1 both? I'm just wondering if we stick with 2.0 version only will this change make any issue or not.

codecov · 2023-10-26T21:37:24Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (2c3b744) 91.47% compared to head (a11b5b2) 90.68%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #316      +/-   ##
==========================================
- Coverage   91.47%   90.68%   -0.79%     
==========================================
  Files          40       40              
  Lines        4339     4339              
==========================================
- Hits         3969     3935      -34     
- Misses        370      404      +34

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

rawwar · 2023-10-27T02:31:29Z

Could you please check if this change works with panda version 2.0 & 2.1 both? I'm just wondering if we stick with 2.0 version only will this change make any issue or not.

From what i see, this is a breaking change only after 2.1. If we stick to 2.0.3, it worked without issues. Can i update nox tests to just use the pandas from requirements.txt instead of pinning it to 1.5.3?

dhrubo-os · 2023-10-27T04:03:24Z

Could you please check if this change works with panda version 2.0 & 2.1 both? I'm just wondering if we stick with 2.0 version only will this change make any issue or not.

From what i see, this is a breaking change only after 2.1. If we stick to 2.0.3, it worked without issues. Can i update nox tests to just use the pandas from requirements.txt instead of pinning it to 1.5.3?

Looks like this should work with 2.1.0 version also: https://pandas.pydata.org/docs/reference/api/pandas.api.types.is_timedelta64_dtype.html. That's good then.

dhrubo-os · 2023-10-27T04:05:42Z

I think this is a vital change. Could you please update the changelog. Section 1.2.0 and add details in the Update section.

Signed-off-by: kalyan <[email protected]>

rawwar · 2023-10-31T02:33:34Z

I think this is a vital change. Could you please update the changelog. Section 1.2.0 and add details in the Update section.

Updated changelog.

…ch-py-ml into kalyan/263-fix-pandas-import-issue Signed-off-by: kalyanr <[email protected]>

Jon-AtAWS · 2024-03-24T18:50:22Z

Hitting this now

> python --version
Python 3.11.5

> pip --version
pip 24.0 from /Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/pip (python 3.11)

> pip freeze                
boto3==1.33.8
botocore==1.33.8
certifi==2023.11.17
charset-normalizer==3.3.2
contourpy==1.2.0
cycler==0.12.1
Faker==20.1.0
fonttools==4.50.0
idna==3.6
jmespath==1.0.1
kiwisolver==1.4.5
matplotlib==3.8.3
numpy==1.26.4
opensearch-py==2.4.2
opensearch-py-ml==1.1.0
packaging==24.0
pandas==2.2.1
pillow==10.2.0
pyparsing==3.1.2
python-dateutil==2.8.2
pytz==2024.1
requests==2.31.0
requests-aws4auth==1.2.3
s3transfer==0.8.2
six==1.16.0
tzdata==2024.1
urllib3==2.0.7

Python 3.11.5 (v3.11.5:cce6ba91b3, Aug 24 2023, 10:50:31) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from opensearch_py_ml import ml_commons
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/opensearch_py_ml/__init__.py", line 28, in <module>
    from .dataframe import DataFrame
  File "/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/opensearch_py_ml/dataframe.py", line 47, in <module>
    from opensearch_py_ml.groupby import DataFrameGroupBy
  File "/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/opensearch_py_ml/groupby.py", line 28, in <module>
    from opensearch_py_ml.query_compiler import QueryCompiler
  File "/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/opensearch_py_ml/query_compiler.py", line 44, in <module>
    from opensearch_py_ml.field_mappings import FieldMappings
  File "/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/opensearch_py_ml/field_mappings.py", line 44, in <module>
    from pandas.core.dtypes.common import (
ImportError: cannot import name 'is_datetime_or_timedelta_dtype' from 'pandas.core.dtypes.common' (/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/pandas/core/dtypes/common.py)

Not sure whether it's my environment or something else. I also tried various versions of Pandas from 2.0.0 through 2.2.1, same problem. I seem to have the latest version of opensearch-py-ml (and opensearch-py)

yuye-aws · 2024-05-21T07:17:15Z

Also hitting the same issue. This PR gets merged after 1.1.0 release of opensearch-py-ml. Do we have a newer version to include this PR?

yuye-aws · 2024-05-21T07:20:25Z

Hitting this now

> python --version
Python 3.11.5

> pip --version
pip 24.0 from /Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/pip (python 3.11)

> pip freeze                
boto3==1.33.8
botocore==1.33.8
certifi==2023.11.17
charset-normalizer==3.3.2
contourpy==1.2.0
cycler==0.12.1
Faker==20.1.0
fonttools==4.50.0
idna==3.6
jmespath==1.0.1
kiwisolver==1.4.5
matplotlib==3.8.3
numpy==1.26.4
opensearch-py==2.4.2
opensearch-py-ml==1.1.0
packaging==24.0
pandas==2.2.1
pillow==10.2.0
pyparsing==3.1.2
python-dateutil==2.8.2
pytz==2024.1
requests==2.31.0
requests-aws4auth==1.2.3
s3transfer==0.8.2
six==1.16.0
tzdata==2024.1
urllib3==2.0.7

Python 3.11.5 (v3.11.5:cce6ba91b3, Aug 24 2023, 10:50:31) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from opensearch_py_ml import ml_commons
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/opensearch_py_ml/__init__.py", line 28, in <module>
    from .dataframe import DataFrame
  File "/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/opensearch_py_ml/dataframe.py", line 47, in <module>
    from opensearch_py_ml.groupby import DataFrameGroupBy
  File "/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/opensearch_py_ml/groupby.py", line 28, in <module>
    from opensearch_py_ml.query_compiler import QueryCompiler
  File "/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/opensearch_py_ml/query_compiler.py", line 44, in <module>
    from opensearch_py_ml.field_mappings import FieldMappings
  File "/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/opensearch_py_ml/field_mappings.py", line 44, in <module>
    from pandas.core.dtypes.common import (
ImportError: cannot import name 'is_datetime_or_timedelta_dtype' from 'pandas.core.dtypes.common' (/Users/handler/code/amazon-pqa/.venv/lib/python3.11/site-packages/pandas/core/dtypes/common.py)

Not sure whether it's my environment or something else. I also tried various versions of Pandas from 2.0.0 through 2.2.1, same problem. I seem to have the latest version of opensearch-py-ml (and opensearch-py)

I tried to manually pick up the changes in this PR to file opensearch_py_ml/field_mappings.py. It works for me.

yuye-aws · 2024-05-22T02:57:32Z

This solution also works for me: #306 (comment)

caroline717 · 2024-11-07T15:28:44Z

I'm still getting this issue on opensearch-py-ml==1.1.0. Is the suggestion still to just lock pandas? :/

yuye-aws · 2024-11-07T23:54:55Z

I'm still getting this issue on opensearch-py-ml==1.1.0. Is the suggestion still to just lock pandas? :/

Because 1.1.0 version does not include this pull request. You can check my comment above.

rawwar added 3 commits October 13, 2023 10:54

replace is_datetime_or_timedelta_dtype

da9f744

Signed-off-by: kalyan <[email protected]>

fix formatting

00a3dbe

Signed-off-by: kalyan <[email protected]>

fix import order

71ab696

Signed-off-by: kalyan <[email protected]>

rawwar requested review from dhrubo-os, greaa-aws, ylwu-amzn, b4sjoo, jngz-es and rbhavna as code owners October 13, 2023 05:31

dhrubo-os approved these changes Oct 27, 2023

View reviewed changes

update changelog

a4cfd47

Signed-off-by: kalyan <[email protected]>

Merge branch 'main' of https://github.com/opensearch-project/opensear…

a11b5b2

…ch-py-ml into kalyan/263-fix-pandas-import-issue Signed-off-by: kalyanr <[email protected]>

rawwar requested review from dhrubo-os November 13, 2023 22:12

dhrubo-os approved these changes Nov 13, 2023

View reviewed changes

b4sjoo approved these changes Nov 13, 2023

View reviewed changes

dhrubo-os merged commit 7c2528b into opensearch-project:main Nov 13, 2023
15 of 16 checks passed

dhrubo-os mentioned this pull request Feb 13, 2024

[BUG] is_datetime_or_timedelta removed in Pandas 2.1.0 #263

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

replace `is_datetime_or_timedelta_dtype` #316

replace `is_datetime_or_timedelta_dtype` #316

rawwar commented Oct 13, 2023

dhrubo-os commented Oct 19, 2023

codecov bot commented Oct 26, 2023 •

edited

Loading

rawwar commented Oct 27, 2023

dhrubo-os commented Oct 27, 2023

dhrubo-os commented Oct 27, 2023 •

edited

Loading

rawwar commented Oct 31, 2023

Jon-AtAWS commented Mar 24, 2024

yuye-aws commented May 21, 2024

yuye-aws commented May 21, 2024

yuye-aws commented May 22, 2024

caroline717 commented Nov 7, 2024

yuye-aws commented Nov 7, 2024

replace is_datetime_or_timedelta_dtype #316

replace is_datetime_or_timedelta_dtype #316

Conversation

rawwar commented Oct 13, 2023

Description

Issues Resolved

Check List

dhrubo-os commented Oct 19, 2023

codecov bot commented Oct 26, 2023 • edited Loading

Codecov Report

rawwar commented Oct 27, 2023

dhrubo-os commented Oct 27, 2023

dhrubo-os commented Oct 27, 2023 • edited Loading

rawwar commented Oct 31, 2023

Jon-AtAWS commented Mar 24, 2024

yuye-aws commented May 21, 2024

yuye-aws commented May 21, 2024

yuye-aws commented May 22, 2024

caroline717 commented Nov 7, 2024

yuye-aws commented Nov 7, 2024

replace `is_datetime_or_timedelta_dtype` #316

replace `is_datetime_or_timedelta_dtype` #316

codecov bot commented Oct 26, 2023 •

edited

Loading

dhrubo-os commented Oct 27, 2023 •

edited

Loading