Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Downcasting neither warned against nor discussed in documentation #50214

Closed
2 of 3 tasks
mcp292 opened this issue Dec 12, 2022 · 10 comments
Closed
2 of 3 tasks

BUG: Downcasting neither warned against nor discussed in documentation #50214

mcp292 opened this issue Dec 12, 2022 · 10 comments
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member

Comments

@mcp292
Copy link

mcp292 commented Dec 12, 2022

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd


data = [0, 10.25, 11.11, 5.555, 50.55, 70.75, 98.99, 99.98, 99.99, 100.00]

df = pd.DataFrame(data)
f16_df = df.astype("float16").round(2)
f32_df = df.astype("float32").round(2)
f64_df = df.astype("float64").round(2)
f_df = df.astype(float).round(2)
f16t32_df = f16_df.astype("float32").round(2)

print(f"df:\n{df}\n")
print(f"f16:\n{f16_df}\n")
print(f"f32:\n{f32_df}\n")
print(f"f64:\n{f64_df}\n")
print(f"f:\n{f_df}\n")
print(f"f16t32:\n{f16t32_df}\n")
df:
         0
0    0.000
1   10.250
2   11.110
3    5.555
4   50.550
5   70.750
6   98.990
7   99.980
8   99.990
9  100.000

f16:
            0
0    0.000000
1   10.250000
2   11.109375
3    5.558594
4   50.562500
5   70.750000
6   99.062500
7  100.000000
8  100.000000
9  100.000000

f32:
            0
0    0.000000
1   10.250000
2   11.110000
3    5.560000
4   50.549999
5   70.750000
6   98.989998
7   99.980003
8   99.989998
9  100.000000

f64:
        0
0    0.00
1   10.25
2   11.11
3    5.56
4   50.55
5   70.75
6   98.99
7   99.98
8   99.99
9  100.00

f:
        0
0    0.00
1   10.25
2   11.11
3    5.56
4   50.55
5   70.75
6   98.99
7   99.98
8   99.99
9  100.00

f16t32:
            0
0    0.000000
1   10.250000
2   11.110000
3    5.560000
4   50.560001
5   70.750000
6   99.059998
7  100.000000
8  100.000000
9  100.000000

Issue Description

Downcasting neither warned against nor discussed in the documentation. Upcasting has a section under dytpes, and downcasting is mentioned only within the scope of pd.to_numeric().

Further, I'm not sure if the behavior is undefined because for float16 the mantissa is 10 bits which means it should max out at 1023, yet 10.25 (1025e-2) and up are preserved.

pd.DataFrame.round() does not seem to take effect when using NumPy
datatypes
.

Expected Behavior

  1. Receive warning.
  2. Truncate values.
  3. Discussed in documentation.

Installed Versions

INSTALLED VERSIONS

commit : 8dab54d
python : 3.10.8.final.0
python-bits : 64
OS : Linux
OS-release : 6.0.12-arch1-1
Version : #1 SMP PREEMPT_DYNAMIC Thu, 08 Dec 2022 11:03:38 +0000
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.5.2
numpy : 1.23.5
pytz : 2021.3
dateutil : 2.8.2
setuptools : 58.1.0
pip : 22.3.1
Cython : 0.29.32
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.4.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.5.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.1
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None
None

@mcp292 mcp292 added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Dec 12, 2022
@phofl phofl closed this as completed Dec 12, 2022
@mcp292
Copy link
Author

mcp292 commented Dec 12, 2022

This is not discussed in the documentation, yet the issue is closed.

@phofl
Copy link
Member

phofl commented Dec 12, 2022

PRs are welcome I guess. Not sure where to add this though. It should be more or less clear that we return python dtypes based on the examples in the docs

@mcp292
Copy link
Author

mcp292 commented Dec 12, 2022

In the dtypes section (linked above), they use float32 and int8 in the first example. Are you suggesting this is not good practice nor supported?

@mcp292
Copy link
Author

mcp292 commented Dec 12, 2022

Related to #50213.

@phofl
Copy link
Member

phofl commented Dec 12, 2022

This is perfectly fine, but they get converted to python floats instead of np.float32() objects when writing them out, like printing or csv.

@mcp292
Copy link
Author

mcp292 commented Dec 12, 2022

Okay, I'll take it.

This is still unanswered though:

Further, I'm not sure if the behavior is undefined because for float16 the mantissa is 10 bits which means it should max out at 1023, yet 10.25 (1025e-2) and up are preserved.

@phofl
Copy link
Member

phofl commented Dec 12, 2022

Please comment only in one issue

@mcp292
Copy link
Author

mcp292 commented Dec 12, 2022

I just realized I didn't name this issue. These issues are separate, but use the same example.

@mcp292 mcp292 changed the title BUG: BUG: Downcasting neither warned against nor discussed in documentation Dec 12, 2022
@phofl
Copy link
Member

phofl commented Dec 12, 2022

Not sure what you mean with downcasting, but you can cast whatever you want, we don’t warn about loss of data right now

@mcp292
Copy link
Author

mcp292 commented Dec 12, 2022

Opposite of upcasting.

There would only need to be a warning if risking data loss, which you say is not implemented.

This is fine to not implement, so long as it is mentioned in the docs. I feel a downcasting section could be added to the link in this comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member
Projects
None yet
Development

No branches or pull requests

2 participants