Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get precise precision for Decimal literal in FieldToDataType and DecimalField::getPrec #5007

Closed
windtalker opened this issue May 26, 2022 · 0 comments · Fixed by #5014
Closed
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@windtalker
Copy link
Contributor

Enhancement

In FieldToDataType

DataTypePtr operator()(const DecimalField<T> & x) const
    {
        PrecType prec = maxDecimalPrecision<T>();
        return std::make_shared<DataTypeDecimal<T>>(prec, x.getScale());
    }

For a Decimal literal, its precision is always the max precision of the Decimal type.(9 for Decimal32, 18 for Decimal64, 38 for Decimal128 and 65 for Decimal256). But, for a decimal literal, we can always get its precise precision, no need to use the upper bound.

In DecimalField::getPrec

UInt32 getPrec() const
    {
        UInt32 cnt = 0;
        auto x = dec.value;
        while (x != 0)
        {
            x /= 10;
            cnt++;
        }
        if (cnt == 0)
            cnt = 1;
        return cnt;
    }

As we can see, if the dec.value == 0, the prec is 1, if the scale is 2, then the (prec, scale) will be (1,2) which is not valid, so we should return std::max(cnt, scale) as its prec.

@windtalker windtalker added the type/enhancement The issue or PR belongs to an enhancement. label May 26, 2022
ti-chi-bot pushed a commit that referenced this issue May 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant