Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Floating value literals without postfix should be parsed as decimal #4072

Open
Tracked by #3480
viirya opened this issue Nov 2, 2022 · 4 comments
Open
Tracked by #3480

Floating value literals without postfix should be parsed as decimal #4072

viirya opened this issue Nov 2, 2022 · 4 comments
Labels
enhancement New feature or request
Milestone

Comments

@viirya
Copy link
Member

viirya commented Nov 2, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and why for this feature, in addition to the what)

Related to #4024, #4071.

A literal like 0.06 is parsed as double in DataFusion. It causes some counter-intuitive result as 0.06 - 0.01 = 0.049999999, 0.06 + 0.01 = 0.069999999 in DataFusion. This result is correct, though. (If you ask Spark to treat them as double (i.e., 0.06f), you will get same result).

Such literals are parsed as decimal in Spark. I think for floating literals without postfix, we should parse them as decimal in DataFusion

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

@viirya viirya added the enhancement New feature or request label Nov 2, 2022
@andygrove
Copy link
Member

Thanks @viirya I had forgotten about this ... this is the relevant code in sql/planner.rs, for reference

// Parse number in sql string, convert to Expr::Literal
fn parse_sql_number(n: &str) -> Result<Expr> {
    // parse first as i64
    n.parse::<i64>()
        .map(lit)
        // if parsing as i64 fails try f64
        .or_else(|_| n.parse::<f64>().map(lit))
        .map_err(|_| {
            DataFusionError::from(ParserError(format!(
                "Cannot parse {} as i64 or f64",
                n
            )))
        })
}

@andygrove andygrove added this to the 14.0.0 milestone Nov 2, 2022
@viirya
Copy link
Member Author

viirya commented Nov 2, 2022

Yes, I played this a bit yesterday. With some tweak, I can make it parsed as decimal and get 0.06 - 0.01 = 0.05 correctly.

@andygrove
Copy link
Member

Here is the spec for SQL numeric literals:

<signed numeric literal> ::=
[ <sign> ] <unsigned numeric literal>

<unsigned numeric literal> ::=
<exact numeric literal>
| <approximate numeric literal>

<exact numeric literal> ::=
<unsigned integer> [ <period> [ <unsigned integer> ] ]
| <period> <unsigned integer>

<sign> ::=
<plus sign>
| <minus sign>

<approximate numeric literal> ::=
<mantissa> E <exponent>

<mantissa> ::=
<exact numeric literal>

<exponent> ::=
<signed integer>

<signed integer> ::=
[ <sign> ] <unsigned integer>

<unsigned integer> ::=
<digit>...

@liukun4515
Copy link
Contributor

agree with your suggestion that if there is no postfix, we will convert the data to decimal by default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants