Large numbers parsed as strings

[slisaasquatch]

Here's the sample code:

  @Test
  public void testLargeNum() {
    final JSONObject j = new JSONObject("{\r\n" +
        "  \"foo\": 10000000000000000000000000000000\r\n" +
        "}");
    // The following line prints class java.lang.String!!!
    System.out.println(j.get("foo").getClass());
  }

[stleary]

@slisaasquatch Thanks for bringing up this issue; interesting that numbers are handled differently depending on whether chars like '.' or 'e' are present. Do you want to present a case that the code is not in compliance with ECMA-404 and/or RFC8259? Otherwise, I believe this behavior has been in the code since the first commit for this repo (see stringToValue()).

[johnjaylward]

I was pretty sure we changed this behavior a few years ago to use BigInteger instead for these cases. Let me try with a recent build

[johnjaylward]

Ah, we never made the full switch. I left notes in the code on where it would be handled though:

https://github.com/stleary/JSON-java/blob/master/JSONObject.java#L2173-L2218

[stleary]

I think that code review is still open, if you mean #453, due to changes to existing behavior.

[slisaasquatch]

@slisaasquatch Thanks for bringing up this issue; interesting that numbers are handled differently depending on whether chars like '.' or 'e' are present. Do you want to present a case that the code is not in compliance with ECMA-404 and/or RFC8259? Otherwise, I believe this behavior has been in the code since the first commit for this repo (see stringToValue()).

Thanks for your quick response.

I checked the specs for numbers here, and all it says about really large numbers is that implementations may approximate the numbers to fit double precision floating points, and says nothing about turning large number into strings. And in this case, the number actually does fit into a double precision floating point.

[johnjaylward]

@stleary, yes. that PR would fix this and open up the parser to have more exact representation of what was in the original JSON text.

[stleary]

My best guess is that this implementation only guarantees integer interoperability to 2 ** 53 per the spec via Java Long type, and Double was only used to support decimal and scientific notation. I think if you have access to the software that generates very large numbers, it is probably better to just serialize them as strings.

[stleary]

@johnjaylward Agreed, exact representation is preferred. I wanted to keep the PR open to see if there is a way to implement it as an option, to ensure existing applications don't break.

[johnjaylward]

@stleary, gotcha. If I have time, I'll see if I can come up with a "parser config" type object that could be used to configure how items are parsed.

@slisaasquatch as a work around for your specific example, you can represent the number with a decimal point, or use scientific notation to force the parser to try to use a double as the backing type instead of an integer.

@Test
  public void testLargeNum() {
    final JSONObject j = new JSONObject("{\r\n" +
        "  \"foo\": 10000000000000000000000000000000.0\r\n" +
        "}");
    // The following line prints class Double!!!
    System.out.println(j.get("foo").getClass());
  }

OR:

@Test
  public void testLargeNum() {
    final JSONObject j = new JSONObject("{\r\n" +
        "  \"foo\": 1e31\r\n" +
        "}");
    // The following line prints class Double!!!
    System.out.println(j.get("foo").getClass());
  }

[javadev]

  @Test
  public void testLargeNum() {
    final JSONObject j = new JSONObject("{\r\n" +
        "  \"foo\": 10000000000000000000000000000000\r\n" +
        "}");
    // The following line prints class java.lang.String!!!
    System.out.println(j.get("foo").getClass());
  }

The expected value is class java.math.BigInteger.

[johnjaylward]

@javadev yes. This has been a known limitation of the library for quite some time. When a number can not be parsed as an integer or a double, it stores it as a string internally. In our case, we only parse using a double if the value "looks" like a double (contains a . or e), otherwise we parse it as a long.

In cases where the caller requires a number, this can be worked around by using the getNumber, getDouble, getBigInteger or getBigDecimal, or their opt* equivalent methods to get a consistent numeric type.

According to the JSON specifications, a parser has no obligation to handle values that don't fit in 64-bit integer or floating point data type (long, or double in Java). This parser historically tried to maintain the value as close as possible by keeping the value as a string instead of ignoring it or throwing an exception.

[stleary]

Closed due to working as designed.