Skip to content

Commit

Permalink
GH-43956: [Format] Allow Decimal32/Decimal64 in format (#43976)
Browse files Browse the repository at this point in the history
### Rationale for this change
Widening the Decimal128/256 type to allow for bitwidths of 32 and 64
allows for more interoperability with other libraries and utilities
which already support these types. This provides even more opportunities
for zero-copy interactions between things such as libcudf and various
databases.

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

### What changes are included in this PR?
Updating the documentation in Schema.fbs to explicitly state that 32-bit
and 64-bit is now allowed for bitwidths of Decimal types. This is the
only area in the the spec that mentions the allowed decimal bitwidths.
* GitHub Issue: #43956

---------

Co-authored-by: Antoine Pitrou <[email protected]>
  • Loading branch information
zeroshade and pitrou authored Sep 10, 2024
1 parent 44b72d5 commit c4b68ee
Show file tree
Hide file tree
Showing 3 changed files with 11 additions and 5 deletions.
2 changes: 1 addition & 1 deletion docs/source/format/Columnar.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
Arrow Columnar Format
*********************

*Version: 1.4*
*Version: 1.5*

.. seealso:: :ref:`Additions to the Arrow columnar format since version 1.0.0
<post-1-0-0-format-versions>`
Expand Down
5 changes: 5 additions & 0 deletions docs/source/format/Versioning.rst
Original file line number Diff line number Diff line change
Expand Up @@ -105,3 +105,8 @@ Version 1.4
* Added :ref:`listview-layout` and the associated ListView and LargeListView
types.
* Added :ref:`variadic-buffers`.

Version 1.5
-----------

* Expanded Decimal type bit widths to allow 32-bit and 64-bit types.
9 changes: 5 additions & 4 deletions format/Schema.fbs
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
/// Version 1.3 - Add Run-End Encoded.
/// Version 1.4 - Add BinaryView, Utf8View, variadicBufferCounts, ListView, and
/// LargeListView.
/// Version 1.5 - Add 32-bit and 64-bit as allowed bit widths for Decimal

namespace org.apache.arrow.flatbuf;

Expand Down Expand Up @@ -222,17 +223,17 @@ table RunEndEncoded {
}

/// Exact decimal value represented as an integer value in two's
/// complement. Currently only 128-bit (16-byte) and 256-bit (32-byte) integers
/// are used. The representation uses the endianness indicated
/// in the Schema.
/// complement. Currently 32-bit (4-byte), 64-bit (8-byte),
/// 128-bit (16-byte) and 256-bit (32-byte) integers are used.
/// The representation uses the endianness indicated in the Schema.
table Decimal {
/// Total number of decimal digits
precision: int;

/// Number of digits after the decimal point "."
scale: int;

/// Number of bits per value. The only accepted widths are 128 and 256.
/// Number of bits per value. The accepted widths are 32, 64, 128 and 256.
/// We use bitWidth for consistency with Int::bitWidth.
bitWidth: int = 128;
}
Expand Down

0 comments on commit c4b68ee

Please sign in to comment.