Skip to content

apache/arrow-rs

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

9f12eca · Jul 2, 2024
Jul 1, 2024
Jul 1, 2024
Jul 1, 2024
Jan 17, 2024
Jun 26, 2024
Jul 2, 2024
Jun 17, 2024
Jun 26, 2024
Jun 29, 2024
May 20, 2024
Jun 29, 2024
Jun 29, 2024
Jun 13, 2024
Jun 26, 2024
Apr 5, 2024
Jun 26, 2024
Jun 13, 2024
Jun 21, 2024
Jun 24, 2024
Jun 26, 2024
Feb 22, 2022
Jul 2, 2024
Mar 26, 2024
Jul 2, 2024
Apr 30, 2024
Jul 2, 2024
Jul 1, 2024
Jul 1, 2024
Jan 17, 2024
Jun 27, 2023
Oct 25, 2022
Jan 30, 2024
Jan 4, 2024
Apr 18, 2021
Apr 18, 2021
Jul 2, 2024
Jul 2, 2024
Jun 5, 2021
Jun 26, 2024
Jul 2, 2024
Aug 16, 2022
Jul 15, 2019
Jun 26, 2024
Aug 18, 2016
Oct 25, 2023
Oct 19, 2023

Repository files navigation

Native Rust implementation of Apache Arrow and Apache Parquet

Coverage Status

Welcome to the Rust implementation of Apache Arrow, the popular in-memory columnar format.

This repo contains the following main components:

Crate Description Latest API Docs README
arrow Core functionality (memory layout, arrays, low level computations) docs.rs (README)
arrow-flight Support for Arrow-Flight IPC protocol docs.rs (README)
object-store Support for object store interactions (aws, azure, gcp, local, in-memory) docs.rs (README)
parquet Support for Parquet columnar file format docs.rs (README)
parquet_derive A crate for deriving RecordWriter/RecordReader for arbitrary, simple structs docs.rs (README)

The current development version the API documentation in this repo can be found here.

Release Versioning and Schedule

arrow and parquet crates

The Arrow Rust project releases approximately monthly and follows Semantic Versioning.

Due to available maintainer and testing bandwidth, arrow crates (arrow, arrow-flight, etc.) are released on the same schedule with the same versions as the parquet and [parquet-derive] crates.

Starting June 2024, we plan to release new major versions with potentially breaking API changes at most once a quarter, and release incremental minor versions in the intervening months. See this ticket for more details.

To keep our maintenance burden down, we do regularly scheduled releases (major and minor) from the master branch. How we handle PRs with breaking API changes is described in the contributing guide.

For example:

Approximate Date Version Notes
Jun 2024 52.0.0 Major, potentially breaking API changes
Jul 2024 52.1.0 Minor, NO breaking API changes
Aug 2024 52.2.0 Minor, NO breaking API changes
Sep 2024 53.0.0 Major, potentially breaking API changes

object_store crate

The object_store crate is released independently of the arrow and parquet crates and follows Semantic Versioning. We aim to release new versions approximately every 2 months.

Related Projects

There are several related crates in different repositories

Crate Description Documentation
datafusion In-memory query engine with SQL support (README)
ballista Distributed query execution (README)
object_store_opendal Use opendal as object_store backend (README)

Collectively, these crates support a wider array of functionality for analytic computations in Rust.

For example, you can write SQL queries or a DataFrame (using the datafusion crate) to read a parquet file (using the parquet crate), evaluate it in-memory using Arrow's columnar format (using the arrow crate), and send to another process (using the arrow-flight crate).

Generally speaking, the arrow crate offers functionality for using Arrow arrays, and datafusion offers most operations typically found in SQL, including joins and window functions.

You can find more details about each crate in their respective READMEs.

Arrow Rust Community

The dev@arrow.apache.org mailing list serves as the core communication channel for the Arrow community. Instructions for signing up and links to the archives can be found on the Arrow Community page. All major announcements and communications happen there.

The Rust Arrow community also uses the official ASF Slack for informal discussions and coordination. This is a great place to meet other contributors and get guidance on where to contribute. Join us in the #arrow-rust channel and feel free to ask for an invite via:

  1. the dev@arrow.apache.org mailing list
  2. the GitHub Discussions
  3. the Discord channel

The Rust implementation uses GitHub issues as the system of record for new features and bug fixes and this plays a critical role in the release process.

For design discussions we generally collaborate on Google documents and file a GitHub issue linking to the document.

There is more information in the contributing guide.