Skip to content

apache/datafusion

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

01f12d9 · Sep 3, 2022
Sep 1, 2022
Aug 30, 2022
Aug 19, 2022
Mar 12, 2022
Sep 3, 2022
Aug 30, 2022
Sep 3, 2022
Aug 25, 2022
Sep 3, 2022
Aug 8, 2022
May 26, 2021
Aug 30, 2022
Sep 15, 2021
May 22, 2022
May 3, 2021
Apr 14, 2022
Nov 14, 2021
May 22, 2022
Apr 19, 2021
Mar 30, 2021
Jun 10, 2022
May 24, 2021
Aug 13, 2022
Jul 14, 2022
Aug 15, 2022
Jul 15, 2019
Aug 22, 2022
Aug 18, 2016
Aug 24, 2021
Oct 27, 2021

Repository files navigation

DataFusion

logo

DataFusion is an extensible query planning, optimization, and execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

Coverage Status

Features

  • SQL query planner with support for multiple SQL dialects
  • DataFrame API
  • Parquet, CSV, JSON, and Avro file formats are supported natively. Custom file formats can be supported by implementing a TableProvider trait.
  • Supports popular object stores, including AWS S3, Azure Blob Storage, and Google Cloud Storage. There are extension points for implementing custom object stores.

Use Cases

DataFusion is modular in design with many extension points and can be used without modification as an embedded query engine and can also provide a foundation for building new systems. Here are some example use cases:

  • DataFusion can be used as a SQL query planner and query optimizer, providing optimized logical plans that can then be mapped to other execution engines.
  • DataFusion is used to create modern, fast and efficient data pipelines, ETL processes, and database systems, which need the performance of Rust and Apache Arrow and want to provide their users the convenience of an SQL interface or a DataFrame API.

Why DataFusion?

  • High Performance: Leveraging Rust and Arrow's memory model, DataFusion achieves very high performance
  • Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
  • Easy to Embed: Allowing extension at almost any point in its design, DataFusion can be tailored for your specific use case
  • High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can be used as the foundation for production systems.

DataFusion Community Extensions

There are a number of community projects that extend DataFusion or provide integrations with other systems.

Language Bindings

Integrations

Known Uses

Here are some of the projects known to use DataFusion:

(if you know of another project, please submit a PR to add a link!)

Example Usage

Please see example usage to find how to use DataFusion.

Roadmap

Please see Roadmap for information of where the project is headed.

Architecture Overview

There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.

  • (July 2022): DataFusion and Arrow: Supercharge Your Data Analytical Tool with a Rusty Query Engine: recording and slides
  • (March 2021): The DataFusion architecture is described in Query Engine Design and the Rust-Based DataFusion in Apache Arrow: recording (DataFusion content starts ~ 15 minutes in) and slides
  • (February 2021): How DataFusion is used within the Ballista Project is described in *Ballista: Distributed Compute with Rust and Apache Arrow: recording

User Guide

Please see User Guide for more information about DataFusion.

Contributor Guide

Please see Contributor Guide for information about contributing to DataFusion.