Include DuckDB and use it for previewing tabular data (parquet, csv, json, ...)

[keen85]

Preflight Checklist

• I have installed the latest version of Storage Explorer.
• I have checked existing resources, including the troubleshooting guide and the release notes.
• I have searched for similar issues.

Problem

Current preview functionality for data files (parquet, csv, json, ...) is limited:

• preview only works for small files
• data is downloaded to the client
• preview has problems with specific data types (e.g. timestamps in parquet)

Desired Solution

DuckDB is a lightweight analytical engine. It is able to query various file types and also features an extension so it works with Azure Storage seamlessly. Using DuckDB for previewing data from Azure Storage will have several benefits:

• less traffic: DuckDB does not require downloading the entire file but reads only parts of it
• DuckDB is already able to handle even complex data types
• integrating DuckDB into Azrue Storage Explorer theoretically would also allow users writing and executing more sophisticated queries than SELECT * FROM blob LIMIT 100

Alternatives and Workarounds

Users can install DuckDB manually, use Azure Storage Explorer to obtain a blob url, open DuckDB CLI and use the blob url to query a file from Azure Storage.

Additional Context

• DuckDB:
  • Main documentation https://duckdb.org
  • DuckDB's client APIs: https://duckdb.org/docs/api/overview
• Feature Requests that potentially would be solved by integrating DuckDB
  • Incorrect preview of parquet files with decimals #7957
  • Parquet Preview: Add support for Delta Encoding #7506
  • Investigate downloading partial parquet data for preview #7043
  • Support ZSTD on Parquet Preview #7675