Add stats format specification

landing-page/content/common/statistics-format-spec.md

@@ -0,0 +1,111 @@
+---
+url: statistics-format
+toc: false
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements.  See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License.  You may obtain a copy of the License at
+ -
+ -   http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Plain Format for Iceberg Statistics
+
+This is a specification for the Plain Format for Iceberg Statistics, a file
+format designed to store information such as statistics about data managed in an
+Iceberg table that cannot be stored directly within the Iceberg manifest. A
+statistics file contains arbitrary pieces of information (here called "blobs"),
+along with metadata necessary to interpret them.
+
+## Format specification
+
+A file conforming to the format specification should have the structure as
+described below.
+
+### File structure
+
+The file has the following structure
+
+```
+Magic Blob₁ Blob₂ ... Blobₙ Footer
+```
+
+where
+
+- `Magic` is four bytes 0x80, 0x70, 0x73, 0x83 (short for: Plain Format for
+  Iceberg Statistics),
+- `Blobᵢ` is i-th blob contained in the file, to be interpreted by application
+  according to the footer,
+- `Footer` is defined below.
+
+### Footer structure
+
+Footer has the following structure
+
+```
+FooterPayload FooterPayloadSize Reserved FileFormatVersion Magic
+```
+
+where
+
+- `FooterPayload` LZ4-compressed, UTF-8 encoded JSON payload describing the
+  blobs in the file, with the structure described below,
+- `FooterPayloadSize` is a length in bytes of the `FooterPayload`, stored as 4
+  byte integer, little-endian,
+- `Reserved` is 4 bytes reserved for future use, currently should be written as
+  0x00, 0x00, 0x00, 0x00,
+- `FileFormatVersion` is a number, stored as 4 byte integer, little-endian.
+
+### Footer Payload
+
+Footer payload bytes is LZ4-compressed, UTF-8 encoded JSON payload representing
+a single `FileMetadata` object.
+
+#### FileMetadata
+
+`FileMetadata` has the following fields
+
+
+| Field Name | Field Type             | Required | Description |
+| ---------- | ---------------------- | -------- | ----------- |
+| blobs      | list of Blob objects   | yes      |
+| properties | map {string: string}   | no       | storage for arbitrary meta-information, like writer identification/version
+
+#### Blob
+
+`Blob` has the following fields
+
+| Field Name | Field Type             | Required | Description |
+| ---------- | ---------------------- | -------- | ----------- |
+| type       | string                 | yes      | See [Blob types](#blob-types)
+| columns    | list of integer numbers| yes      | list of column IDs the blob was computed for
+| offset     | integer number (long)  | yes      | The offset in the file where the blob contents start. Reader should assume the value can be more than 2^32.
+| length     | integer number         | yes      | The length of the blob stored in the file
+| compression_codec | string          | no       | See [Compression codecs](#compression-codecs). If omitted, the data is assumed to be uncompressed.
+
+### Blob types
+
+The file format makes no assumptions about the information stored in it. The
+applications using the format need to agree on type of information stored, the
+format of the information and type designators. The list below is not part of
+the file format specification itself, but is provided for readers' convenience.
+
+| Blob type                      | Description |
+| ------------------------------ | ----------- |
+| ndv-long-little-endian         | 8-bytes integer stored little-endian and representing number of distinct values
+| apache-datasketches-theta-v1   | A compact [Apache DataSketches](https://datasketches.apache.org/)'s Theta sketch representation.
+
+### Compression codecs
+
+The compression codec used should be one of: ZSTD, LZ4. The data can also be
+uncompressed. For maximal interoperability, other codecs are not supported.