firepubauditsource

Publishes Firestore data changes to Pub/Sub as JSON audit records for downstream processing.

References

• firepubauditsource - Publishes Firestore data changes to Pub/Sub as JSON audit records for downstream processing.
• firepubauditsource-tofu - A module for OpenTofu that deploys firepubauditsource to GCP Cloud Run, along with configuring essential services including Eventarc for Firestore and Pub/Sub.
• bqpubauditsink - Ingests Pub/Sub audit JSON events and inserts the records into BigQuery.
• bqpubauditsink-tofu - A module for OpenTofu that deploys bqpubauditsink to GCP Cloud Run, along with configuring essential services including the Pub/Sub subscription and BigQuery dataset and table.

Overview

The purpose of this application is to take all record changes from a Firestore table and publish them to a Pub/Sub topic. This allows for downstream applications to process the complete database record changes including the old and new values.

This application is designed to run in Cloud Run and is triggered by changes to Firestore records using Eventarc.

Use Case: BigQuery

• A companion application bqpubauditsink takes the Pub/Sub messages and writes them to BigQuery in a way that allows BigQuery to be a direct replica of the data stored in Firestore.

Configuration

This application is run as a docker container and requires the following environment variables to be set:

• PROJECT_ID: The GCP project ID where the Firestore and Pub/Sub resources are located.
• PUBSUB_TOPIC: The Pub/Sub topic to publish the audit records to.

Example Pub/Sub Message

The following show what the JSON message will look like when published to Pub/Sub. The oldValue field will be null for inserts, and value will be null for deletes.

Inserting a Record:

{
  "timestamp": "2024-10-27 12:00:00.000000",
  "database": "(default)",
  "documentPath": "mycollection/mydoc",
  "value": {
    "foo": "new"
  },
  "oldValue": null
}

Updating a Record:

{
  "timestamp": "2024-10-27 12:00:10.000000",
  "database": "(default)",
  "documentPath": "mycollection/mydoc",
  "value": {
    "foo": "updated"
  },
  "oldValue": {
    "foo": "bar"
  }
}

Deleting a Record:

{
  "timestamp": "2024-10-27 12:00:20.000000",
  "database": "(default)",
  "documentPath": "mycollection/mydoc",
  "value": null,
  "oldValue": {
    "foo": "bar"
  }
}

Pub/Sub Attributes

In order to assist with subscriptions that may want to only process a subset of messages, the following attributes are added to the Pub/Sub message:

• database: The Firestore database ID.
• action: The Firestore action that triggered the message. This will be one of create, update, or delete.

The document path is also included as an attribute split into parts on the / character. This allows for subscriptions to filter on specific collections or documents. The attribute names are path0, path1, etc. so that the first part of the path is path0 represents the collection and path1 represents the document, but additional parts are also included for nested documents.

Limitations

• The translation from the Firestore document to JSON is not perfect, it uses firestoreproto2json with the default settings to convert the Firestore document to JSON. This means that some data types may not be converted correctly.