Skip to content

Commit

Permalink
docs(firestore-bigquery-export): add documentation on reconfig workar…
Browse files Browse the repository at this point in the history
…ound
  • Loading branch information
cabljac committed Jan 6, 2025
1 parent ed34bd6 commit 3e8a5fd
Show file tree
Hide file tree
Showing 3 changed files with 42 additions and 0 deletions.
4 changes: 4 additions & 0 deletions firestore-bigquery-export/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
## Version 0.1.57

docs - add documentation on workarounds to mitigate data loss during extension updates

## Version 0.1.56

feat - improve sync strategy by immediately writing to BQ, and using cloud tasks only as a last resort
Expand Down
19 changes: 19 additions & 0 deletions firestore-bigquery-export/PREINSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,25 @@ For PowerShell script:

**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts.

#### Mitigating Data Loss During Extension Updates

When updating or reconfiguring this extension, there may be a brief period where data streaming from Firestore to BigQuery is interrupted. While this limitation exists within the Extensions platform, we provide two strategies to mitigate potential data loss.

##### Strategy 1: Post-Update Import
After reconfiguring the extension, run the import script on your collection to ensure all data is captured. Refer to the "Import Existing Documents" section above for detailed steps.

##### Strategy 2: Parallel Instance Method
1. Install a second instance of the extension that streams to a new BigQuery table
2. Reconfigure the original extension
3. Once the original extension is properly configured and streaming events
4. Uninstall the second instance
5. Run a BigQuery merge job to combine the data from both tables

##### Considerations
- Strategy 1 is simpler but may result in duplicate records that need to be deduplicated
- Strategy 2 requires more setup but provides better data continuity
- Choose the strategy that best aligns with your data consistency requirements and operational constraints

#### Billing
To install an extension, your project must be on the [Blaze (pay as you go) plan](https://firebase.google.com/pricing)

Expand Down
19 changes: 19 additions & 0 deletions firestore-bigquery-export/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,25 @@ For PowerShell script:

**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts.

#### Mitigating Data Loss During Extension Updates

When updating or reconfiguring this extension, there may be a brief period where data streaming from Firestore to BigQuery is interrupted. While this limitation exists within the Extensions platform, we provide two strategies to mitigate potential data loss.

##### Strategy 1: Post-Update Import
After reconfiguring the extension, run the import script on your collection to ensure all data is captured. Refer to the "Import Existing Documents" section above for detailed steps.

##### Strategy 2: Parallel Instance Method
1. Install a second instance of the extension that streams to a new BigQuery table
2. Reconfigure the original extension
3. Once the original extension is properly configured and streaming events
4. Uninstall the second instance
5. Run a BigQuery merge job to combine the data from both tables

##### Considerations
- Strategy 1 is simpler but may result in duplicate records that need to be deduplicated
- Strategy 2 requires more setup but provides better data continuity
- Choose the strategy that best aligns with your data consistency requirements and operational constraints

#### Billing
To install an extension, your project must be on the [Blaze (pay as you go) plan](https://firebase.google.com/pricing)

Expand Down

0 comments on commit 3e8a5fd

Please sign in to comment.