[Task]: Build a cleaner for assets in the GCP test environment

[pabloem]

What needs to happen?

Many tests and examples in the Beam codebase create GCP assets that are not cleaned up. Some examples:

• BQ datasets
• Pubsub topics, subscriptions
• BigTable instances?
• Others?

The task is to build automation that can safely disable-then-clean-up leftover assets from the GCP environment. The tool would:

• List existing inventory of various kinds of assets
• Figure out which ones are meant to be cleaned up (i.e. short-lived test assets, vs. long-lived project assets like base sources, base buckets, etc).
• Wait a week or so
• Disable and (eventually?) delete these assets

Why?

• A lot of these assets have billing implications for the project
• A lot of these assets make navigating the project more difficult (e.g. reviewing Bigquery datasets)

Risks:

• We need to roll this out carefully to make sure we don't delete important assets. We'd probably start with strict lists of inclusions (i.e. everything that's not included will be excluded).

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[pabloem]

Sample of this issue: #19760