Skip to content

Commit

Permalink
docs: Document how to configure nested stream maps values with enviro…
Browse files Browse the repository at this point in the history
…nment variables in Meltano
  • Loading branch information
edgarrmondragon committed Dec 3, 2024
1 parent 87da56d commit ec774c1
Showing 1 changed file with 28 additions and 0 deletions.
28 changes: 28 additions & 0 deletions docs/stream_maps.md
Original file line number Diff line number Diff line change
Expand Up @@ -778,3 +778,31 @@ the `key_properties` in an extract-load pipeline. For instance, it is common to
"append-only" loading behavior in certain targets, as may be required for historical reporting. This does not change the
underlying nature of the `primary_key` configuration in the upstream source data, only how it will be landed or deduped
in the downstream source.


### Q: How do I use Meltano environment variables to configure stream maps?

**Answer:** Environment variables in Meltano can be used to configure stream maps, but you first need to add the corresponding settings
to your plugins `settings` option. For example:

```yaml
plugins:
extractors:
- name: tap-csv
variant: meltanolabs
pip_url: git+https://github.com/MeltanoLabs/tap-csv.git
settings:
- name: stream_maps.customers.email
- name: stream_maps.customers.email_domain
- name: stream_maps.customers.email_hash
- name: stream_maps.customers.__else__
- name: stream_maps.stream_map_config
```

Then, you can set the following environment variables:

```shell
TAP_CSV_STREAM_MAPS_CUSTOMERS_EMAIL_DOMAIN='email.split("@")[-1]'
TAP_CSV_STREAM_MAPS_CUSTOMERS_EMAIL_HASH='md5(config["hash_seed"] + email)'
TAP_CSV_STREAM_MAP_CONFIG_HASH_SEED='01AWZh7A6DzGm6iJZZ2T'
```

0 comments on commit ec774c1

Please sign in to comment.