Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add explanation and recommendations for context usage #1060

Merged
merged 3 commits into from
Oct 15, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/context_object.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# The Context Object

Many of the methods in the [Stream](classes/singer_sdk.Stream) class and its subclasses accept
a `context` parameter, which is a dictionary that contains information about the stream
partition or parent stream.

## Best practices for using context

- The context object MUST NOT contain any sensitive information, such as API keys or secrets.
This is because the context is<br><br>

1) sent to the target,
2) stored in the state file and
3) logged to the console as a tag in metrics and logs.<br><br>

- The context object SHOULD NOT be mutated during the stream's lifecycle. This is because the
context is stored in the state file, and mutating it will cause the state file to be
inconsistent with the actual state of the stream.
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ Advanced Topics

parent_streams
partitioning
context_object
stream_maps
batch
porting
Expand Down
25 changes: 14 additions & 11 deletions docs/partitioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,23 @@ which each have their own state and their own distinct queryable domain.

## If you do not require partitioning

In general, developers can simply ignore the `context` arguments in methods like
`Stream.get_records()` if partitioning is not required.
In general, developers can simply ignore the [`context`](./context_object.md) arguments
in methods like [`Stream.get_records()`](singer_sdk.Stream.get_records) if partitioning
is not required.

## If you do want to utilize partitioning

To take advantage of partitioning, first override the `Stream.partitions` property,
returning a list of dictionaries, where each dictionary uniquely defines the construct of
a partition. For instance, a regionally partitioned stream may return the following:
To take advantage of partitioning, first override the
[`Stream.partitions`](singer_sdk.Stream.partitions) property, returning a list of
dictionaries, where each dictionary uniquely defines the construct of a partition.
For instance, a regionally partitioned stream may return the following:

`[{"region": "us-east"}, {"region": "us-west"}, ...]`

For any streams which define the `partitions` property, the individual partitions will be
passed one at a time through the `partition` argument of methods which reference the
partition, such as `Stream.get_records()`.
For any streams which define the [`partitions`](singer_sdk.Stream.partitions) property,
the individual partitions will be passed one at a time through the `context` argument
of methods which reference the partition, such as
[`Stream.get_records()`](singer_sdk.Stream.get_records).

## If you are unsure if partitioning will be needed

Expand All @@ -28,9 +31,9 @@ work regardless of whether partition is an actual partition context or `None`, m
no partition is specified.

When dealing with state, for example, developers may always call
`Stream.get_context_state(context)` even if `context` is not set.
The method will automatically return the state that is appropriate, either for the partition
or for the stream.
[`Stream.get_context_state(context)`](singer_sdk.Stream.get_context_state) even if
aaronsteers marked this conversation as resolved.
Show resolved Hide resolved
`context` is not set. The method will automatically return the state that is appropriate,
either for the partition or for the stream.

## Additional State Partitioning References

Expand Down