diff --git a/docs/context_object.md b/docs/context_object.md new file mode 100644 index 000000000..09e5ed469 --- /dev/null +++ b/docs/context_object.md @@ -0,0 +1,18 @@ +# The Context Object + +Many of the methods in the [Stream](classes/singer_sdk.Stream) class and its subclasses accept +a `context` parameter, which is a dictionary that contains information about the stream +partition or parent stream. + +## Best practices for using context + +- The context object MUST NOT contain any sensitive information, such as API keys or secrets. + This is because the context is

+ + 1) sent to the target, + 2) stored in the state file and + 3) logged to the console as a tag in metrics and logs.

+ +- The context object SHOULD NOT be mutated during the stream's lifecycle. This is because the + context is stored in the state file, and mutating it will cause the state file to be + inconsistent with the actual state of the stream. diff --git a/docs/index.rst b/docs/index.rst index 599d54f4b..5f3d0d448 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -63,6 +63,7 @@ Advanced Topics parent_streams partitioning + context_object stream_maps batch porting diff --git a/docs/partitioning.md b/docs/partitioning.md index 37a2f99e2..95103dc45 100644 --- a/docs/partitioning.md +++ b/docs/partitioning.md @@ -5,20 +5,23 @@ which each have their own state and their own distinct queryable domain. ## If you do not require partitioning -In general, developers can simply ignore the `context` arguments in methods like -`Stream.get_records()` if partitioning is not required. +In general, developers can simply ignore the [`context`](./context_object.md) arguments +in methods like [`Stream.get_records()`](singer_sdk.Stream.get_records) if partitioning +is not required. ## If you do want to utilize partitioning -To take advantage of partitioning, first override the `Stream.partitions` property, -returning a list of dictionaries, where each dictionary uniquely defines the construct of -a partition. For instance, a regionally partitioned stream may return the following: +To take advantage of partitioning, first override the +[`Stream.partitions`](singer_sdk.Stream.partitions) property, returning a list of +dictionaries, where each dictionary uniquely defines the construct of a partition. +For instance, a regionally partitioned stream may return the following: `[{"region": "us-east"}, {"region": "us-west"}, ...]` -For any streams which define the `partitions` property, the individual partitions will be -passed one at a time through the `partition` argument of methods which reference the -partition, such as `Stream.get_records()`. +For any streams which define the [`partitions`](singer_sdk.Stream.partitions) property, +the individual partitions will be passed one at a time through the `context` argument +of methods which reference the partition, such as +[`Stream.get_records()`](singer_sdk.Stream.get_records). ## If you are unsure if partitioning will be needed @@ -28,9 +31,9 @@ work regardless of whether partition is an actual partition context or `None`, m no partition is specified. When dealing with state, for example, developers may always call -`Stream.get_context_state(context)` even if `context` is not set. -The method will automatically return the state that is appropriate, either for the partition -or for the stream. +[`Stream.get_context_state(context)`](singer_sdk.Stream.get_context_state) even if +`context` is not set. The method will automatically return the state that is appropriate, +either for the partition or for the stream. ## Additional State Partitioning References