REST Endpoints

Contents

• Datastream Data Model
• Datastream Management Service (DMS)
  • Get a datastream
  • Get all datastreams
  • Create a datastream
  • Update one or more datastreams
  • Delete a datastream
  • Pause a datastream
  • Resume a datastream
  • Pause source partitions
  • Resume source partitions
• Health Monitoring
  • Get status
  • Get all status

Datastream Data Model

Field Name	Description
name	Datastream name
connectorName	Connector to use for consuming data from source Name must match a connector in brooklin.server.connectorNames
transportProviderName	TransportProvider to use for producing data to destination Name must match a transport provider in brooklin.server.transportProviderNames
DatastreamStatus	Datastream status Can be one of "INITIALIZING" "READY" "PAUSED" "DELETING"
source.connectionString	Connection string representing source to consume from
source.partitions	Number of partitions of source data
destination.connectionString	Connection string representing destination to produce to
destination.partitions	Number of partitions of destination data
destination.keySerDe	Fully-qualified class name of serializer/deserializer Used with the key of destination data Must implement com.linkedin.datastream.serde.SerDe
destination.payloadSerDe	Same as destination.keySerDe Used with the payload of destination data
destination.envelopeSerDe	Same as destination.keySerDe Used with the envelope of destination data
metadata	Generic datastream metadata User-defined key/value pair Example usages: ownership, expiration …​ etc.

Datastream Management Service (DMS)

The definitive reference of this endpoints is:

• com.linkedin.datastream.DatastreamRestClient

• com.linkedin.datastream.server.dms.DatastreamResources

This endpoint offers the following operations on Datastreams:

Get a datastream

URL	GET /datastream/:datastreamName
URL Params	Required datastreamName=[string]
Response	Description Datastream metadata Content-Type application/json

Get all datastreams

URL	GET /datastream/
Response	Description List of datastream metadata Content-Type application/json

Create a datastream

URL	POST /datastream/
Request Body	Description Datastream metadata Content-Type application/json Definition { "name" : "[string]", "connectorName" : "[string]", "source": { "connectionString" : "[string]", "partitions": "[int]" }, "destination": { "connectionString": "[string]", "partitions": "[int]", "keySerDe": "[string]", "payloadSerDe": "[string]", "envelopeSerDe": "[string]" }, "metadata": { "owner": "[string]", "system.taskPrefix": "[string]" } } Required Fields name connectorName source.connectionString metadata.owner Constraints New datastream name cannot match that of an existing datastream
Success Response	201 Created

Update one or more datastreams

URL	PUT /datastream?ids=List(:datastreamName1,:datastreamName2,…​)
URL Params	Constraints At least one datastream has to be specified Specified datastreams must all exist All specified datastreams must belong to the same group (i.e. have identical metadata.system.taskPrefix values) All datastreams in the same group must be updated together (no partial group updates). In the case of updating multiple datastreams, they must all have identical source and destination properties. Updating any of the following fields of a datastream is prohibited: Connector Transport Provider Destination Status Updating any datastream is subject to the approval of its Connector (and is not enabled by default).
Request Body	Description List of datastream metadata Content-Type application/json Definition { "entities": { "datastreamName1": { "key1": "value1", "key2": "value2" }, "datastreamName2": { "key1": "value1", "key2": "value2" } } } Note Keys and values specified in the request body under entities.datastreamName1, entities.datastreamName2 …​ etc., have the same Datastream schema specified in Create a Datastream.
Success Response	200 OK

Delete a datastream

URL	DELETE /datastream/:datastreamName
URL Params	Required datastreamName=[string] Constraints Datastream must exist
Success Response	200 OK

Pause a datastream

URL	POST /datastream/:datastreamName?action=pause
URL Params	Required datastreamName=[string] Constraints Datastream must exist
Request Body	Description Flag indicating if the entire datastream group (i.e. all datastreams with the same system.taskPrefix metadata property value) should be paused (optional) Content-Type application/json Definition { "force" : "[boolean]" }
Success Response	200 OK

Resume a datastream

URL	POST /datastream/:datastreamName?action=resume
URL Params	Required datastreamName=[string] Constraints Datastream must exist
Request Body	Description Flag indicating if the entire datastream group (i.e. all datastreams with the same system.taskPrefix metadata property value) should be resumed (optional) Content-Type application/json Definition { "force" : "[boolean]" }
Success Response	200 OK

Pause source partitions

URL	POST datastream/:datastreamName?action=pauseSourcePartitions
URL Params	Required datastreamName=[string] Constraints Datastream must exist Datastream state must be READY Datastream must not be part of any datastream group Datastream Connector must support this type of update
Request Body	Description Content-Type application/json Definition { "sourcePartitions" : { "key1": "value1", "key2": "value2" } } Notes Keys and values under sourcePartitions are expected to be of type string Key should be datastream source names Value can either be: a comma-separated list of partitions (e.g. "0,4,6") an asterisk "*" (indicating all partitions)
Success Response	200 OK

Resume source partitions

URL	POST datastream/:datastreamName?action=resumeSourcePartitions
URL Params	Required datastreamName=[string] Constraints Datastream must exist Datastream state must be READY Datastream must not be part of any datastream group Datastream Connector must support this type of update
Request Body	Description Content-Type application/json Definition { "sourcePartitions" : { "key1": "value1", "key2": "value2" } } Notes Keys and values under sourcePartitions are expected to be of type string Key should be datastream source names Value can either be: a comma-separated list of partitions (e.g. "0,4,6") an asterisk "*" (indicating all partitions)
Success Response	200 OK

Health Monitoring

The definitive reference of this endpoints is com.linkedin.datastream.server.diagnostics.ServerComponentHealthResources.

This endpoints offers the following operations:

Get status

URL	GET /health/

Get all status