REST Endpoints

1. Datastream Management Service (DMS)

This endpoint offers the following operations on Datastreams:

1.1. Get all datastreams

URL	GET /datastream/

1.2. Get a datastream

URL	GET /datastream/:datastreamName
URL Params	Required datastreamName=[string]

1.3. Create a datastream

URL

POST /datastream/

Request Body

Description Datastream metadata Content-Type application/json Example { "name" : "[string]", "connectorName" : "[string]", "source": { "connectionString" : "[string]", "partitions": "[int]" }, "destination": { "connectionString": "[string]", "partitions": "[int]", "keySerDe": "[string]", "payloadSerDe": "[string]", "envelopeSerDe": "[string]" }, "metadata": { "owner": "[string]", "system.taskPrefix": "[string]" } } Required Fields name connectorName source.connectionString metadata.owner Constraints New datastream name cannot match that of an existing datastream

Field Description

Field Name	Description
name	Datastream name
connectorName	Connector to use for consuming data from source Name must match a connector in brooklin.server.connectorNames
transportProviderName	TransportProvider to use for producing data to destination Name must match a transport provider in brooklin.server.transportProviderNames
source.connectionString	Connection string representing source to consume from
source.partitions	Number of partitions of source data
destination.connectionString	Connection string representing destination to produce to
destination.partitions	Number of partitions of destination data
destination.keySerDe	Fully-qualified class name of serializer/deserializer Used with the key of destination data Must implement com.linkedin.datastream.serde.SerDe
destination.payloadSerDe	Same as destination.keySerDe Used with the payload of destination data
destination.envelopeSerDe	Same as destination.keySerDe Used with the envelope of destination data
metadata	Generic datastream metadata User-defined key/value pair Example usages: ownership, expiration …​ etc.

1.4. Update one or more datastreams

URL	PUT /datastream?ids=List(:datastreamName1,:datastreamName2,…​)
URL Params	Constraints At least one datastream has to be specified Specified datastreams must all exist All specified datastreams must belong to the same group (i.e. have identical metadata.system.taskPrefix values) All datastreams in the same group must be updated together (no partial group updates). In the case of updating multiple datastreams, they must all have identical source and destination properties. Updating any of the following fields of a datastream is prohibited: Connector Transport Provider Destination Status
Request Body	Description List of datastream metadata Content-Type application/json Example { "entities": { "datastreamName1": { "key1": "value1", "key2": "value2" }, "datastreamName2": { "key1": "value1", "key2": "value2" } } } Note Keys and values specified in the request body under entities.datastreamName1, entities.datastreamName2 …​ etc., have the same Datastream schema specified in Create a Datastream.

1.5. Delete a datastream

URL	DELETE /datastream/:datastreamName
URL Params	Required datastreamName=[string] Constraints Datastream must exist