Skip to content

REST Endpoints

Ahmed Elbahtemy edited this page Apr 19, 2019 · 24 revisions

Contents

Data Models

Datastream

Field Name Type Description

name

string

Datastream name

connectorName

string

transportProviderName

string

Status

DatastreamStatus (enum)

  • Datastream status

  • Can be one of

    • "INITIALIZING"

    • "READY"

    • "PAUSED"

    • "DELETING"

source

DatastreamSource

Source from which the Connector consumes

destination

DatastreamDestination

Destination to which the Transport Provider will produce

metadata

map

  • Generic datastream metadata

  • User-defined key/value pair

  • Example usages: ownership, expiration …​ etc.

DatastreamSource

Field Name Type Description

connectionString

string

Connection string representing source to consume from

partitions

int

Number of partitions of source data

DatastreamDestination

Field Name Type Description

connectionString

string

Connection string representing destination to produce to

partitions

int

Number of partitions of destination data

keySerDe

string

payloadSerDe

string

  • Same as keySerDe

  • Used with the payload of destination data

envelopeSerDe

string

  • Same as keySerDe

  • Used with the envelope of destination data

ServerHealth

Field Name Type Description

clusterName

string

Brooklin cluster name

instanceName

string

Brooklin instance name

connectors

array of [Connector Health]

Connectors loaded in Brooklin instance

ConnectorHealth

Field Name Type Description

connectorName

string

Connector name

strategy

string

Name of AssignmentStrategy impl used by the Connector

tasks

array of [Task Health]

Tasks assigned to the Connector instance

TaskHealth

Field Name Type Description

name

string

Task name

datastreams

string

Comma-separated list of datastream names associated with the task

partitions

string

Array literal of partitions associated with the task (e.g. [0, 1, 5])

source

string

Datastream source connection string

destination

string

Datastream destination connection string

statusCode

string (enum)

  • Datastream task status code

  • Can be one of

    • "OK"

    • "ERROR"

    • "COMPLETE"

    • "PAUSED"

statusMessage

string

Datastream task status message

sourceCheckpoint

string

Datastream source checkpoint

Endpoints

Datastream Management Service (DMS)

Reference source files:

Get a datastream

URL

GET /datastream/:datastreamName

URL Params

Required

datastreamName=[string]

Response

Schema

Datastream

Content-Type

application/json

Get all datastreams

URL

GET /datastream/

Response

Schema

Array of Datastreams

Content-Type

application/json

Create a datastream

URL

POST /datastream/

Request Body

Description

Datastream metadata

Content-Type

application/json

Schema

Datastream

Required Fields
  • name

  • connectorName

  • source.connectionString

  • metadata.owner

Constraints
  • New datastream name cannot match

    that of an existing datastream

Success Response

201 Created

Update one or more datastreams

URL

PUT /datastream?ids=List(:datastreamName1,:datastreamName2,…​)

URL Params

Constraints
  • At least one datastream has to be specified

  • Specified datastreams must all exist

  • All specified datastreams must belong to the same group

    (i.e. have identical metadata.system.taskPrefix values)

  • All datastreams in the same group must

    be updated together (no partial group updates).

  • In the case of updating multiple datastreams, they must

    all have identical source and destination properties.

  • Updating any of the following fields of a datastream

    is prohibited:

    • connectorName

    • transportProviderName

    • destination

    • Status

  • Updating any datastream is subject to the approval

    of its Connector (and is not enabled by default).

Request Body

Description

Map of datastream name to Datastream

Content-Type

application/json

Schema
{
  "entities": {
    "datastreamName1": {

    },
    "datastreamName2": {

    }
  }
}
Note

Datastream names should have values

Success Response

200 OK

Delete a datastream

URL

DELETE /datastream/:datastreamName

URL Params

Required

datastreamName=[string]

Constraints
  • Datastream must exist

Success Response

200 OK

Pause a datastream

URL

POST /datastream/:datastreamName?action=pause

URL Params

Required

datastreamName=[string]

Constraints
  • Datastream must exist

Request Body

Description

Flag indicating if the entire datastream group

(i.e. all datastreams with the same system.taskPrefix

metadata property value) should be paused (optional)

Content-Type

application/json

Schema
{
  "force" : "[boolean]"
}

Success Response

200 OK

Resume a datastream

URL

POST /datastream/:datastreamName?action=resume

URL Params

Required

datastreamName=[string]

Constraints
  • Datastream must exist

Request Body

Description

Flag indicating if the entire datastream group

(i.e. all datastreams with the same system.taskPrefix

metadata property value) should be resumed (optional)

Content-Type

application/json

Schema
{
  "force" : "[boolean]"
}

Success Response

200 OK

Pause source partitions

URL

POST datastream/:datastreamName?action=pauseSourcePartitions

URL Params

Required

datastreamName=[string]

Constraints
  • Datastream must exist

  • Datastream Status must be READY

  • Datastream must not be part of any datastream group

  • Datastream Connector must support this type of update

Request Body

Description
Content-Type

application/json

Schema
{
  "sourcePartitions" : {
    "key1": "value1",
    "key2": "value2"
  }
}
Notes
  • Keys and values under sourcePartitions

    are expected to be of type string

  • Key should be datastream source names

  • Value can either be:

    • a comma-separated list of partitions (e.g. "0,4,6")

    • an asterisk "*" (indicating all partitions)

Success Response

200 OK

Resume source partitions

URL

POST datastream/:datastreamName?action=resumeSourcePartitions

URL Params

Required

datastreamName=[string]

Constraints
  • Datastream must exist

  • Datastream Status must be READY

  • Datastream must not be part of any datastream group

  • Datastream Connector must support this type of update

Request Body

Description
Content-Type

application/json

Schema
{
  "sourcePartitions" : {
    "key1": "value1",
    "key2": "value2"
  }
}
Notes
  • Keys and values under sourcePartitions

    are expected to be of type string

  • Key should be datastream source names

  • Value can either be:

    • a comma-separated list of partitions (e.g. "0,4,6")

    • an asterisk "*" (indicating all partitions)

Success Response

200 OK

Health Monitoring Endpoint

Get status

URL

GET /health/

Response

Description

Datastream metadata

Content-Type

application/json

Get all status