From b49ae829adcae1111c49ee3f725c47d8e6cb3edd Mon Sep 17 00:00:00 2001 From: Manjunath Davanam Date: Mon, 29 Jan 2024 14:57:12 +0530 Subject: [PATCH] Develop to 1.0.2-GA (#93) * Issue #4 feat: enhance the ingest event to add obsrv and source meta * issue#223: feat: Updated API endpoints with verbs * issue#223: fix: Test case modification * issue #84 : fix: API swagger doc update * #99 fix: upgrade packages to fix vulnerabilities * Command-service Vulnerabilities fixes (#56) * Release 1.3.0 into Main (#53) * issue#223: feat: Updated API endpoints with verbs * issue#223: fix: Test case modification * issue #84 : fix: API swagger doc update * #99 fix: upgrade packages to fix vulnerabilities --------- Co-authored-by: Jerald Co-authored-by: harishkumar gangula Co-authored-by: Praveen <66662436+pveleneni@users.noreply.github.com> * command-service vulnerabilities fixes --------- Co-authored-by: Manjunath Davanam Co-authored-by: Jerald Co-authored-by: harishkumar gangula Co-authored-by: Praveen <66662436+pveleneni@users.noreply.github.com> * #90 fix: Resolve API Issues (#52) * #90 fix: format error messages, code cleanup * #90 fix: validate extraction config during ingest * #90 fix: add test cases for extraction key validation * #90 fix: move error handler to helpers for standard handling * #90 fix: update telemetry audit event set function (#58) * Build and deployment (#57) * build api image * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * build api image * Update build_and_deploy.yaml * build api image * build api image * build api image * build and deploy api image * build and deploy api image * build and deploy api image * build and deploy of api service * build and deploy of api service * modify docker file * modify docker file * modify docker file * update build and deployment * update build and deployment * Update build_and_deploy.yaml * api service build and deployment * api service build and deployment * feat: obsrv api service build and deployment github actions configuration * #0 fix: update the tag condition in actions --------- Co-authored-by: ManojCKrishna Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintauri@users.noreply.github.com> Co-authored-by: Harish Kumar Gangula Co-authored-by: Manjunath Davanam Co-authored-by: Praveen <66662436+pveleneni@users.noreply.github.com> * #0 fix: add azure exhaust support (#44) * #0 fix: add azure exhaust support * #0 fix: update azure exhaust service without async * develop into release 1.3.0 (#59) * Issue #4 feat: enhance the ingest event to add obsrv and source meta * #90 fix: Resolve API Issues (#52) * #90 fix: format error messages, code cleanup * #90 fix: validate extraction config during ingest * #90 fix: add test cases for extraction key validation * #90 fix: move error handler to helpers for standard handling * #90 fix: update telemetry audit event set function (#58) * Build and deployment (#57) * build api image * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * build api image * Update build_and_deploy.yaml * build api image * build api image * build api image * build and deploy api image * build and deploy api image * build and deploy api image * build and deploy of api service * build and deploy of api service * modify docker file * modify docker file * modify docker file * update build and deployment * update build and deployment * Update build_and_deploy.yaml * api service build and deployment * api service build and deployment * feat: obsrv api service build and deployment github actions configuration * #0 fix: update the tag condition in actions --------- Co-authored-by: ManojCKrishna Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintauri@users.noreply.github.com> Co-authored-by: Harish Kumar Gangula Co-authored-by: Manjunath Davanam Co-authored-by: Praveen <66662436+pveleneni@users.noreply.github.com> * #0 fix: add azure exhaust support (#44) * #0 fix: add azure exhaust support * #0 fix: update azure exhaust service without async --------- Co-authored-by: shiva-rakshith Co-authored-by: Praveen Veleneni <66662436+pveleneni@users.noreply.github.com> Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintaluri@users.noreply.github.com> Co-authored-by: ManojCKrishna Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintauri@users.noreply.github.com> Co-authored-by: Harish Kumar Gangula Co-authored-by: Manjunath Davanam * #126 Feat: Add Querying on Aggregated datasources (#60) * #126 feat: enable querying on aggregated datasources * #126 feat: Fix validation issues and update routes for aggregate queries * #126 fix: remove unused methods * #126 fix: remove aggregates from endpoint * #126 fix: add validation for granularity options * #126 fix: Remove unused routes and validators * #126 feat: add new property to datasources structure and update rollup querying * #305 Feat: Add and Update dataset status (#62) * #305 feat: add retire apis and update dataset status * #305 fix: remove unused import * #305 fix: add command service api call to restart jobs * #305 fix: fix command service payload * #305 fix: update api doc * #305 fix: clear unused code * #305 fix: clear unused code * #305 fix: undo change submit ingestion * #305 fix: update test case to use enums * #305 fix: add test env file for test cases * #305 fix: add enum for string in test case * Issue #305 feat: exclude system-events data source in validation (#63) * Issue #165 feat: generate AUDIT events (#71) * Issue #fix-344 fix: defect fix (#76) * Issue #193 Feat: postman collection for Obsrv APIs (#77) * #198 feat: add validation for denorm out fields, to avoid duplicates (#78) * #198 feat: add validation for denorm out fields, to avoid duplicate out fields * #198 fix: update error message for denorm fields duplicate * #198 feat: add test cases for validation of denorm validation * #198 fix: remove console statement * Tenant ingestion (#81) * #0 feat: Add tenant_ingest API endpoint * #0 fix: Update API ID in RoutesConfig.ts * #0 fix: Update datasetId format in IngestorService.ts * #0 fix: Swap datasetId and tenantId in IngestorService.ts (#82) * Changed status enum according to latest workflow (#80) * Issue #193 Fix: changed status enum according to latest workflow * Issue #193 Fix: changed status enum according to latest workflow * Issue #0000: fix: update Dockerfile (#85) * issue #210 refactor: api endpoints refactored [removal of obsrv prefix] (#88) * issue #210 refactor: removal of obsrv prefix from API endpoints * issue #210 refactor: updated api endpoints postman and swagger documentation * issue #210 refactor: html doc for api endpoints --------- Co-authored-by: shiva-rakshith Co-authored-by: Jerald Co-authored-by: harishkumar gangula Co-authored-by: Praveen <66662436+pveleneni@users.noreply.github.com> Co-authored-by: Sowmya N Dixit Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintaluri@users.noreply.github.com> Co-authored-by: ManojCKrishna Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintauri@users.noreply.github.com> Co-authored-by: Ravi Mula Co-authored-by: Ravinder Kumar Co-authored-by: GayathriSrividya Co-authored-by: Shreyas Bhaktharam <121869503+shreyasb22@users.noreply.github.com> Co-authored-by: Jerald <127138957+JeraldJF@users.noreply.github.com> --- .../Obsrv API Service.postman_collection.json | 134 +- api-service/redoc-static.html | 538 -------- api-service/src/configs/RoutesConfig.ts | 90 +- api-service/swagger-doc/openapi.yaml | 34 +- api-service/swagger-doc/redoc-static.html | 1094 +++++++++-------- 5 files changed, 765 insertions(+), 1125 deletions(-) delete mode 100644 api-service/redoc-static.html diff --git a/api-service/postman-collection/Obsrv API Service.postman_collection.json b/api-service/postman-collection/Obsrv API Service.postman_collection.json index fb0c4a7f..56354c48 100644 --- a/api-service/postman-collection/Obsrv API Service.postman_collection.json +++ b/api-service/postman-collection/Obsrv API Service.postman_collection.json @@ -1,10 +1,10 @@ { "info": { - "_postman_id": "a3107544-5d3b-4af9-8002-74ce81357719", + "_postman_id": "eb1ecc94-340f-4cdb-a978-e52e1f454aab", "name": "Obsrv API Service", "schema": "https://schema.getpostman.com/json/collection/v2.0.0/collection.json", - "_exporter_id": "23908122", - "_collection_link": "https://restless-water-97719.postman.co/workspace/temp-workspace~5901a80f-1ada-4758-bf49-6f0fe63041c8/collection/23908122-a3107544-5d3b-4af9-8002-74ce81357719?action=share&source=collection_link&creator=23908122" + "_exporter_id": "26192103", + "_collection_link": "https://speeding-star-177775.postman.co/workspace/sanketika-obsrv~2ce96556-12e2-48bd-8e42-9c1dba428cc8/collection/26192103-eb1ecc94-340f-4cdb-a978-e52e1f454aab?action=share&source=collection_link&creator=26192103" }, "item": [ { @@ -27,7 +27,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/datasets/list" + "url": "{{HOST_IP}}/datasets/v1/list" }, "response": [] }, @@ -49,14 +49,13 @@ } }, "url": { - "raw": "{{HOST_IP}}/obsrv/v1/datasets/create?status=Live", + "raw": "{{HOST_IP}}/datasets/v1/create?status=Live", "host": [ "{{HOST_IP}}" ], "path": [ - "obsrv", - "v1", "datasets", + "v1", "create" ], "query": [ @@ -86,7 +85,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/datasets/create" + "url": "{{HOST_IP}}/datasets/v1/create" }, "response": [] }, @@ -107,7 +106,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/datasets/update" + "url": "{{HOST_IP}}/datasets/v1/update" }, "response": [] }, @@ -120,16 +119,15 @@ "method": "GET", "header": [], "url": { - "raw": "{{HOST_IP}}/obsrv/v1/datasets/get/?status=Live", + "raw": "{{HOST_IP}}/datasets/v1/get/dataset?status=Live", "host": [ "{{HOST_IP}}" ], "path": [ - "obsrv", - "v1", "datasets", + "v1", "get", - "" + "dataset" ], "query": [ { @@ -163,7 +161,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/datasources/list" + "url": "{{HOST_IP}}/datasources/v1/list" }, "response": [] }, @@ -184,7 +182,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/datasources/create" + "url": "{{HOST_IP}}/datasources/v1/create" }, "response": [] }, @@ -205,7 +203,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/datasources/update" + "url": "{{HOST_IP}}/datasources/v1/update" }, "response": [] }, @@ -218,14 +216,13 @@ "method": "GET", "header": [], "url": { - "raw": "{{HOST_IP}}/obsrv/v1/datasources/get/sb-telemetry_sb-telemetry?status=Live", + "raw": "{{HOST_IP}}/datasources/v1/get/sb-telemetry_sb-telemetry?status=Live", "host": [ "{{HOST_IP}}" ], "path": [ - "obsrv", - "v1", "datasources", + "v1", "get", "sb-telemetry_sb-telemetry" ], @@ -261,7 +258,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/datasets/source/config/list" + "url": "{{HOST_IP}}/datasets/v1/source/config/list" }, "response": [] }, @@ -286,14 +283,13 @@ } }, "url": { - "raw": "{{HOST_IP}}/obsrv/v1/datasets/source/config/get/6c3fc8c2-357d-489b-b0c9-afdde6e5c6c0?status=Live", + "raw": "{{HOST_IP}}/datasets/v1/source/config/get/6c3fc8c2-357d-489b-b0c9-afdde6e5c6c0?status=Live", "host": [ "{{HOST_IP}}" ], "path": [ - "obsrv", - "v1", "datasets", + "v1", "source", "config", "get", @@ -326,7 +322,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/datasets/source/config/create" + "url": "{{HOST_IP}}/datasets/v1/source/config/create" }, "response": [] }, @@ -347,7 +343,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/datasets/source/config/update" + "url": "{{HOST_IP}}/datasets/v1/source/config/update" }, "response": [] } @@ -357,7 +353,7 @@ "name": "Query (Data OUT APIs)", "item": [ { - "name": "native query request", + "name": "native query with params", "request": { "auth": { "type": "noauth" @@ -373,12 +369,12 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/data/query/dataset-1_dataset-1.1_DAY" + "url": "{{HOST_IP}}/data/v1/query/dataset-1_dataset-1.1_DAY" }, "response": [] }, { - "name": "sql query request", + "name": "sql query with params", "event": [ { "listen": "prerequest", @@ -405,7 +401,43 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/data/sql-query/dataset-1_dataset-1.1_DA" + "url": "{{HOST_IP}}/data/v1/sql-query/dataset-1_dataset-1.1_DA" + }, + "response": [] + }, + { + "name": "native query", + "request": { + "method": "POST", + "header": [ + { + "key": "Content-Type", + "value": "application/json" + } + ], + "body": { + "mode": "raw", + "raw": "{\n \"context\": {\n \"dataSource\": \"dataset-1\"\n },\n \"query\": {\n \"queryType\": \"timeseries\",\n \"dataSource\": \"dataset-1\",\n \"intervals\": {\n \"type\": \"intervals\",\n \"intervals\": [\n \"2023-06-15/2023-07-11\"\n ]\n },\n \"granularity\": \"week\",\n \"aggregations\": [\n {\n \"type\": \"count\",\n \"name\": \"total_records\"\n }\n ]\n }\n}" + }, + "url": "{{HOST_IP}}/data/v1/query" + }, + "response": [] + }, + { + "name": "sql-query", + "request": { + "method": "POST", + "header": [ + { + "key": "Content-Type", + "value": "application/json" + } + ], + "body": { + "mode": "raw", + "raw": "{\n \"context\": {\n \"dataSource\": \"dataset-1\"\n },\n \"querySql\": {\n \"query\": \"SELECT COUNT(*) AS totalRatingsCount FROM \\\"dataset-1\\\" LIMIT 100\"\n }\n}" + }, + "url": "{{HOST_IP}}/data/v1/sql-query" }, "response": [] } @@ -431,7 +463,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/data/create/sb-telemetry" + "url": "{{HOST_IP}}/data/v1/in/sb-telemetry" }, "response": [] }, @@ -452,7 +484,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/data/create/sb-telemetry" + "url": "{{HOST_IP}}/data/v1/in/sb-telemetry" }, "response": [] }, @@ -473,7 +505,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/data/create/sb-telemetry-user" + "url": "{{HOST_IP}}/data/v1/in/sb-telemetry-user" }, "response": [] }, @@ -491,7 +523,7 @@ } } }, - "url": "{{HOST_IP}}/obsrv/v1/data/submit/ingestion" + "url": "{{HOST_IP}}/data/v1/submit/ingestion" }, "response": [] } @@ -501,7 +533,7 @@ "name": "Exhaust APIs", "item": [ { - "name": "New Request", + "name": "Dataset exhaust", "request": { "method": "GET", "header": [ @@ -511,16 +543,15 @@ } ], "url": { - "raw": "{{HOST_IP}}/obsrv/v1/data/exhaust/sb-telemetry?from=2023-07-01&to=2023-07-31&type=transformed", + "raw": "{{HOST_IP}}/data/v1/exhaust/dataset-1?from=2023-07-01&to=2023-07-31&type=transformed", "host": [ "{{HOST_IP}}" ], "path": [ - "obsrv", - "v1", "data", + "v1", "exhaust", - "sb-telemetry" + "dataset-1" ], "query": [ { @@ -542,5 +573,32 @@ } ] } + ], + "event": [ + { + "listen": "prerequest", + "script": { + "type": "text/javascript", + "exec": [ + "" + ] + } + }, + { + "listen": "test", + "script": { + "type": "text/javascript", + "exec": [ + "" + ] + } + } + ], + "variable": [ + { + "key": "HOST_IP", + "value": "localhost:3005", + "type": "string" + } ] } \ No newline at end of file diff --git a/api-service/redoc-static.html b/api-service/redoc-static.html deleted file mode 100644 index 034bd56d..00000000 --- a/api-service/redoc-static.html +++ /dev/null @@ -1,538 +0,0 @@ - - - - - - Obsrv APIS - - - - - - - - - -
API docs by Redocly

Obsrv APIS (1.0.0)

Download OpenAPI specification:Download

Sunbird Obsrv: example@gmail.com URL: # License: All rights reserved

Obsrv is a set of APIs that provide access to a variety of data sources and datasets. These APIs can be used to query and analyze different types of events, as well as to manage data sources and datasets.

-

This OpenAPI document provides a detailed specification for the Obsrv APIs,including information about the endpoints, parameters, responses, and authentication requirements.

-

Obsrv Query APIs

The Query API provides the the ability to query various type of events from the underlying data sources.

-

Perform a native query to fetch data from analytical data source

This endpoint of the READ API provides a way to construction of the queries to read data such as a Native -JSON query. The READ API also allows filters on the date ranges within which the query will be applied to. -The context information in the READ API is required.

-
Request Body schema: application/json
object
query
object

Responses

Request samples

Content type
application/json
{
  • "context": {
    },
  • "query": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.native.query",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Perform a sql query to fetch data from analytical data source

This endpoint of the READ API provides a way to construction of the queries to read data such as a Native -JSON query. The READ API also allows filters on the date ranges within which the query will be applied to. -The context information in the READ API is optional.

-
Request Body schema: application/json
object
object

Responses

Request samples

Content type
application/json
{
  • "context": {
    },
  • "querySql": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.sql.query",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Obsrv Data IN API

Ingests data from the request body into a Kafka topic.

-

The API designed for ingesting data into Kafka.

"When processing batch events, the API expects the key to be labeled as 'events.' However, when dealing with individual events, the key should be identified as 'event.'"

-
path Parameters
datasetId
required
string
Request Body schema: application/json
data
object

Responses

Request samples

Content type
application/json
Example
{
  • "data": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.dataset.data.in",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

Obsrv Dataset APIs

The Dataset APIs provide a way to interact with a database and allow you to create, read, update records in the database, as well as list multiple records based on certain criteria.

-

creating datasets into postgres tables

This API allows you to create new datasets in the PostgreSQL database used by the analytical data source.

-
Request Body schema: application/json
id
string
dataset_id
string
name
string
type
string
Enum: "dataset" "master-dataset"
object
object
object
object
object
object
tags
Array of strings
status
string
Enum: "Live" "Retired"
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "name": "obsrv-telemetry-events",
  • "type": "dataset",
  • "router_config": {
    },
  • "tags": [ ],
  • "data_schema": {
    },
  • "status": "Live",
  • "published_date": "2023-03-24 12:19:32.091544"
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.save",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to update existing records in database

This API allows you to update existing datasets in the PostgreSQL database used by the analytical data source.

-
Request Body schema: application/json
id
string
dataset_id
string
name
string
type
string
Enum: "dataset" "master-dataset"
object
object
object
object
object
object
tags
Array of strings
status
string
Enum: "Live" "Retired"
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "type": "dataset",
  • "name": "obsrv-telemetry-events",
  • "router_config": {
    },
  • "denorm_config": {
    },
  • "dataset_config": {
    },
  • "extraction_config": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.update",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to read records from database

This API allows you to retrieve records from a dataset in the PostgreSQL. A successful response will return a status code of 200. If the dataset does not exist, a status code of 404 will be returned.

-
path Parameters
datasetId
required
string <uuid>

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.read",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to list records from database

This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results.

-
Request Body schema: application/json
filters
object

Responses

Request samples

Content type
application/json
{
  • "filters": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.list",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Obsrv Datasource APIs

The Datsources APIs provide a way to interact with a database and allow you to create, read, update records in the database, as well as list multiple records based on certain criteria.

-

creating datasources into postgres tables

This API allows you to create new datasource in the PostgreSQL database. While saving a datasource, it is required that the datasource field in ingestion spec must match with datasource_ref.

-
Request Body schema: application/json
id
string
dataset_id
string
ingestion_spec
object
datasource
string
datasource_ref
string
retention_period
object
archival_policy
object
purge_policy
object
backup_config
object
status
string
Enum: "Live" "Retired"
created_by
string
updated_by
string
published_date
string
metadata
object

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "ingestion_spec": {
    },
  • "datasource": "obsrv-telemetry-events",
  • "datasource_ref": "obsrv-telemetry-events",
  • "published_date": "2023-03-27T11:40:43.269Z",
  • "status": "Live",
  • "metadata": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.create",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to update existing records in database

This API allows you to update existing datasource in the PostgreSQL database. While saving a datasource, it is required that the datasource field in ingestion spec must match with datasource_ref.

-
Request Body schema: application/json
id
string
dataset_id
string
ingestion_spec
object
datasource
string
datasource_ref
string
retention_period
object
archival_policy
object
purge_policy
object
backup_config
object
status
string
Enum: "Live" "Retired"
created_by
string
updated_by
string
published_date
string
metadata
object

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "datasource": "obsrv-telemetry-events",
  • "backup_config": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.update",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to read records from database

This API allows you to retrieve records from a datasource in the PostgreSQL. A successful response will return a status code of 200. If the datasource does not exist, a status code of 404 will be returned

-
path Parameters
datasourceId
required
string <uuid>

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.read",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to list records from database

This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results.

-
Request Body schema: application/json
filters
object

Responses

Request samples

Content type
application/json
{
  • "filters": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.list",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

API to submit ingestion spec to analytical database

Request Body schema: application/json
object (SubmitIngestion)

Responses

Request samples

Content type
application/json
{
  • "type": "kafka",
  • "spec": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.submit.ingestion",
  • "ver": "v1",
  • "ts": 1693891173180,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

Obsrv Dataset Source Config APIs

The Dataset Source Config APIs provide a way to interact with a database and allow you to create, read, update records in the database, as well as list multiple records based on certain criteria.

-

creating datasets into postgres tables

This API allows you to create new datasets in the PostgreSQL database used by the analytical data source.

-
Request Body schema: application/json
id
string
dataset_id
string
connector_type
string
connector_config
object
status
string
connector_stats
object
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "connector_type": "kafka",
  • "connector_config": {
    },
  • "status": "Live",
  • "published_date": "2023-03-24 12:19:32.091544"
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.create",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to update existing records in database

This API allows you to update existing datasets in the PostgreSQL database used by the analytical data source.

-
Request Body schema: application/json
id
string
dataset_id
string
connector_type
string
connector_config
object
status
string
connector_stats
object
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "connector_type": "kafka",
  • "connector_config": {
    },
  • "status": "Live"
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.update",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to read records from database

This API allows you to retrieve records from a dataset in the PostgreSQL. A successful response will return a status code of 200. If the dataset does not exist, a status code of 404 will be returned.

-
path Parameters
datasetId
required
string

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.read",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to list records from database

This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results.

-
Request Body schema: application/json
filters
object

Responses

Request samples

Content type
application/json
{
  • "filters": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.list",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Obsrv Data Exhaust APIs

Data Exhaust APIs are used for accessing Backup files from Cloud providers

-

API to access backup files from Cloud providers

path Parameters
datasetId
required
string
query Parameters
required
object or object (DataExhaust)
Example: from=2023-06-01&to=2023-06-30&type=transformed

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.dataset.data.exhaust",
  • "ver": "v1",
  • "ts": 1693891173180,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}
- - - - diff --git a/api-service/src/configs/RoutesConfig.ts b/api-service/src/configs/RoutesConfig.ts index 60e1e27b..da7632f1 100644 --- a/api-service/src/configs/RoutesConfig.ts +++ b/api-service/src/configs/RoutesConfig.ts @@ -1,119 +1,119 @@ export const routesConfig = { default: { - api_id: "obsrv.api", + api_id: "api", validation_schema: null, }, query: { native_query: { - api_id: "obsrv.native.query", + api_id: "native.query", method: "post", - path: "/obsrv/v1/data/query", + path: "/data/v1/query", validation_schema: "QueryRequest.json", }, native_query_with_params: { - api_id: "obsrv.native.query", + api_id: "native.query", method: "post", - path: "/obsrv/v1/data/query/:datasetId", + path: "/data/v1/query/:datasetId", validation_schema: "QueryRequest.json", }, sql_query: { - api_id: "obsrv.sql.query", + api_id: "sql.query", method: "post", - path: "/obsrv/v1/data/sql-query", + path: "/data/v1/sql-query", validation_schema: "QueryRequest.json", }, sql_query_with_params: { - api_id: "obsrv.sql.query", + api_id: "sql.query", method: "post", - path: "/obsrv/v1/data/sql-query/:datasetId", + path: "/data/v1/sql-query/:datasetId", validation_schema: "QueryRequest.json", }, }, config: { dataset: { save: { - api_id: "obsrv.config.dataset.create", + api_id: "config.dataset.create", method: "post", - path: "/obsrv/v1/datasets/create", + path: "/datasets/v1/create", validation_schema: "DatasetCreateReq.json", }, read: { - api_id: "obsrv.config.dataset.read", + api_id: "config.dataset.read", method: "get", - path: "/obsrv/v1/datasets/get/:datasetId", + path: "/datasets/v1/get/:datasetId", validation_schema: null, }, update: { - api_id: "obsrv.config.dataset.update", + api_id: "config.dataset.update", method: "patch", - path: "/obsrv/v1/datasets/update", + path: "/datasets/v1/update", validation_schema: "DatasetUpdateReq.json", }, list: { - api_id: "obsrv.config.dataset.list", + api_id: "config.dataset.list", method: "post", - path: "/obsrv/v1/datasets/list", + path: "/datasets/v1/list", validation_schema: "DatasetListReq.json", }, }, datasource: { save: { - api_id: "obsrv.config.datasource.create", + api_id: "config.datasource.create", method: "post", - path: "/obsrv/v1/datasources/create", + path: "/datasources/v1/create", validation_schema: "DatasourceSaveReq.json", }, read: { - api_id: "obsrv.config.datasource.read", + api_id: "config.datasource.read", method: "get", - path: "/obsrv/v1/datasources/get/:datasourceId", + path: "/datasources/v1/get/:datasourceId", validation_schema: null, }, update: { - api_id: "obsrv.config.datasource.update", + api_id: "config.datasource.update", method: "patch", - path: "/obsrv/v1/datasources/update", + path: "/datasources/v1/update", validation_schema: "DatasourceUpdateReq.json", }, list: { - api_id: "obsrv.config.datasource.list", + api_id: "config.datasource.list", method: "post", - path: "/obsrv/v1/datasources/list", + path: "/datasources/v1/list", validation_schema: "DatasetListReq.json", }, }, dataset_source_config: { save: { - api_id: "obsrv.config.dataset.source.config.create", + api_id: "config.dataset.source.config.create", method: "post", - path: "/obsrv/v1/datasets/source/config/create", + path: "/datasets/v1/source/config/create", validation_schema: "DatasetSourceConfigSaveReq.json", }, read: { - api_id: "obsrv.config.dataset.source.config.read", + api_id: "config.dataset.source.config.read", method: "get", - path: "/obsrv/v1/datasets/source/config/get/:datasetId", + path: "/datasets/v1/source/config/get/:datasetId", validation_schema: null, }, update: { - api_id: "obsrv.config.dataset.source.config.update", + api_id: "config.dataset.source.config.update", method: "patch", - path: "/obsrv/v1/datasets/source/config/update", + path: "/datasets/v1/source/config/update", validation_schema: "DatasetSourceConfigUpdateReq.json", }, list: { - api_id: "obsrv.config.dataset.source.config.list", + api_id: "config.dataset.source.config.list", method: "post", - path: "/obsrv/v1/datasets/source/config/list", + path: "/datasets/v1/source/config/list", validation_schema: "DatasetListReq.json", }, } }, data_ingest: { - api_id: "obsrv.dataset.data.in", + api_id: "dataset.data.in", method: "post", - path: "/obsrv/v1/data/create/:datasetId", + path: "/data/v1/in/:datasetId", validation_schema: "DataIngestionReq.json", }, tenant_ingest: { @@ -123,9 +123,9 @@ export const routesConfig = { validation_schema: "DataIngestionReq.json", }, exhaust: { - api_id: "obsrv.dataset.data.exhaust", + api_id: "dataset.data.exhaust", method: "get", - path: "/obsrv/v1/data/exhaust/:datasetId", + path: "/data/v1/exhaust/:datasetId", validation_schema: "DataExhaustReq.json" }, prometheus: { @@ -134,34 +134,34 @@ export const routesConfig = { validation_schema: null, }, submit_ingestion: { - api_id: "obsrv.submit.ingestion", + api_id: "submit.ingestion", method: "post", - path: "/obsrv/v1/data/submit/ingestion", + path: "/data/v1/submit/ingestion", validation_schema: "SubmitIngestionReq.json" }, query_wrapper: { sql_wrapper: { - api_id: "obsrv.query.wrapper.sql.query", + api_id: "query.wrapper.sql.query", method: "post", - path: "/obsrv/v1/sql", + path: "/v1/sql", }, native_post: { - api_id: "obsrv.query.wrapper.native.post", + api_id: "query.wrapper.native.post", method: "post", path: /\/druid\/v2.*/, }, native_get: { - api_id: "obsrv.query.wrapper.native.get", + api_id: "query.wrapper.native.get", method: "get", path: /\/druid\/v2.*/ }, native_delete: { - api_id: "obsrv.query.wrapper.native.delete", + api_id: "query.wrapper.native.delete", method: "delete", path: "/druid/v2/:queryId" }, druid_status: { - api_id: "obsrv.query.wrapper.status", + api_id: "query.wrapper.status", method: "get", path: "/status" } diff --git a/api-service/swagger-doc/openapi.yaml b/api-service/swagger-doc/openapi.yaml index fd5b5c23..d3dc82fa 100644 --- a/api-service/swagger-doc/openapi.yaml +++ b/api-service/swagger-doc/openapi.yaml @@ -4424,7 +4424,7 @@ components: type: object paths: - /obsrv/v1/data/query: + /data/v1/query: post: summary: Perform a native query to fetch data from analytical data source description: | @@ -4447,7 +4447,7 @@ paths: 400: $ref: "#/components/responses/BadRequestResponseSchema" - /obsrv/v1/data/sql-query: + /data/v1/sql-query: post: summary: Perform a sql query to fetch data from analytical data source description: | @@ -4470,7 +4470,7 @@ paths: 400: $ref: "#/components/responses/BadRequestResponseSchemaSql" - /obsrv/v1/data/create/{datasetId}: + /data/v1/in/{datasetId}: post: summary: The API designed for ingesting data into Kafka. description: | @@ -4501,7 +4501,7 @@ paths: 404: $ref: "#/components/responses/DataIngestNotFound" - /obsrv/v1/datasets/create: + /datasets/v1/create: post: summary: creating datasets into postgres tables description: This API allows you to create new datasets in the PostgreSQL database used by the analytical data source. @@ -4523,7 +4523,7 @@ paths: 409: $ref: "#/components/responses/DatasetCreateDuplicate" - /obsrv/v1/datasets/update: + /datasets/v1/update: patch: summary: API to update existing records in database description: This API allows you to update existing datasets in the PostgreSQL database used by the analytical data source. @@ -4543,7 +4543,7 @@ paths: 400: $ref: "#/components/responses/DatasetUpdateBadRequest" - /obsrv/v1/datasets/get/{datasetId}: + /datasets/v1/get/{datasetId}: get: summary: API to read records from database description: This API allows you to retrieve records from a dataset in the PostgreSQL. A successful response will return a status code of 200. If the dataset does not exist, a status code of 404 will be returned. @@ -4561,7 +4561,7 @@ paths: $ref: "#/components/responses/DatasetReadResponse" 404: $ref: "#/components/responses/DatasetReadNotFound" - /obsrv/v1/datasets/list: + /datasets/v1/list: post: summary: API to list records from database description: This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results. @@ -4581,7 +4581,7 @@ paths: 400: $ref: "#/components/responses/DatasetListBadRequest" - /obsrv/v1/datasources/create: + /datasources/v1/create: post: summary: creating datasources into postgres tables description: This API allows you to create new datasource in the PostgreSQL database. While saving a datasource, it is required that the datasource field in ingestion spec must match with datasource_ref. @@ -4603,7 +4603,7 @@ paths: 409: $ref: "#/components/responses/DatasourceCreateDuplicate" - /obsrv/v1/datasources/update: + /datasources/v1/update: patch: summary: API to update existing records in database description: This API allows you to update existing datasource in the PostgreSQL database. While saving a datasource, it is required that the datasource field in ingestion spec must match with datasource_ref. @@ -4622,7 +4622,7 @@ paths: $ref: "#/components/responses/DatasourceUpdateResponseSchema" 400: $ref: "#/components/responses/DatasourceUpdateBadRequest" - /obsrv/v1/datasources/get/{datasourceId}: + /datasources/v1/get/{datasourceId}: get: summary: API to read records from database description: This API allows you to retrieve records from a datasource in the PostgreSQL. A successful response will return a status code of 200. If the datasource does not exist, a status code of 404 will be returned @@ -4639,7 +4639,7 @@ paths: $ref: "#/components/responses/DatasourceReadResponse" 404: $ref: "#/components/responses/DatasourceReadNotFound" - /obsrv/v1/datasources/list: + /datasources/v1/list: post: summary: API to list records from database description: This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results. @@ -4658,7 +4658,7 @@ paths: $ref: "#/components/responses/DatasourceListResponse" 400: $ref: "#/components/responses/DatasourceListBadRequest" - /obsrv/v1/datasets/source/config/create: + /datasets/v1/source/config/create: post: summary: creating datasets into postgres tables description: This API allows you to create new datasets in the PostgreSQL database used by the analytical data source. @@ -4680,7 +4680,7 @@ paths: 409: $ref: "#/components/responses/DatasetSourceConfigCreateDuplicate" - /obsrv/v1/datasets/source/config/update: + /datasets/v1/source/config/update: patch: summary: API to update existing records in database description: This API allows you to update existing datasets in the PostgreSQL database used by the analytical data source. @@ -4699,7 +4699,7 @@ paths: $ref: "#/components/responses/DatasetSourceConfigUpdateResponseSchema" 400: $ref: "#/components/responses/DatasetSourceConfigUpdateBadRequest" - /obsrv/v1/datasets/source/config/get/{datasetId}: + /datasets/v1/source/config/get/{datasetId}: get: summary: API to read records from database description: This API allows you to retrieve records from a dataset in the PostgreSQL. A successful response will return a status code of 200. If the dataset does not exist, a status code of 404 will be returned. @@ -4717,7 +4717,7 @@ paths: $ref: "#/components/responses/DatasetSourceConfigReadResponse" 404: $ref: "#/components/responses/DatasetSoureConfigReadNotFound" - /obsrv/v1/datasets/source/config/list: + /datasets/v1/source/config/list: post: summary: API to list records from database description: This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results. @@ -4736,7 +4736,7 @@ paths: $ref: "#/components/responses/DatasetSoureConfigListResponse" 400: $ref: "#/components/responses/DatasetSourceConfigListBadRequest" - /obsrv/v1/data/submit/ingestion: + /data/v1/submit/ingestion: post: summary: API to submit ingestion spec to analytical database tags: [Obsrv Datasource APIs] @@ -4755,7 +4755,7 @@ paths: 400: $ref: "#/components/responses/SubmitIngestionBadRequestResponseSchema" - /obsrv/v1/data/exhaust/{datasetId}: + /data/v1/exhaust/{datasetId}: get: summary: API to access backup files from Cloud providers tags: [Obsrv Data Exhaust APIs] diff --git a/api-service/swagger-doc/redoc-static.html b/api-service/swagger-doc/redoc-static.html index d50c4c3a..2debfc2e 100644 --- a/api-service/swagger-doc/redoc-static.html +++ b/api-service/swagger-doc/redoc-static.html @@ -12,369 +12,356 @@ margin: 0; } - -
API docs by Redocly
API docs by Redocly

Obsrv APIS (1.0.0)

Download OpenAPI specification:Download

Sunbird Obsrv: example@gmail.com URL: # License: All rights reserved

Obsrv is a set of APIs that provide access to a variety of data sources and datasets. These APIs can be used to query and analyze different types of events, as well as to manage data sources and datasets.

+ " fill="currentColor">

Obsrv APIS (1.0.0)

Download OpenAPI specification:Download

Sunbird Obsrv: example@gmail.com URL: # License: All rights reserved

Obsrv is a set of APIs that provide access to a variety of data sources and datasets. These APIs can be used to query and analyze different types of events, as well as to manage data sources and datasets.

This OpenAPI document provides a detailed specification for the Obsrv APIs,including information about the endpoints, parameters, responses, and authentication requirements.

-

Obsrv Query APIs

The Query API provides the the ability to query various type of events from the underlying data sources.

-

Perform a native query to fetch data from analytical data source

This endpoint of the READ API provides a way to construction of the queries to read data such as a Native +

Obsrv Query APIs

The Query API provides the the ability to query various type of events from the underlying data sources.

+

Perform a native query to fetch data from analytical data source

This endpoint of the READ API provides a way to construction of the queries to read data such as a Native JSON query. The READ API also allows filters on the date ranges within which the query will be applied to. The context information in the READ API is required.

-
Request Body schema: application/json
object
query
object

Responses

Request samples

Content type
application/json
{
  • "context": {
    },
  • "query": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.native.query",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Perform a sql query to fetch data from analytical data source

This endpoint of the READ API provides a way to construction of the queries to read data such as a Native +

Request Body schema: application/json
required
object
query
object

Responses

Request samples

Content type
application/json
{
  • "context": {
    },
  • "query": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.native.query",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Perform a sql query to fetch data from analytical data source

This endpoint of the READ API provides a way to construction of the queries to read data such as a Native JSON query. The READ API also allows filters on the date ranges within which the query will be applied to. The context information in the READ API is optional.

-
Request Body schema: application/json
object
object

Responses

Request samples

Content type
application/json
{
  • "context": {
    },
  • "querySql": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.sql.query",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Obsrv Data IN API

Ingests data from the request body into a Kafka topic.

-

The API designed for ingesting data into Kafka.

"When processing batch events, the API expects the key to be labeled as 'events.' However, when dealing with individual events, the key should be identified as 'event.'"

-
path Parameters
datasetId
required
string
Request Body schema: application/json
data
object

Responses

Request samples

Content type
application/json
Example
{
  • "data": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.dataset.data.in",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

Obsrv Dataset APIs

The Dataset APIs provide a way to interact with a database and allow you to create, read, update records in the database, as well as list multiple records based on certain criteria.

-

creating datasets into postgres tables

This API allows you to create new datasets in the PostgreSQL database used by the analytical data source.

-
Request Body schema: application/json
id
string
dataset_id
string
name
string
type
string
Enum: "dataset" "master-dataset"
object
object
object
object
object
object
tags
Array of strings
status
string
Enum: "ACTIVE" "DISABLED"
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "name": "obsrv-telemetry-events",
  • "type": "dataset",
  • "router_config": {
    },
  • "tags": [ ],
  • "data_schema": {
    },
  • "status": "ACTIVE",
  • "published_date": "2023-03-24 12:19:32.091544"
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.save",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to update existing records in database

This API allows you to update existing datasets in the PostgreSQL database used by the analytical data source.

-
Request Body schema: application/json
id
string
dataset_id
string
name
string
type
string
Enum: "dataset" "master-dataset"
object
object
object
object
object
object
tags
Array of strings
status
string
Enum: "ACTIVE" "DISABLED"
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "type": "dataset",
  • "name": "obsrv-telemetry-events",
  • "router_config": {
    },
  • "denorm_config": {
    },
  • "dataset_config": {
    },
  • "extraction_config": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.update",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to read records from database

This API allows you to retrieve records from a dataset in the PostgreSQL. A successful response will return a status code of 200. If the dataset does not exist, a status code of 404 will be returned.

-
path Parameters
datasetId
required
string <uuid>

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.read",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to list records from database

This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results.

-
Request Body schema: application/json
filters
object

Responses

Request samples

Content type
application/json
{
  • "filters": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.list",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Obsrv Datasource APIs

The Datsources APIs provide a way to interact with a database and allow you to create, read, update records in the database, as well as list multiple records based on certain criteria.

-

creating datasources into postgres tables

This API allows you to create new datasource in the PostgreSQL database. While saving a datasource, it is required that the datasource field in ingestion spec must match with datasource_ref.

-
Request Body schema: application/json
id
string
dataset_id
string
ingestion_spec
object
datasource
string
datasource_ref
string
retention_period
object
archival_policy
object
purge_policy
object
backup_config
object
status
string
Enum: "ACTIVE" "DISABLED"
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "ingestion_spec": {
    },
  • "datasource": "obsrv-telemetry-events",
  • "datasource_ref": "obsrv-telemetry-events",
  • "published_date": "2023-03-27T11:40:43.269Z",
  • "status": "ACTIVE"
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.create",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to update existing records in database

This API allows you to update existing datasource in the PostgreSQL database. While saving a datasource, it is required that the datasource field in ingestion spec must match with datasource_ref.

-
Request Body schema: application/json
id
string
dataset_id
string
ingestion_spec
object
datasource
string
datasource_ref
string
retention_period
object
archival_policy
object
purge_policy
object
backup_config
object
status
string
Enum: "ACTIVE" "DISABLED"
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "datasource": "obsrv-telemetry-events",
  • "backup_config": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.update",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to read records from database

This API allows you to retrieve records from a datasource in the PostgreSQL. A successful response will return a status code of 200. If the datasource does not exist, a status code of 404 will be returned

-
path Parameters
datasourceId
required
string <uuid>

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.read",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to list records from database

This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results.

-
Request Body schema: application/json
filters
object

Responses

Request samples

Content type
application/json
{
  • "filters": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.list",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

API to submit ingestion spec to analytical database

Request Body schema: application/json
object (SubmitIngestion)

Responses

Request samples

Content type
application/json
{
  • "type": "kafka",
  • "spec": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.submit.ingestion",
  • "ver": "v1",
  • "ts": 1693891173180,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

Obsrv Dataset Source Config APIs

The Dataset Source Config APIs provide a way to interact with a database and allow you to create, read, update records in the database, as well as list multiple records based on certain criteria.

-

creating datasets into postgres tables

This API allows you to create new datasets in the PostgreSQL database used by the analytical data source.

-
Request Body schema: application/json
id
string
dataset_id
string
connector_type
string
connector_config
object
status
string
connector_stats
object
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "connector_type": "kafka",
  • "connector_config": {
    },
  • "status": "ACTIVE",
  • "published_date": "2023-03-24 12:19:32.091544"
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.create",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to update existing records in database

This API allows you to update existing datasets in the PostgreSQL database used by the analytical data source.

-
Request Body schema: application/json
id
string
dataset_id
string
connector_type
string
connector_config
object
status
string
connector_stats
object
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "connector_type": "kafka",
  • "connector_config": {
    },
  • "status": "ACTIVE"
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.update",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to read records from database

This API allows you to retrieve records from a dataset in the PostgreSQL. A successful response will return a status code of 200. If the dataset does not exist, a status code of 404 will be returned.

-
path Parameters
datasetId
required
string

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.read",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to list records from database

This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results.

-
Request Body schema: application/json
filters
object

Responses

Request samples

Content type
application/json
{
  • "filters": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.list",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Obsrv Data Exhaust APIs

Data Exhaust APIs are used for accessing Backup files from Cloud providers

-

API to access backup files from Cloud providers

path Parameters
datasetId
required
string
query Parameters
required
object or object (DataExhaust)
Example: from=2023-06-01&to=2023-06-30&type=transformed

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.dataset.data.exhaust",
  • "ver": "v1",
  • "ts": 1693891173180,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}
+
Request Body schema: application/json
required
object
object

Responses

Request samples

Content type
application/json
{
  • "context": {
    },
  • "querySql": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.sql.query",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Obsrv Data IN API

Ingests data from the request body into a Kafka topic.

+

The API designed for ingesting data into Kafka.

"When processing batch events, the API expects the key to be labeled as 'events.' However, when dealing with individual events, the key should be identified as 'event.'"

+
path Parameters
datasetId
required
string
Request Body schema: application/json
required
data
object

Responses

Request samples

Content type
application/json
Example
{
  • "data": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.dataset.data.in",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

Obsrv Dataset APIs

The Dataset APIs provide a way to interact with a database and allow you to create, read, update records in the database, as well as list multiple records based on certain criteria.

+

creating datasets into postgres tables

This API allows you to create new datasets in the PostgreSQL database used by the analytical data source.

+
Request Body schema: application/json
required
id
string
dataset_id
string
name
string
type
string
Enum: "dataset" "master-dataset"
object
object
object
object
object
object
tags
Array of strings
status
string
Enum: "Live" "Retired"
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "name": "obsrv-telemetry-events",
  • "type": "dataset",
  • "router_config": {
    },
  • "tags": [ ],
  • "data_schema": {
    },
  • "status": "Live",
  • "published_date": "2023-03-24 12:19:32.091544"
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.save",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to update existing records in database

This API allows you to update existing datasets in the PostgreSQL database used by the analytical data source.

+
Request Body schema: application/json
required
id
string
dataset_id
string
name
string
type
string
Enum: "dataset" "master-dataset"
object
object
object
object
object
object
tags
Array of strings
status
string
Enum: "Live" "Retired"
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "type": "dataset",
  • "name": "obsrv-telemetry-events",
  • "router_config": {
    },
  • "denorm_config": {
    },
  • "dataset_config": {
    },
  • "extraction_config": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.update",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to read records from database

This API allows you to retrieve records from a dataset in the PostgreSQL. A successful response will return a status code of 200. If the dataset does not exist, a status code of 404 will be returned.

+
path Parameters
datasetId
required
string <uuid>

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.read",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to list records from database

This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results.

+
Request Body schema: application/json
required
filters
object

Responses

Request samples

Content type
application/json
{
  • "filters": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.list",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Obsrv Datasource APIs

The Datsources APIs provide a way to interact with a database and allow you to create, read, update records in the database, as well as list multiple records based on certain criteria.

+

creating datasources into postgres tables

This API allows you to create new datasource in the PostgreSQL database. While saving a datasource, it is required that the datasource field in ingestion spec must match with datasource_ref.

+
Request Body schema: application/json
required
id
string
dataset_id
string
ingestion_spec
object
datasource
string
datasource_ref
string
retention_period
object
archival_policy
object
purge_policy
object
backup_config
object
status
string
Enum: "Live" "Retired"
created_by
string
updated_by
string
published_date
string
metadata
object

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "ingestion_spec": {
    },
  • "datasource": "obsrv-telemetry-events",
  • "datasource_ref": "obsrv-telemetry-events",
  • "published_date": "2023-03-27T11:40:43.269Z",
  • "status": "Live",
  • "metadata": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.create",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to update existing records in database

This API allows you to update existing datasource in the PostgreSQL database. While saving a datasource, it is required that the datasource field in ingestion spec must match with datasource_ref.

+
Request Body schema: application/json
required
id
string
dataset_id
string
ingestion_spec
object
datasource
string
datasource_ref
string
retention_period
object
archival_policy
object
purge_policy
object
backup_config
object
status
string
Enum: "Live" "Retired"
created_by
string
updated_by
string
published_date
string
metadata
object

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "datasource": "obsrv-telemetry-events",
  • "backup_config": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.update",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to read records from database

This API allows you to retrieve records from a datasource in the PostgreSQL. A successful response will return a status code of 200. If the datasource does not exist, a status code of 404 will be returned

+
path Parameters
datasourceId
required
string <uuid>

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.read",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to list records from database

This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results.

+
Request Body schema: application/json
required
filters
object

Responses

Request samples

Content type
application/json
{
  • "filters": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.datasource.list",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

API to submit ingestion spec to analytical database

Request Body schema: application/json
required
object (SubmitIngestion)

Responses

Request samples

Content type
application/json
{
  • "type": "kafka",
  • "spec": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.submit.ingestion",
  • "ver": "v1",
  • "ts": 1693891173180,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

Obsrv Dataset Source Config APIs

The Dataset Source Config APIs provide a way to interact with a database and allow you to create, read, update records in the database, as well as list multiple records based on certain criteria.

+

creating datasets into postgres tables

This API allows you to create new datasets in the PostgreSQL database used by the analytical data source.

+
Request Body schema: application/json
required
id
string
dataset_id
string
connector_type
string
connector_config
object
status
string
connector_stats
object
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "connector_type": "kafka",
  • "connector_config": {
    },
  • "status": "Live",
  • "published_date": "2023-03-24 12:19:32.091544"
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.create",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to update existing records in database

This API allows you to update existing datasets in the PostgreSQL database used by the analytical data source.

+
Request Body schema: application/json
required
id
string
dataset_id
string
connector_type
string
connector_config
object
status
string
connector_stats
object
created_by
string
updated_by
string
published_date
string

Responses

Request samples

Content type
application/json
{
  • "dataset_id": "obsrv-telemetry-events",
  • "connector_type": "kafka",
  • "connector_config": {
    },
  • "status": "Live"
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.update",
  • "ver": "v1",
  • "ts": 1679916559382,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to read records from database

This API allows you to retrieve records from a dataset in the PostgreSQL. A successful response will return a status code of 200. If the dataset does not exist, a status code of 404 will be returned.

+
path Parameters
datasetId
required
string

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.read",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}

API to list records from database

This API allows you to retrieve a list of records from a dataset in the PostgreSQL database used by the analytical data source. You can specify filters to narrow down the results.

+
Request Body schema: application/json
required
filters
object

Responses

Request samples

Content type
application/json
{
  • "filters": {
    }
}

Response samples

Content type
application/json
{
  • "id": "obsrv.config.dataset.source.config.list",
  • "ver": "v1",
  • "ts": 1672679516875,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": [
    ]
}

Obsrv Data Exhaust APIs

Data Exhaust APIs are used for accessing Backup files from Cloud providers

+

API to access backup files from Cloud providers

path Parameters
datasetId
required
string
query Parameters
required
object or object (DataExhaust)
Example: from=2023-06-01&to=2023-06-30&type=transformed

Responses

Response samples

Content type
application/json
{
  • "id": "obsrv.dataset.data.exhaust",
  • "ver": "v1",
  • "ts": 1693891173180,
  • "params": {
    },
  • "responseCode": "OK",
  • "result": {
    }
}