Query api #289

Koustavd18 · 2024-08-09T03:23:50Z

Update: Use of enhanced query API

using enhanced query api which will fetch the table headers along with data

src/@types/parseable/api/stream.ts

src/api/query.ts

src/hooks/useQueryLogs.ts

src/@types/parseable/api/stream.ts

src/api/constants.ts

balaji-jr · 2024-08-12T05:47:57Z

src/utils/URLParamsParser.ts

@@ -0,0 +1,6 @@
+export function paramsParser(queryParams: Record<string, string>) {
+	for (let i in queryParams) {


Try sending multiple items in the queryParams object and verify whether you get the expected result.

current implemention will take a singe param which will append to the url with "?" in front of the param. For multiple params we can chain them using '&'

balaji-jr · 2024-08-12T11:14:02Z

src/utils/URLParamsParser.ts

@@ -0,0 +1,14 @@
+export function paramsParser(queryParams: Record<string, string>) {
+	let parsedParams = '';


Use reduce then we do not have to append an extra '&' and rmv it

This function should accept null | undefined and return an empty string for invalid params so that we don't have to write conditional operators on each endpoint where we need a query string. Like this: https://github.com/parseablehq/console/pull/289/files#diff-dc3796761041251e6580415307413937f036d7143b52eb0a254085147ac758bcR8

The '?' itself should be emitted from this function.

If we make all these changes, the fn name is inappropriate. It should mention what it receives and what it returns.

balaji-jr · 2024-08-12T11:17:54Z

src/@types/parseable/api/stream.ts

@@ -7,6 +8,11 @@ export type LogStreamSchemaData = {
 	metadata: Record<string, string>;
 };

+export type LogStreamQueryWithFields = {


This is not a query. Rename the type.

balaji-jr · 2024-08-12T11:21:55Z

src/api/query.ts

 // to optimize performace, it has been decided to round off the time at the given level
 // so making the end-time inclusive
 const optimizeEndTime = (endTime: Date) => {
 	return dayjs(endTime).add(1, 'minute').toDate();
 };

-export const getQueryLogs = (logsQuery: QueryLogs) => {
+export const getQueryLogs = (logsQuery: QueryLogs, queryParams?: QueryParams) => {


this should always receive an object. Empty {} if no params.

balaji-jr · 2024-08-12T11:22:04Z

src/api/query.ts

@@ -33,11 +33,11 @@ export const getQueryLogs = (logsQuery: QueryLogs) => {
 	);
 };

-export const getQueryResult = (logsQuery: LogsQuery, query = '') => {
+export const getQueryResult = (logsQuery: LogsQuery, query = '', queryParams?: QueryParams) => {


this should always receive an object. Empty {} if no params.

balaji-jr · 2024-08-12T11:26:33Z

src/hooks/useQueryLogs.ts


-			const data = logsQueryRes.data;
+			const data = logsQueryRes.data.records;


There is a possibility where server could just send you {} or null.
Verify you always get a result that contains a records array even if there is no data.

balaji-jr · 2024-08-12T11:33:53Z

src/pages/Stream/utils.ts

+export const makeHeadersFromQueryFields = (queryResponse: LogStreamQueryWithFields | null): string[] => {
+	if (queryResponse) {
+		const { fields } = queryResponse;
+		return fields;


If its straight forward, why'd you need a separate function ?

balaji-jr · 2024-08-12T11:35:29Z

src/pages/Stream/providers/LogsProvider.tsx

+	setData: (
+		store: LogsStore,
+		data: Log[],
+		queryResponse: LogStreamQueryWithFields | null,


data: Log[], queryResponse: LogStreamQueryWithFields | null

With logs provider context, both deal with almost the same values. Rmv one with less significant.

balaji-jr · 2024-08-12T11:36:15Z

src/pages/Stream/providers/LogsProvider.tsx

@@ -272,7 +277,7 @@ type LogsStoreReducers = {
 	getUniqueValues: (data: Log[], key: string) => string[];
 	makeExportData: (data: Log[], headers: string[], type: string) => Log[];
 	setRetention: (store: LogsStore, retention: { description: string; duration: string }) => ReducerOutput;
-	setTableHeaders: (store: LogsStore, schema: LogStreamSchemaData) => ReducerOutput;
+	setTableHeaders: (store: LogsStore, queryResponse: LogStreamQueryWithFields) => ReducerOutput;


for setting table headers, why'd you need entire LogStreamQueryWithFields

balaji-jr · 2024-08-12T11:36:51Z

src/pages/Stream/providers/LogsProvider.tsx

 	const { data: existingData, custQuerySearchState, tableOpts } = store;
 	const { filteredData } = existingData;
 	const newHeaders =
 		filteredData && custQuerySearchState.isQuerySearchActive
 			? makeHeadersfromData(filteredData)
-			: makeHeadersFromSchema(schema);
+			: makeHeadersFromQueryFields(queryResponse);


It's straight forward. You do not need a function here.

balaji-jr · 2024-08-12T11:37:32Z

src/pages/Stream/providers/LogsProvider.tsx

@@ -563,7 +573,7 @@ const setData = (store: LogsStore, data: Log[], schema: LogStreamSchemaData | nu
 			: filterAndSortData(tableOpts, data);
 	const newPageSlice = filteredData && getPageSlice(currentPage, tableOpts.perPage, filteredData);
 	const newHeaders =
-		isQuerySearchActive && activeMode === 'sql' ? makeHeadersfromData(data) : makeHeadersFromSchema(schema);
+		isQuerySearchActive && activeMode === 'sql' ? makeHeadersfromData(data) : makeHeadersFromQueryFields(queryResponse);


why'd you want to make headers from data when
isQuerySearchActive && activeMode === 'sql'

balaji-jr · 2024-08-12T13:09:30Z

@Koustavd18
Also, we need a change to fetch the schema whenever we fetch logs.

balaji-jr · 2024-08-12T13:19:02Z

@Koustavd18
To determine the exact type definition for LogStreamQueryWithFields,
Try different scenarios (no-data, diff time range, non-existing streams, etc) and send multiple queries, and examine the results. This change is critical since a normal response without any params is an array but with fields=true it is an object.
This object may or may not contain fields and data keys.

Stating the above, it's better to have a common query function, that will always return an object {fields: string[], records: Log[] }. Just to prevent when we expect an array where its going to be an object response.

balaji-jr · 2024-08-13T20:17:30Z

This will be addressed in a separate pr.

Koustavd18 added 3 commits August 8, 2024 21:49

Enhanced Query Api

2196f64

Build issue fixed

2cabd50

Update: use of enhanced query api

60a309b

Koustavd18 requested review from balaji-jr, nitisht and nikhilsinhaparseable August 9, 2024 03:23

Koustavd18 added 2 commits August 9, 2024 13:22

Fixed type declaration

4954fc2

Refactor

e855aa8

balaji-jr reviewed Aug 10, 2024

View reviewed changes

src/@types/parseable/api/stream.ts Outdated Show resolved Hide resolved

balaji-jr suggested changes Aug 10, 2024

View reviewed changes

src/api/query.ts Outdated Show resolved Hide resolved

src/hooks/useQueryLogs.ts Outdated Show resolved Hide resolved

balaji-jr assigned Koustavd18 Aug 10, 2024

Comments resolved

d6b1a11

balaji-jr suggested changes Aug 10, 2024

View reviewed changes

src/@types/parseable/api/stream.ts Outdated Show resolved Hide resolved

Fixed Type declarations

de68e81

Koustavd18 requested a review from balaji-jr August 10, 2024 09:39

balaji-jr suggested changes Aug 10, 2024

View reviewed changes

src/api/constants.ts Outdated Show resolved Hide resolved

Koustavd18 added 5 commits August 10, 2024 15:39

Decoupled params parser to an independent function

1f0a8a5

Decoupled params parser to an independent function

322743e

Decoupled params parser to an independent function

4fe1672

Prettier formatted

e025d10

Changed LOG_QUERY_URL to a function

49b523f

nitisht closed this Aug 10, 2024

nitisht reopened this Aug 10, 2024

github-actions bot locked and limited conversation to collaborators Aug 10, 2024

Merge branch 'main' into queryApi

d67716d

Koustavd18 requested a review from balaji-jr August 12, 2024 05:37

balaji-jr suggested changes Aug 12, 2024

View reviewed changes

URL parser can parse to multiple query params

d2bf1c0

Koustavd18 requested a review from balaji-jr August 12, 2024 08:37

URL parser can parse to multiple query params

fc1ee1f

Merge branch 'main' into queryApi

76bef80

balaji-jr suggested changes Aug 12, 2024

View reviewed changes

balaji-jr closed this Aug 13, 2024

Koustavd18 deleted the queryApi branch August 19, 2024 05:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Query api #289

Query api #289

Koustavd18 commented Aug 9, 2024

balaji-jr Aug 12, 2024

Koustavd18 Aug 12, 2024

balaji-jr Aug 12, 2024 •

edited

Loading

balaji-jr Aug 12, 2024

balaji-jr Aug 12, 2024

balaji-jr Aug 12, 2024

balaji-jr Aug 12, 2024

balaji-jr Aug 12, 2024

balaji-jr Aug 12, 2024 •

edited

Loading

balaji-jr Aug 12, 2024

balaji-jr Aug 12, 2024

balaji-jr Aug 12, 2024

balaji-jr commented Aug 12, 2024

balaji-jr commented Aug 12, 2024

balaji-jr commented Aug 13, 2024

		@@ -0,0 +1,6 @@
		export function paramsParser(queryParams: Record<string, string>) {
		for (let i in queryParams) {

		@@ -0,0 +1,14 @@
		export function paramsParser(queryParams: Record<string, string>) {
		let parsedParams = '';


		const data = logsQueryRes.data;
		const data = logsQueryRes.data.records;

Query api #289

Query api #289

Conversation

Koustavd18 commented Aug 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

balaji-jr Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

balaji-jr Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

balaji-jr commented Aug 12, 2024

balaji-jr commented Aug 12, 2024

balaji-jr commented Aug 13, 2024

balaji-jr Aug 12, 2024 •

edited

Loading

balaji-jr Aug 12, 2024 •

edited

Loading