Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function 'LATEST_BY_OFFSET' does not accept parameters (TIMESTAMP) #8226

Closed
rmoff opened this issue Oct 5, 2021 · 1 comment
Closed

Function 'LATEST_BY_OFFSET' does not accept parameters (TIMESTAMP) #8226

rmoff opened this issue Oct 5, 2021 · 1 comment
Labels
bug needs-triage streaming-engine Tickets owned by the ksqlDB Streaming Team

Comments

@rmoff
Copy link
Member

rmoff commented Oct 5, 2021

tl;dr

Please make LATEST_BY_OFFSET support TIMESTAMP type.

Why?

STRINGTOTIMESTAMP is marked as deprecated in favour of PARSE_TIMESTAMP, but if I use PARSE_TIMESTAMP I get a TIMESTAMP type (as expected) which then does not work with existing LATEST_BY_OFFSET statements which did using the previous BIGINT type that STRINGTOTIMESTAMP emits.

>CREATE STREAM CARPARK_EVENTS WITH (VALUE_FORMAT='PROTOBUF') AS
>SELECT PARSE_TIMESTAMP(DATE + ' ' + TIME ,'yyyy-MM-dd HH:mm','Europe/London' ) AS TS,
>       *,
>       (CAST((CAPACITY - EMPTY_PLACES) AS DOUBLE) /
>        CAST(CAPACITY AS DOUBLE)) * 100 AS PCT_FULL,
>       STRUCT("lat" := LATITUDE, "lon":= LONGITUDE) AS "location",
>       'v2/Council/CarParkCurrent' as SOURCE
>  FROM CARPARK_SRC
>  EMIT CHANGES;

>CREATE TABLE CARPARK AS
>SELECT NAME,
>       FORMAT_TIMESTAMP(LATEST_BY_OFFSET(TS),
>                         'yyyy-MM-dd HH:mm:ss','Europe/London') AS LATEST_TS,
>       LATEST_BY_OFFSET(CAPACITY)      AS CAPACITY,
>       LATEST_BY_OFFSET(EMPTY_PLACES)  AS CURRENT_EMPTY_PLACES,
>       LATEST_BY_OFFSET(PCT_FULL)      AS CURRENT_PCT_FULL,
>       AVG(EMPTY_PLACES)               AS AVG_EMPTY_PLACES,
>       LATEST_BY_OFFSET(STATUS)        AS CURRENT_STATUS,
>       LATEST_BY_OFFSET(LATITUDE)      AS LATITUDE,
>       LATEST_BY_OFFSET(LONGITUDE)     AS LONGITUDE,
>       LATEST_BY_OFFSET(DIRECTIONSURL) AS DIRECTIONSURL
>    FROM CARPARK_EVENTS
>    GROUP BY NAME;
Successfully changed local property 'auto.offset.reset' from 'earliest' to 'earliest'.
Could not determine output schema for query due to error: Function 'LATEST_BY_OFFSET' does not accept parameters (TIMESTAMP).

Instead I have to revert to previous code

ksql> CREATE STREAM CARPARK_EVENTS WITH (VALUE_FORMAT='PROTOBUF') AS
>SELECT STRINGTOTIMESTAMP(DATE + ' ' + TIME ,'yyyy-MM-dd HH:mm','Europe/London' ) AS TS,
>       *,
>       (CAST((CAPACITY - EMPTY_PLACES) AS DOUBLE) /
>        CAST(CAPACITY AS DOUBLE)) * 100 AS PCT_FULL,
>       STRUCT("lat" := LATITUDE, "lon":= LONGITUDE) AS "location",
>       'v2/Council/CarParkCurrent' as SOURCE
>  FROM CARPARK_SRC
>  EMIT CHANGES;

>CREATE TABLE CARPARK AS
>SELECT NAME,
>       TIMESTAMPTOSTRING(LATEST_BY_OFFSET(TS),
>                         'yyyy-MM-dd HH:mm:ss','Europe/London') AS LATEST_TS,
>       LATEST_BY_OFFSET(CAPACITY)      AS CAPACITY,
>       LATEST_BY_OFFSET(EMPTY_PLACES)  AS CURRENT_EMPTY_PLACES,
>       LATEST_BY_OFFSET(PCT_FULL)      AS CURRENT_PCT_FULL,
>       AVG(EMPTY_PLACES)               AS AVG_EMPTY_PLACES,
>       LATEST_BY_OFFSET(STATUS)        AS CURRENT_STATUS,
>       LATEST_BY_OFFSET(LATITUDE)      AS LATITUDE,
>       LATEST_BY_OFFSET(LONGITUDE)     AS LONGITUDE,
>       LATEST_BY_OFFSET(DIRECTIONSURL) AS DIRECTIONSURL
>    FROM CARPARK_EVENTS2
>    GROUP BY NAME;
Successfully changed local property 'auto.offset.reset' from 'earliest' to 'earliest'.

 Message
--------------------------------------
 Created query with ID CTAS_CARPARK_7
--------------------------------------
@agavra agavra added the streaming-engine Tickets owned by the ksqlDB Streaming Team label Oct 8, 2021
@jzaralim
Copy link
Contributor

Duplicate of #7787

@jzaralim jzaralim marked this as a duplicate of #7787 Oct 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug needs-triage streaming-engine Tickets owned by the ksqlDB Streaming Team
Projects
None yet
Development

No branches or pull requests

3 participants