Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add create and drop materialized view SQL support #73

Merged

Conversation

dai-chen
Copy link
Collaborator

@dai-chen dai-chen commented Oct 16, 2023

Description

Add SQL support for CREATE / DROP MATERIALIZED VIEW. To avoid dependency on entire Spark grammar file and also support any language (either SQL or PPL), the grammar rule will accept any text in AS clause.

Documentation

Please see user manual for details: https://github.com/dai-chen/opensearch-spark/blob/add-mv-syntax-with-empty-impl/docs/index.md#materialized-view

Example

spark-sql> 
CREATE MATERIALIZED VIEW http_logs_metrics
AS
SELECT
  window.start AS startTime,
  COUNT(*) AS count
FROM ds_tables.http_logs
WHERE year = 1998 AND month = 6 AND day = 11 
  AND status BETWEEN 400 AND 599
GROUP BY TUMBLE(`@timestamp`, '1 Hour')
WITH (
  auto_refresh = true,
  checkpoint_location = "s3://test/"
);

DROP MATERIALIZED VIEW http_logs_metrics

OpenSearch Index:

GET flint_myglue_default_http_logs_metrics/_mapping
{
  "flint_myglue_default_http_logs_metrics": {
    "mappings": {
      "_meta": {
        "kind": "mv",
        "indexedColumns": [
          {
            "columnType": "timestamp",
            "columnName": "startTime"
          },
          {
            "columnType": "long",
            "columnName": "count"
          }
        ],
        "name": "myglue.default.http_logs_metrics",
        "options": {
          "auto_refresh": "true",
          "checkpoint_location": "s3://test"
        },
        "source": """SELECT
  window.start AS startTime,
  COUNT(*) AS count
FROM ds_tables.http_logs
WHERE year = 1998 AND month = 6 AND day = 11
  AND status BETWEEN 400 AND 599
GROUP BY TUMBLE(`@timestamp`, '1 Hour')""",
        "version": "0.1.0",
        "properties": {}
      },
      "properties": {
        "count": {
          "type": "long"
        },
        "startTime": {
          "type": "date",
          "format": "strict_date_optional_time_nanos"
        }
      }
    }
  }
}

GET flint_myglue_default_http_logs_metrics/_search
{
    ...
    "hits": [
      {
        "_source": {
          "startTime": "1998-06-11T12:00:00.000000+0000",
          "count": 14801
        }
      },
      {
        "_source": {
          "startTime": "1998-06-11T20:00:00.000000+0000",
          "count": 9490
        }
      },
      ...

Issues Resolved

#25

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dai-chen dai-chen self-assigned this Oct 16, 2023
@dai-chen dai-chen added the enhancement New feature or request label Oct 16, 2023
@dai-chen dai-chen changed the title Add create, drop and refresh materialized view SQL support Add create and drop materialized view SQL support Oct 18, 2023
@dai-chen dai-chen force-pushed the add-mv-syntax-with-empty-impl branch from c5efc62 to 90c6d3f Compare October 18, 2023 17:06
@dai-chen dai-chen merged commit ea012af into opensearch-project:main Oct 19, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants