-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add cancel and status queries to server-side async execution #192
Changes from all commits
cfd856b
321b021
370e5cf
6ad3be4
1bea472
bf7f1f1
4196204
7ff71a7
0bace5b
c69f585
15df8cc
bec0804
37f2373
cb04815
fc7d116
ec0fd55
f7b9d09
d2ae886
f77ba35
8cae1d2
95f5e4d
9d1f1ff
442c420
e76a545
224f7c4
9d20bba
f403673
16ed40d
b58f881
8753a99
3d1c5b2
1abaaa4
9d4ffe8
c9e8947
2ba105f
52bd72c
22e4540
3320495
aef7dff
65b61da
3a4655d
ba6fd52
e9c70f8
7671506
c3a02af
2bc20db
d482caf
04a9e2f
cc749a3
3650755
3a33790
e373cf0
18d8fbe
36d0bbe
50b89f9
66604c0
bdb1063
4888d3f
a37d250
6939e28
634541d
3278e94
f7dc846
4072793
8830361
df3709d
0e2c553
7a798be
b428ba9
3a98c18
e9f2fc4
98c3252
c840433
58807a2
99540ed
d8d6d8f
0ab70cc
3704b9a
18d7c83
8d4ce91
c93bc08
ef3b668
7ad25c9
b5efbf4
143b0bd
3090dc1
b24a263
923d931
316f50a
ad18500
51fb204
0e816a7
5398f77
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,23 +3,23 @@ | |
Connecting and running queries | ||
############################### | ||
|
||
This topic provides a walkthrough and examples for how to use the Firebolt Python SDK to connect to Firebolt resources to run commands and query data. | ||
This topic provides a walkthrough and examples for how to use the Firebolt Python SDK to connect to Firebolt resources to run commands and query data. | ||
|
||
|
||
Setting up a connection | ||
========================= | ||
|
||
To connect to a Firebolt database to run queries or command, you must provide your account credentials through a connection request. | ||
To connect to a Firebolt database to run queries or command, you must provide your account credentials through a connection request. | ||
|
||
To get started, follow the steps below: | ||
To get started, follow the steps below: | ||
|
||
**1. Import modules** | ||
|
||
The Firebolt Python SDK requires you to import the following modules before making any command or query requests to your Firebolt database. | ||
The Firebolt Python SDK requires you to import the following modules before making any command or query requests to your Firebolt database. | ||
|
||
.. _required_connection_imports: | ||
|
||
:: | ||
:: | ||
|
||
from firebolt.db import connect | ||
from firebolt.client import DEFAULT_API_URL | ||
|
@@ -30,9 +30,9 @@ To get started, follow the steps below: | |
**2. Connect to your database and engine** | ||
|
||
|
||
Your account information can be provided as parameters in a ``connection()`` function. | ||
Your account information can be provided as parameters in a ``connection()`` function. | ||
|
||
A connection requires the following parameters: | ||
A connection requires the following parameters: | ||
|
||
+------------------------------------+-------------------------------------------------------------------+ | ||
| ``username`` | The email address associated with your Firebolt user. | | ||
|
@@ -50,9 +50,9 @@ To get started, follow the steps below: | |
|
||
* **Set credentials manually** | ||
|
||
You can manually include your account information in a connection object in your code for any queries you want to request. | ||
You can manually include your account information in a connection object in your code for any queries you want to request. | ||
|
||
Replace the values in the example code below with your Firebolt account credentials as appropriate. | ||
Replace the values in the example code below with your Firebolt account credentials as appropriate. | ||
|
||
:: | ||
|
||
|
@@ -61,19 +61,19 @@ To get started, follow the steps below: | |
engine_name = "your_engine" | ||
database_name = "your_database" | ||
|
||
connection = connect( | ||
connection = connect( | ||
engine_name=engine_name, | ||
database=database_name, | ||
username=username, | ||
password=password, | ||
) | ||
|
||
cursor = connection.cursor() | ||
|
||
|
||
* **Use an .env file** | ||
|
||
Consolidating all of your Firebolt credentials into a ``.env`` file can help protect sensitive information from exposure. Create an ``.env`` file with the following key-value pairs, and replace the values with your information. | ||
Consolidating all of your Firebolt credentials into a ``.env`` file can help protect sensitive information from exposure. Create an ``.env`` file with the following key-value pairs, and replace the values with your information. | ||
|
||
:: | ||
|
||
|
@@ -82,9 +82,9 @@ To get started, follow the steps below: | |
FIREBOLT_ENGINE="your_engine" | ||
FIREBOLT_DB="your_database" | ||
|
||
Be sure to place this ``.env`` file into your root directory. | ||
Be sure to place this ``.env`` file into your root directory. | ||
|
||
Your connection script can load these environmental variables from the ``.env`` file by using the `python-dotenv <https://pypi.org/project/python-dotenv/>`_ package. Note that the example below imports the ``os`` and ``dotenv`` modules in order to load the environmental variables. | ||
Your connection script can load these environmental variables from the ``.env`` file by using the `python-dotenv <https://pypi.org/project/python-dotenv/>`_ package. Note that the example below imports the ``os`` and ``dotenv`` modules in order to load the environmental variables. | ||
|
||
:: | ||
|
||
|
@@ -105,35 +105,35 @@ To get started, follow the steps below: | |
|
||
**3. Execute commands using the cursor** | ||
|
||
The ``cursor`` object can be used to send queries and commands to your Firebolt database and engine. See below for examples of functions using the ``cursor`` object. | ||
The ``cursor`` object can be used to send queries and commands to your Firebolt database and engine. See below for examples of functions using the ``cursor`` object. | ||
|
||
Command and query examples | ||
Server-side synchronous command and query examples | ||
============================ | ||
|
||
This section includes Python examples of various SQL commands and queries. | ||
This section includes Python examples of various SQL commands and queries. | ||
|
||
|
||
Inserting and selecting data | ||
----------------------------- | ||
|
||
.. _basic_execute_example: | ||
|
||
The example below uses ``cursor`` to create a new table called ``test_table``, insert rows into it, and then select the table's contents. | ||
The example below uses ``cursor`` to create a new table called ``test_table``, insert rows into it, and then select the table's contents. | ||
|
||
The engine attached to your specified database must be started before executing any queries. For help, see :ref:`starting an engine`. | ||
The engine attached to your specified database must be started before executing any queries. For help, see :ref:`starting an engine`. | ||
|
||
:: | ||
|
||
cursor.execute( | ||
'''CREATE FACT TABLE IF NOT EXISTS test_table ( | ||
id INT, | ||
name TEXT | ||
) | ||
id INT, | ||
name TEXT | ||
) | ||
PRIMARY INDEX id;''' | ||
) | ||
|
||
cursor.execute( | ||
'''INSERT INTO test_table VALUES | ||
'''INSERT INTO test_table VALUES | ||
(1, 'hello'), | ||
(2, 'world'), | ||
(3, '!');''' | ||
|
@@ -145,23 +145,23 @@ The engine attached to your specified database must be started before executing | |
|
||
cursor.close() | ||
|
||
.. note:: | ||
.. note:: | ||
|
||
For reference documentation on ``cursor`` functions, see :ref:`Db.cursor` | ||
For reference documentation on ``cursor`` functions, see :ref:`Db.cursor` | ||
|
||
|
||
Fetching query results | ||
----------------------- | ||
|
||
After running a query, you can fetch the results using a ``cursor`` object. The examples below use the data queried from ``test_table`` created in the :ref:`Inserting and selecting data`. | ||
After running a query, you can fetch the results using a ``cursor`` object. The examples below use the data queried from ``test_table`` created in the :ref:`Inserting and selecting data`. | ||
|
||
.. _fetch_example: | ||
|
||
:: | ||
|
||
print(cursor.fetchone()) | ||
|
||
**Returns**: ``[2, 'world']`` | ||
**Returns**: ``[2, 'world']`` | ||
|
||
:: | ||
|
||
|
@@ -181,25 +181,25 @@ Executing parameterized queries | |
|
||
.. _parameterized_query_execute_example: | ||
|
||
Parameterized queries (also known as “prepared statements”) format a SQL query with placeholders and then pass values into those placeholders when the query is run. This protects against SQL injection attacks and also helps manage dynamic queries that are likely to change, such as filter UIs or access control. | ||
Parameterized queries (also known as “prepared statements”) format a SQL query with placeholders and then pass values into those placeholders when the query is run. This protects against SQL injection attacks and also helps manage dynamic queries that are likely to change, such as filter UIs or access control. | ||
|
||
To run a parameterized query, use the ``execute()`` cursor method. Add placeholders to your statement using question marks ``?``, and in the second argument pass a tuple of parameters equal in length to the number of ``?`` in the statement. | ||
|
||
|
||
:: | ||
:: | ||
|
||
cursor.execute( | ||
'''CREATE FACT TABLE IF NOT EXISTS test_table2 ( | ||
id INT, | ||
name TEXT, | ||
name TEXT, | ||
date_value DATE | ||
) | ||
PRIMARY INDEX id;''' | ||
) | ||
|
||
|
||
:: | ||
|
||
cursor.execute( | ||
"INSERT INTO test_table2 VALUES (?, ?, ?)", | ||
(1, "apple", "2018-01-01"), | ||
|
@@ -216,8 +216,8 @@ If you need to run the same statement multiple times with different parameter in | |
cursor.executemany( | ||
"INSERT INTO test_table2 VALUES (?, ?, ?)", | ||
( | ||
(2, "banana", "2019-01-01"), | ||
(3, "carrot", "2020-01-01"), | ||
(2, "banana", "2019-01-01"), | ||
(3, "carrot", "2020-01-01"), | ||
(4, "donut", "2021-01-01") | ||
) | ||
) | ||
|
@@ -231,7 +231,7 @@ Executing multiple-statement queries | |
|
||
Multiple-statement queries allow you to run a series of SQL statements sequentially with just one method call. Statements are separated using a semicolon ``;``, similar to making SQL statements in the Firebolt UI. | ||
|
||
:: | ||
:: | ||
|
||
cursor.execute( | ||
""" | ||
|
@@ -246,32 +246,110 @@ Multiple-statement queries allow you to run a series of SQL statements sequentia | |
|
||
cursor.close() | ||
|
||
**Returns**: | ||
**Returns**: | ||
|
||
:: | ||
:: | ||
|
||
First query: [[2, 'banana', datetime.date(2019, 1, 1)], [3, 'carrot', datetime.date(2020, 1, 1)], [1, 'apple', datetime.date(2018, 1, 1)]] | ||
Second query: [[3, 'carrot', datetime.date(2020, 1, 1)], [4, 'donut', datetime.date(2021, 1, 1)]] | ||
|
||
.. note:: | ||
.. note:: | ||
|
||
Multiple statement queries are not able to use placeholder values for parameterized queries. | ||
|
||
|
||
|
||
Server-side asynchronous query execution | ||
========================================== | ||
|
||
Server-side asynchronous query execution allows you to run a long query in the background while executing other asynchronous or synchronous queries. An additional benefit of server-side async execution that can free up a connection, close a connection while running a query, or potentially even spin down an entire service (such as AWS Lambda) while a long-running database job is still underway. Note that it is not possible to retrieve the results of a server-side asynchronous query, so these queries are best used for running DMLs and DDLs. SELECTs should be used only for warming the cache. | ||
|
||
Running DDL commands | ||
----------------------------- | ||
|
||
.. _basic_execute_example: | ||
|
||
Running queries server-side asynchronously is similar to running server-side asynchronous queries, but the ``execute()`` command receives an extra parameter, ``async_execution=True``. The example below uses ``cursor`` to create a new table called ``test_table``. ``execute(query, async_execution=True)`` will return a query ID, which can subsequently be used to check the query status. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you mean that it's similar to client-side async? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @ericf-firebolt I see you merged the PR, but what about this comment above? |
||
|
||
:: | ||
|
||
query_id = cursor.execute( | ||
'''CREATE FACT TABLE IF NOT EXISTS test_table ( | ||
id INT, | ||
name TEXT | ||
) | ||
PRIMARY INDEX id;''', | ||
async_execution=True | ||
) | ||
|
||
|
||
To check the status of a query, send the query ID to ```get_status()``` to receive a QueryStatus enumeration object. Possible statuses are: | ||
|
||
|
||
* ``RUNNING`` | ||
* ``ENDED_SUCCESSFULLY`` | ||
* ``ENDED_UNSUCCESSFULLY`` | ||
* ``NOT_READY`` | ||
* ``STARTED_EXECUTION`` | ||
* ``PARSE_ERROR`` | ||
* ``CANCELED_EXECUTION`` | ||
* ``EXECUTION_ERROR`` | ||
|
||
|
||
Once the status of the table creation is ``ENDED_SUCCESSFULLY`` created, data can be inserted into it: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. delete "created" in this sentence? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @ericf-firebolt I see you merged the PR, but what about this comment above? |
||
|
||
:: | ||
|
||
from firebolt.async_db.cursor import QueryStatus | ||
|
||
query_status = cursor.get_status(query_id) | ||
|
||
if query_status == QueryStatus.ENDED_SUCCESSFULLY: | ||
cursor.execute( | ||
'''INSERT INTO test_table VALUES | ||
(1, 'hello'), | ||
(2, 'world'), | ||
(3, '!');''' | ||
) | ||
|
||
|
||
In addition, server-side asynchronous queries can be cancelled calling ``cancel()``. | ||
|
||
:: | ||
|
||
query_id = cursor.execute( | ||
'''CREATE FACT TABLE IF NOT EXISTS test_table ( | ||
id INT, | ||
name TEXT | ||
) | ||
PRIMARY INDEX id;''', | ||
async_execution=True | ||
) | ||
|
||
cursor.cancel(query_id) | ||
|
||
query_status = cursor.get_status(query_id) | ||
|
||
print(query_status) | ||
|
||
**Returns**: ``CANCELED_EXECUTION`` | ||
|
||
Multiple statement queries are not able to use placeholder values for parameterized queries. | ||
|
||
|
||
Using DATE and DATETIME values | ||
--------------------------------- | ||
============================== | ||
|
||
DATE, DATETIME and TIMESTAMP values used in SQL insertion statements must be provided in a specific format; otherwise they could be read incorrectly. | ||
DATE, DATETIME and TIMESTAMP values used in SQL insertion statements must be provided in a specific format; otherwise they could be read incorrectly. | ||
|
||
* DATE values should be formatted as **YYYY-MM-DD** | ||
* DATE values should be formatted as **YYYY-MM-DD** | ||
|
||
* DATETIME and TIMESTAMP values should be formatted as **YYYY-MM-DD HH:MM:SS.SSSSSS** | ||
|
||
The `datetime <https://docs.python.org/3/library/datetime.html>`_ module from the Python standard library contains various classes and methods to format DATE, TIMESTAMP and DATETIME data types. | ||
The `datetime <https://docs.python.org/3/library/datetime.html>`_ module from the Python standard library contains various classes and methods to format DATE, TIMESTAMP and DATETIME data types. | ||
|
||
You can import this module as follows: | ||
You can import this module as follows: | ||
|
||
:: | ||
:: | ||
|
||
from datetime import datetime | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think about adding a brief paragraph here explaining the differences between server-side async vs. client-side async?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ericf-firebolt I see you merged the PR, but what about this comment above?