Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve example docs around SQLExecuteQueryOperator in all SQLrelated providers #44807

Open
1 of 12 tasks
eladkal opened this issue Dec 10, 2024 · 4 comments
Open
1 of 12 tasks
Labels
area:providers good first issue kind:documentation kind:meta High-level information important to the community

Comments

@eladkal
Copy link
Contributor

eladkal commented Dec 10, 2024

Body

Context: #44707 (comment)

In Sqlite provider we have operators.rst which explains how to leverage SQLExecuteQueryOperator from common.sql provider to be used with Sqlite. We should have similar doc and examples for all SQL related operators that were removed as part of #44559

The Task:
verify that we have operators.rst in each of the above providers that explain how to use SQLExecuteQueryOperator with the specific provider See sqlite provider operators.rst for example template:

.. _howto/operator:SqliteOperator:

SQLExecuteQueryOperator to connect to Sqlite
============================================

Use the :class:`SQLExecuteQueryOperator<airflow.providers.common.sql.operators.sql>` to execute
Sqlite commands in a `Sqlite <https://sqlite.org/lang.html>`__ database.

.. note::
    Previously, ``SqliteOperator`` was used to perform this kind of operation. After deprecation this has been removed. Please use ``SQLExecuteQueryOperator`` instead.

Using the Operator
^^^^^^^^^^^^^^^^^^

Use the ``conn_id`` argument to connect to your Sqlite instance where
the connection metadata is structured as follows:

.. list-table:: Sqlite Airflow Connection Metadata
   :widths: 25 25
   :header-rows: 1

   * - Parameter
     - Input
   * - Host: string
     - Sqlite database file

An example usage of the SQLExecuteQueryOperator to connect to Sqlite is as follows:

.. exampleinclude:: /../../providers/tests/system/sqlite/example_sqlite.py
    :language: python
    :start-after: [START howto_operator_sqlite]
    :end-before: [END howto_operator_sqlite]

Furthermore, you can use an external file to execute the SQL commands. Script folder must be at the same level as DAG.py file.

.. exampleinclude:: /../../providers/tests/system/sqlite/example_sqlite.py
    :language: python
    :start-after: [START howto_operator_sqlite_external_file]
    :end-before: [END howto_operator_sqlite_external_file]

If possible accommodate for the specific parameters that can be passed from the provider to the hook.
For example in Postgres we can pass hook_params for enabling logging so we explain about this explicitly with example

Screenshot 2024-12-10 at 10 42 52

The items:

  • Exasol (Previously had ExasolOperator)
  • apache.drill (Previously had DrillOperator)
  • jdbc (Previously had JdbcOperator)
  • microsoft.mssql (Previously had MsSqlOperator)
  • mysql (Previously had MySQLOperator)
  • oracle (Previously had OracleOperator)
  • presto Previously had PrestoOperator)
  • snowflake (Previously had SnowflakeOperator)
  • trino (Previously had TrinoOperator)
  • vertica (Previously had VerticaOperator)
  • postgres (Previously had PostgresOperator) Docs already exists but requires tweaks to align the style

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.
@eladkal eladkal added area:providers good first issue kind:documentation kind:meta High-level information important to the community labels Dec 10, 2024
@geraj1010
Copy link
Contributor

To confirm, this is for all operators that were deprecated (and merged) in #44559? Are you looking for additional HOW-TO guides for the The Items?

@eladkal
Copy link
Contributor Author

eladkal commented Dec 11, 2024

To confirm, this is for all operators that were deprecated (and merged) in #44559? Are you looking for additional HOW-TO guides for the The Items?

Yes, get what we have for sqlite to the other providers. I assume it's mostly the same with difference of conn_id

@geraj1010
Copy link
Contributor

For Amazon, I did find mention of SQLExecuteQueryOperator for Redshift and Athena, but not RDS.

Redshift: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/redshift/redshift_sql.html
Athena: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/athena/athena_sql.html

Should I replace the existing files with operators.rst and create a new one for RDS (if needed)?

@eladkal
Copy link
Contributor Author

eladkal commented Dec 15, 2024

Amazon is not on the list because the docs there are OK (I looked only on Redshift)

If RDS can be read with this operator then you are welcome to add it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers good first issue kind:documentation kind:meta High-level information important to the community
Projects
None yet
Development

No branches or pull requests

2 participants