[Format][Flight SQL] Add Flight SQL Command for Bulk Ingestion #38255
Comments
kou
added a commit
to kou/arrow-flight-sql-postgresql
that referenced
this issue
Nov 16, 2023
Closes apacheGH-169 Apache Arrow Flight SQL is slow because the current implementation inserts records one by one. We will be able to improve performance by using a new bulk ingestion Flight SQL command. See also: apache/arrow#38255
kou
added a commit
to kou/arrow-flight-sql-postgresql
that referenced
this issue
Nov 16, 2023
Closes apacheGH-169 Apache Arrow Flight SQL is slow because the current implementation inserts records one by one. We will be able to improve performance by using a new bulk ingestion Flight SQL command. See also: apache/arrow#38255
kou
added a commit
to apache/arrow-flight-sql-postgresql
that referenced
this issue
Nov 17, 2023
Closes GH-169 Apache Arrow Flight SQL is slow because the current implementation inserts records one by one. We will be able to improve performance by using a new bulk ingestion Flight SQL command. See also: apache/arrow#38255
zeroshade
pushed a commit
that referenced
this issue
Apr 17, 2024
### Rationale for this change It was suggested in the discussion around apache/arrow-adbc#1107 for the Flight SQL ADBC driver that an "Ingest" command would be a helpful addition to the Flight SQL specification. This command would enable a Flight SQL client to provide a FlightData stream to the server without needing to know its SQL syntax, and have that stream loaded into a target table by whichever means the server deems appropriate. ### What changes are included in this PR? - Format: - Add CommandStatementIngest message type to Flight SQL proto definition - Add FLIGHT_SQL_SERVER_BULK_INGESTION and FLIGHT_SQL_SERVER_INGEST_TRANSACTIONS_SUPPORTED options for SqlInfo - Go: - Generate pb - Server-side implementation - Client-side implementation - Unit + integration tests - C++: - Server-side implementation - Client-side implementation - Integration tests ### Are these changes tested? Yes, see `server_test.go`, `scenario.go`, and `test_integration.cc`. ### Are there any user-facing changes? Yes, new Flight SQL client and server functionality is being added. Changes are not expected to break existing users. * Closes: #38255 Lead-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Signed-off-by: Matt Topol <[email protected]>
tolleybot
pushed a commit
to tmct/arrow
that referenced
this issue
May 2, 2024
…#38385) ### Rationale for this change It was suggested in the discussion around apache/arrow-adbc#1107 for the Flight SQL ADBC driver that an "Ingest" command would be a helpful addition to the Flight SQL specification. This command would enable a Flight SQL client to provide a FlightData stream to the server without needing to know its SQL syntax, and have that stream loaded into a target table by whichever means the server deems appropriate. ### What changes are included in this PR? - Format: - Add CommandStatementIngest message type to Flight SQL proto definition - Add FLIGHT_SQL_SERVER_BULK_INGESTION and FLIGHT_SQL_SERVER_INGEST_TRANSACTIONS_SUPPORTED options for SqlInfo - Go: - Generate pb - Server-side implementation - Client-side implementation - Unit + integration tests - C++: - Server-side implementation - Client-side implementation - Integration tests ### Are these changes tested? Yes, see `server_test.go`, `scenario.go`, and `test_integration.cc`. ### Are there any user-facing changes? Yes, new Flight SQL client and server functionality is being added. Changes are not expected to break existing users. * Closes: apache#38255 Lead-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Signed-off-by: Matt Topol <[email protected]>
tolleybot
pushed a commit
to tmct/arrow
that referenced
this issue
May 4, 2024
…#38385) ### Rationale for this change It was suggested in the discussion around apache/arrow-adbc#1107 for the Flight SQL ADBC driver that an "Ingest" command would be a helpful addition to the Flight SQL specification. This command would enable a Flight SQL client to provide a FlightData stream to the server without needing to know its SQL syntax, and have that stream loaded into a target table by whichever means the server deems appropriate. ### What changes are included in this PR? - Format: - Add CommandStatementIngest message type to Flight SQL proto definition - Add FLIGHT_SQL_SERVER_BULK_INGESTION and FLIGHT_SQL_SERVER_INGEST_TRANSACTIONS_SUPPORTED options for SqlInfo - Go: - Generate pb - Server-side implementation - Client-side implementation - Unit + integration tests - C++: - Server-side implementation - Client-side implementation - Integration tests ### Are these changes tested? Yes, see `server_test.go`, `scenario.go`, and `test_integration.cc`. ### Are there any user-facing changes? Yes, new Flight SQL client and server functionality is being added. Changes are not expected to break existing users. * Closes: apache#38255 Lead-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Signed-off-by: Matt Topol <[email protected]>
rok
pushed a commit
to tmct/arrow
that referenced
this issue
May 8, 2024
…#38385) ### Rationale for this change It was suggested in the discussion around apache/arrow-adbc#1107 for the Flight SQL ADBC driver that an "Ingest" command would be a helpful addition to the Flight SQL specification. This command would enable a Flight SQL client to provide a FlightData stream to the server without needing to know its SQL syntax, and have that stream loaded into a target table by whichever means the server deems appropriate. ### What changes are included in this PR? - Format: - Add CommandStatementIngest message type to Flight SQL proto definition - Add FLIGHT_SQL_SERVER_BULK_INGESTION and FLIGHT_SQL_SERVER_INGEST_TRANSACTIONS_SUPPORTED options for SqlInfo - Go: - Generate pb - Server-side implementation - Client-side implementation - Unit + integration tests - C++: - Server-side implementation - Client-side implementation - Integration tests ### Are these changes tested? Yes, see `server_test.go`, `scenario.go`, and `test_integration.cc`. ### Are there any user-facing changes? Yes, new Flight SQL client and server functionality is being added. Changes are not expected to break existing users. * Closes: apache#38255 Lead-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Signed-off-by: Matt Topol <[email protected]>
rok
pushed a commit
to tmct/arrow
that referenced
this issue
May 8, 2024
…#38385) ### Rationale for this change It was suggested in the discussion around apache/arrow-adbc#1107 for the Flight SQL ADBC driver that an "Ingest" command would be a helpful addition to the Flight SQL specification. This command would enable a Flight SQL client to provide a FlightData stream to the server without needing to know its SQL syntax, and have that stream loaded into a target table by whichever means the server deems appropriate. ### What changes are included in this PR? - Format: - Add CommandStatementIngest message type to Flight SQL proto definition - Add FLIGHT_SQL_SERVER_BULK_INGESTION and FLIGHT_SQL_SERVER_INGEST_TRANSACTIONS_SUPPORTED options for SqlInfo - Go: - Generate pb - Server-side implementation - Client-side implementation - Unit + integration tests - C++: - Server-side implementation - Client-side implementation - Integration tests ### Are these changes tested? Yes, see `server_test.go`, `scenario.go`, and `test_integration.cc`. ### Are there any user-facing changes? Yes, new Flight SQL client and server functionality is being added. Changes are not expected to break existing users. * Closes: apache#38255 Lead-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Signed-off-by: Matt Topol <[email protected]>
vibhatha
pushed a commit
to vibhatha/arrow
that referenced
this issue
May 25, 2024
…#38385) ### Rationale for this change It was suggested in the discussion around apache/arrow-adbc#1107 for the Flight SQL ADBC driver that an "Ingest" command would be a helpful addition to the Flight SQL specification. This command would enable a Flight SQL client to provide a FlightData stream to the server without needing to know its SQL syntax, and have that stream loaded into a target table by whichever means the server deems appropriate. ### What changes are included in this PR? - Format: - Add CommandStatementIngest message type to Flight SQL proto definition - Add FLIGHT_SQL_SERVER_BULK_INGESTION and FLIGHT_SQL_SERVER_INGEST_TRANSACTIONS_SUPPORTED options for SqlInfo - Go: - Generate pb - Server-side implementation - Client-side implementation - Unit + integration tests - C++: - Server-side implementation - Client-side implementation - Integration tests ### Are these changes tested? Yes, see `server_test.go`, `scenario.go`, and `test_integration.cc`. ### Are there any user-facing changes? Yes, new Flight SQL client and server functionality is being added. Changes are not expected to break existing users. * Closes: apache#38255 Lead-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Signed-off-by: Matt Topol <[email protected]>
lidavidm
pushed a commit
that referenced
this issue
Sep 5, 2024
Please look at #38255 for details on this functionality. Support for Go and C++ was added as part of #38385. This pull request is to add the required support for Java. * GitHub Issue: #38255 Lead-authored-by: Amit Mittal <[email protected]> Co-authored-by: Amit Mittal <[email protected]> Signed-off-by: David Li <[email protected]>
zanmato1984
pushed a commit
to zanmato1984/arrow
that referenced
this issue
Sep 6, 2024
) Please look at apache#38255 for details on this functionality. Support for Go and C++ was added as part of apache#38385. This pull request is to add the required support for Java. * GitHub Issue: apache#38255 Lead-authored-by: Amit Mittal <[email protected]> Co-authored-by: Amit Mittal <[email protected]> Signed-off-by: David Li <[email protected]>
khwilson
pushed a commit
to khwilson/arrow
that referenced
this issue
Sep 14, 2024
) Please look at apache#38255 for details on this functionality. Support for Go and C++ was added as part of apache#38385. This pull request is to add the required support for Java. * GitHub Issue: apache#38255 Lead-authored-by: Amit Mittal <[email protected]> Co-authored-by: Amit Mittal <[email protected]> Signed-off-by: David Li <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the enhancement requested
Creating an issue here to link the ongoing discussion in the arrow-adbc repository regarding bulk ingestion support in the Flight SQL spec: apache/arrow-adbc#1107
More context can be found within the issue's discussion, but to summarize the current proposal is to extend the Flight SQL spec with a new proto message type
CommandStatementIngest. The semantics of the command are intended to follow those of ADBC bulk ingestion closely, which is in part why the discussion started in that repo initially.Component(s)
FlightRPC, Format
The text was updated successfully, but these errors were encountered: