You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Different query requires different Spark job submit parameters.
What solution would you like?
CREATE statement:
a. Set execution-timeout-minutes to 0 so EMR-S won't time out the streaming job at background
b. Disable DRA (Spark dynamic resource allocator) because it only works well for long-running batch job
c. [TBD] Flint config that specifies which Env variable to populate to Flint metadata, such as EMR-S job info
SELECT statement
Also think about if any flexible way to manage these parameters in case of changes in future.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem?
Different query requires different Spark job submit parameters.
What solution would you like?
CREATE
statement:a. Set
execution-timeout-minutes
to 0 so EMR-S won't time out the streaming job at backgroundb. Disable DRA (Spark dynamic resource allocator) because it only works well for long-running batch job
c. [TBD] Flint config that specifies which Env variable to populate to Flint metadata, such as EMR-S job info
SELECT
statementAlso think about if any flexible way to manage these parameters in case of changes in future.
The text was updated successfully, but these errors were encountered: