You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
I would like to have a possibility to use the data of the registered saved_dataset in a different spark session. Now I have only a name in the feast registry without the data if I create a new spark session.
Part of persist function:
"""
Run the retrieval and persist the results in the same offline store used for read.
Please note the persisting is done only within the scope of the spark session.
"""
assert isinstance(storage, SavedDatasetSparkStorage)
table_name = storage.spark_options.table
if not table_name:
raise ValueError("Cannot persist, table_name is not defined")
self.to_spark_df().createOrReplaceTempView(table_name)
Describe the solution you'd like
Add possibility to save dataset as a table, for example when Spark Session config is included info about remote storage (hive, s3 path, etc)
Describe alternatives you've considered
Add an optional parameter for SparkOptions, which allows to save dataset as a table in any spark session configurations.
Additional context
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
I would like to have a possibility to use the data of the registered saved_dataset in a different spark session. Now I have only a name in the feast registry without the data if I create a new spark session.
Part of persist function:
Describe the solution you'd like
Add possibility to save dataset as a table, for example when Spark Session config is included info about remote storage (hive, s3 path, etc)
Describe alternatives you've considered
Add an optional parameter for SparkOptions, which allows to save dataset as a table in any spark session configurations.
Additional context
The text was updated successfully, but these errors were encountered: