SNOW-1865926: Infer schema for StructType columns from nested Rows

[sfc-gh-jrose]

1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

   Fixes SNOW-1865926

2. Fill out the following pre-review checklist:

   • I am adding a new automated test(s) to verify correctness of my new code
     • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
   • I am adding new logging messages
   • I am adding a new telemetry message
   • I am adding new credentials
   • I am adding a new dependency
   • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
   • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines

3. Please describe how your code solves the related issue.

   With the new semantic mode enabled structured MapType and ArrayType are already used by default. This PR makes it so that it's possible to express StructType objects in a create_dataframe call by passing them as Row objects similar to how pyspark would handle such data.

[github-actions]

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

[sfc-gh-aling]

is there a chance that the given datatype is not StructType while users still input a Row as data? if so what would happen, do we error out?

or Row data always auto inferred as StructType?

[sfc-gh-jrose]

I'm not sure what that scenario would look like. Can you give an example?

[sfc-gh-aling]

sorry I didn't make it clear, actually I had two questions:

1. when the infer schema logic would be triggered for Row values -- is it only when schema is not explicitly set?

2. for my own learning purposes, will following Row input + MapType be a valid input?

    struct = Row(f1="v1", f2=2)
    df = structured_type_session.create_dataframe(
        [
            (struct),
        ],
        schema=[StructureType(MapType(xxx)],
    )

[sfc-gh-jrose]

1. Yes, the schema is only inferred if no explicitly set.
2. Today this example would give an error like this:

>>> struct = Row(f1="v1", f2=2)
>>> df = session.create_dataframe([(struct,),], schema=StructType([StructField("test", MapType())]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src/snowflake/snowpark/_internal/utils.py", line 960, in func_call_wrapper
    return func(*args, **kwargs)
  File "src/snowflake/snowpark/session.py", line 3318, in create_dataframe
    raise TypeError(
TypeError: Cannot cast <class 'snowflake.snowpark.row.Row'>(Row(f1='v1', f2=2)) to MapType(StringType(), StringType()).

Currently createDataFrame does not know how to handle casting Rows to Maps. You could get it to work by calling Row.as_dict if you wanted it to be a MapType.

After this change it's improved slightly:

# Inferred
>>> df = session.create_dataframe([(struct),])
>>> df.schema
StructType([StructField('F1', StringType(), nullable=False), StructField('F2', LongType(), nullable=False)])
 
# or explicit
>>> df = session.create_dataframe([(struct),], schema=StructType([StructField('F1', StringType(), nullable=False), StructField('F2', LongType(), nullable=False)]))

Without this change the explicit schema still works.