SNOW-1829870: Allow structured types to be enabled by default

[sfc-gh-jrose]

1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

   Fixes SNOW-1829870

2. Fill out the following pre-review checklist:

   • I am adding a new automated test(s) to verify correctness of my new code
     • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
   • I am adding new logging messages
   • I am adding a new telemetry message
   • I am adding new credentials
   • I am adding a new dependency
   • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
   • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines

3. Please describe how your code solves the related issue.

   This PR addresses a few misc issue involved with enabling structured types by default.

• Change to_sql to support structured types. There were some edge cases where it was generating incorrect sql.
• When the structured type semantic mode is enabled structured types are assumed to be structured and to have semi-structured types you have to set the element types to None for ArrayType, MapType, and StructType.
• In cases where StringType was being passed to init for those types None is now passed by default allowing the type to decide if it should default to StringType.
• When the new mode is enabled types will prefer to be structured if element typing is available.

Before this change the in order to have a structured schema you would have to set the structured parameter to True on each type that was expected to be structured:

structured_schema = StructType([
    StructField("array", ArrayType(IntegerType(), structured=True)),
    StructField("map", MapType(IntegerType(), IntegerType(), structured=True)),
    StructField("struct", StructType([StructField("field", IntegerType())], structured=True)),
])

semi_structured_schema = StructType([
    StructField("array", ArrayType(IntegerType())),
    StructField("map", MapType(IntegerType(), IntegerType())),
    StructField("struct", StructType([StructField("field", IntegerType())])),
])

also_semi_structured_schema = StructType([
    StructField("array", ArrayType()),
    StructField("map", MapType()),
    StructField("struct", StructType()),
])

# Both of the semi-structured schemas actually materialize to this:
StructType([
	StructField('ARRAY', ArrayType(StringType()), nullable=True),
	StructField('MAP', MapType(StringType(), StringType()), nullable=True),
	StructField('STRUCT', MapType(StringType(), StringType()), nullable=True)
])

After this change the second case above will instead be inferred to be structured which should be more understandable to users. We've seen confusion in the past with the inner types being converted to StringType being non-inuitive.

After the change the definitions will look like this:

structured_schema = StructType([
    StructField("array", ArrayType(IntegerType())),
    StructField("map", MapType(IntegerType(), IntegerType())),
    StructField("struct", StructType([StructField("field", IntegerType())])),
])

semi_structured_schema = StructType([
    StructField("array", ArrayType()),
    StructField("map", MapType()),
    StructField("struct", StructType()),
])

# Semi structured actually materializes to what it is set to.

[github-actions]

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

[github-actions]

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

[sfc-gh-aalam]

can element_type be None here? What does it mean for the column type to be so?

[sfc-gh-jrose]

If element_type is None than this is a semi-structured array column and could contain anything.

[github-actions]

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing