Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compat issue on COLLECTION_DELIM in Hive 3.x #1714

Closed
7c00 opened this issue Oct 10, 2019 · 3 comments
Closed

Compat issue on COLLECTION_DELIM in Hive 3.x #1714

7c00 opened this issue Oct 10, 2019 · 3 comments
Labels
bug Something isn't working good first issue Good for newcomers
Milestone

Comments

@7c00
Copy link
Member

7c00 commented Oct 10, 2019

HIVE-16922 fixed the typo of org.apache.hadoop.hive.serde.serdeConstants#COLLECTION_DELIM, but also introduced a breaking change.

#1321 fixed this compatibility issue for RC files, while leaving TEXT files out.

We walk around it by changing the io.prestosql.plugin.hive.HiveUtil#getDeserializer

public static Deserializer getDeserializer(Configuration configuration, Properties schema)
    {
        String name = getDeserializerClassName(schema);

        // fix compat issue: colelction.delim in Hive 2.x, but collection.delim in Hive 3.x
        // see also https://issues.apache.org/jira/browse/HIVE-16922
        if (name.equals("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe")) {
            if (schema.containsKey("colelction.delim") && !schema.containsKey(COLLECTION_DELIM)) {
                schema.put(COLLECTION_DELIM, schema.getProperty("colelction.delim"));
            }
        }

        Deserializer deserializer = createDeserializer(getDeserializerClass(name));
        initializeDeserializer(configuration, deserializer, schema);
        return deserializer;
    }
@findepi findepi added bug Something isn't working good first issue Good for newcomers labels Oct 10, 2019
@Anurag870
Copy link
Contributor

Created PR #1733

@JustanotherN8
Copy link

You guys are doing a great job

@findepi findepi added this to the 321 milestone Oct 15, 2019
@findepi
Copy link
Member

findepi commented Oct 15, 2019

Solved by #1733

@findepi findepi closed this as completed Oct 15, 2019
This was referenced Oct 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers
Development

Successfully merging a pull request may close this issue.

4 participants