Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix avro tests (#2570) #2571

Merged
merged 1 commit into from
May 18, 2022
Merged

Fix avro tests (#2570) #2571

merged 1 commit into from
May 18, 2022

Conversation

tustvold
Copy link
Contributor

Which issue does this PR close?

Closes #2570

Rationale for this change

See ticket

What changes are included in this PR?

  • Fix compilation of tests
  • Fix a mistake in avro -> arrow conversion that was being caught by the new arrow-rs validation logic 🎉
  • Adds avro feature to CI

Are there any user-facing changes?

The previous logic was likely broken for lists

@@ -861,6 +861,7 @@ fn flatten_string_values(values: &[&Value]) -> Vec<Option<String>> {
values
.iter()
.flat_map(|row| {
let row = maybe_resolve_union(row);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We must do this here, to match the logic used to deduce the offsets. Without this the offsets would not match the flattened values.

@github-actions github-actions bot added the datafusion Changes in the datafusion crate label May 18, 2022
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tustvold -- I thought avro was tested, but it looks like just the example is:

https://github.com/alamb/arrow-datafusion/blob/alamb%2Fmerge_benchmark/.github/workflows/rust.yml#L123

So while that is better than nothing, testing via cargo test --features avro is even better

cc @Igosuki

@tustvold tustvold merged commit a08d26e into apache:master May 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datafusion Changes in the datafusion crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Avro Tests Fail To Compile
2 participants