Support for Zipping multiple array

[rohitjoshi]

I am currently using rayon iterator for zipping two arrays. I need to zip multiple arrays and a combination of rayon and itertools will definitely help. for now, I had to switch to itertools which is sequential.

See the similar request on itertools

[cuviper]

Combining rayon with generic iterators (including itertools) is a hard problem -- #46.

But specifically for zipping, I suppose you mean something like itertools' izip!() and multizip()? I think this would be feasible to recreate in rayon.

[rohitjoshi]

yes, izip!() and multizip()

[MaikKlein]

@cuviper Is anyone working on it? If not I could probably implement it, I don't think it would require much work. I'd probably change the signature from Zip<A, B> to Zip<Tuple>, and then implement iterator for the various zips Zip<(A, B)> etc.

[cuviper]

@MaikKlein I don't know of anyone working on it. I don't think we would want to make a breaking change to the existing Zip, so this should probably be added as a new thing.

[cuviper]

I'm working on this now...

[dineshadepu]

Hello @cuviper, is there any chance of adding to rayon in future?

[cuviper]

Well, clearly I stopped working on it, sorry about that. I just found my WIP branch and pushed it to my fork though, if anyone would like to pick that up and finish it:

branch: https://github.com/cuviper/rayon/tree/multizip
comparison: master...cuviper:multizip

[dineshadepu]

It looks complete, and the documentation states all the use cases. I think I can use it in my code base. Can I know what else is there to add?

[cuviper]

I honestly don't know what I thought was remaining... 😕

[dineshadepu]

Hi Josh,

I am using it in my crate as in here https://github.com/multiphase-rs/mulltiphase_rs/blob/master/src/dem/mod.rs#L50 and many other places which is not yet open source, by creating another package https://github.com/multiphase-rs/rayon.

Please try to add it to the upcoming release of rayon.

Thank you.

[cuviper]

Now that I look again, I'm not really happy with that function signature:

pub fn multizip<T, I>(tuple: T) -> MultiZip<I>
where
    T: IntoParallelIterator<Iter = MultiZip<I>>,
    MultiZip<I>: ParallelIterator<Item = T::Item>,
{
    tuple.into_par_iter()
}

Having I as a type parameter implies that it can be chosen, but it's actually locked to the exact type from T's Iter = MultiZip<I>. The design of itertools::multizip is similar, but they have an additional constraint. They can't implement a foreign trait (IntoIterator) for a foreign type (tuples), which is why theirs ends up based on From.

In our case, we defined IntoParallelIterator, so we're free to implement it directly on tuples and their references, thus you can call into_par_iter(), par_iter(), and par_iter_mut() directly on the tuple. I think we don't really need a multizip function at all.

I'm going to remove that and clean up the examples to match, then I'll open a PR. We can always add such a function later if there's a compelling case.

[rohitjoshi]

Thanks