[BUG]: vertex pair not properly shuffled

[jnke2016]

Version

22.12

Which installation method(s) does this occur on?

Docker, Conda, Pip, Source

Describe the bug.

The vertex pairs are not properly shuffled to the appropriate GPUs leading to undefined behaviors such as illegal memory accesses. This causes all the MG similarity algorithms (jaccard, sorensen, overlap) to fail at 8+ GPUs

Minimum reproducible example

Run the script below or any MG similarity algo test with 8+ GPUs

    setup_objs = setup()
    client = setup_objs[0]
    num_workers = len(client.scheduler_info()['workers'])

    df = karate.get_edgelist()
    
    ddf = dask_cudf.from_cudf(df, npartitions=num_workers)

    # Create MG Graph
    dg = cugraph.Graph(directed=False)
    dg.from_dask_cudf_edgelist(
        ddf, source='src', destination='dst',
        legacy_renum_only=True)

    # Get vertex_pair by computing the two_hop_neighbors
    vertex_pair = dg.get_two_hop_neighbors()
    vertex_pair = vertex_pair.compute.head()

    # Call jaccard
    df = dcg.jaccard(dg, vertex_pair)    

    teardown(*setup_objs)

Relevant log output

Exception: "RuntimeError('non-success value returned from cugraph_jaccard_coefficients: CUGRAPH_UNKNOWN_ERROR std::bad_alloc: out_of_memory: CUDA error at: /gpfs/fs1/projects/sw_rapids/users/jnke/miniconda3/envs/cugraph_test/include/rmm/mr/device/cuda_memory_resource.hpp')"

Code of Conduct

• I agree to follow cuGraph's Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report