Skip to content

Commit

Permalink
Fix device synchronization in local_cupy benchmark (#518)
Browse files Browse the repository at this point in the history
The way the code was written would pass the client's device to each worker and synchronize on that, which is wrong. This was catched due to recent changes to Distributed, where the following was raised:

```python
Task exception was never retrieved
future: <Task finished coro=<Scheduler.broadcast.<locals>.send_message() done, defined at /datasets/pentschev/miniconda3/envs/gdf/lib/python3.7/site-packages/distributed/scheduler.py:4965> exception=Exception("unhashable type: 'cupy.cuda.device.Device'")>
Traceback (most recent call last):
  File "/datasets/pentschev/miniconda3/envs/gdf/lib/python3.7/site-packages/distributed/scheduler.py", line 4969, in send_message
    resp = await send_recv(comm, close=True, serializers=serializers, **msg)
  File "/datasets/pentschev/miniconda3/envs/gdf/lib/python3.7/site-packages/distributed/core.py", line 662, in send_recv
    raise Exception(response["text"])
Exception: unhashable type: 'cupy.cuda.device.Device'
```

Authors:
  - Peter Andreas Entschev (@pentschev)

Approvers:
  - Benjamin Zaitlen (@quasiben)

URL: #518
  • Loading branch information
pentschev authored Feb 9, 2021
1 parent 531e186 commit f9ce83a
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion dask_cuda/benchmarks/local_cupy.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ async def _run(client, args):
res = client.compute(func(*func_args))
await client.gather(res)
if args.type == "gpu":
await client.run(xp.cuda.Device().synchronize)
await client.run(lambda xp: xp.cuda.Device().synchronize(), xp)
took = clock() - t1

return {
Expand Down

0 comments on commit f9ce83a

Please sign in to comment.