what dataloader to use for torchdata.nodes nodes?

[keunwoochoi]

hi, thanks for reviving torchdata. i was able to move on to 0.10.1 for lots of my existing datapipes. it seems to work pretty nicely.

question - am i supposed to use torchdata.nodes.Loader or torchdata.stateful_dataloader.StatefulDataLoader for my data nodes? or just torch.utils.data.DataLoader? i'm getting confused a bit after reading the docs and code. currently Loader works for my iterable data nodes, but with some caveats (no multi processing).

[ramanishsingh]

Hi @keunwoochoi ,
Thanks for checking out nodes.
Loader works pretty well, and you can check out some examples here: https://github.com/pytorch/data/tree/main/examples/nodes
You can also check the migration guide here: https://pytorch.org/data/beta/migrate_to_nodes_from_utils.html

For your failing examples, can you share a minimum working example, and we can look into it.

[keunwoochoi]

i see. yes Loader works pretty well.

but, to clarify @ramanishsingh - you meant, StatefulDataLoader is supposed to work with nodes? so are torch.utils.data.DataLoader ?

[keunwoochoi]

from #1389, it seems like Loader is the only choice.

[andrewkho]

@keunwoochoi thanks for trying this out! We shoudl clarify this in the documentation, but right now the idea is that torchdata.nodes is a super-set of StatefulDataLoader, ie nodes should be able to do everything torch.utils.DataLoader and StatefulDataLoader should do, but nodes are not designed to be plugged into StatefulDataLoader. cc @scotts on confusion around torchdata vs dataloader v1.