Chain export inefficiencies

[austinabell]

Describe the bug
A clear and concise description of what the bug is.

When walking a snapshot, ipld nodes are loaded 2x more than they need to, as well as redundant data structures (2 hashsets and 1 array to do what it seems 1 hashset could)

For extra blockstore reads:
Whenever the callback happens, the Cid is loaded from the blockstore (

lotus/chain/store/store.go

Line 1360 in a63ef1d

blk, err := cs.bs.Get(c)

), but also in addition to this, it's also read when it's used during walking (

lotus/chain/store/store.go

Line 1393 in a63ef1d

data, err := cs.bs.Get(blk)

,

lotus/chain/store/store.go

Line 1318 in a63ef1d

data, err := bs.Get(root)

, ...)

Extra data structures:

• seen and walked hashsets (

  lotus/chain/store/store.go

  Line 1378 in a63ef1d

  seen := cid.NewSet()

  ) They are updated in lockstep, but why would you need two hashsets for this, either it's been seen and it should be ignored, or it hasn't and it should be recursed.

• cids and out slices (

  lotus/chain/store/store.go

  Line 1410 in a63ef1d

  var cids []cid.Cid

  ,

  lotus/chain/store/store.go

  Line 1430 in a63ef1d

  out := cids

  ) My memory is failing as to if this would reallocate a new slice or it's just assigning the pointer in go, but neither of these should be needed. They are being added to this vector, then checked against the seen hashset, when this could be done before it's added

Let me know if I'm missing something here, I've implemented without these (using the callback to pass back the loaded bytes and only one hashset) and it seems to work the same, but I could be missing an edge case. This also doesn't matter that much to optimize, so probably a low priority ticket.

To Reproduce
Steps to reproduce the behavior:

1. Run '...'
2. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Version (run lotus version):

Additional context
Add any other context about the problem here.

[raulk]

@austinabell ditto. I've ran into these inefficiencies when trying to produce a full snapshot. The current logic tries to remove duplicates, but it could be done differently.

[jennijuju]

@vyzo mentioned he may be interested in this😉

[Kubuxu]

Another thing of note is that lotus doesn't seem to stream the export CARs.