ccl/changefeedccl: respect filesystem walk errors in cloudFeed

[nvanbenschoten]

Closes #42979.

This commit properly handles errors in cloudFeed's filepath.WalkFunc
implementation. The documentation says:

If there was a problem walking to the file or directory named by
path, the incoming error will describe the problem and the function
can decide how to handle that error (and Walk will not descend into
that directory). If an error is returned, processing stops.

I stressed the test for 10,000 iterations and never saw anything, so it's
possible that this was a fluke. It's not clear what filesystem error this was
throwing so we might still see this pop up again, but at least we'll now
correctly propagate the error and surface it instead of hitting an NPE.
Because of that, I'm closing the issue for now.

Release note: None

[cockroach-teamcity]

This change is 

[danhhz]

This happens when the file disappears between getting the filenames in the directory and lstat-ing the info to pass it to walkFn. In our tests, this happens because export storage first writes the file under a temporary filename and then renames it to the final path. Previously, the temporary file was written in the system temp, but as of a recent change, it's written in the final dir. This was causing issues, so this morning knz changed the temp files to have a ".tmp" suffix which this walkFn already ignores. If you rebased this commit back to just before #42987, then I bet it would fail under stress.

[nvanbenschoten]

Ah, I missed that fly by. This still seems like a nice complementary solution. TFTR.

bors r+

[craig]

Build failed (retrying...)

• GitHub CI (Cockroach)

[otan]

i think this broke something (/ hid something that is broken):

[run] {"Time":"2019-12-05T18:17:11.027965988-05:00","Action":"output","Package":"github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl","Test":"TestChangefeedNemeses/cloudstorage","Output":"        nemeses_test.go:38: lstat /tmp/TestChangefeedNemeses_cloudstorage598908568/0/2019-12-05/201912052317095153334660000000001-ee65cbd3711bd858-1-20-00000004-foo-12.ndjson518464510.tmp: no such file or directory\n"}

https://teamcity.cockroachdb.com/viewLog.html?tab=buildLog&logTab=tree&filter=debug&expand=all&buildId=1629211&_focus=11613

[nvanbenschoten]

bors r-

[craig]

Canceled

[nvanbenschoten]

Yeah, I think this hit the error case Dan mentioned and broke

a ".tmp" suffix which this walkFn already ignores

The error handling code should have fired only after checking for and ignoring .tmp files. Sorry for the disruption. Done.

bors r+

[craig]

Build succeeded

• GitHub CI (Cockroach)