NetworkWriter

[makslevental]

I have experiments that run on various machines. I would like to centralize all tensorboard logs on one dashboard machine. In my imagination this should be possible by simple sending the protobufs over the wire rather than writing to disk.

[stephanwlee]

We did consider that but it leads to other complexities when network is flaky (should we fallback to local filesystem while offline? then how do we sync?) and when throughput is not enough (we don't want to slow down TensorFlow). We do actively think about this problem but it currently is not our priority.

+cc @nfelt.

[nfelt]

My recommendation would be using an existing general solution for exposing files across the network, e.g. you might consider things like rsync or sshfs mounts (for rsync you'll want the --inplace option per #349).

We've considered ways to make it easier to user TensorBoard with remote jobs, but in reality making a robust distributed summary writing system is not a particularly simple task, and in the short term it would be better to rely on tools that have already solved the distributed filesystem problem.

[makslevental]

@nfelt @stephanwlee sshfs occurred to me but it felt hacky. but thanks for the input.