Feature: Sparse Access Memory

[ixaxaar]

Implement sparse access memory for DNCs.
Related paper: https://arxiv.org/abs/1610.09027

[MaxwellRebo]

Are you already diving into this, or do you need some help?

[ixaxaar]

Not looking at this at the moment. Trying to make DNCs work for language modeling & seq2seq.

[MaxwellRebo]

I've done DNCs for seq2seq before. Does anything about the base implementation need to change, or is it just a matter of structuring the inputs and outputs?

[ixaxaar]

I actually wrote this repo because the other ones were not torch.nn-like. Making the APIs more like torch.nn makes it easier to replace nn.LSTM etc with DNC otherwise keeping implementations intact, potentially requiring one line of code change.

The ones I tried, I had to also write this piece of code for LMs to detach hidden states: https://gist.github.com/ixaxaar/02b65312eb1a840cc14e74d93b35178f
For seq2seq I took the openNMT repo and forked it here.

Other than that it was pretty much a few liners.

[ixaxaar]

Is implemented in #16 and #18 , though code review etc are pending.