Recurrent weighted average

[MartinuzziFrancesco]

https://arxiv.org/abs/1703.01253

[MartinuzziFrancesco]

$$\begin{align} h_0 &= f(s_0), \quad n_0 = 0, \quad d_0 = 0 \\\ u(x_t) &= W_u \cdot x_t + b_u \\\ g(x_t, h_{t-1}) &= W_g \cdot [x_t, h_{t-1}] + b_g \\\ a(x_t, h_{t-1}) &= W_a \cdot [x_t, h_{t-1}] \\\ z(x_t, h_{t-1}) &= u(x_t) \circ \tanh g(x_t, h_{t-1}) \\\ n_t &= n_{t-1} + z(x_t, h_{t-1}) \circ e^{a(x_t, h_{t-1})} \\\ d_t &= d_{t-1} + e^{a(x_t, h_{t-1})} \\\ h_t &= f \left( \frac{n_t}{d_t} \right) \end{align}$$