Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Recurrent neural network additions (rnn, lstm, and gru) #29

Merged
merged 67 commits into from
Dec 6, 2016
Merged

Recurrent neural network additions (rnn, lstm, and gru) #29

merged 67 commits into from
Dec 6, 2016

Conversation

robertleeplummerjr
Copy link
Contributor

@robertleeplummerjr robertleeplummerjr commented Sep 10, 2016 •
edited
Loading

Ready for merge.

@robertleeplummerjr
start on #16 using https://github.com/karpathy/recurrentjs as a basis…
8448f50 
… to start from
@robertleeplummerjr
A bit further, merged solver and graph into rnn, which lstm extends f…
e447789 
…or reuse
@robertleeplummerjr
stepCache doesn't belong in defaults
8eeb800
@robertleeplummerjr
Merge branch 'master' into recurrent
7bd8182
@robertleeplummerjr
nearing POC
82763f8
@robertleeplummerjr
start with the unit testing
e24ba22
@robertleeplummerjr
button up pr, ready for merge
5b665aa
@robertleeplummerjr
fix some porting errors
b62dbb4
@robertleeplummerjr
make terminology more english oriented
eb8df13
@robertleeplummerjr
start putting together the phrase writer test
e449226
@robertleeplummerjr
further with predict, finally
369eafa
@robertleeplummerjr
rnn seemingly working
cdfd348
@robertleeplummerjr
bring lstm network online
8235a65
@robertleeplummerjr
added math, but really just toying
22694c4
@robertleeplummerjr
Merge branch 'master' into recurrent
6bb3e8e
@robertleeplummerjr
unnest
d08e6f3
@robertleeplummerjr
working rnn
83102f0 
added utility methods for going to and from a vocab or indexes
@robertleeplummerjr
join array output for consistency
c1a49f3
@robertleeplummerjr
test lstm
b3bfe84
@robertleeplummerjr
start working on gru
3c4387e
@robertleeplummerjr
ensure recurrent is tested, and log times of execution for fun
5b98029
@robertleeplummerjr
nearly working with new equation building technique
dadfbca
@robertleeplummerjr
Fully running, problems still with exploding recurrence
b1ba89e
@robertleeplummerjr
recurrence now working, and infinite issue fixed
ff0c22c
@robertleeplummerjr
now working with multi-depth states
957fab8
@robertleeplummerjr
fully working lstm and rnn, though may need some tuning
aeedec5 
gru is close
simplify equation so that following equations are inside the neural net, and not the model
@robertleeplummerjr
gru online
0709d92 
housekeeping
@robertleeplummerjr
rename gru's matrixes for better clarification
79165a7
@robertleeplummerjr
bring rnn/lstm/gru toJSON and fromJSON methods online and tested (for…
1879e7d 
… the most part)
@robertleeplummerjr
partial code, mostly just getting ready for es6
7e42d7e
@robertleeplummerjr robertleeplummerjr mentioned this pull request Sep 11, 2016
Closed
@robertleeplummerjr
Copy link
Contributor Author

looking for feedback at #24

@robertleeplummerjr
second draft with toFunction
dfeb84a
@robertleeplummerjr
toFunction running, without prediction
427f54a
@robertleeplummerjr
more refactoring found a few flaws with backpropagation
5027c1d
@robertleeplummerjr
fixes, eureka moments, and code
84f0ea7
@robertleeplummerjr
more organization and refactoring
2ba84c0
@robertleeplummerjr
got the exploding weights issue solved, now the network seems to not …
29fb5b6 
…learn very fast
@robertleeplummerjr
found that recursion was not being set in transition, discovered that…
64930ce 
… it was attempting to be set in the wrong order, by moving it to result, it acts more like the actual network it is modeled after, push, not pull.
@robertleeplummerjr
added perplexity test, and looks like prediction is what is failing
adc2f21
@robertleeplummerjr
fully working rnn, lstm, and gru
d9dc977
@robertleeplummerjr
syntax
6f31698
@robertleeplummerjr
tests
c17f606
@robertleeplummerjr
remove comment
7818f45
@robertleeplummerjr
gru tests and fat arrow functions
438dbc9
@robertleeplummerjr
fat arrow functions, tweak rnn tests so they aren't so fragile
4b96078
@robertleeplummerjr
added printable characters to vocab
9cb8b3f
@robertleeplummerjr
move result so that it may be extended like input/output
f88ed76
@robertleeplummerjr
Merge branch 'master' into recurrent
da8daa7
@robertleeplummerjr
toFunction, working in rnn
d972d6f
@robertleeplummerjr
toFunction tests
d2a4283
@robertleeplummerjr
up training sessions
7647914
@robertleeplummerjr
Copy link
Contributor Author

The moment has come. .toFunction is working great, unit tests could be improved overall, but that will be a separate strategy. I believe we are merge worthy.

@robertleeplummerjr robertleeplummerjr merged commit 1a18515 into master Dec 6, 2016
@robertleeplummerjr robertleeplummerjr deleted the recurrent branch December 6, 2016 02:53
@IonicaBizau
Copy link
Member

Amazing! 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@robertleeplummerjr @IonicaBizau