Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MoE #639

Open
wants to merge 144 commits into
base: main
Choose a base branch
from
Open

MoE #639

Changes from 1 commit
Commits
Show all changes
144 commits
Select commit Hold shift + click to select a range
e725eb9
Clean MoE implementation
Muennighoff Jun 20, 2024
db24750
Add conf
Muennighoff Jun 20, 2024
18450de
Fix return args
Muennighoff Jun 20, 2024
4ab7f77
Rmv outdated kwarg
Muennighoff Jun 20, 2024
dba42fd
Rmv legacy kwarg
Muennighoff Jun 20, 2024
6c5f8a3
Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennig…
Muennighoff Jun 20, 2024
6a8e089
Add distributed_strategy
Muennighoff Jun 20, 2024
1a9a317
Allow w/o weight attr
Muennighoff Jun 20, 2024
ddf6fd4
Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennig…
Muennighoff Jun 20, 2024
ab55e07
Allow w/o weight attr
Muennighoff Jun 20, 2024
7aeefd4
Add MoE params
Muennighoff Jun 20, 2024
3eab45c
Rmv kwarg
Muennighoff Jun 20, 2024
6d736da
Reduce lb & moe losses
Muennighoff Jun 21, 2024
d07c638
LN & Emb Dec
Muennighoff Jun 21, 2024
cdb592f
Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennig…
Muennighoff Jun 21, 2024
1399841
Do not decay emb
Muennighoff Jun 21, 2024
a13b5b8
Tmp - debug throughput
Muennighoff Jun 21, 2024
935167e
Fix
Muennighoff Jun 21, 2024
b96972d
Fix
Muennighoff Jun 21, 2024
0079490
maintain init order
Muennighoff Jun 21, 2024
8b1c441
Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennig…
Muennighoff Jun 21, 2024
e2c7286
Decay emb
Muennighoff Jun 21, 2024
d39a37c
Keep EA on CPU
Muennighoff Jun 21, 2024
3acfc04
Do not decay emb
Muennighoff Jun 21, 2024
4432261
Change norm
Muennighoff Jun 21, 2024
cef7707
Confs
Muennighoff Jun 22, 2024
2a6df33
Adapt wrap
Muennighoff Jun 22, 2024
7421890
Add conf
Muennighoff Jun 23, 2024
021974e
decemb conf
Muennighoff Jun 24, 2024
d5a0626
Updates
Muennighoff Jun 24, 2024
fce086f
up
Muennighoff Jun 25, 2024
daa7c91
Update weka
Muennighoff Jun 25, 2024
3a40b7f
Rev weka
Muennighoff Jun 25, 2024
2676d03
Adapt
Muennighoff Jun 25, 2024
2fb4c96
Add fine
Muennighoff Jun 25, 2024
448a9a8
Adapt
Muennighoff Jun 26, 2024
e361918
Adapt
Muennighoff Jun 26, 2024
9377aa5
Add eps
Muennighoff Jun 26, 2024
bad3a34
Update confs
Muennighoff Jun 27, 2024
3dbd156
Change eps
Muennighoff Jun 27, 2024
a6496d6
Add conf
Muennighoff Jun 28, 2024
5dd6135
Confs
Muennighoff Jun 28, 2024
195c77d
Add conf
Muennighoff Jun 29, 2024
4ea9f0a
Fix path
Muennighoff Jun 29, 2024
44fa5ae
Fix path
Muennighoff Jun 29, 2024
b7658eb
Adapt confs
Muennighoff Jun 30, 2024
16edece
add s3 conf
Muennighoff Jun 30, 2024
f8e061f
fix paths
Muennighoff Jun 30, 2024
c8a51cf
Update confs
Muennighoff Jul 1, 2024
b95a05f
Increase bs
Muennighoff Jul 1, 2024
20103ec
Adjust paths
Muennighoff Jul 1, 2024
459ef27
conf
Muennighoff Jul 1, 2024
84f21db
confs
Muennighoff Jul 1, 2024
08d3253
confs
Muennighoff Jul 1, 2024
4b07140
Update confs
Muennighoff Jul 1, 2024
cf6fa33
Update conf
Muennighoff Jul 1, 2024
d95f978
Conf
Muennighoff Jul 1, 2024
3c15abf
Finegrained
Muennighoff Jul 2, 2024
4163e70
Cx5
Muennighoff Jul 2, 2024
aae0e0b
Add conf
Muennighoff Jul 2, 2024
76e6e6d
Add cx5
Muennighoff Jul 2, 2024
31e387b
Up confs
Muennighoff Jul 2, 2024
f631d8e
Up confs
Muennighoff Jul 2, 2024
142720b
Dfix
Muennighoff Jul 4, 2024
412a55e
reddit
Muennighoff Jul 4, 2024
669bba5
flan
Muennighoff Jul 4, 2024
7d90908
fine
Muennighoff Jul 4, 2024
3c97519
indent
Muennighoff Jul 4, 2024
6e08b09
Add conf
Muennighoff Jul 4, 2024
1a6a2e2
Conf
Muennighoff Jul 5, 2024
103e450
Add shared
Muennighoff Jul 5, 2024
48b6c14
Add
Muennighoff Jul 5, 2024
8a387fa
Add conf
Muennighoff Jul 5, 2024
3203724
Add conf
Muennighoff Jul 6, 2024
5309137
Add conf
Muennighoff Jul 6, 2024
bf66e68
Addqk
Muennighoff Jul 6, 2024
63c12e1
Make QK Norm parametric
Muennighoff Jul 6, 2024
1c4aa8d
dense comp
Muennighoff Jul 7, 2024
6a94263
add conf
Muennighoff Jul 7, 2024
b58c316
conf
Muennighoff Jul 7, 2024
3bbfaed
conf
Muennighoff Jul 7, 2024
364659f
Change torch version
Muennighoff Jul 18, 2024
15f5503
Add anneal
Muennighoff Jul 20, 2024
6c516d8
Resets
Muennighoff Jul 20, 2024
8a0758e
add conf
Muennighoff Jul 21, 2024
c23b048
conf
Muennighoff Jul 21, 2024
285ff10
Adjust mlp hidden
Muennighoff Jul 21, 2024
99aec31
Change ratio
Muennighoff Jul 21, 2024
a43eae8
Add conf
Muennighoff Jul 21, 2024
aaefc58
Conf
Muennighoff Jul 21, 2024
270271f
Adjust
Muennighoff Jul 21, 2024
c084d34
add conf
Muennighoff Jul 21, 2024
6789ee2
Fix conf
Muennighoff Jul 21, 2024
f084fa0
Adjust
Muennighoff Jul 21, 2024
14ee7e4
Add alt
Muennighoff Jul 21, 2024
f831adf
Fix typo; update script
Muennighoff Jul 23, 2024
26eb3f3
merge main
Muennighoff Jul 23, 2024
0a3b076
add moe reorder
Muennighoff Jul 23, 2024
d6ccbf0
Add datafix
Muennighoff Jul 23, 2024
6a2c17e
Max doc len MoE
Muennighoff Jul 23, 2024
9ccb2f1
add fa varlen
Muennighoff Jul 23, 2024
f5291ec
Add conf
Muennighoff Jul 23, 2024
ed571a8
add conf
Muennighoff Jul 24, 2024
71d2d2a
fix conf
Muennighoff Jul 24, 2024
8b72521
fixconf
Muennighoff Jul 24, 2024
fc822a0
fixonf
Muennighoff Jul 24, 2024
1ccaf9a
fix
Muennighoff Jul 24, 2024
c157441
Add 8k
Muennighoff Jul 25, 2024
06b8010
add conf
Muennighoff Jul 26, 2024
cf31e53
conf
Muennighoff Jul 26, 2024
a750fde
Adapt
Muennighoff Jul 31, 2024
c853a43
Merge branch 'main' into Muennighoff/MoE
Muennighoff Jul 31, 2024
bea80ec
Sort
Muennighoff Jul 31, 2024
dfdcfc5
Simplify configs for merge
Muennighoff Jul 31, 2024
4606598
Update megablocks installation
Muennighoff Aug 1, 2024
f6c707d
Simplify
Muennighoff Aug 1, 2024
76f0376
Simplify
Muennighoff Aug 1, 2024
9631c80
typing & changes
Muennighoff Aug 1, 2024
2fa2acb
Format
Muennighoff Aug 1, 2024
c517703
Change torch.tensor to Any so type check passes?
Muennighoff Aug 1, 2024
b4eb33f
Simplify
Muennighoff Aug 1, 2024
ae6f16a
Simplify
Muennighoff Aug 1, 2024
02781be
Sort
Muennighoff Aug 1, 2024
a62a1ee
Fix type checks
Muennighoff Aug 1, 2024
0ecd4b8
Format
Muennighoff Aug 1, 2024
d8452a0
Fix typo; MoEArgs func
Muennighoff Aug 3, 2024
8a28ced
Format
Muennighoff Aug 3, 2024
91f5553
Check for act ckpt strategy & moe; fix typo
Muennighoff Aug 3, 2024
61ac104
fix import
Muennighoff Aug 3, 2024
f4faf8a
Sort impot
Muennighoff Aug 20, 2024
fdc1021
Merge branch 'main' into Muennighoff/MoE
Muennighoff Aug 20, 2024
ed82181
Fix typo
Muennighoff Aug 21, 2024
b0cc754
Simplify isinstance
Muennighoff Aug 21, 2024
ca9b41f
Clean conf & move constructor
Muennighoff Aug 21, 2024
215c0f5
Add ref
Muennighoff Sep 4, 2024
43baf74
Merge main
Muennighoff Sep 4, 2024
775e514
Sort imports
Muennighoff Sep 4, 2024
cd0004b
Format
Muennighoff Sep 4, 2024
acb23dd
No exp ass
Muennighoff Sep 12, 2024
a143469
Revert
Muennighoff Sep 12, 2024
1a4bdae
Simplify
Muennighoff Sep 12, 2024
410064a
Rmv interleave
Muennighoff Oct 3, 2024
671bc8e
Merge branch 'main' into Muennighoff/MoE
Muennighoff Oct 3, 2024
04a2da5
expert_assignments on gpu
Muennighoff Oct 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add conf
Muennighoff committed Jun 29, 2024
commit 195c77d49c01ecf3d7b9c4c5296d5f2bb4c2649e
1,861 changes: 1,861 additions & 0 deletions configs/olmoe17/olmoe17-8x1b-final-eps-noqk.yml

Large diffs are not rendered by default.