Code

Llama#270

Code

Merged

Ivan-Zhou merged 60 commits intomainstanford-crfm/levanter:mainfrom llamastanford-crfm/levanter:llamaCopy head branch name to clipboardAug 30, 2023

+893-14

Commits

Commits on Jul 30, 2023

• Add Llama config, Mlp, Attention, and RotaryEmb

  Ivan-Zhou

  committed
  ca6ada5

Commits on Jul 31, 2023

• address integration test

  Ivan-Zhou

  committed
  667a56c

Commits on Aug 1, 2023

• test compare with hf implementation

  Ivan-Zhou

  committed
  0e77701

• Finish LlamaRotaryEmbedding

  Ivan-Zhou

  committed
  ed10ee1

• Add LlamaLinearScalingRotaryEmbedding

  Ivan-Zhou

  committed
  6034aee

• Add LlamaDynamicNTKScalingRotaryEmbedding

  Ivan-Zhou

  committed
  e9a7c56

• Refactor to simplified class differences

  Ivan-Zhou

  committed
  6e97f5f

• Implement _get_rotary_emb

  Ivan-Zhou

  committed
  271be4b

Commits on Aug 2, 2023

• work on attention

  Ivan-Zhou

  committed
  669e6e7

Commits on Aug 6, 2023

• test initialize attention

  Ivan-Zhou

  committed
  e45b39b

Commits on Aug 8, 2023

• _apply_rotary_pos_emb

  Ivan-Zhou

  committed
  bbcd0ec

• update llama and test

  Ivan-Zhou

  committed
  442b93d

Commits on Aug 13, 2023

• Merge branch 'main' into llama

  Ivan-Zhou

  committed
  673fa17

• update llama and test

  Ivan-Zhou

  committed
  40bb7b4

• Finish Llama Attention

  Ivan-Zhou

  committed
  466c1db

Commits on Aug 14, 2023

• Finish implementing LlamaLMHeadModel

  Ivan-Zhou

  committed
  f9db049

• fix build

  Ivan-Zhou

  committed
  6781a8c

• fix build

  Ivan-Zhou

  committed
  abb70fa

Commits on Aug 16, 2023

• remove max_position_embeddings

  Ivan-Zhou

  committed
  f21cd1c

• Fix issues found from testing

  Ivan-Zhou

  committed
  6e222b7

Commits on Aug 20, 2023

• Fix issues found from end-to-end tests

  Ivan-Zhou

  committed
  182327e

• Fix torch import issue

  Ivan-Zhou

  committed
  46939d5

• Refactor RoPE

  Ivan-Zhou

  committed
  92a7f23

• remove ()

  Ivan-Zhou

  committed
  65f5888

• NamedArray type hint

  Ivan-Zhou

  committed
  3f028b9

• Remove position_ids

  Ivan-Zhou

  committed
  2fb1a3f

• from/to HF config

  Ivan-Zhou

  committed
  97dadce

Commits on Aug 22, 2023

• Update to state_dict

  Ivan-Zhou

  committed
  fefb93a

• ignore type in default_hf_checkpoint_converter

  Ivan-Zhou

  committed
  8239cc3

• attn to self_attn in LlamaDecoderLayer

  Ivan-Zhou

  committed
  73f1c90

Commits on Aug 23, 2023

• Remove position embed from LlamaEmbedding

  Ivan-Zhou

  committed
  8f16ec0

• Set out_dims_first_in_dict

  Ivan-Zhou

  committed
  5578db2

• Add (incomplete) roundtrip test

  Ivan-Zhou

  committed
  d73c6f7

Commits on Aug 24, 2023

• remove unused Axis form mlp

  Ivan-Zhou

  committed
  79f4fa9

• Rename v's position

  Ivan-Zhou

  committed
  ee6e11c

• pretty sure this is the problem: weights weren't being deserialized/reserialized properly

  dlwh

  committed
  f06e3fa

• document when to use out_dims_first_in_dict

  dlwh

  committed
  41b103c

Commits on Aug 28, 2023

• Add LlamaRMSNorm and add more consistency

  Ivan-Zhou

  committed
  6ed1ee5

• Fix issues from pre-commit tests

  Ivan-Zhou

  committed
  9c669fd

• tie llama weights by default

  dlwh

  committed
  6245f9b

• add a todo for ivan

  dlwh

  committed
  190cba3

• make test pass even without auth token

  dlwh

  committed
  a022bb1

Commits on Aug 29, 2023

• Intermediate -> Mlp

  Ivan-Zhou

  committed
  ec742c4

• Update src/levanter/models/llama.py

  Show description for a64e63a
  

  Ivan-Zhou

  and

  dlwh

  authored
  a64e63a

• Merge branch 'llama' of github.com:stanford-crfm/levanter into llama

  Ivan-Zhou

  committed
  1820ea9

• Update src/levanter/models/llama.py

  Show description for 1bd25ca
  

  Ivan-Zhou

  and

  dlwh

  authored
  1bd25ca

• Update src/levanter/models/llama.py

  Show description for d7412d4
  

  Ivan-Zhou

  and

  dlwh

  authored
  d7412d4

• Update src/levanter/models/llama.py

  Show description for 2e3c1fb
  

  Ivan-Zhou

  and

  dlwh

  authored
  2e3c1fb

• Fix issues from pre-commit checks

  Ivan-Zhou

  committed
  05f684f

• Start from llama 2 hf in roundtrip

  Ivan-Zhou

  committed
  f6157df

• Update model_id in the round trip test

  Ivan-Zhou

  committed
  5109b55

Commits on Aug 30, 2023

• Untie weight at LMHead Linear Layer

  Ivan-Zhou

  committed
  0a5d36d

• fix round trip test, use compile time eval for the cos/sin cache

  dlwh

  committed
  c056942

• Update src/levanter/models/llama.py

  dlwh

  authored
  8380283

• Update tests/test_llama.py

  dlwh

  authored
  93eeebe

• let's just use llama names where reasonable

  dlwh

  committed
  dca9157

• implement LmHeadModel in LLama

  dlwh

  committed
  9ccd795

• use haliax's built in attention

  dlwh

  committed
  5d8a102

• Merge remote-tracking branch 'origin/main' into llama

  dlwh

  committed
  d6e780c

• update for latest main: tokenizer resizing

  dlwh

  committed
  8c6a20d