Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modifier Refactor OBCQ Implementation #1737

Merged
merged 189 commits into from
Oct 11, 2023
Merged

Modifier Refactor OBCQ Implementation #1737

merged 189 commits into from
Oct 11, 2023

Conversation

Satrat
Copy link

@Satrat Satrat commented Sep 26, 2023 •
edited
Loading

Summary

Implementation of OBCQ in the new modifiers framework, tested for OPT and Llama so far. This shouldn't be merged to main until #1713 is merged in. Main changes are in src/sparseml/modifiers/obcq

  • A new modifier is added for the SparseGPT algorithm, building off the code in src/sparseml/experimental/sparsegpt.
  • Added quantization modifier to the new modifier framework. Currently this modifier only supports one-shot
  • Added post one-shot calibration parameter for recalibrating quantization statistics after OBCQ
  • Dataset registry for c4,ptb,wikitext2,open_platypus

Instructions

To run for OPT:

python src/sparseml/transformers/sparsification/obcq/obcq.py facebook/opt-1.3b c4
    --recipe src/sparseml/transformers/sparsification/obcq/example.yaml

To run for Llama, you'll need a local copy of the model

python src/sparseml/transformers/sparsification/obcq/obcq.py /local/path/to/llama open_platypus
    --recipe src/sparseml/transformers/sparsification/obcq/example_llama.yaml

Testing

To compare the perplexity evaluation between the experimental implementation and productionized version of OPT run:

python src/sparseml/experimental/sparsegpt/examples/opt/compare_obcq.py

To compare perplexity for Llama run

python src/sparseml/experimental/sparsegpt/examples/llama2/compare_obcq.py

natuan and others added 30 commits August 17, 2023 23:28
@natuan
First abstract structure with OPT, MPT
bd729f3
Merge branch 'main' into tuan/sparsegpt
0476a14
scaffolding
e9c4e95
@natuan
First abstract structure with OPT, MPT
07a7273
@natuan
Updated
73ccd2e
@natuan
Updated
f742f4a
@anmarques
Initial implementation of Llama2 integration
7c2d3d4
@anmarques
Initial implementation of Llama2 integration
3f4c5d8
@anmarques
Fix llama-2 key
c181380
@anmarques
Add modelutils
dd0f5f8
@anmarques
Make smoothquant optional
99f8bbe
@anmarques
probe sequence length from model
db901e4
@anmarques
probe sequence length from model
48ebe81
@anmarques
Fix typo
2be7def
add in recipe and modifier logic
fb346ad
@anmarques
Catch additional arguments to load_model and load_data
c4fc152
@anmarques
Redifinition of seqlen
2b50607
@natuan
Runable OPT main code path
15b2c10
mpt modifier
34c3d84
Merge branch 'tuan/sparsegpt' into sa/sgpt_modifier
4cac803
@markurtz
Initial start implementation
a3934c2
@anmarques
State before rebasing
6a35b7f
@anmarques
State before rebasing
e02abb2
@anmarques
Merge remote-tracking branch 'origin/tuan/sparsegpt' into research/sp…
2fb9e66 
…arsegpt/llama2

# Conflicts:
#	src/sparseml/experimental/sparsegpt/dispatch.py
#	src/sparseml/experimental/sparsegpt/layer_compressor.py
#	src/sparseml/experimental/sparsegpt/main.py
#	src/sparseml/experimental/sparsegpt/model_preprocessor.py
#	src/sparseml/experimental/sparsegpt/opt.py
#	src/sparseml/experimental/sparsegpt/sequential.py
#	src/sparseml/experimental/sparsegpt/sparsegpt.py
@natuan
Initial support for MPT
466213c
@markurtz
add in further completion state for session and events
d953be4
@natuan
Initial commit for Llama2
7fc49ac
@natuan
Return model after override Llama attn
ff5b9a1
working refactor
090af1c
@natuan
More memory-efficient layer compression
594339f
@Satrat Satrat requested review from dsikka and rahul-tuli October 4, 2023 14:10
Base automatically changed from sparsification-refactor to main October 4, 2023 19:08
bfineran
bfineran suggested changes Oct 4, 2023
View reviewed changes
@Satrat Satrat requested a review from bfineran October 6, 2023 19:32
bfineran
bfineran previously approved these changes Oct 6, 2023
View reviewed changes
Copy link
Contributor

@bfineran bfineran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

latest commits look great. good to go from my end

bfineran
bfineran previously approved these changes Oct 9, 2023
View reviewed changes
@Satrat Satrat mentioned this pull request Oct 11, 2023
Merged
@Satrat Satrat requested review from natuan and bfineran October 11, 2023 14:35
@bfineran bfineran merged commit 6c3f054 into main Oct 11, 2023
@bfineran bfineran deleted the refactor_obcq branch October 11, 2023 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bfineran bfineran bfineran approved these changes

@anmarques anmarques Awaiting requested review from anmarques

@dsikka dsikka Awaiting requested review from dsikka

@rahul-tuli rahul-tuli Awaiting requested review from rahul-tuli

@natuan natuan Awaiting requested review from natuan

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Satrat @bfineran @natuan @anmarques @markurtz @rahul-tuli