Layerwise Sparsity Support for SparseGPT #1777

Satrat · 2023-10-19T14:03:20Z

Support for providing a list of sparsities to the SparseGPTModifier to enable layer-wise sparsity configuration. The length must match up with the number of target layers to compress, otherwise an error is thrown.

Testing

Example Recipe

test_stage:
  obcq_modifiers:
    SparseGPTModifier:
      sparsity: [
        0.1,
        0.2,
        0.3,
        0.4,
        0.5
      ]
      block_size: 128
      sequential_update: False
      quantize: False
      percdamp: 0.01
      prunen: 0
      prunem: 0
      compress_layers: [
        "model.decoder.layers.0",
        "model.decoder.layers.1",
        "model.decoder.layers.2",
        "model.decoder.layers.3",
        "model.decoder.layers.4"
      ]
      target_ids: ["attention_mask"]
      layer_prefix: "decoder"

To Run

python src/sparseml/transformers/sparsification/obcq/obcq.py facebook/opt-1.3b c4 --recipe example_recipe.yaml --eval wikitext2

Output

2023-10-19 09:56:37 sparseml.modifiers.obcq.pytorch INFO     
===== Compressing layer 1/5 to sparsity 0.1 =====
2023-10-19 09:56:37 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.k_proj...
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.47
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 201.29
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.v_proj...
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.37
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 48.39
2023-10-19 09:56:38 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.q_proj...
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 210.59
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.out_proj...
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 0.02
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc1...
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 189.64
2023-10-19 09:56:39 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc2...
2023-10-19 09:56:41 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 1.58
2023-10-19 09:56:41 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 1.16
2023-10-19 09:56:41 sparseml.modifiers.obcq.pytorch INFO     
===== Compressing layer 2/5 to sparsity 0.2 =====
2023-10-19 09:56:42 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.k_proj...
2023-10-19 09:56:42 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.39
2023-10-19 09:56:42 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 444.99
2023-10-19 09:56:42 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.v_proj...
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.37
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 87.64
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.q_proj...
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 348.68
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.out_proj...
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 0.02
2023-10-19 09:56:43 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc1...
2023-10-19 09:56:44 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:44 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 1242.68
2023-10-19 09:56:44 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc2...
2023-10-19 09:56:45 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 1.57
2023-10-19 09:56:45 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 0.08
2023-10-19 09:56:46 sparseml.modifiers.obcq.pytorch INFO     
===== Compressing layer 3/5 to sparsity 0.3 =====
2023-10-19 09:56:46 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.k_proj...
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.39
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 1737.14
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.v_proj...
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 567.80
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.q_proj...
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 1402.33
2023-10-19 09:56:47 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.out_proj...
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 2.21
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc1...
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 5431.26
2023-10-19 09:56:48 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc2...
2023-10-19 09:56:50 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 1.58
2023-10-19 09:56:50 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 6.83
2023-10-19 09:56:50 sparseml.modifiers.obcq.pytorch INFO     
===== Compressing layer 4/5 to sparsity 0.4 =====
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.k_proj...
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.39
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 5309.78
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.v_proj...
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.37
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 2328.10
2023-10-19 09:56:51 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.q_proj...
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 4082.74
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.out_proj...
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 12.44
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc1...
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 16928.26
2023-10-19 09:56:52 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc2...
2023-10-19 09:56:54 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 1.58
2023-10-19 09:56:54 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 26.82
2023-10-19 09:56:54 sparseml.modifiers.obcq.pytorch INFO     
===== Compressing layer 5/5 to sparsity 0.5 =====
2023-10-19 09:56:55 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.k_proj...
2023-10-19 09:56:55 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.39
2023-10-19 09:56:55 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 13890.12
2023-10-19 09:56:55 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.v_proj...
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.37
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 6673.23
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.q_proj...
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 12242.67
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing self_attn.out_proj...
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 44.67
2023-10-19 09:56:56 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc1...
2023-10-19 09:56:57 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 0.36
2023-10-19 09:56:57 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 45602.01
2023-10-19 09:56:57 sparseml.modifiers.obcq.utils.layer_compressor INFO     Compressing fc2...
2023-10-19 09:56:58 sparseml.modifiers.obcq.utils.sparsegpt INFO     time 1.58
2023-10-19 09:56:58 sparseml.modifiers.obcq.utils.sparsegpt INFO     error 85.04
Found cached dataset wikitext (/home/sadkins/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
2023-10-19 09:57:00 sparseml.modifiers.obcq.utils.helpers INFO     Evaluating perplexity...
2023-10-19 09:58:02 sparseml.modifiers.obcq.utils.helpers INFO     Perplexity: 14.667972

src/sparseml/modifiers/obcq/base.py

rahul-tuli

LGTM pending comment!

src/sparseml/modifiers/obcq/base.py

Sara Adkins added 2 commits October 19, 2023 09:48

layerwise sparsity support

52672ee

change example recipe back

eeb26f5

Satrat requested review from bfineran, dsikka, rahul-tuli and dbogunowicz October 19, 2023 14:03

horheynm reviewed Oct 19, 2023

View reviewed changes

src/sparseml/modifiers/obcq/base.py Outdated Show resolved Hide resolved

rahul-tuli reviewed Oct 19, 2023

View reviewed changes

src/sparseml/modifiers/obcq/base.py Outdated Show resolved Hide resolved

Sara Adkins added 4 commits October 19, 2023 10:21

unit tests

367dc09

clean regex check

8b412e9

Merge branch 'main' into layer-wise-sparsity

ae6096a

additional unit tests

33bd856

Satrat requested review from rahul-tuli and horheynm October 19, 2023 15:20

fix unit test

3187e8d

bfineran previously approved these changes Oct 20, 2023

View reviewed changes

src/sparseml/modifiers/obcq/base.py Show resolved Hide resolved

Sara Adkins added 2 commits October 23, 2023 12:23

Merge branch 'main' into layer-wise-sparsity

3b3fb2f

bug fix for mixed quantization

ad4159b

rahul-tuli previously approved these changes Oct 26, 2023

View reviewed changes

Sara Adkins added 2 commits October 26, 2023 16:57

Merge branch 'main' of github.com:neuralmagic/sparseml into main

972529c

Merge branch 'main' into layer-wise-sparsity

1c30bd8

Satrat dismissed stale reviews from rahul-tuli and bfineran via 1c30bd8 October 26, 2023 21:09

fix for fn rename

9dc4877

bfineran merged commit e20927f into main Oct 26, 2023

bfineran deleted the layer-wise-sparsity branch October 26, 2023 22:31

bfineran pushed a commit that referenced this pull request Nov 16, 2023

Layerwise Sparsity Support for SparseGPT (#1777)

f6ab063

bfineran pushed a commit that referenced this pull request Nov 16, 2023

Layerwise Sparsity Support for SparseGPT (#1777)

1f66cc9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Layerwise Sparsity Support for SparseGPT #1777

Layerwise Sparsity Support for SparseGPT #1777

Satrat commented Oct 19, 2023 •

edited

Loading

rahul-tuli left a comment

Layerwise Sparsity Support for SparseGPT #1777

Layerwise Sparsity Support for SparseGPT #1777

Conversation

Satrat commented Oct 19, 2023 • edited Loading

Testing

Example Recipe

To Run

Output

rahul-tuli left a comment

Choose a reason for hiding this comment

Satrat commented Oct 19, 2023 •

edited

Loading