Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry pick v1.17.0 #1964

Merged
merged 58 commits into from
Aug 10, 2024
Merged
Changes from 1 commit
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
a9df7da
[SW-184941] INC CI, CD and Promotion
Yantom1 May 19, 2024
14f031e
[SW-183320]updated setup.py
RonBenMosheHabana Jun 6, 2024
ee7e5c8
[SW-177474] add HQT FP8 porting code
zyuwen-habana May 22, 2024
ca1444b
[SW-189361] Fix white list extend
ulivne Jun 19, 2024
dfec104
[SW-191317] Raise exception according to hqt config object
ulivne Jul 3, 2024
216d94b
[SW-184714] Port HQT code into INC
ulivne Jul 6, 2024
96bffd9
[SW-184714] Add internal folder to fp8 quant
ulivne Jul 7, 2024
90838a4
[SW-177468] Removed unused code + cleanup
HolyFalafel Jun 20, 2024
b76f002
Fix errors in regression_detection
smarkovichgolan Jul 3, 2024
90b10d3
[SW-187731] Save orig module as member of patched module
ulivne Jun 23, 2024
62026c2
[SW-190899] Install packages according to configuration
ulivne Jul 8, 2024
4a0d704
[SW-184689] use finalize_calibration intrenaly for one step flow
ulivne Jul 9, 2024
dfa8833
[SW-191945] align requirement_pt.txt in gerrit INC with Github INC
Jul 9, 2024
604d664
[SW-192358] Remove HQT reference in INC
ulivne Jul 11, 2024
a493d7c
[SW-191415] update fp8 maxAbs observer using torch.copy_
dudilester Jul 11, 2024
8803808
[SW-184943] Enhance INC WOQ model loading
zyuwen-habana Jun 13, 2024
a14c5c6
[SW-190303] Implement HPUWeightOnlyLinear class in INC
Yantom1 Jul 9, 2024
35b5bd2
[SW-192809] fix json_file bug when instantiating FP8Config class
zyuwen-habana Jul 15, 2024
f45e0aa
[SW-192931] align setup.py with github INC and remove fp8_convert
Jul 16, 2024
165ce63
[SW-192917] Update all HQT logic files with pre-commit check
Jul 16, 2024
853bb8d
update docstring
yuwenzho Jul 26, 2024
86d8dfa
add fp8 example and document (#1639)
xin3he Jul 29, 2024
051fee8
Update settings to be compatible with gerrit
xin3he Jul 30, 2024
2737870
enhance ut
yuwenzho Jul 30, 2024
402c16f
move fp8 sample to helloworld folder
yuwenzho Aug 1, 2024
34855fa
update torch version of habana docker
Aug 6, 2024
8c57adb
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 6, 2024
7fbceaf
update readme demo
Aug 6, 2024
3e5552e
update WeightOnlyLinear to INCWeightOnlyLinear
Aug 6, 2024
e200364
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 6, 2024
2496153
Merge branch 'master' into cherry_pick_v1.17.0
xin3he Aug 6, 2024
ec45a27
add docstring for FP8Config
Aug 6, 2024
ab212c8
fix pylint
Aug 6, 2024
bfde945
Merge branch 'master' into cherry_pick_v1.17.0
xin3he Aug 7, 2024
d42660a
update fp8 test scripts
chensuyue Aug 7, 2024
b675220
Merge branch 'cherry_pick_v1.17.0' of https://github.com/intel/neural…
chensuyue Aug 7, 2024
1b01e61
delete deps
chensuyue Aug 8, 2024
60b98f8
update container into v1.17.0
chensuyue Aug 8, 2024
8437a65
update docker version
Aug 8, 2024
7fadea9
update pt ut
chensuyue Aug 8, 2024
8fe70fe
Merge branch 'cherry_pick_v1.17.0' of https://github.com/intel/neural…
chensuyue Aug 8, 2024
5a055c1
add lib path
chensuyue Aug 9, 2024
45406db
fix dir issue
Aug 9, 2024
cb4735b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 9, 2024
f5b74b9
Merge branch 'cherry_pick_v1.17.0' of https://github.com/intel/neural…
chensuyue Aug 9, 2024
62addf2
update fp8 test scope
chensuyue Aug 9, 2024
f569f21
Merge branch 'master' into cherry_pick_v1.17.0
chensuyue Aug 9, 2024
0df24a6
fix typo
Aug 9, 2024
93e4aa0
update fp8 test scope
chensuyue Aug 9, 2024
a586ae3
Merge branch 'cherry_pick_v1.17.0' of https://github.com/intel/neural…
chensuyue Aug 9, 2024
f087298
update pre-commit-ci
chensuyue Aug 9, 2024
a858bab
work around for hpu
Aug 9, 2024
9ab10e6
fix UT
Aug 9, 2024
6642962
fix parameter
chensuyue Aug 9, 2024
02d490e
omit some test
chensuyue Aug 9, 2024
5e02321
update main page example to llm loading
Aug 9, 2024
084e244
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 9, 2024
3763723
fix autotune
Aug 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update main page example to llm loading
Signed-off-by: xinhe3 <[email protected]>
  • Loading branch information
xinhe3 committed Aug 9, 2024
commit 5e023210a01b57bd0fb5cc7ff18d820da24a44ca
25 changes: 11 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,27 +93,24 @@ calib_func(model)
model = convert(model)
```

### [Weight-Only Quantization (LLMs)](./examples/3.x_api/pytorch/nlp/huggingface_models/language-modeling/quantization/weight_only/)
### Weight-Only Large Language Model Loading (LLMs)

Following example code demonstrates Weight-Only Quantization on LLMs, it supports Intel CPU, Intel Gaudi2 AI Accelerator, Nvidia GPU, best device will be selected automatically.
Following example code demonstrates weight-only large language model loading on Intel Gaudi2 AI Accelerator.

```python
from transformers import AutoModelForCausalLM
from neural_compressor.torch.quantization import prepare, convert, AutoRoundConfig

model_name = "EleutherAI/gpt-neo-125m"
model = AutoModelForCausalLM.from_pretrained(model_name)

quant_config = AutoRoundConfig()
model = prepare(model, quant_config)
# customer defined calibration
run_fn(model) # calibration
model = convert(model)
from neural_compressor.torch.quantization import load
model_name = "TheBloke/Llama-2-7B-GPTQ"
model = load(
model_name_or_path=model_name,
format="huggingface",
device="hpu",
torch_dtype=torch.bfloat16,
)
```

**Note:**

To try INT4 model inference, please directly use [Intel Extension for Transformers](https://github.com/intel/intel-extension-for-transformers), which leverages Intel Neural Compressor for model quantization.
Intel Neural Compressor will convert the model format from auto-gptq to hpu format on the first load and save hpu_model.safetensors to the local cache directory for the next load. So it may take a while to load for the first time.

## Documentation

Expand Down
Loading