Optimize quantized masked fill

[lucasavila00]

Closes #161

This MR:

Prompt T/s = 70.70707
Completion T/s = 61.244022

Master:

Prompt T/s = 59.82906
Completion T/s = 61.538464

[github-actions]

Code Metrics Report

  ───────────────────────────────────────────────────────────────────────────────
Language                 Files     Lines   Blanks  Comments     Code Complexity
───────────────────────────────────────────────────────────────────────────────
Rust                        60     19995     1439       821    17735       1128
───────────────────────────────────────────────────────────────────────────────
Total                       60     19995     1439       821    17735       1128
───────────────────────────────────────────────────────────────────────────────
Estimated Cost to Develop 53,182
Estimated Schedule Effort 10.982569 months
Estimated People Required 4.474866
───────────────────────────────────────────────────────────────────────────────
Processed 676190 bytes, 0.676 megabytes (SI)
───────────────────────────────────────────────────────────────────────────────
  

[lucasavila00]

It reduces completion speed too much.

I wonder if that's because of bad measurements or real. I'm profiling it.

[lucasavila00]

It was a measurement issue, fixed by #163

[EricLBuehler]

@lucasavila00, I just merged #163. Can you please update the benchmarks?

[lucasavila00]

@EricLBuehler I just did.

I'm having a hard time measuring small changes, significantly.

I ran it a bunch of times and completion speed is unchanged. Prompt speed improved.

I think we need something like https://github.com/ggerganov/llama.cpp/blob/master/examples/llama-bench/README.md instead of the --prompt setup to benchmark.

I'm also using my local GPU, with bad cooling and other processes using it etc

The llama-bench setup runs a but of repetitions to remove this noise

I created an issue for it #164

[EricLBuehler]

That is a great idea, I think that we should add that in light of future improvements. I'll merge this as I think the performance gains are very significant!