Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add snapshot tests #31

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Conversation

DhruvDh
Copy link

@DhruvDh DhruvDh commented Mar 7, 2024

This draft PR adds snapshot tests. I am trying to incrementally rewrite operations in KernelAbstractions.jl, and am using these tests to ensure consistent outputs at temp 0.

Currently, it doesn't run on GPUs, for that I think a method that moves weights to a chosen backend would be needed.
Right now, I just want to write correct kernels that run on the CPU and produce outputs consistent to the original version.

Here is a kernel for rmsnorm - DhruvDh@1d49d42. It makes the whole thing some 2-5% slower but produces correct outputs. This way, we can try incrementally writing kernels for operations and later look at the results and decide if GPU acceleration is needed, how to maintain readability, etc.

I can also keep a draft PR open where we do this incremental addition of kernels

I am also having trouble with formatting, if a .JuliaFormatter.toml can be added to the project, that would help a lot.

@cafaxo
Copy link
Owner

cafaxo commented Mar 22, 2024

GPU support would be awesome. I am not sure if testing for exact reproduction is that useful since language models are very sensitive to rounding/quantization errors.
Even llama.cpp produces different outputs depending on the backend (see ggerganov/llama.cpp#4755).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants