Missing Windows support

[TimDettmers]

Currently, the library uses precompiled Linux binaries. I am unsure how compatible these are with standard PyTorch installs on Windows. It might be that the binaries need to be compiled against mingw32/64 to create functional binaries for Windows.

The most helpful would be a case where a person is able to compile from source and use the library. This will require altering the Makefile file. If this works, we can add instructions on compiling for Windows as a first step before doing a full-scale Windows deployment of binaries on pip.

Since I do not have a Windows machine, any help is wanted on this!

[DeXtmL]

I am able to compile the csrc part on Windows but fail to link (Visual Studio)
FYI, this is the error message. I'm no expert on cuda, so I'm not quite sure where it goes wrong.

To successfully compile on windows MSVC, some parts need to be patched as following:

1. rename pythonInterface.c to pythonInterface.cpp, or visual studio will try using a C compiler for it.

2. add one missing template instantiation like this: (in SIMD.h)
   

3. get unistd.h and getopt.h for windows

4. get pthread for windows

finally, this is just a build test, so I'm not using the Makefile.
That's all for now.
Tested on CUDA toolkit 11.6, windows 11

[DeXtmL]

After some really tedious debugging and tackling various hidden problems, I managed to compile the whole module.
This is the end result:
https://github.com/DeXtmL/bitsandbytes-win-prebuilt

The binaries are compiled against CUDAToolkit 11.6 and Visual Studio 2022.
I am able to make inferences nearly identical to "normal" fp16 version. So this is kind of a confirmation for "it works" shout-out. No vigorous testing was conducted though. @TimDettmers
Finally, the "cuda_setup" part of the source code is entirely incompatible with Windows, there are loads of hardcoded routines; so I used a quick makeshift patch instead of making it proper, that's also why I'm not posting my changes or making PR for now. If you are eager to test:

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

That should do the trick.

Hopefully this can help someone in the Windows territory; let's hope the official windows support come fast.

[TemporalLabsLLC-SOL]

Thank you. This might be the key.

[korakoe]

After some really tedious debugging and tackling various hidden problems, I managed to compile the whole module. This is the end result: https://github.com/DeXtmL/bitsandbytes-win-prebuilt

The binaries are compiled against CUDAToolkit 11.6 and Visual Studio 2022. I am able to make inferences nearly identical to "normal" fp16 version. So this is kind of a confirmation for "it works" shout-out. No vigorous testing was conducted though. @TimDettmers Finally, the "cuda_setup" part of the source code is entirely incompatible with Windows, there are loads of hardcoded routines; so I used a quick makeshift patch instead of making it proper, that's also why I'm not posting my changes or making PR for now. If you are eager to test:

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

That should do the trick.

Hopefully this can help someone in the Windows territory; let's hope the official windows support come fast.

could you provide a makefile for this?

[bmaltais]

After some really tedious debugging and tackling various hidden problems, I managed to compile the whole module. This is the end result: https://github.com/DeXtmL/bitsandbytes-win-prebuilt

The binaries are compiled against CUDAToolkit 11.6 and Visual Studio 2022. I am able to make inferences nearly identical to "normal" fp16 version. So this is kind of a confirmation for "it works" shout-out. No vigorous testing was conducted though. @TimDettmers Finally, the "cuda_setup" part of the source code is entirely incompatible with Windows, there are loads of hardcoded routines; so I used a quick makeshift patch instead of making it proper, that's also why I'm not posting my changes or making PR for now. If you are eager to test:

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

That should do the trick.

Hopefully this can help someone in the Windows territory; let's hope the official windows support come fast.

Where do you put the pre-built file to activate adam?

[davidvfx07]

After some really tedious debugging and tackling various hidden problems, I managed to compile the whole module. This is the end result: https://github.com/DeXtmL/bitsandbytes-win-prebuilt

The binaries are compiled against CUDAToolkit 11.6 and Visual Studio 2022. I am able to make inferences nearly identical to "normal" fp16 version. So this is kind of a confirmation for "it works" shout-out. No vigorous testing was conducted though. @TimDettmers Finally, the "cuda_setup" part of the source code is entirely incompatible with Windows, there are loads of hardcoded routines; so I used a quick makeshift patch instead of making it proper, that's also why I'm not posting my changes or making PR for now. If you are eager to test:

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

That should do the trick.

Hopefully this can help someone in the Windows territory; let's hope the official windows support come fast.

An easy way to always return libbitsandbytes_cuda116.dll would be to insert
if torch.cuda.is_available(): return 'libbitsandbytes_cuda116.dll', None, None, None, None
above
119: if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None

[davidvfx07]

After some really tedious debugging and tackling various hidden problems, I managed to compile the whole module. This is the end result: https://github.com/DeXtmL/bitsandbytes-win-prebuilt
The binaries are compiled against CUDAToolkit 11.6 and Visual Studio 2022. I am able to make inferences nearly identical to "normal" fp16 version. So this is kind of a confirmation for "it works" shout-out. No vigorous testing was conducted though. @TimDettmers Finally, the "cuda_setup" part of the source code is entirely incompatible with Windows, there are loads of hardcoded routines; so I used a quick makeshift patch instead of making it proper, that's also why I'm not posting my changes or making PR for now. If you are eager to test:

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

That should do the trick.
Hopefully this can help someone in the Windows territory; let's hope the official windows support come fast.

Where do you put the pre-built file to activate adam?

You put them in site-packages\bitsandbytes

[TinyBeeman]

Appreciate you doing this work, helped unblock me in a big way. I hope bitsandbytes supports Windows directly sooner rather than later, but this is a great stopgap.

[desis123]

@PinPointPing Thanks a lot, it just worked like a charm

[FloydianSound]

Same issue for me, if someone can compile cuda 11.8 binaries for me i can test them :)
using StableDiffusion-WebUI + Dreambooth, would love to give Adam a spin !

btw under Windows, the environment variable is CUDA_PATH and CUDA_PATH_V11_8

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
================================================================================
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
CUDA SETUP: Loading binary G:\Visions of Chaos\MachineLearning\Text To Image\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
Exception importing 8bit adam: argument of type 'WindowsPath' is not iterable
 Scheduler Loaded 
 Allocated: 2.3GB 
 Reserved: 2.4GB 

[FloydianSound]

after following some advice and making the edits in this thread, i got Adam to run on my CUDA 11.8 setup/ it does indeed work for me :)

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
================================================================================
CUDA SETUP: Loading binary G:\Visions of Chaos\MachineLearning\Text To Image\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda116.dll...
 Scheduler Loaded 
 Allocated: 0.3GB 
 Reserved: 0.4GB 

[geocine]

would you mind sharing your cuda 11.8 binary

[FloydianSound]

would you mind sharing your cuda 11.8 binary

using the libs cuda 11.6 from [(https://github.com/DeXtmL/bitsandbytes-win-prebuilt)]
and cuda 11.8 on windows 10

[FloydianSound]

@TimDettmers any updates on this ? 😄

[centerionware]

I just got this building with cmake.

First thing I did was make a directory called dependencies , then download https://github.com/GerHobbelt/pthread-win32 and extract the whole thing there so it's /project_root/dependencies/pthread-win32-main/pthread.h

(I lied, I didn't really do that first, did that last but I suggest anyone following this do it first to avoid the error)

I did apply the patch above by @DeXtmL (Does not compile without it, missing vec_t error).
As for unistd.h/getopt, I literally just commented out the #include <unistd.h> in the file it gave the error on and it now fully compiles so not sure anything relies on unistd.h (At least in windows with VC2019.. ), we'll see when I go to test the .dll (Didn't give any errors about getopt)

CMakeLists.txt
CMakeLists.txt **** (New files in next reply) ****

The first CMakeLists.txt is in the root, the second in csrc.
I do:

cd /project_root
mkdir build
cd build
cmake ..

Then can open the .sln, right click bitsandbytes hit build and it goes to town.

I imagine for linux the same cmake file would actually work just fine. Only thing I did special for windows was add the include path for the pthreads, adding an include path to a folder that doesn't exist probably won't hurt right?

GL

[centerionware]

Quick final follow up, it built fine in Debug with the above, i actually don't know quite how. For release mode I did actually have to build the pthread library (just another mkdir build, cmake .., open solution, build all in release mode), then slightly modify the cmake file. (Probably could have just done cmake .. ; cmake --build . -j4 --config Release ; to build pthread)
CMakeLists.txt

I don't know why it worked before in debug mode at all, because I had link_libraries wrong (don't have the -l in cmake in front), and for release mode had to fix that and include pthreadVC3.lib .. here's the final final from me CMakeLists.txt for the csrc folder
CMakeLists.txt
**EDIT
I got it to load up, but it says it compiled without GPU support so i'm still working on it.
**EDIT2
Still working on it. Added "add_compile_definitions(BUILD_CUDA)" then checked the resulting vc files and it does enable the BUILD_CUDA define, and in pythonInterface.cpp visual studio says BUILD_CUDA is defined, I can see where the cadam32bit_g32 is generated via that macro, but not quite sure why when it loads the .dll - lib.cadam32bit_g32 throws an attribute error.
**EDIT3
I finally got it to work. It took a couple hours (long compile times) but I finally got one that exports all symbols. the trick was putting the thing in a different cmake file ffs. The final two
root/CMakeLists.txt
CMakeLists.txt

root/csrc/CMakeLists.txt
CMakeLists.txt

mkdir build, cd build, cmake .. , cmake --build ./ -j4 --config Release , .dll is put into build/csrc/Release/bitsandbyts.dll

just for fun here's it running gpt-j-6b on a RTX3080 on windows 11 with cuda 113

import torch
import transformers

access_token = "hf_"

from transformers.models.gptj import GPTJForCausalLM


device = 'cuda' if torch.cuda.is_available() else 'cpu'

tokenizer = transformers.AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B", use_auth_token=access_token, device_map='auto')
gpt = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", use_auth_token=access_token, device_map='auto', load_in_8bit=True, low_cpu_mem_usage=True).to(device)

prompt = tokenizer("A cat sat on a mat", return_tensors='pt')
prompt = {key: value.to(device) for key, value in prompt.items()}
out = gpt.generate(**prompt, min_length=128, max_length=128, do_sample=True)
tokenizer.decode(out[0])

>>> tokenizer.decode(out[0])
"A cat sat on a mat, staring at me, his back legs tucked under him, tail swerving in quick little circles.\n\nI squatted next to him and leaned against the cold wooden wall. I'd come down here to feed the cat, but I'd been too tired and cold and my stomach still ached and my hands and feet were numb from spending the night in a tree. Besides, this was not my house, not my town, not my time. The cat stared at me, his green eyes the only proof he knew I was the intruder he was protecting.\n\nI wished I'd brought a blanket"

[james-things]

Quick final follow up...

You are a hero, thank you so much for posting this!

[ZQ-Dev8]

Quick final follow up...

Any chance you could hang some instructions somewhere to help others replicate the process? I'm trying to follow along but have run into some issues. A step-by-step would be awesome.

[centerionware]

From my not so great memory it's something like:

To build the bitsandbytes project for Windows, you will need two programs: cmake and nvcc. You can a build environment such as Visual Studio and Miniconda.

Open the command line interface (CLI) for your build environment. (Start menu/visual studio/ one of them consoles)
Activate your chosen environment(Miniconda) and install necessary packages. ( cuda-nvcc iirc? Probably a cuda environment like https://pytorch.org/get-started/locally/ )
Place cmake files in the right location.
Build pthreads (if necessary) using cmake (The same commands as below).
Edit1
( download https://github.com/GerHobbelt/pthread-win32 and extract the whole thing there so it's /project_root/dependencies/pthread-win32-main/pthread.h )
END EDIT1

Run the following commands: (Note -j4 means use 4 cores to build. If you don't have 4 cores, or you have a lot more, change that number.)

(Assuming on C:\ drive, if on other drive, change letter on first and second line appropriately.

C:
cd C:\PROJECT_ROOT\  ( or cd C:\PROJECT_ROOT\dependencies\pthread-win32-main )
mkdir build
cd build
cmake ..
cmake --build ./ -j4 --config Release

The resulting dll file will be in build/csrc/Release/bitsandbytes.dll.
Edit2
When it errors about unistd.h or getopt.h, open that file and comment out the #include - although a more proper way would to be change to detect _MSC_VER, if found just don't include unistd.h ( wouldn't test against WIN32 cause can be true in mingw, WSL,(etc) environment where using unistd.h would be required still, where _MSC_VER indicates the microsoft visual studio compiler version )
EG:

#ifndef _MSC_VER
#include <unistd.h>
#endif

END EDIT2

[healthyfat]

Is it possible to run bitsandbytes with 2060RTX 6GB on Windows 10?

[centerionware]

Is it possible to run bitsandbytes with 2060RTX 6GB on Windows 10?

I don't see why it wouldn't run on a 2060, just be aware it doesn't eliminate vram requirements, just reduces them. Still wouldn't be able to run chatgpt for example with it's 800GB+ 32-bit precision vram requirement (if had access to that). Any model that takes <24gb of vram in 32bit, or <12gb of vram in 16bit mode, should be able to fit in 6gb at 8bit.

[healthyfat]

@centerionware

Thanks for the answer.

I think I get confused about bitsandbytes. I was thinking it only works on Linux and repository such as kohya-ss/sd-scripts or bmaltais/kohya_ss won't work on Windows, because of the lack of compatibility and that is the reason I am not able to run them. But this is not true, as I manage to make them work on my machine eventually.

[FurkanGozukara]

why we still cant use on windows

this is 2023

[stoperro]

@FurkanGozukara we can use it on Windows as many people do (me including https://github.com/stoperro/bitsandbytes_windows) with this community effort. It's just less convenient as support is not yet merged into official branch and you need to compile it yourself.

Note that even Microsoft doesn't care about Windows support of their own AI tools (microsoft/DeepSpeed#2427), so having some support here when authors don't necessarily have Windows machine is heartening.

[FurkanGozukara]

@FurkanGozukara we can use it on Windows as many people do (me including https://github.com/stoperro/bitsandbytes_windows) with this community effort. It's just less convenient as support is not yet merged into official branch and you need to compile it yourself.

Note that even Microsoft doesn't care about Windows support of their own AI tools (microsoft/DeepSpeed#2427), so having some support here when authors don't necessarily have Windows machine is heartening.

I know that branch really good but really older commit

I am making tutorials for regular people. It is not an option for them to "compile it yourself"

I hope windows support gets added

[stoperro]

"compile it yourself" can be a blocker indeed, especially as this is lib for Python and compilation errors one may get will be from C++. I could find time to make a PR with whatever was done till now and polish it where necessary, though not sure if someone is doing it already.

[YoungPhlo]

If anyone else is still searching for a Windows solution and doesn't want to lose a few hours to the same issue just use this repo:
https://github.com/jllllll/bitsandbytes-windows-webui

The README even includes a pip install command and (as of 0.41.1) installs the newest version of bitsandbytes... only difference is that it's compatible with windows. Includes .dll instead of .so and cuda_setup\main.py works with us

[FurkanGozukara]

If anyone else is still searching for a Windows solution and doesn't want to lose a few hours to the same issue just use this repo: https://github.com/jllllll/bitsandbytes-windows-webui

The README even includes a pip install command and (as of 0.41.1) installs the newest version of bitsandbytes... only difference is that it's compatible with windows. Includes .dll instead of .so and cuda_setup\main.py works with us

wow very nice

[WilsonRoblesTafco]

If anyone else is still searching for a Windows solution and doesn't want to lose a few hours to the same issue just use this repo: https://github.com/jllllll/bitsandbytes-windows-webui

The README even includes a pip install command and (as of 0.41.1) installs the newest version of bitsandbytes... only difference is that it's compatible with windows. Includes .dll instead of .so and cuda_setup\main.py works with us

Legend it worked for me I was one of those looking for hours xDD

[swumagic]

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang）
1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui
2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

[GanizaniSitara]

pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Excellent! On Anaconda same thing applies - has to be root of the venv

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.