Skip to content

Latest commit

 

History

History
466 lines (326 loc) · 17.6 KB

README.md

File metadata and controls

466 lines (326 loc) · 17.6 KB

🖼️ Python Bindings for stable-diffusion.cpp

Simple Python bindings for @leejet's stable-diffusion.cpp library.

License: MIT PyPi version Downloads

This package provides:

  • Low-level access to C API via ctypes interface.
  • High-level Python API for Stable Diffusion and FLUX image generation.

Installation

Requirements:

  • Python 3.8+
  • C compiler
    • Linux: gcc or clang
    • Windows: Visual Studio or MinGW
    • MacOS: Xcode

To install the package, run:

pip install stable-diffusion-cpp-python

This will also build stable-diffusion.cpp from source and install it alongside this python package.

If this fails, add --verbose to the pip install to see the full cmake build log.

Installation Configuration

stable-diffusion.cpp supports a number of hardware acceleration backends to speed up inference as well as backend specific options. See the stable-diffusion.cpp README for a full list.

All stable-diffusion.cpp cmake build options can be set via the CMAKE_ARGS environment variable or via the --config-settings / -C cli flag during installation.

Environment Variables 
# Linux and Mac
CMAKE_ARGS="-DSD_CUDA=ON" pip install stable-diffusion-cpp-python
# Windows
$env:CMAKE_ARGS="-DSD_CUDA=ON"
pip install stable-diffusion-cpp-python
CLI / requirements.txt

They can also be set via pip install -C / --config-settings command and saved to a requirements.txt file:

pip install --upgrade pip # ensure pip is up to date
pip install stable-diffusion-cpp-python -C cmake.args="-DSD_CUDA=ON"
# requirements.txt

stable-diffusion-cpp-python -C cmake.args="-DSD_CUDA=ON"

Supported Backends

Below are some common backends, their build commands and any additional environment variables required.

Using CUDA (CUBLAS)

This provides BLAS acceleration using the CUDA cores of your Nvidia GPU. Make sure you have the CUDA toolkit installed. You can download it from your Linux distro's package manager (e.g. apt install nvidia-cuda-toolkit) or from here: CUDA Toolkit. You can check your installed CUDA toolkit version by running nvcc --version.

  • It is recommended you have at least 4 GB of VRAM.
CMAKE_ARGS="-DSD_CUDA=ON" pip install stable-diffusion-cpp-python
Using HIPBLAS (ROCm)

This provides BLAS acceleration using the ROCm cores of your AMD GPU. Make sure you have the ROCm toolkit installed and that you replace the -DAMDGPU_TARGETS= value with that of your GPU architecture. Windows users refer to docs/hipBLAS_on_Windows.md for a comprehensive guide and troubleshooting tips.

CMAKE_ARGS="-G Ninja -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DSD_HIPBLAS=ON -DCMAKE_BUILD_TYPE=Release -DAMDGPU_TARGETS=gfx1101" pip install stable-diffusion-cpp-python
Using Metal

Using Metal makes the computation run on the GPU. Currently, there are some issues with Metal when performing operations on very large matrices, making it highly inefficient at the moment. Performance improvements are expected in the near future.

CMAKE_ARGS="-DSD_METAL=ON" pip install stable-diffusion-cpp-python
Using Vulkan Install Vulkan SDK from https://www.lunarg.com/vulkan-sdk/. 
CMAKE_ARGS="-DSD_VULKAN=ON" pip install stable-diffusion-cpp-python
Using SYCL

Using SYCL makes the computation run on the Intel GPU. Please make sure you have installed the related driver and Intel® oneAPI Base toolkit before start. More details and steps can refer to llama.cpp SYCL backend.

# Export relevant ENV variables
source /opt/intel/oneapi/setvars.sh

# Option 1: Use FP32 (recommended for better performance in most cases)
CMAKE_ARGS="-DSD_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx" pip install stable-diffusion-cpp-python

# Option 2: Use FP16
CMAKE_ARGS="-DSD_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_SYCL_F16=ON" pip install stable-diffusion-cpp-python
Using Flash Attention

Enabling flash attention reduces memory usage by at least 400 MB. At the moment, it is not supported when CUDA (CUBLAS) is enabled because the kernel implementation is missing.

CMAKE_ARGS="-DSD_FLASH_ATTN=ON" pip install stable-diffusion-cpp-python
Using OpenBLAS 
CMAKE_ARGS="-DGGML_OPENBLAS=ON" pip install stable-diffusion-cpp-python
Using MUSA

This provides BLAS acceleration using the MUSA cores of your Moore Threads GPU. Make sure to have the MUSA toolkit installed.

CMAKE_ARGS="-DCMAKE_C_COMPILER=/usr/local/musa/bin/clang -DCMAKE_CXX_COMPILER=/usr/local/musa/bin/clang++ -DSD_MUSA=ON -DCMAKE_BUILD_TYPE=Release" pip install stable-diffusion-cpp-python

Upgrading and Reinstalling

To upgrade and rebuild stable-diffusion-cpp-python add --upgrade --force-reinstall --no-cache-dir flags to the pip install command to ensure the package is rebuilt from source.

High-level API

The high-level API provides a simple managed interface through the StableDiffusion class.

Below is a short example demonstrating how to use the high-level API to generate a simple image:

Text to Image

from stable_diffusion_cpp import StableDiffusion

def callback(step: int, steps: int, time: float):
    print("Completed step: {} of {}".format(step, steps))

stable_diffusion = StableDiffusion(
      model_path="../models/v1-5-pruned-emaonly.safetensors",
      # wtype="default", # Weight type (e.g. "q8_0", "f16", etc) (The "default" setting is automatically applied and determines the weight type of a model file)
)
output = stable_diffusion.txt_to_img(
      prompt="a lovely cat",
      width=512, # Must be a multiple of 64
      height=512, # Must be a multiple of 64
      progress_callback=callback,
      # seed=1337, # Uncomment to set a specific seed (use -1 for a random seed)
)
output[0].save("output.png") # Output returned as list of PIL Images

With LoRA (Stable Diffusion)

You can specify the directory where the lora weights are stored via lora_model_dir. If not specified, the default is the current working directory.

  • LoRA is specified via prompt, just like stable-diffusion-webui. (e.g. <lora:marblesh:1>)
  • LoRAs will not work when using quantized models. You must instead use a full precision .safetensors model.

Here's a simple example:

from stable_diffusion_cpp import StableDiffusion

stable_diffusion = StableDiffusion(
      model_path="../models/v1-5-pruned-emaonly.safetensors",
      lora_model_dir="../models/", # This should point to folder where LoRA weights are stored (not an individual file)
)
output = stable_diffusion.txt_to_img(
      prompt="a lovely cat<lora:marblesh:1>",
)
  • The lora_model_dir argument is used in the same way for FLUX image generation.

FLUX Image Generation

FLUX models should be run using the same implementation as the stable-diffusion.cpp FLUX documentation where the diffusion_model_path argument is used in place of the model_path. The clip_l_path, t5xxl_path, and vae_path arguments are also required for inference to function.

Download the weights from the links below:

from stable_diffusion_cpp import StableDiffusion

stable_diffusion = StableDiffusion(
    diffusion_model_path="../models/flux1-schnell-q3_k.gguf", # In place of model_path
    clip_l_path="../models/clip_l.safetensors",
    t5xxl_path="../models/t5xxl_fp16.safetensors",
    vae_path="../models/ae.safetensors",
    vae_decode_only=True, # Can be True if we dont use img_to_img
)
output = stable_diffusion.txt_to_img(
      prompt="a lovely cat holding a sign says 'flux.cpp'",
      sample_steps=4,
      cfg_scale=1.0, # a cfg_scale of 1 is recommended for FLUX
      sample_method="euler", # euler is recommended for FLUX
)

With LoRA (FLUX)

LoRAs can be used with FLUX models in the same way as Stable Diffusion models (as shown above).

Note that:

SD3.5 Image Generation

Download the weights from the links below:

from stable_diffusion_cpp import StableDiffusion

stable_diffusion = StableDiffusion(
    model_path="../models/sd3.5_large.safetensors",
    clip_l_path="../models/clip_l.safetensors",
    clip_g_path="../models/clip_g.safetensors",
    t5xxl_path="../models/t5xxl_fp16.safetensors",
)
output = stable_diffusion.txt_to_img(
      prompt="a lovely cat holding a sign says 'Stable diffusion 3.5 Large'",
      height=1024,
      width=1024,
      cfg_scale=4.5,
      sample_method="euler",
)

Image to Image

from stable_diffusion_cpp import StableDiffusion

INPUT_IMAGE = "../input.png"
# INPUT_IMAGE = Image.open("../input.png") # or alternatively, pass as PIL Image

stable_diffusion = StableDiffusion(model_path="../models/v1-5-pruned-emaonly.safetensors")

output = stable_diffusion.img_to_img(
      prompt="blue eyes",
      image=INPUT_IMAGE, # Note: The input image will be automatically resized to the match the width and height arguments (default: 512x512)
      strength=0.4,
)

Inpainting

from stable_diffusion_cpp import StableDiffusion

# Note: Inpainting with a base model gives poor results. A model fine-tuned for inpainting is recommended.
stable_diffusion = StableDiffusion(model_path="../models/v1-5-pruned-emaonly.safetensors")

output = stable_diffusion.img_to_img(
      prompt="blue eyes",
      image="../input.png",
      mask_image="../mask.png", # A grayscale image where 0 is masked and 255 is unmasked
      strength=0.4,
)

PhotoMaker

You can use PhotoMaker to personalize generated images with your own ID.

NOTE, currently PhotoMaker ONLY works with SDXL (any SDXL model files will work). The VAE in SDXL encounters NaN issues. You can find a fixed VAE here: SDXL VAE FP16 Fix.

Download PhotoMaker model file (in safetensor format) here. The official release of the model file (in .bin format) does not work with stablediffusion.cpp.

In prompt, make sure you have a class word followed by the trigger word "img" (hard-coded for now). The class word could be one of "man, woman, girl, boy". If input ID images contain asian faces, add Asian before the class word.

from stable_diffusion_cpp import StableDiffusion

stable_diffusion = StableDiffusion(
      model_path="../models/sdxl.vae.safetensors",
      vae_path="../models/sdxl.vae.safetensors",
      stacked_id_embed_dir="../models/photomaker-v1.safetensors",
      # keep_vae_on_cpu=True,  # If on low memory GPUs (<= 8GB), setting this to True is recommended to get artifact free images
)

output = stable_diffusion.txt_to_img(
      cfg_scale=5.0, # a cfg_scale of 5.0 is recommended for PhotoMaker
      height=1024,
      width=1024,
      style_strength=10,  # (0-100)% Default is 20 and 10-20 typically gets good results. Lower ratio means more faithfully following input ID (not necessarily better quality).
      sample_method="euler",
      prompt="a man img, retro futurism, retro game art style but extremely beautiful, intricate details, masterpiece, best quality, space-themed, cosmic, celestial, stars, galaxies, nebulas, planets, science fiction, highly detailed",
      negative_prompt="realistic, photo-realistic, worst quality, greyscale, bad anatomy, bad hands, error, text",
      input_id_images_path="../assets/newton_man",
)

PhotoMaker Version 2

PhotoMaker Version 2 (PMV2) has some key improvements. Unfortunately it has a very heavy dependency which makes running it a bit involved in SD.cpp.

Running PMV2 Requires running a python script face_detect.py (found here) to obtain id_embeds for the given input images.

python face_detect.py <input_image_dir>

An id_embeds.safetensors file will be generated in input_images_dir.

Note: this step is only needed to run once; the same id_embeds can be reused

  • Run the same command as in version 1 but replacing photomaker-v1.safetensors with photomaker-v2.safetensors.

    You can download photomaker-v2.safetensors from here.

  • All the other parameters from Version 1 remain the same for Version 2.

Listing GGML model and RNG types, schedulers and sample methods

Access the GGML model and RNG types, schedulers, and sample methods via the following maps:

from stable_diffusion_cpp import GGML_TYPE_MAP, RNG_TYPE_MAP, SCHEDULE_MAP, SAMPLE_METHOD_MAP

print("GGML model types:", list(GGML_TYPE_MAP))
print("RNG types:", list(RNG_TYPE_MAP))
print("Schedulers:", list(SCHEDULE_MAP))
print("Sample methods:", list(SAMPLE_METHOD_MAP))

Other High-level API Examples

Other examples for the high-level API (such as upscaling and model conversion) can be found in the tests directory.

Low-level API

The low-level API is a direct ctypes binding to the C API provided by stable-diffusion.cpp. The entire low-level API can be found in stable_diffusion_cpp/stable_diffusion_cpp.py and directly mirrors the C API in stable-diffusion.h.

Below is a short example demonstrating how to use the low-level API:

import stable_diffusion_cpp as sd_cpp
import ctypes
from PIL import Image

img = Image.open("path/to/image.png")
img_bytes = img.tobytes()

c_image = sd_cpp.sd_image_t(
      width=img.width,
      height=img.height,
      channel=channel,
      data=ctypes.cast(
            (ctypes.c_byte * len(img_bytes))(*img_bytes),
            ctypes.POINTER(ctypes.c_uint8),
      ),
) # Create a new C sd_image_t

img = sd_cpp.upscale(
      self.upscaler,
      image_bytes,
      upscale_factor,
) # Upscale the image

Development

To get started, clone the repository and install the package in editable / development mode.

git clone --recurse-submodules https://github.com/william-murray1204/stable-diffusion-cpp-python.git
cd stable-diffusion-cpp-python

# Upgrade pip (required for editable mode)
pip install --upgrade pip

# Install with pip
pip install -e .

Now you can make changes to the code within the stable_diffusion_cpp directory and test them in your python environment.

References

License

This project is licensed under the terms of the MIT license. See LICENSE for details.