Public Backend API

[carson-katri]

Closes #604

The API is slightly different than it that issue. You can see an example of a backend in diffusers_backend.py, which is the default Dream Textures backend.

class MyCustomBackend(Backend):
    name = "My Backend Name"
    description = "A short description of this backend"

    custom_optimization: BoolProperty(name="My Custom Optimization")

    def list_models(self, context) -> List[Model]:
        return [...]

    def list_schedulers(self, context) -> List[str]:
        return [...]

    def generate(self, task: Task, model: Model, prompt: Prompt, size: Tuple[int, int] | None, seed: int, steps: int, guidance_scale: float, scheduler: str, seamless_axes: SeamlessAxes, step_preview_mode: StepPreviewMode, iterations: int, step_callback: StepCallback, callback: Callback):
        match task:
            case PromptToImage():
                ...
            case ImageToImage(image=image, strength=strength, fit=fit):
                ...
            case Inpaint(image=image, fit=fit, strength=strength, mask_source=mask_source, mask_prompt=mask_prompt, confidence=confidence):
                ...
            case DepthToImage(depth=depth, image=image, strength=strength):
                ...
            case Outpaint(image=image, origin=origin):
                ...
            case _:
                raise NotImplementedError()

    def draw_speed_optimizations(self, layout, context):
        layout.prop(self, "custom_optimization")

A backend could be as simple as a single file Blender add-on, so I'm considering adding a folder to this repo to contain scripts for additional backends that would be maintained by different community members. This could contain backends for sources like Automatic1111, DreamStudio, InvokeAI, StableHorde, etc.

[NullSenseStudio]

Looks like you've accidentally pushed the packaged CoreML backend in 47118a0

[carson-katri]

Should be fixed now.

[carson-katri]

Basic image generation is working. Here's a list of features that need to be tested/updated:

• Prompt to image
• Batched generation
• Generation iterations
• Cancellation
• Image to image
• Inpainting
• Depth to image
• ControlNet
• Texture Projection
• Nodes
• Render Pass
• Upscaling

[Made-0f-Ice]

I just tried to test this tool, I have "automatic1111" server running on 127.0.0.1:7860 (working via web-interface), but on the "Dream" panel in Blender I see only "Huggingface Diffusers" backend and no available models… (I'm on Debian Linux)
Does this approch yet to implement or I've made some error configuring?
Thank you in advance. I'm not hurry or so, I'm just interested in this feature.

[carson-katri]

This PR doesn’t add an Automatic1111 backend, it just makes it possible for one to be created in the future.

Contributions for new backends would be welcome once this is merged!

[macronomicus]

So you dont plan to use ckpt/safetensors files, but rather allow for other backend? Is there performance loss with those types? Or just too much refactoring? I've read a few posts & was curious.

I had dreamtexture installed on last pc but have yet to dive back in on new setup, im glad you're still progressing & its looking great, blender is by far my favorite app. It'd be nice to use our model files from automatic1111, but regardless Dreamtextures stands on its own without it. I am fine with the two being separate, its simple enough to jump back & forth between them with stuff. But you're showing such great & deeper integrations its very tempting!

I imagine blender official will get in on the ai stuff too, its all still fast moving but at some point you might get some official specialized blender api stuff to work with too. Im keeping an eye out for when they start talking more ai.

[carson-katri]

@macronomicus Diffusers added a new from_single_file loader, which can load .ckpt/safetensor files. With that we may be able to easily support separate lookup directories for single-file models.

[NullSenseStudio]

I'll take a crack at adding from_single_file since I'm already handling related code in #703

[NullSenseStudio]

Took a bit more than I thought to fix VAE tiling. It's a bit of a mess with how it's juggling the image between torch and numpy, but it's functional for now. I'll get that in better order soon since I have some changes planned for VAE tiling.

Outpainting sort of works, but the outpainted regions remain black. Seems to be caused by changes made to the inpaint pipeline. You can color a portion of an image then mark that area for inpainting, and it will still resemble that color after inpainting now.

[carson-katri]

Could the strength value be incorrect, so it takes some of the black/previous image into account?

[NullSenseStudio]

That was it!

[TimWolfram]

@carson-katri

This PR doesn’t add an Automatic1111 backend, it just makes it possible for one to be created in the future.

Contributions for new backends would be welcome once this is merged!

Has nobody implemented an A1111 backend yet? I can't find any mentions of it existing but considering it's been a year now since the backend API and A1111 is the most popular way to use stable diffusion, I'd assume someone would have implemented it by now. If not, I can try to implement it myself but I thought I might ask here before wasting my time.

[carson-katri]

@TimWolfram As far as I know, no one has built an A1111 backend yet. But I'd be happy to answer any questions you have if you build one.