Skip to content

Commit

Permalink
Add documentation on various features (carson-katri#252)
Browse files Browse the repository at this point in the history
  • Loading branch information
@carson-katri @JasonHoku
carson-katri authored and JasonHoku committed Dec 20, 2022
1 parent e802bf6 commit f96197c
Show file tree
Hide file tree
Showing 25 changed files with 116 additions and 138 deletions.
75 changes: 19 additions & 56 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,84 +1,47 @@
# Dream Textures

> Stable Diffusion built-in to the Blender shader editor.
![Dream Textures, subtitle: Stable Diffusion built-in to Blender](readme_assets/banner.png)

* Create textures, concept art, background assets, and more with a simple text prompt
* Use the 'Seamless' option to create textures that tile perfectly with no visible seam
* Quickly create variations on an existing texture
* Experiment with AI image generation
* Re-style animations with the Cycles render pass
* Run the models on your machine to iterate without slowdowns from a service

## Installation
Download the [latest release](https://github.com/carson-katri/dream-textures/releases/tag/0.0.6) and follow the instructions there to get up and running.
# Installation
Download the [latest release](https://github.com/carson-katri/dream-textures/releases/tag/0.0.7) and follow the instructions there to get up and running.

> On macOS, it is possible you will run into a quarantine issue with the dependencies. To work around this, run the following command in the app `Terminal`: `xattr -r -d com.apple.quarantine ~/Library/Application\ Support/Blender/3.3/scripts/addons/dream_textures/.python_dependencies`. This will allow the PyTorch `.dylib`s and `.so`s to load without having to manually allow each one in System Preferences.
## Usage

| Enter a prompt | Generate a unique texture in a few seconds |
| -------------- | ------------------------------------------ |
| ![](readme_assets/brick_wall_texture_prompt.png) | ![](readme_assets/brick_wall_texture.png) |
| ![](readme_assets/uneven_stone_path_prompt.png) | ![](readme_assets/uneven_stone_path.png) |

| Take an existing texture | Modify it with a text prompt |
| ------------------------ | ---------------------------- |
| ![](readme_assets/marble.jpg) | ![](readme_assets/marble_brick_wall_texture.png) |

Dream Textures provides most of the configuration options available when using Stable Diffusion directly.
# Usage

1. Open the Shader or Image editor
2. Select the 'Dream Textures' tab in the side panel. You may need to press 'N' to open this panel.
3. Enter a prompt. There are some presets you can use to automatically fine tune your result:
1. *Texture* - simply adds the word *texture* to the end of your prompt to help the model create a texture.
2. *Photography* - provides many options for specifying a photorealistic result.
3. *Concept Art* - create environments, characters, and more
4. *Custom* - passes the prompt directly to SD with no additional keywords.
4. Click 'Generate'. The generation runs asynchronously, so Blender won't completely freeze up.
5. If you have an Image Editor open, the current progress of the model will appear there with the current step # as the image name (for example, *Step 1/25*).
6. After the image finishes generating, a new Image Texture node will be added to the open shader editor. The image is packed, so you may want to export it to a separate file.
Here's a few quick guides:

> The name of the generated image is the random seed used to create it. If you want to use the same seed, copy the name of the image into the 'Seed' parameter under the 'Advanced' section.
## [Image Generation](docs/IMAGE_GENERATION.md)
Create textures, concept art, and more with text prompts. Learn how to use the various configuration options to get exactly what you're looking for.

## Init Image
Use an init image to create a variation of an existing texture.
## [Inpainting](docs/INPAINTING.md)
Fix up images and convert existing textures into seamless ones automatically.

![](readme_assets/init_image.png)
## [Render Pass](docs/RENDER_PASS.md)
Perform style transfer and create novel animations with Stable Diffusion as a post processing step.

1. Enable 'Init Image'
2. Select an image from your filesystem
> Currently, this doesn't work with packed textures. So select an image from your file system, or export the packed texture to a file and select that.
3. You can change the strength and specify if it should fit to the set size, or keep the original image size.
4. Enter a prompt that tells SD how to modify the image.
5. Click 'OK' and wait for it to finish.
## [AI Upscaling](docs/AI_UPSCALING.md)
Convert your low-res generations to 2K, 4K, and higher with Real-ESRGAN built-in.

## Advanced Configuration
These options match those of SD.
## [History](docs/HISTORY.md)
Recall, export, and import history entries for later use.

![](readme_assets/advanced_configuration.png)

* *Full Precision* - more VRAM intensive, but creates better results. Disable this if you have a lower-end graphics card.
* *Seed* - a random seed if set to `-1`, otherwise it uses the value you input. You can copy the random seed from the generated image's name to use the same seed again.
* *Iterations* - how many images to generate. This doesn't quite work yet, so keep it at `1`.
* *Steps* - higher is generally better, but I wouldn't recommend going past 50 until you've refined your prompt.
* *CFG Scale* - essentially how much the model takes your prompt into consideration. `7.5` is a good default for a collaborative experience, but if you want to force the model to follow instructions crank it up to `15-20`.
* *Sampler* - KLMS is a good speed/quality default, DDIM is generally faster, and you'll get different results with each so play around with it.

## Compatibility
# Compatibility
Dream Textures has been tested with CUDA and Apple Silicon GPUs.

If you have an issue with a supported GPU, please create an issue.

## Future Directions
* Other image map types (normal, roughness, displacement, etc.) using a new LDM checkpoint and vocabulary.
* AI upscaling and face fixing with ESRGAN and GFPGAN

## Contributing
# Contributing
After cloning the repository, there a few more steps you need to complete to setup your development environment:
1. Install submodules:
```sh
git submodule update --init --recursive
```
2. I recommend the [Blender Development](https://marketplace.visualstudio.com/items?itemName=JacquesLucke.blender-development) extension for VS Code.
2. I recommend the [Blender Development](https://marketplace.visualstudio.com/items?itemName=JacquesLucke.blender-development) extension for VS Code for debugging. If you just want to install manually though, you can put the `dream_textures` repo folder in Blender's addon directory.
3. After running the local add-on in Blender, setup the model weights like normal.
4. Install dependencies locally
* Open Blender's preferences window
Expand Down
24 changes: 7 additions & 17 deletions docs/AI_UPSCALING.md
View file
Original file line number Diff line number Diff line change
@@ -1,24 +1,14 @@
# AI Upscaling
Use the Stable Diffusion upscaler to increase images 4x in size while retaining detail. You can guide the upscaler with a text prompt.
Real-ESRGAN is built-in to the addon to upscale any generated image 2-4x the original size.

> Upscaling uses the model `stabilityai/stable-diffusion-4x-upscaler`. This model will automatically be downloaded when the operator is first run.
> You must setup the Real-ESRGAN weights separately from the Stable Diffusion weights before upscaling. The *AI Upscaling* panel contains instructions for downloading them.
Use the AI Upscaling panel to access this tool.

1. Open the image to upscale in an *Image Editor* space
1. Open the image to upscale an *Image Editor* space
2. Expand the *AI Upscaling* panel, located in the *Dream* sidebar tab
3. Type a prompt to subtly influence the generation.
4. Optionally configure the tile size, blend, and other advanced options.

![](assets/ai_upscaling/panel.png)

The upscaled image will be opened in the *Image Editor*. The image will be named `Source Image Name (Upscaled)`.

## Tile Size
Due to the large VRAM consumption of the `stabilityai/stable-diffusion-4x-upscaler` model, the input image is split into tiles with each tile being upscaled independently, then stitched back together.
3. Choose a target size and click *Upscale*

The default tile size is 128x128, which will result in an image of size 512x512. These 512x512 images are stitched back together to form the final image.
> Some GPUs will require Full Precision to be enabled.
You can increase or decrease the tile size depending on your GPU's capabilities.
![A screenshot of the AI Upscaling panel set to 2 times target size and full precision enabled](../readme_assets/upscaling.png)

The *Blend* parameter controls how much overlap is included in the tiles to help reduce visible seams.
The upscaled image will be opened in the *Image Editor*. The image will be named `Source Image Name (Upscaled)`.
4 changes: 2 additions & 2 deletions docs/HISTORY.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@ You can also export the selected prompt to JSON for later import. This is a more
2. Click the export icon button
3. Save the JSON file to your computer

![A screenshot of the History panel with the Export icon button highlighted](assets/history/history-export.png)
![A screenshot of the History panel with the Export icon button highlighted](../readme_assets/history-export.png)

### Import
1. Select the import icon button in the header of the *Dream Texture* panel
2. Open a valid prompt JSON file
3. Every configuration option will be loaded in

![A screenshot of the Dream Texture panel with the Import icon button highlighted](assets/history/history-import.png)
![A screenshot of the Dream Texture panel with the Import icon button highlighted](../readme_assets/history-import.png)
62 changes: 18 additions & 44 deletions docs/IMAGE_GENERATION.md
View file
Original file line number Diff line number Diff line change
@@ -1,26 +1,14 @@
# Image Generation
1. To open Dream Textures, go to an Image Editor or Shader Editor
1. Ensure the sidebar is visible by pressing *N* or checking *View* > *Sidebar*
2. Select the *Dream* panel to open the interface
2. Select the 'Dream' panel to open the interface

![A screenshot showing the 'Dream' panel in an Image Editor space](assets/image_generation/opening-ui.png)
![A screenshot showing the 'Dream' panel in an Image Editor space](../readme_assets/opening-ui.png)

Enter a prompt then click *Generate*. It can take anywhere from a few seconds to a few minutes to generate, depending on your GPU.
Enter a prompt then click *Generate*. It can take anywhere from a few seconds to a few minutes to generate, depending on your graphics card.

## Options

### Pipeline
Two options are currently available:
* Stable Diffusion - for local generation
* DreamStudio - for cloud processing

Only the options available for the version you installed and the keys provided in the add-on preferences will be available.

### Model
Choose from any installed model. Some options require specific kinds of model.

For example, []

### Prompt

A few presets are available to help you create great prompts. They work by asking you to fill in a few simple fields, then generate a full prompt string that is passed to Stable Diffusion.
Expand All @@ -30,8 +18,6 @@ The default preset is *Texture*. It asks for a subject, and adds the word `textu
### Seamless
Checking seamless will use a circular convolution to create a perfectly seamless image, which works great for textures.

You can also specify which axes should be seamless.

### Negative
Enabling negative prompts gives you finer control over your image. For example, if you asked for a `cloud city`, but you wanted to remove the buildings it added, you could enter the negative prompt `building`. This would tell Stable Diffusion to avoid drawing buildings. You can add as much content you want to the negative prompt, and it will avoid everything entered.

Expand All @@ -42,40 +28,28 @@ Most graphics cards with 4+GB of VRAM should be able to generate 512x512 images.

> Stable Diffusion was trained on 512x512 images, so you will get the best results at this size (or at least when leaving one dimensions at 512).
### Source Image
Choose an image from a specific *File* or use the *Open Image*.
### Inpaint Open Image
See [Inpainting](INPAINTING.md) for more information.

Three actions are available that work on a source image.

#### Modify
Mixes the image with the noise with the ratio specified by the *Noise Strength*. This will make Stable Diffusion match the style, composition, etc. from it.
### Init Image
Specifies an image to mix with the latent noise. Open any image, and Stable Diffusion will match the style, composition, etc. from it.

Stength specifies how much latent noise to mix with the image. A higher strength means more latent noise, and more deviation from the init image. If you want it to stick to the image more, decrease the strength.

> Depending on the strength value, some steps will be skipped. For example, if you specified `10` steps and set strength to `0.5`, only `5` steps would be used.
Fit to width/height will ensure the image is contained within the configured size.

The *Image Type* option has a few options:
1. Color - Mixes the image with noise

> The following options require a depth model to be selected, such as `stabilityai/stable-diffusion-2-depth`. Follow the instructions to [download a model](setup.md#download-a-model).
2. Color and Generated Depth - Uses MiDaS to infer the depth of the initial image and includes it in the conditioning. Can give results that more closely match the composition of the source image.
3. Color and Depth Map - Specify a secondary image to use as the depth map, instead of generating one with MiDaS.
4. Depth - Treats the intial image as a depth map, and ignores any color. The generated image will match the composition but not colors of the original.

### Advanced
You can have more control over the generation by trying different values for these parameters:

* Random Seed - When enabled, a seed will be selected for you
* Seed - The value used to seed RNG, if text is input instead of a number its hash will be used
* Steps - Number of sampler steps, higher steps will give the sampler more time to converge and clear up artifacts
* CFG Scale - How strongly the prompt influences the output
* Scheduler - Some schedulers take fewer steps to produce a good result than others. Try each one and see what you prefer.
* Step Preview - Whether to show each step in the image editor. Defaults to 'Fast', which samples the latents without using the VAE. 'Accurrate' will run the latents through the VAE at each step and slow down generation significantly.
* Speed Optimizations - Various optimizations to increase generation speed, some at the cost of VRAM. Recommended default is *Half Precision*.
* Memory Optimizations - Various optimizations to reduce VRAM consumption, some at the cost of speed. Recommended default is *Attention Slicing* with *Automatic* slice size.

### Iterations
How many images to generate. This is only particularly useful when *Random Seed* is enabled.
* Precision - the math precision
* Automatic - chooses the best option for your GPU
* Full Precision - uses 32-bit floats, required on some GPUs
* Half Precision - uses 16-bit floats, faster
* Autocast - uses the correct precision for each PyTorch operation
* Random Seed - when enabled, a seed will be selected for you
* Seed - the value used to seed RNG, if text is input instead of a number its hash will be used
* Steps - number of sampler steps, higher steps will give the sampler more time to converge and clear up artifacts
* CFG Scale - how strongly the prompt influences the output
* Sampler - the sampling method to use, all samplers (except for KEULER_A and KDPM_2A) will produce the same image if given enough steps
* Show Steps - whether to show each step in the Image Editor, can slow down generation significantly
22 changes: 22 additions & 0 deletions docs/INPAINTING.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Inpainting
Use Stable Diffusion to fill in gaps in images, add new content, fix artifacting, make existing textures seamless, and more.

1. Open an image in the Image Editor
2. Select the *Paint* mode
3. Use the *Mark Inpaint Area* brush to erase the alpha channel from the desired part of the image
4. Check *Inpaint Open Image*
5. Enter a prompt describing what you want and click *Generate*

![A screenshot of an Image Editor space in 'Paint' mode with the 'Mark Inpaint Area' brush active, a section of the image alpha erased, and the 'Inpaint Open Image' option checked in the Dream Textures' UI](../readme_assets/inpainting.png)

## Making Textures Seamless
Inpainting can also be used to make an existing texture seamless.

1. Use the *Mark Inpaint Area* brush to remove the edges of the image
2. Enter a prompt that describes the texture, and check *Seamless*
3. Click *Generate*

![A screenshot of an Image Editor space with the edges of a brick texture at 0% alpha, and the 'Seamless' and 'Inpaint Open Image' options checked in the Dream Textures' UI](../readme_assets/inpainting-seamless.png)

## Adding New Content
Note that adding new content with inpainting may require a bit of manual work at this time. Filling the area to inpaint with noise before attempting to generate something new in that spot can help prevent Stable Diffusion from just copying what was there before. There is no option to do this automatically at this time.
Loading

0 comments on commit f96197c

Please sign in to comment.