Unity ControlNet Rig

A rigged Unity character controller with inverse kinematics built using the Animation Rigging Package, that works with OpenPose ControlNet in Automatic1111 WebUI via the API.

Example video showing usage:

Table of Contents

1. Getting Started
   • Prerequisites
   • Installation
2. Usage
3. License
4. Contact
5. Acknowledgments

Getting Started

This project should be considered experimental since it relies on Automatic1111 WebUI API which changes often and this repo is not guaranteed to be updated to reflect any future conflicts. Feel free to use it as a base template to develop other interesting features and enhancements. I used code from some third party projects (links at the bottom) to initially set it up - primarily the WebUI API integration by dobrado76 which I modified to enable ControlNet use and live previews.

Prerequisites

This project is Windows based. You will need Unity, a local installation of Automatic1111 WebUI, the ControlNet extension and OpenPose model in order for this project to function.

• Unity URP 2021.3.13f1 LTS https://unity.com/download

  Download and install Unity Hub
  

  Install the Unity 2021.3.13 LTS version of the Editor from the archive
  https://unity.com/releases/editor/archive
  

• Automatic1111 WebUI https://github.com/AUTOMATIC1111/stable-diffusion-webui

  Follow the installation instructions below *Features* in the ReadMe
  

• ControlNet Extension https://github.com/Mikubill/sd-webui-controlnet

  Follow the installation instructions from the ReadMe
  

• ControlNet OpenPose Model https://huggingface.co/lllyasviel/ControlNet/blob/main/models/control_sd15_openpose.pth Download the model and place it in the extension folder

  stable-diffusion-webui/extensions/sd-webui-controlnet/models/
  

Installation

Follow these steps to get the project up and running.

1. Clone this repo to your local drive

   git clone https://github.com/hack-mans/Unity-ControlNet-Rig.git
   

2. Add the project folder to Unity Hub

   Projects > OPEN drop down > Add project from disk > Choose the cloned project folder
   

   Open the project using Unity 2021.3.13f1 (tested and working) or similar version
   

3. Wait for Unity to install the package dependencies and build the Library
4. Open the scene once in Unity

   Assets > HACKMANS > OpenPoseUnity > Scenes > OpenPoseUnity
   

5. Make sure the Game View is in 1080x1080 resolution

   GAME tab > Resolution drop down > Plus (to add new resolution) > Width & Height 1080 x 1080
   

6. Open the WebUI startup batch file

   Open webui-user.bat file in the WebUI root directory in Notepad or an IDE
   

7. Change the following line to add --api

   set COMMANDLINE_ARGS=--api
   

8. Launch WebUI using the edited webui-user.bat file
9. In Unity run List Models from the Stable Diffusion Configuration script

   Scene Hierarchy > StableDiffusionGenerator > Stable Diffusion Generator > List Models
   

10. Configure settings on Open Pose Control Net script

    Scene Hierarchy > OpenPoseController > Open Pose Control Net > select sampler + model + change prompt
    

11. Enter Play Mode to run, usage instructions are below

(back to top)

Usage

The UI panel in the top left allows you to change resolution, preview the raw view of the OpenPose rig, generate and save images. Images are saved to the OutputImages folder in Assets by default but can be configured in the Open Pose Control Net script along with prompt and generation settings. The camera is controlled using WASD + QE while holding down right mouse button. The controls for manipulating the rig are similar to Unity (W E R for transforms) but all keyboard commands can be viewed or changed in the Transform Gizmo script attached to the Main Camera. Pressing Spacebar toggles the view of the generated image so that you can switch back to posing the rig.

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Contact

Twitter - @hackmans

Project Link: https://github.com/hack-mans/Unity-ControlNet-Rig

(back to top)

Acknowledgments

Code from these projects was adapted for posing of the character rig and to enable ControlNet and live previews with Automatic1111 WebUI API

• Unity3DRuntimeTransformGizmo by HiddenMonk
• Stable Diffusion Unity Integration by dobrado76
• Unity3d Network by ididdidi
• Simple Disk Utils by Dikra Prasetya

Special shout out to these projects based on ControlNet that inspired the creation of this project

• 3DCinetv Riggify Blender integration
• toyxyz Blender integration

(back to top)