-
Notifications
You must be signed in to change notification settings - Fork 7.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support system prompt and cfg renorm in Lumina2 #6795
Conversation
Some issues with the ruff style check, see the failing test. |
Thank you for checking, we have made modifications according to your comments. @comfyanonymous |
Thank you for reviewing @comfyanonymous. Another thing is whether you can update the official workflow in https://comfyanonymous.github.io/ComfyUI_examples/lumina2/ as our provided workflow. Our workflow supports the merged functions and uses euler sampling of 50 steps and shifting 6. |
The CFG Renorm doesn't work with batch size > 1.
In a different codebase I fixed it by max_new_norms = cond_norm * float(renorm_cfg)
noise_norms = torch.linalg.vector_norm(
noise_pred, dim=tuple(range(1, len(noise_pred.shape))), keepdim=True
)
# Iterate through batch
for noise_norm, max_new_norm, noise in zip(noise_norms, max_new_norms, noise_pred):
if noise_norm >= max_new_norm:
noise = noise * (max_new_norm / noise_norm) I could make a PR but I might not be able to for a little bit. |
We support CFG Renorm official codes and two system prompts (superior and alignment.)
Meanwhile, we modify the workflow using our recommended settings.
The CFG Renorm is implemented through a new node, RenormCFG, in
comfy_extras/nodes_lumina2.py
.The system prompts are implemented through a new node, CLIPTextEncodeLumina2, in
comfy_extras/nodes_lumina2.py.
This node is used to prevent users from deleting or forgetting system prompts.