Flux - Slower generation time P40 Pascal REDUX

[JimikoSK]

UPDATE: 11/26/2024 - generation time is slow again.

I'm not using super hardware, Pascal Tesla P40, but my generation times before updating yesterday with FP8 Dev Flux were about 2minutes 7.17it/s not speedy, but usable.

Now the generation time is up to 5-7 minutes.

Image size 512x512 btw.

Anything significantly change that I should be aware of to get back to smaller gen time?

[realmalikov]

Same issue here!
#1166

[JimikoSK]

Thank you for confirming, I was driving myself crazy. :P

[navarisun1982]

same here, i thought the latest version will fasten thing but it do the opposite, any simple image (20 steps - 512x512) will take at least 7 minutes

[JimikoSK]

Same here with the NB4_v2 it seemed like things were gonne be fast!

[Moussack]

Yup me too nf4 v2,
5 minutes per image, after update 8 minutes per image

[JimikoSK]

Bumping this because performance has degraded again.

Latest Update slow again, it was around 7 s/it, now up to 25s/it - 30s/it.

################################################################
Launching launch.py...
################################################################
glibc version is 2.39
Cannot locate TCMalloc. Do you have tcmalloc or google-perftool installed on your system? (improves CPU memory usage)
Python 3.10.6 (main, Oct 24 2022, 16:07:47) [GCC 11.2.0]
Version: f2.0.1v1.10.1-previous-626-ga332f7cc
Commit hash: a332f7c
Installing xformers
Legacy Preprocessor init warning: Unable to install insightface automatically. Please try run pip install insightface manually.
Launching Web UI with arguments: --lora-dir /SDFiles/models/LORA --ckpt-dir /SDFiles/models/Stable-Diffusion --embeddings-dir /SDFiles/embeddings --no-gradio-queue --listen --server-name 192.168.1.107 --enable-insecure-extension-access --cuda-malloc --xformers
Using cudaMallocAsync backend.
Total VRAM 24446 MB, total RAM 80128 MB
pytorch version: 2.3.1+cu121
xformers version: 0.0.27
Set vram state to: NORMAL_VRAM
Device: cuda:0 Tesla P40 : cudaMallocAsync
VAE dtype preferences: [torch.float32] -> torch.float32
CUDA Using Stream: False
Using xformers cross attention
Using xformers attention for VAE
ControlNet preprocessor location: /stable-diffusion-webui-forge/models/ControlNetPreprocessor
Loading additional modules ... done.
2024-11-27 04:49:59,665 - ControlNet - INFO - ControlNet UI callback registered.
Model selected: {'checkpoint_info': {'filename': '/models/Stable-Diffusion/flux1-dev-fp8.safetensors', 'hash': '26acbda5'}, 'additional_modules': ['/stable-diffusion-webui-forge/models/text_encoder/clip_l.safetensors', '/stable-diffusion-webui-forge/models/VAE/ae.safetensors', '/stable-diffusion-webui-forge/models/text_encoder/t5xxl_fp8_e4m3fn.safetensors'], 'unet_storage_dtype': None}
Using online LoRAs in FP16: False
Running on local URL: localhost

To create a public link, set share=True in launch().
Startup time: 107.0s (prepare environment: 17.4s, launcher: 1.2s, import torch: 15.2s, initialize shared: 0.1s, other imports: 0.4s, load scripts: 6.5s, initialize google blockly: 53.9s, create ui: 6.4s, gradio launch: 5.8s).
Environment vars changed: {'stream': False, 'inference_memory': 1024.0, 'pin_shared_memory': False}
[GPU Setting] You will use 95.81% GPU memory (23421.00 MB) to load weights, and use 4.19% GPU memory (1024.00 MB) to do matrix computation.
Loading Model: {'checkpoint_info': {'filename': '/models/Stable-Diffusion/flux1-dev-fp8.safetensors', 'hash': '26acbda5'}, 'additional_modules': ['/stable-diffusion-webui-forge/models/text_encoder/clip_l.safetensors', '/stable-diffusion-webui-forge/models/VAE/ae.safetensors', '/stable-diffusion-webui-forge/models/text_encoder/t5xxl_fp8_e4m3fn.safetensors'], 'unet_storage_dtype': None}
[Unload] Trying to free all memory for cuda:0 with 0 models keep loaded ... Done.
StateDict Keys: {'transformer': 780, 'vae': 244, 'text_encoder': 196, 'text_encoder_2': 220, 'ignore': 0}
Using Detected T5 Data Type: torch.float8_e4m3fn
Using Detected UNet Type: torch.float8_e4m3fn
Working with z of shape (1, 16, 32, 32) = 16384 dimensions.
K-Model Created: {'storage_dtype': torch.float8_e4m3fn, 'computation_dtype': torch.float32}
Model loaded in 1.5s (unload existing model: 0.4s, forge model load: 1.1s).
Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored.
All loaded to GPU.
Moving model(s) has taken 0.03 seconds
Distilled CFG Scale: 3.5
[Unload] Trying to free 16721.17 MB for cuda:0 with 0 models keep loaded ... Current free memory is 24260.87 MB ... Done.
[Memory Management] Target: KModel, Free GPU: 24260.87 MB, Model Require: 11350.07 MB, Previously Loaded: 0.00 MB, Inference Require: 1024.00 MB, Remaining: 11886.81 MB, All loaded to GPU.
Moving model(s) has taken 3.33 seconds
15%|############6 | 3/20 [01:08<06:46, 23.93s/it]Total progress: 15%|██████████▏ | 3/20 [00:52<05:17, 18.66s/it]