patch model patcher + torch compile

[codexq123]

CompilationError happens while using torch.compile.

System:Window11
torch:2.4.1
triton:3.0.0
python:3.11
gpu:3090ti

Error message as bellow.

got prompt
Using pytorch attention in VAE
Using pytorch attention in VAE
model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
model_type FLUX
C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: clean_up_tokenization_spaces was not set. It will be set to True by default. This behavior will be depracted in transformers v4.45, and will be then set to False by default. For more details check this issue: huggingface/transformers#31884
warnings.warn(
Requested to load FluxClipModel_
Loading 1 new model
loaded completely 0.0 9319.23095703125 True
clip missing: ['text_projection.weight']
C:\bilibili_project\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\attention.py:407: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:555.)
out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False)
Requested to load Flux
Loading 1 new model
loaded completely 0.0 11350.048889160156 True
0%| | 0/20 [00:00<?, ?it/s]ptxas info : 0 bytes gmem
ptxas info : Compiling entry function 'triton_' for 'sm_86'
ptxas info : Function properties for triton_
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 14 registers, used 0 barriers, 380 bytes cmem[0]
main.c
正在创建库 main.lib 和对象 main.exp
ptxas info : 0 bytes gmem
ptxas info : Compiling entry function 'triton_' for 'sm_86'
ptxas info : Function properties for triton_
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 11 registers, used 0 barriers, 380 bytes cmem[0]
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] Triton compilation failed: triton_poi_fused__to_copy_1
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] def triton_(in_ptr0, out_ptr0, xnumel, XBLOCK : tl.constexpr):
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] xnumel = 56623104
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] xoffset = tl.program_id(0) * XBLOCK
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] xindex = xoffset + tl.arange(0, XBLOCK)[:]
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] xmask = xindex < xnumel
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] x0 = xindex
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] tmp0 = tl.load(in_ptr0 + (x0), None)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] tmp1 = tmp0.to(tl.float32)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] tl.store(out_ptr0 + (x0), tmp1, None)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0]
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] metadata: {'signature': {0: '*fp8e4nv', 1: '*bf16', 2: 'i32'}, 'device': 0, 'constants': {3: 1024}, 'configs': [AttrsDescriptor(divisible_by_16=(0, 1, 2), equal_to_1=())], 'device_type': 'cuda', 'num_warps': 4, 'num_stages': 1, 'debug': True, 'cc': 86}
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] Traceback (most recent call last):
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\code_generator.py", line 1116, in visit_Call
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] return fn(*args, **extra_kwargs, **kws)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\language\core.py", line 35, in wrapper
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] return fn(*args, **kwargs)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\language\core.py", line 993, in to
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] return semantic.cast(self, dtype, builder, fp_downcast_rounding)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\language\semantic.py", line 759, in cast
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] assert builder.options.allow_fp8e4nv, "fp8e4nv data type is not supported on CUDA arch < 89"
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] AssertionError: fp8e4nv data type is not supported on CUDA arch < 89
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0]
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] The above exception was the direct cause of the following exception:
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0]
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] Traceback (most recent call last):
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_inductor\runtime\triton_heuristics.py", line 431, in precompile_config
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] binary = triton.compile(*compile_args, **compile_kwargs)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\compiler.py", line 280, in compile
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] module = src.make_ir(options, codegen_fns, context)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\compiler.py", line 113, in make_ir
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\code_generator.py", line 1297, in ast_to_ttir
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] generator.visit(fn.parse())
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\code_generator.py", line 1204, in visit
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ret = super().visit(node)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "ast.py", line 418, in visit
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\code_generator.py", line 359, in visit_Module
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ast.NodeVisitor.generic_visit(self, node)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "ast.py", line 426, in generic_visit
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\code_generator.py", line 1204, in visit
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ret = super().visit(node)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "ast.py", line 418, in visit
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\code_generator.py", line 443, in visit_FunctionDef
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] self.visit_compound_statement(node.body)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\code_generator.py", line 351, in visit_compound_statement
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] self.visit(stmt)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\code_generator.py", line 1204, in visit
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ret = super().visit(node)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "ast.py", line 418, in visit
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\code_generator.py", line 496, in visit_Assign
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] values = self.visit(node.value)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\code_generator.py", line 1204, in visit
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ret = super().visit(node)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^^^^^^^^^^^^^^^^^^^
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "ast.py", line 418, in visit
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] File "C:\bilibili_project\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\code_generator.py", line 1124, in visit_Call
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] raise CompilationError(self.jit_fn.src, node, None) from e
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] triton.compiler.errors.CompilationError: at 8:11:
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] def triton(in_ptr0, out_ptr0, xnumel, XBLOCK : tl.constexpr):
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] xnumel = 56623104
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] xoffset = tl.program_id(0) * XBLOCK
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] xindex = xoffset + tl.arange(0, XBLOCK)[:]
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] xmask = xindex < xnumel
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] x0 = xindex
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] tmp0 = tl.load(in_ptr0 + (x0), None)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] tmp1 = tmp0.to(tl.float32)
E1106 09:11:30.390000 6556 torch_inductor\runtime\triton_heuristics.py:433] [0/0] ^
0%| | 0/20 [00:06<?, ?it/s]
!!! Exception during processing !!! backend='inductor' raised:
CompilationError: at 8:11:
def triton(in_ptr0, out_ptr0, xnumel, XBLOCK : tl.constexpr):
xnumel = 56623104
xoffset = tl.program_id(0) * XBLOCK
xindex = xoffset + tl.arange(0, XBLOCK)[:]
xmask = xindex < xnumel
x0 = xindex
tmp0 = tl.load(in_ptr0 + (x0), None)
tmp1 = tmp0.to(tl.float32)

[1506086927]

+1