We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
muDNN(v2400) 2024-04-02 15:54:04.407026 0d:0h:0m:35s TID=0x7128101b227cf70d GPU=0 Handle=0xaacc340 ERROR# INVALID_PARAMETER in ScaledDotProductAttention::RunMath, Reason: Internal error 4d mask shape /home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/torch/nn/functional.py:5334: UserWarning: Flash attention only supports architecture with mp version 2.2. But now attempts to run on a mp 2.1 gpu. (Triggered internally at /home/torch_musa/torch_musa/csrc/aten/ops/attention/mudnn/SDPUtils.h:28.) attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal) 0%| | 0/509 [00:13<?, ?it/s] Traceback (most recent call last): File "/home/kakaru/ReFlow-VAE-SVC-main/preprocess.py", line 207, in <module> preprocess(args.data.train_path, f0_extractor, volume_extractor, mel_extractor, units_encoder, sample_rate, hop_size, device = device, use_pitch_aug = use_pitch_aug, extensions = extensions) File "/home/kakaru/ReFlow-VAE-SVC-main/preprocess.py", line 137, in preprocess process(file) File "/home/kakaru/ReFlow-VAE-SVC-main/preprocess.py", line 102, in process units_t = units_encoder.encode(audio_t, sample_rate, hop_size) File "/home/kakaru/ReFlow-VAE-SVC-main/reflow/extractors.py", line 238, in encode units = self.model(audio_res) File "/home/kakaru/ReFlow-VAE-SVC-main/reflow/extractors.py", line 343, in __call__ logits = self.hubert.extract_features(**inputs) File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/models/hubert/hubert.py", line 535, in extract_features res = self.forward( File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/models/hubert/hubert.py", line 467, in forward x, _ = self.encoder( File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/models/wav2vec/wav2vec2.py", line 1003, in forward x, layer_results = self.extract_features(x, padding_mask, layer) File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/models/wav2vec/wav2vec2.py", line 1049, in extract_features x, (z, lr) = layer( File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/models/wav2vec/wav2vec2.py", line 1260, in forward x, attn = self.self_attn( File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/modules/multihead_attention.py", line 538, in forward return F.multi_head_attention_forward( File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/torch/nn/functional.py", line 5334, in multi_head_attention_forward attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal) RuntimeError: MuDNNMathSDPAFwd MUDNN failed in: Run SDPA
不使用fairseq去推理实际上是没问题的,或许是个bug?
The text was updated successfully, but these errors were encountered:
nn.TransformerEncoderLayer基本都有这个问题
Sorry, something went wrong.
请提供复现源码
https://github.com/yxlllc/ReFlow-VAE-SVC/blob/main/reflow/extractors.py
可以参考这里,line335开始
可能是SDP接口不适配导致的,麻烦提供下具体运行的代码(不是module源码),下个release中会修复
No branches or pull requests
不使用fairseq去推理实际上是没问题的,或许是个bug?
The text was updated successfully, but these errors were encountered: