Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

problems about keeping training using the pretrained model #5

Open
yyhlvdl opened this issue Mar 12, 2019 · 1 comment
Open

problems about keeping training using the pretrained model #5

yyhlvdl opened this issue Mar 12, 2019 · 1 comment

Comments

@yyhlvdl
Copy link

yyhlvdl commented Mar 12, 2019

@bill9800 It is really a amazing work! Thanks a lot for sharing the code. Howerver,I meet a problem when I try to keep training using the pretrained model. AS follows:
Traceback (most recent call last):
File "/home/yyh/pycharm-2018.3.4/helpers/pydev/pydevd.py", line 1741, in
main()
File "/home/yyh/pycharm-2018.3.4/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/yyh/pycharm-2018.3.4/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/yyh/pycharm-2018.3.4/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/yyh/xym/speech_separation-master/model/model_v2/AV_train.py", line 74, in
AV_model = load_model(latest_file,custom_objects={"tf": tf})
File "/home/yyh/anaconda3/envs/xym/lib/python3.6/site-packages/keras/engine/saving.py", line 289, in load_model
sample_weight_mode=sample_weight_mode)
File "/home/yyh/anaconda3/envs/xym/lib/python3.6/site-packages/keras/engine/training.py", line 139, in compile
loss_function = losses.get(loss)
File "/home/yyh/anaconda3/envs/xym/lib/python3.6/site-packages/keras/losses.py", line 133, in get
return deserialize(identifier)
File "/home/yyh/anaconda3/envs/xym/lib/python3.6/site-packages/keras/losses.py", line 114, in deserialize
printable_module_name='loss function')
File "/home/yyh/anaconda3/envs/xym/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 165, in deserialize_keras_object
':' + function_name)
ValueError: Unknown loss function:loss_func

my environment is tensorflow-gpu==1.8.0. keras ==2.2.2. python=3.6.
cloud you give me some advices?

@bill9800
Copy link
Owner

bill9800 commented Mar 14, 2019

My environment is tensorflow-gpu == 1.12.0 keras-gpu == 2.2.4 python = 3.6.

Some possible solutions:
1.Are you using one GPU or 2 GPU? If the pretrained model is for 2-GPU , it cannot be trained later using one GPU. (Need to change the parameter in line 31 to NUM_GPU=2, you can use nvidia-smi to check the utility)
2.Maybe the pretrained model use different customized loss function 1. Try to change line 11 from from model_loss import audio_discriminate_loss2 as audio_loss to from model_loss import audio_discriminate_loss as audio_loss in AV_train.py. (I also change a little bit for audio_discriminate_loss2, you can just pull it again. Not quite sure how keras get the customized loss function)
3.Keras is not good at multiprocessing. Suggest not to use it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants