Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

keras create/load model fails on Apple M2 cpu. Same code works on Ryzen #46

Closed
drmatthewclark opened this issue Jul 6, 2023 · 31 comments
Assignees
Labels

Comments

@drmatthewclark
Copy link

tf_env.txt

You can obtain the TensorFlow version with:
python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
result:
v2.13.0-rc2-7-g1cb1a030a62 2.13.0

Can't load a keras model that I just created. The issue is not specific to Adam; all of the optimizers have the same issue.

Describe the problem.

code:

  from tensorflow import keras
  
  
  optimizer = keras.optimizers.Adam()
  
  vh  = keras.Input(shape=(2,3), name = 'vh')
  v1  = keras.layers.Dense(512)(vh)
  
  output  = keras.layers.Dense(1, activation='softmax', name='prediction')(v1)
  
  model = keras.Model(inputs=vh, outputs=[output], name="antibody_model")
  
  model.compile(optimizer=optimizer )
  
  model.save('nn_model.keras')
  test = keras.models.load_model('nn_model.keras')

output:

the legacy Adam is missing the method "build". This same code works on non-mac platforms.

WARNING:absl:At this time, the v2.11+ optimizer tf.keras.optimizers.Adam runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at tf.keras.optimizers.legacy.Adam.
WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., tf.keras.optimizers.legacy.Adam.
WARNING:absl:At this time, the v2.11+ optimizer tf.keras.optimizers.Adam runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at tf.keras.optimizers.legacy.Adam.
WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., tf.keras.optimizers.legacy.Adam.
Traceback (most recent call last):
File "/Users/matthewclark/eclipse-workspace/antibody_classification/test/test.py", line 17, in
test = keras.models.load_model('nn_model.keras')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/saving/saving_api.py", line 230, in load_model
return saving_lib.load_model(
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/saving/saving_lib.py", line 275, in load_model
raise e
File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/saving/saving_lib.py", line 240, in load_model
model = deserialize_keras_object(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/saving/serialization_lib.py", line 710, in deserialize_keras_object
instance.compile_from_config(compile_config)
File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/engine/training.py", line 3582, in compile_from_config
self.optimizer.build(self.trainable_variables)
^^^^^^^^^^^^^^^^^^^^
File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/optimizers/legacy/optimizer_v2.py", line 997, in getattribute
raise e
File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/optimizers/legacy/optimizer_v2.py", line 987, in getattribute
return super().getattribute(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Adam' object has no attribute 'build'

@tilakrayal
Copy link
Collaborator

@drmatthewclark,
I tried to execute the code on both tensorflow v2.12 and v2.13 and it was executed without any error/issue. Kindly find the gist of it here and try to test your code in the virtual environment. Thank you!

@jtpotato
Copy link

Can confirm having the same issue (M1). Haven't had the chance to test on non-Apple Silicon hardware (that I own) yet, but it appears to be fine on Colab...

@bossebandowski
Copy link

Same here, on M1, python3.10, and tensorflow 2.13.0. Training, inference, and saving works fine. Loading fails.

Code extract:

train(model, train_ds, val_ds)
acc = model.evaluate(test_ds)[1]
print(f"test accuracy: {acc}")
model.save(MODEL_PATH)
test = tf.keras.models.load_model(MODEL_PATH)
print("successfully loaded model")

Output:

[...]
test accuracy: 0.9336022138595581
Traceback (most recent call last):
  File "[my_file]", line 252, in <module>
    test = tf.keras.models.load_model(MODEL_PATH)
  File "[my_venv]/site-packages/keras/src/saving/saving_api.py", line 230, in load_model
    return saving_lib.load_model(
  File "[my_venv]/site-packages/keras/src/saving/saving_lib.py", line 275, in load_model
    raise e
  File "[my_venv]/site-packages/keras/src/saving/saving_lib.py", line 240, in load_model
    model = deserialize_keras_object(
  File "[my_venv]/site-packages/keras/src/saving/serialization_lib.py", line 710, in deserialize_keras_object
    instance.compile_from_config(compile_config)
  File "[my_venv]/site-packages/keras/src/engine/training.py", line 3582, in compile_from_config
    self.optimizer.build(self.trainable_variables)
  File "[my_venv]/site-packages/keras/src/optimizers/legacy/optimizer_v2.py", line 997, in __getattribute__
    raise e
  File "[my_venv]/site-packages/keras/src/optimizers/legacy/optimizer_v2.py", line 987, in __getattribute__
    return super().__getattribute__(name)
AttributeError: 'Adam' object has no attribute 'build'

@tilakrayal
Copy link
Collaborator

@drmatthewclark,
I tried to execute the mentioned above code on both tensorflow v2.12 and tf-nighty(2.14.0-dev20230622), and it was executed without any issue/error. Kindly find the gist of it here
As it is failing only on tensorflow-macos, we request to raise the concern on the macos-apple forum for the quick resolution. Thank you!

@bossebandowski
Copy link

bossebandowski commented Jul 26, 2023

Hey @tilakrayal,

thank you for your input. I did a bit of digging because I found the issue too annoying and it is clearly a tensorflow problem and not a macos problem. So, here's what I found:

  1. As you can easily see from the backtrace in my previous comment, loading a model on the M1/M2 macs causes the legacy optimizers to be loaded instead of the latest code. Sure enough, the legacy code does not have a build method while the latest code does. So, why is the legacy code executed in the first place?
  2. To get an optimizer, the optimizers.get() method is called which is defined here on the latest tag v2.13.1. Whenever we have a mac that runs on the new chips, keras falls back to the legacy optimizers which do not work.
  3. As of commit 5ddf84a, this issue seems to be fixed on master but there are no releases that contain the update.

[EDIT] @drmatthewclark to fix your problem install the nightly version of tensorflow and keras while we wait for a new release.

pip install tf-nightly keras-nightly

@tilakrayal
Copy link
Collaborator

There is an alternative workaround to make AdamW work on Apple Silicon with the latest version of tensorflow, tensorflow-addons.
All you need to do is to import AdamW from tensorflow_addons.optimizers. Could you please try and let us know if this works in your case.

@github-actions
Copy link

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

@mbergwall2222
Copy link

Can confirm same issue on m2 max mac. nightly tensorflow builds don't work and neither does tensorflow-addons.

@tcivie
Copy link

tcivie commented Aug 20, 2023

import AdamW from tensorflow_addons.optimizers

Also tried this and got the same result (M2 Mac):
AttributeError: 'AdamW' object has no attribute 'build'

Tried also the suggested solutions from this thread without any success

@norwend
Copy link

norwend commented Aug 27, 2023

Same error with the SGD omptimizer. M1 Macbook Air.

Traceback (most recent call last):
File "***", line 17, in
model = keras.models.load_model(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/keras/src/saving/saving_api.py", line 230, in load_model
return saving_lib.load_model(
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/keras/src/saving/saving_lib.py", line 275, in load_model
raise e
File "/opt/homebrew/lib/python3.11/site-packages/keras/src/saving/saving_lib.py", line 240, in load_model
model = deserialize_keras_object(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/keras/src/saving/serialization_lib.py", line 710, in deserialize_keras_object
instance.compile_from_config(compile_config)
File "/opt/homebrew/lib/python3.11/site-packages/keras/src/engine/training.py", line 3582, in compile_from_config
self.optimizer.build(self.trainable_variables)
^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/keras/src/optimizers/legacy/optimizer_v2.py", line 997, in getattribute
raise e
File "/opt/homebrew/lib/python3.11/site-packages/keras/src/optimizers/legacy/optimizer_v2.py", line 987, in getattribute
return super().getattribute(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'SGD' object has no attribute 'build'

@RobP-git
Copy link

RobP-git commented Sep 4, 2023

same issue here also. M1 mac.
current workaround is to save and load using the legacy .h5 file extension.

lmoneta referenced this issue in lmoneta/root Sep 4, 2023
Go back to use .h5 files instead of .keras due to a problem in Keras loading .keras files on MacOS ARM (see https://github.com/keras-team/keras/issues/18278 )
lmoneta referenced this issue in root-project/root Sep 6, 2023
Go back to use .h5 files instead of .keras due to a problem in Keras loading .keras files on MacOS ARM (see https://github.com/keras-team/keras/issues/18278 )
@samhva
Copy link

samhva commented Sep 8, 2023

Same issue here, fix plz.
M2 Pro

@ptn24
Copy link

ptn24 commented Sep 9, 2023

Ditto. I am using TensorFlow 2.13 on macOS with M2.

lmoneta referenced this issue in lmoneta/root Sep 11, 2023
Go back to use .h5 files instead of .keras due to a problem in Keras loading .keras files on MacOS ARM (see https://github.com/keras-team/keras/issues/18278 )
@LiWangSH
Copy link

LiWangSH commented Sep 13, 2023

Same issue, using Tensorflow 2.13.0 and Python 3.10.13 on Mac M2.

@LiWangSH
Copy link

Saving the model to a different format (h5) solved the issue^ for me.

lmoneta referenced this issue in lmoneta/root Sep 14, 2023
Go back to use .h5 files instead of .keras due to a problem in Keras loading .keras files on MacOS ARM (see https://github.com/keras-team/keras/issues/18278 )
lmoneta referenced this issue in root-project/root Sep 14, 2023
Go back to use .h5 files instead of .keras due to a problem in Keras loading .keras files on MacOS ARM (see https://github.com/keras-team/keras/issues/18278 )
@palminha
Copy link

any news on this issue?
we are having the same problem here with tensorflow 2.13 in MacOS M1

@palminha
Copy link

palminha commented Sep 19, 2023

Saving the model to a different format (h5) solved the issue^ for me.

This workaround is not suitable for us because the h5 format doesn't support some attributes/parameters of our models

@palminha
Copy link

palminha commented Sep 19, 2023

tf_env.txt

You can obtain the TensorFlow version with: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)" result: v2.13.0-rc2-7-g1cb1a030a62 2.13.0

Can't load a keras model that I just created. The issue is not specific to Adam; all of the optimizers have the same issue.

Describe the problem.

code:

  from tensorflow import keras
  
  
  optimizer = keras.optimizers.Adam()
  
  vh  = keras.Input(shape=(2,3), name = 'vh')
  v1  = keras.layers.Dense(512)(vh)
  
  output  = keras.layers.Dense(1, activation='softmax', name='prediction')(v1)
  
  model = keras.Model(inputs=vh, outputs=[output], name="antibody_model")
  
  model.compile(optimizer=optimizer )
  
  model.save('nn_model.keras')
  test = keras.models.load_model('nn_model.keras')

output:

the legacy Adam is missing the method "build". This same code works on non-mac platforms.

WARNING:absl:At this time, the v2.11+ optimizer tf.keras.optimizers.Adam runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at tf.keras.optimizers.legacy.Adam. WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., tf.keras.optimizers.legacy.Adam. WARNING:absl:At this time, the v2.11+ optimizer tf.keras.optimizers.Adam runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at tf.keras.optimizers.legacy.Adam. WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., tf.keras.optimizers.legacy.Adam. Traceback (most recent call last): File "/Users/matthewclark/eclipse-workspace/antibody_classification/test/test.py", line 17, in test = keras.models.load_model('nn_model.keras') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/saving/saving_api.py", line 230, in load_model return saving_lib.load_model( ^^^^^^^^^^^^^^^^^^^^^^ File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/saving/saving_lib.py", line 275, in load_model raise e File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/saving/saving_lib.py", line 240, in load_model model = deserialize_keras_object( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/saving/serialization_lib.py", line 710, in deserialize_keras_object instance.compile_from_config(compile_config) File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/engine/training.py", line 3582, in compile_from_config self.optimizer.build(self.trainable_variables) ^^^^^^^^^^^^^^^^^^^^ File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/optimizers/legacy/optimizer_v2.py", line 997, in getattribute raise e File "/Users/Shared/anaconda3/envs/ability/lib/python3.11/site-packages/keras/src/optimizers/legacy/optimizer_v2.py", line 987, in getattribute return super().getattribute(name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'Adam' object has no attribute 'build'

I just tried to run this code in my MacOS M1 (using tensorflow==2.13.0 and keras [required: >=2.13.1,<2.14, installed: 2.13.1]) and got the following error:

WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.Adam` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.Adam`.
WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., `tf.keras.optimizers.legacy.Adam`.
WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.Adam` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.Adam`.
WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., `tf.keras.optimizers.legacy.Adam`.
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Cellar/[email protected]/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/palmito/Development/federated-demo/backend/app/test.py", line 15, in <module>
    test = keras.models.load_model('nn_model.keras')
  File "/Users/palmito/.local/share/virtualenvs/backend-tPS_SUas/lib/python3.10/site-packages/keras/src/saving/saving_api.py", line 230, in load_model
    return saving_lib.load_model(
  File "/Users/palmito/.local/share/virtualenvs/backend-tPS_SUas/lib/python3.10/site-packages/keras/src/saving/saving_lib.py", line 275, in load_model
    raise e
  File "/Users/palmito/.local/share/virtualenvs/backend-tPS_SUas/lib/python3.10/site-packages/keras/src/saving/saving_lib.py", line 240, in load_model
    model = deserialize_keras_object(
  File "/Users/palmito/.local/share/virtualenvs/backend-tPS_SUas/lib/python3.10/site-packages/keras/src/saving/serialization_lib.py", line 710, in deserialize_keras_object
    instance.compile_from_config(compile_config)
  File "/Users/palmito/.local/share/virtualenvs/backend-tPS_SUas/lib/python3.10/site-packages/keras/src/engine/training.py", line 3582, in compile_from_config
    self.optimizer.build(self.trainable_variables)
  File "/Users/palmito/.local/share/virtualenvs/backend-tPS_SUas/lib/python3.10/site-packages/keras/src/optimizers/legacy/optimizer_v2.py", line 997, in __getattribute__
    raise e
  File "/Users/palmito/.local/share/virtualenvs/backend-tPS_SUas/lib/python3.10/site-packages/keras/src/optimizers/legacy/optimizer_v2.py", line 987, in __getattribute__
    return super().__getattribute__(name)
AttributeError: 'Adam' object has no attribute 'build'

@qlzh727
Copy link
Member

qlzh727 commented Sep 21, 2023

Adding Neel for the saving issue.

@sachinprasadhs sachinprasadhs transferred this issue from keras-team/keras Sep 22, 2023
@tilakrayal
Copy link
Collaborator

tilakrayal commented Sep 28, 2023

@drmatthewclark,
I tried to execute the mentioned code on Apple M1 with the tf-nightly(2.15.0-dev20230927) and it was executed without any fail or error. Kindly find the screenshot below for the reference. And also please take a look at the PR which is related to the issue.
keras-team/keras#18492
image (20)

Thank you!

@palminha
Copy link

@tilakrayal i confirm that with tf-nightly the error it seems to be fixed

% pipenv graph
...
tf-nightly==2.15.0.dev20230928
└── tf-nightly-macos [required: ==2.15.0-dev20230928, installed: 2.15.0.dev20230928]

it seems it solves the error

% pipenv run adam
WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.Adam` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.Adam`.
WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.Adam` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.Adam`.
WARNING:absl:Skipping variable loading for optimizer 'Adam', because it has 9 variables whereas the saved optimizer has 1 variables. 

I also reported the same finding here: tensorflow/tensorflow#61915 (comment)

@eswar-janjanam
Copy link

eswar-janjanam commented Sep 29, 2023

I'm using mac m1 pro and tensorflow version is 2.14.0. I was facing the same error "AttributeError: 'Adam' object has no attribute 'build'" while loading the saved model, if I save the model as below

model = Sequential()
model.add(Embedding(MAX_FEATURES+1, 32))
model.add(Bidirectional(LSTM(32, activation='tanh')))
model.add(Dense(128, activation='relu'))
model.add(Dense(256, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='BinaryCrossentropy', optimizer='Adam')
model.fit(train, epochs=1, validation_data=val)
model.save('data/model_data.keras')

I updated the optimizer as below and it is working fine now

import tensorflow as tf
model.compile(loss='BinaryCrossentropy', optimizer=tf.keras.optimizers.Adam())

@nkovela1
Copy link
Contributor

@EswarJanjanam The fix is on tf-nightly. Please do pip install tf-nightly and see if your code works, thanks!

@phpmac
Copy link

phpmac commented Oct 2, 2023

$pip install tf-nightly
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
ERROR: Could not find a version that satisfies the requirement tf-nightly (from versions: none)
ERROR: No matching distribution found for tf-nightly

@phpmac
Copy link

phpmac commented Oct 2, 2023

$pip install tf-nightly
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
ERROR: Could not find a version that satisfies the requirement tf-nightly (from versions: none)
ERROR: No matching distribution found for tf-nightly

Crap, it's my mirroring problem, sorry!

But it didn't solve the problem.


AttributeError: 'Adam' object has no attribute 'build'


@hanamurayuki
Copy link

$pip install tf-nightly
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
ERROR: Could not find a version that satisfies the requirement tf-nightly (from versions: none)
ERROR: No matching distribution found for tf-nightly

Same error exists on M2 mac.

AttributeError: 'Adam' object has no attribute 'build'

.h5 format is one of solution instead of .keras format for now.
I hope I can save model as .keras format.

@phpmac
Copy link

phpmac commented Oct 16, 2023

$pip install tf-nightly
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
ERROR: Could not find a version that satisfies the requirement tf-nightly (from versions: none)
ERROR: No matching distribution found for tf-nightly

Same error exists on M2 mac.

AttributeError: 'Adam' object has no attribute 'build'

.h5 format is one of solution instead of .keras format for now. I hope I can save model as .keras format.

Yeah, I mean can't he just automatically adapt to the solution?

@MurtadhaInit
Copy link

Having the exact same issue, and can confirm that tf-nightly is NOT working for me.

@richardokonicha
Copy link

same issue

@saarangbond
Copy link

Having the same issue as well on Mac M1. The same code works on cloud workstation on linux.

@eyash24
Copy link

eyash24 commented Aug 18, 2024

After much trial and error, installing keras-nightly solved the issue and my model started working.
pip install tf-nightly keras-nightly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests