Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue trying to reproduce train_generator notebook #2

Open
fedeotto opened this issue Feb 3, 2022 · 0 comments
Open

Issue trying to reproduce train_generator notebook #2

fedeotto opened this issue Feb 3, 2022 · 0 comments

Comments

@fedeotto
Copy link

fedeotto commented Feb 3, 2022

For now I have been trying to reproduce train_generator.ipnyb notebook. But once I get to run

vae.fit([X_train, y_train], X_train, batch_size=256, epochs=150, validation_split=0.2,callbacks=[reduce_lr,checkpoint,save_loss, DecoderSaveCheckpoint('ding_decoder_best.h5', decoder)])

the following issue appears:

TypeError: in user code:

    File "C:\Users\hp\AppData\Roaming\Python\Python38\site-packages\keras\engine\training.py", line 878, in train_function  *
        return step_function(self, iterator)
    File "C:\Users\hp\AppData\Roaming\Python\Python38\site-packages\keras\engine\training.py", line 867, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "C:\Users\hp\AppData\Roaming\Python\Python38\site-packages\keras\engine\training.py", line 860, in run_step  **
        outputs = model.train_step(data)
    File "C:\Users\hp\AppData\Roaming\Python\Python38\site-packages\keras\engine\training.py", line 809, in train_step
        loss = self.compiled_loss(
    File "C:\Users\hp\AppData\Roaming\Python\Python38\site-packages\keras\engine\compile_utils.py", line 239, in __call__
        self._loss_metric.update_state(
    File "C:\Users\hp\AppData\Roaming\Python\Python38\site-packages\keras\utils\metrics_utils.py", line 73, in decorated
        update_op = update_state_fn(*args, **kwargs)
    File "C:\Users\hp\AppData\Roaming\Python\Python38\site-packages\keras\metrics.py", line 177, in update_state_fn
        return ag_update_state(*args, **kwargs)
    File "C:\Users\hp\AppData\Roaming\Python\Python38\site-packages\keras\metrics.py", line 451, in update_state  **
        sample_weight = tf.__internal__.ops.broadcast_weights(
    File "C:\Users\hp\AppData\Roaming\Python\Python38\site-packages\keras\engine\keras_tensor.py", line 255, in __array__
        raise TypeError(

    TypeError: You are passing KerasTensor(type_spec=TensorSpec(shape=(), dtype=tf.float32, name=None), name='Placeholder:0', description="created by layer 'tf.cast_4'"), an intermediate Keras symbolic input/output, to a TF API that does not allow registering custom dispatchers, such as `tf.cond`, `tf.function`, gradient tapes, or `tf.map_fn`. Keras Functional model construction only supports TF API calls that *do* support dispatching, such as `tf.math.add` or `tf.reshape`. Other APIs cannot be called directly on symbolic Kerasinputs/outputs. You can work around this limitation by putting the operation in a custom Keras layer `call` and calling that layer on this symbolic input/output.

I have actually created a dedicated environment installing all dependencies using requirements.txt .

EDIT:

OK I think I have solved the issue in the following way:

before constructing the model just run

from tensorflow.python.framework.ops import disable_eager_execution

disable_eager_execution()

Since we are using a custom loss function, I have also specified

experimental_run_tf_function=False

in model.compile()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant