Add polynomial_decay and piecewise_decay #8013
Conversation
jacquesqiao
changed the title
… add-piecewise_constant-decay
| end_learning_rate=0.0001, | ||
| power=1.0, | ||
| cycle=False): | ||
| """Applies inverse time decay to the initial learning rate. |
There was a problem hiding this comment.
This should be Applies polynomial decay to the initial learning rate.
… impl-scalar-switch-case-op
… impl-scalar-switch-case-op
…qiao/Paddle into add-piecewise_constant-decay
… add-piecewise_constant-decay
… impl-scalar-switch-case-op
jacquesqiao
changed the title
… impl-scalar-switch-case-op-with-condition-op
…github.com/jacquesqiao/Paddle into add-piecewise_constant-decay
… add-piecewise_constant-decay
| zero_var = layers.fill_constant(shape=[1], dtype='float32', value=0.0) | ||
| one_var = layers.fill_constant(shape=[1], dtype='float32', value=1.0) | ||
|
|
||
| with layers.Switch() as switch: |
There was a problem hiding this comment.
So, our switch operator only support scalar condition now? And our if-else operator support vector condition.
There was a problem hiding this comment.
yes, the current switch operator only supports scalar condition. because it uses conditional_block, it's easy to support tensor as input in the future.
| Args: | ||
| learning_rate: A scalar float32 value or a Variable. This | ||
| will be the initial learning rate during training | ||
| global_step: A Variable that record the training step. |
There was a problem hiding this comment.
So, where is the global_step variable created?
There was a problem hiding this comment.
should be create outside optimizer. like
global_step = fluid.layers.create_global_var(shape=[1], value=0, dtype='float32', force_cpu=True)
sgd_optimizer = fluid.optimizer.SGD(
learning_rate=fluid.learning_rate_decay.exponential_decay(
learning_rate=0.0001,
global_step=global_step,
decay_steps=100000,
decay_rate=0.5,
staircase=True),
global_step=global_step)There was a problem hiding this comment.
If global_step variable is force cpu, does decay_steps also force cpu?
| @@ -975,6 +976,36 @@ def less_than(x, y, cond=None, **ignored): | |||
| return cond | |||
|
|
|||
|
|
|||
| def equal(x, y, cond=None, **ignored): | |||
There was a problem hiding this comment.
Equal is a element-wise operator. And it also can be override in Python.
def __eq__(self):
...
There was a problem hiding this comment.
I find that our equal is not like python eq. Python eq will return a bool, but our equal will return a vector.
project: #7769
This PR will go on after switch_op #8031 is done.