noise shape for dropout

[chengchingwen]

I add the noise shape for dropout, similar to the noise_shape argument in tf.nn.dropout

[MikeInnes]

This looks like a good idea, but it also seems like it's equivalent to a dims keyword argument, which would be more standard in Julia. Any thoughts on doing that API instead?

[chengchingwen]

@MikeInnes Do you mean like using a dims keyword argument for indicating which dimensions should be broadcasted?

[MikeInnes]

Yes exactly. Though it might be better for dims to say which dims are not broadcasted, so that e.g. dims=1 does dropout along columns and dims=2 along rows.

[chengchingwen]

But if the dims is which not broadcasted, then what about higher dimensional input? Like if the input data is with shape of (C, W, H, B) and I want each sample to have the same dropout, than I would need to specified dims=(1,2,3) for the unbroadcasted C, W, H?

[chengchingwen]

@MikeInnes I made a first version of dims API, and currently the dims is the broadcasted dims. For example

julia> Dropout(0.5)(randn(5,4), 1)
5×4 Array{Float64,2}:
 -0.230536  -0.0   1.02677   -0.903341
 -0.605143   0.0   0.748388   0.732854
  2.56266   -0.0  -2.79108   -1.59313 
 -0.613482  -0.0   0.468957  -1.96    
 -0.87279   -0.0   4.01647    0.647282

1 indicate that each row use the same dropout result and dims can also be a tuple for multiple dimensions.

julia> Dropout(0.5)(randn(5,4,2), (1,3))
5×4×2 Array{Float64,3}:
[:, :, 1] =
 -0.0  -0.0  -1.66134    1.97335 
 -0.0   0.0  -0.310311   2.57003 
  0.0  -0.0   1.24803   -3.60845 
 -0.0   0.0  -1.4593    -0.755723
  0.0   0.0   0.8056     4.04177 

[:, :, 2] =
 -0.0   0.0  -0.532319   -0.836303
  0.0   0.0   0.867975   -0.309224
 -0.0  -0.0  -2.63861     1.14548 
 -0.0   0.0  -0.0331286   2.39778 
  0.0   0.0  -2.47692    -0.358082

[chengchingwen]

bump

[chengchingwen]

@MikeInnes

[MikeInnes]

It would be nicer to use dims = : for all dimensions, like the reduction functions do.

[chengchingwen]

Got it. What about the dims question discuss above? I just though it might be more convenient to use dims as the broadcasted dims, but maybe it's not and dims as unbroadcasted dims is more intuitive?

[MikeInnes]

Yes, it's more intuitive if it aligns with how dims is used everywhere else. For example if you wanted to sum across each image you'd likewise do sum(x, dims = (1, 2, 3)).

It should be a keyword argument, too.

[chengchingwen]

Ok, I change the dims as the unbroadcasted dims and also make it a keyword argument.

[MikeInnes]

Ok, one last thing, I think the dims argument should be an argument to Dropout(x), not an argument when you call that layer. Actually, it'd be good if you could split out a dropout function here and use that in the Dropout layer.

We also need to update the Dropout docs and add a news item; but once that's done this will be good to go!

[chengchingwen]

Do you mean that we should make dims a field of the Dropout layer?

[MikeInnes]

Yes.

[chengchingwen]

@MikeInnes where should I add the docs? I can't find the old one in the docs folder

[MikeInnes]

Actually, dropout is part of the docs already so that's fine. Just NEWS.md needs updating.

[MikeInnes]

bors r+

[bors]

Build succeeded

• ci/gitlab/staging