About implementation_fwd and implementation_bwd

[djiajunustc]

• MemCNN version: latest
• Python version: 3.7
• Operating System: Ubuntu16.06

Hi,

In the function called "create_coupling" (from revop.py), there are two implementation mode related variable (i.e. implementation_fwd and implementation_bwd) that we can set to -1, 0 or 1.
But I find that you only take -1 in your example now. Does the other two mode supported now?

And in function called "configure" (from resnet.py, class ResNet), you se the the value of InvertibleModuleWrapper.implementation. It seems to make no effect on the forward nor the backward process.

Could you help me to figure out it?

Thanks a lot!

[silvandeleemput]

@djiajunustc Hi, thanks for your interest in MemCNN. The settings for the implementation_fwd and implementation_bwd have been around since the very beginning of the library, but have been deprecated for a while now. The choice of implementation shouldn't matter much, but I recommend the default setting of -1, which uses a plain PyTorch implementation that doesn't rely on a custom autograd.Function and is more in line with the current workflow of MemCNN.

In the past MemCNN used custom autograd.Function for leveraging the memory saving at the coupling level (hence the different implementations). Nowadays MemCNN uses the InvertibleModuleWrapper class to wrap arbitrary invertible modules (not only the additive/affine couplings) to get the memory savings, which allows for greater flexibility in design and makes it easier to maintain as well.