[SYSTEMDS-3694] Sequence primitive for neural network layers

[Nakroma]

Draft for the student project SYSTEMDS-3694.

• Adds a Layer interface to the Python API which can be extended by nn layers
• Adds the sequence primitive that is able to combine mutliple nn layers

[Baunsgaard]

Hi @Nakroma

The code looks great! The PR is nearly done.

Currently missing

• is backward test
• and i am worried about multi return statements, where some layers return multiple variables. Would they work in the sequences?

best regards
Sebastian

[Nakroma]

Hey @Baunsgaard

I actually completely forgot the backwards pass - sorry about that, it is added now!

About the multi returns - I wasn't 100% sure if returns for forward/backward passes are standardized. So I assumed here how it's handled in affine.py and relu.py is the standard - i.e. I check for a MultiReturn or a single node and return based on that. This might have to be changed depending on how the layer generator will work in the future though. Let me know if you think there is a better solution for that also.

Best,
Tarek

[Baunsgaard]

Great, i like the change. seems like you found a few bugs in the process.

Can we add a few more tests, where

1. a multi return layer goes into a multi return layer,
2. a TestLayerImp goes to Multi to multi

and a few more variations.

Also can we verify that the layers get updated correctly on the backwards pass.

Once these are added I would consider the PR finished and ready for merging.

[Nakroma]

@Baunsgaard just pushed a commit that addresses more variations and the layer update during backwards pass.

[Baunsgaard]

@Baunsgaard just pushed a commit that addresses more variations and the layer update during backwards pass.

Great. I will take a look tomorrow!

[Baunsgaard]

@Nakroma
Thank you very much for the PR. it is now merged.