NIR <> lava-dl support #218

stevenabreu7 · 2023-08-09T11:52:13Z

Issue number: #217 - interoperability of neuromorphic simulators and hardware platforms through NIR.

Objective of pull request: support neuromorphic intermediate representation (NIR) in lava-dl.

Questions

For NIR → lava-dl, I am currently creating a new Network(torch.nn.Module) with slayer blocks for each layer. Should I instead convert the NIR HDF5 file directly to a netx HDF5 file?
How to best handle the discretization or other conversion/optimization issues to make imported networks Loihi-compatible?

Notes

things that change when going lava-dl > NIR > lava-dl
- will create an input layer if it doesn't exist already
- weights will be modified
- vDecay, vThMant, iDecay, refDelay will be replaced by some default values (?)

Things left to do

ensure (and test) that lava-dl <> NIR does not lose information. E.g., converting from lava-dl to NIR and back should still run.
- what gets lost with lava-dl > NIR > lava-dl: simulation.Ts, simulation.tSample,
handle some parameters/cases that are currently ignored (see TODOs in the code).
don't add input layers when nir → lava

Pull request checklist

Your PR fulfills the following requirements:

Issue created that explains the change and why it's needed
Tests are part of the PR (for bug fixes / features)
Docs reviewed and added / updated if needed (for bug fixes / features)
PR conforms to Coding Conventions
PR applys BSD 3-clause or LGPL2.1+ Licenses to all code files
Lint (flakeheaven lint src/lava tests/) and (bandit -r src/lava/.) pass locally
Build tests (pytest) passes locally

Pull request type

Please check your PR type:

What is the current behavior?

lava-dl uses netx to export and import trained networks

What is the new behavior?

lava-dl can now use NIR (in addition to netx) to export and import networks trained in lava-dl or in other simulators.

Does this introduce a breaking change?

Yes
No

Supplemental information

bamsumit · 2023-08-09T18:02:18Z

Thanks, @stevenabreu7 for getting this started. I'll take a close look at the PR soon.

One thing that pops up is the license of https://github.com/neuromorphs/NIR I don't see a license for it. Can we sort out its licensing first? We prefer permissive licensing like BSD-3. Non-permissive licensing models, even GPL, are problematic to get legal approval.

stevenabreu7 · 2023-08-09T18:11:15Z

Thanks for the quick reply @bamsumit. The licensing should not be a problem, I will check with the others. Would MIT or BSD-3 be okay?

The code is still work-in-progress, but if you could take a look and advise on the questions/comments that I added in the PR comment on top, that would be great. Thanks a lot!

bamsumit · 2023-08-09T20:19:08Z

BSD-3 would be great :)

bamsumit

@stevenabreu7 I have left some comments. As you get the PR ready, I'll add more comments on the dostrings and TODOs. General rule is we follow np style docstrings, avoid commented code, and TODO codes as much as possible.

bamsumit · 2023-08-15T20:27:10Z

src/lava/lib/dl/netx/__init__.py


-__all__ = ['hdf5', 'blocks', 'utils']
+__all__ = ['hdf5', 'blocks', 'utils', 'nir_lava']


I prefer just nir, but its up to you. Anyway it needs to be consistent with README which says nir

nir is also the name of the NIR library itself, so I thought nir_lava makes it easier to avoid conflicts (rather than import ... as ...)

Sure. Then lets make it consistent in the README.

bamsumit · 2023-08-15T20:28:25Z

src/lava/lib/dl/netx/nir_lava.py

+PATH_TYPE = typing.Union[str, pathlib.Path]
+
+
+def read_cuba_lif(layer: h5py.Group, shape: Tuple[int] = None) -> nir.NIRNode:


The name should probably be read_neuron as it looks like it is going to handle other neuron types in the future.

bamsumit · 2023-08-15T20:30:14Z

src/lava/lib/dl/netx/nir_lava.py

+
+
+def nir_graph_to_lava_network(graph: nir.NIRGraph) -> NetworkNIR:
+    """Converts a NIRGraph to a Lava network."""


It creates Lava-DL network which is different from Lava network. Lets make that clear.

bamsumit · 2023-08-15T20:32:08Z

tests/lava/lib/dl/netx/test_nir.py

+        netx.nir_lava.convert_to_nir(oxford_net_path, 'oxford.nir')
+        oxford_nir = nir.read('oxford.nir')
+        os.remove('oxford.nir')
+        self.assertIsNotNone(oxford_nir)


I would also further suggest exporting oxford_nir.export_hdf5('temp.net') and checking diff with the original source.

A question on this: do input layers modify the resulting dynamics of the network? in pytorch this is not the case, it's basically just a wrapper for the input data that stores the data shape, but in slayer the input layer also has neuron parameters which makes me think that the input spikes would be fed through an input layer, and the resulting spike train is then different from the original one. or is the slayer input layer just "mock input neurons" and the neuron parameters are irrelevant?

The input layers typically apply neuron dynamics. Exponential decaying dynamics in case of CUBA-LIF, Delta dynamics in case of Sigma-Delta and so on. This is different than PyTorch.

stevenabreu7 · 2023-08-25T11:16:38Z

Thanks for all your help so far @bamsumit. I've been struggling to get the weights and parameters to match between an original lava-dl-generated .net file and the .net file that I get when going to NIR and back. So I have a general question for how to proceed with this.

For lava-dl → NIR, should I 1) use a lava-dl network (i.e. torch.nn.Module with slayer blocks) to go to NIR or 2) use the generated netx hdf5 file to go to NIR? So far I have done 2), but now when I try to go back from the generated NIR graph to a lava-dl network, I need to reverse engineer the quantisation done by lava-dl to make the weights and parameters match.

Similarly, for NIR → lava-dl, it seems that creating a torch.nn.Module with slayer blocks may be better because then we could simply use all the Loihi-specific quantisation schemes that are already part of lava-dl.

With NIR, we want to be able to go lava-dl → NIR → lava-dl easily, but we also want to be able to go from other SNN simulators to NIR, and then to lava-dl. And most SNN simulators will not much/any quantization of the weights/params.

What do you think is the best way to proceed?

bamsumit · 2023-09-25T14:47:03Z

Thanks for all your help so far @bamsumit. I've been struggling to get the weights and parameters to match between an original lava-dl-generated .net file and the .net file that I get when going to NIR and back. So I have a general question for how to proceed with this.

For lava-dl → NIR, should I 1) use a lava-dl network (i.e. torch.nn.Module with slayer blocks) to go to NIR or 2) use the generated netx hdf5 file to go to NIR? So far I have done 2), but now when I try to go back from the generated NIR graph to a lava-dl network, I need to reverse engineer the quantisation done by lava-dl to make the weights and parameters match.

Similarly, for NIR → lava-dl, it seems that creating a torch.nn.Module with slayer blocks may be better because then we could simply use all the Loihi-specific quantisation schemes that are already part of lava-dl.

With NIR, we want to be able to go lava-dl → NIR → lava-dl easily, but we also want to be able to go from other SNN simulators to NIR, and then to lava-dl. And most SNN simulators will not much/any quantization of the weights/params.

What do you think is the best way to proceed?

Hi @stevenabreu7 it is up to you. I am okay with lava-dl -> NIR -> lava-dl route. Only concern is that it adds an extra layer of dependency compared to netx<->NIR translation. Can you elaborate what issue with weight and parameters you are facing? I am guessing its a quantization scaling issue. Perhaps this might be helpful:

Device parameters to neuron parameters translation: https://github.com/lava-nc/lava-dl/blob/main/src/lava/lib/dl/slayer/neuron/cuba.py#L26
Neuron parameters to device parameter translation: https://github.com/lava-nc/lava-dl/blob/main/src/lava/lib/dl/slayer/neuron/cuba.py#L263
Weight quantization method: https://github.com/lava-nc/lava-dl/blob/main/src/lava/lib/dl/slayer/neuron/base.py#L168 descale here will result in device parameters.

stevenabreu7 and others added 6 commits July 26, 2023 14:16

setup + preliminary Lava -> NIR support

2d84e17

Merge branch 'lava-nc:main' into NIR

e8089ff

preliminary NIR <> lava-dl and basic tests

fa74452

refactor for linting + allow assert statements

9e6f3d3

remove assertions for linting + add NIR package

5978836

install NIR

e8f1864

bamsumit linked an issue Aug 9, 2023 that may be closed by this pull request

Support for NIR <> lava-dl #217

Open

bamsumit reviewed Aug 15, 2023

View reviewed changes

stevenabreu7 added 2 commits August 23, 2023 12:27

refactore (PR comments), infer lava-dl net shapes

87e6dfb

make readme consistent

7fcf390

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NIR <> lava-dl support #218

NIR <> lava-dl support #218

stevenabreu7 commented Aug 9, 2023 •

edited

Loading

bamsumit commented Aug 9, 2023 •

edited

Loading

stevenabreu7 commented Aug 9, 2023

bamsumit commented Aug 9, 2023

bamsumit left a comment

bamsumit Aug 15, 2023

stevenabreu7 Aug 23, 2023

bamsumit Aug 24, 2023

bamsumit Aug 15, 2023

bamsumit Aug 15, 2023

bamsumit Aug 15, 2023

stevenabreu7 Aug 23, 2023

stevenabreu7 Aug 23, 2023

bamsumit Aug 24, 2023

stevenabreu7 commented Aug 25, 2023

bamsumit commented Sep 25, 2023


		__all__ = ['hdf5', 'blocks', 'utils']
		__all__ = ['hdf5', 'blocks', 'utils', 'nir_lava']

		PATH_TYPE = typing.Union[str, pathlib.Path]


		def read_cuba_lif(layer: h5py.Group, shape: Tuple[int] = None) -> nir.NIRNode:



		def nir_graph_to_lava_network(graph: nir.NIRGraph) -> NetworkNIR:
		"""Converts a NIRGraph to a Lava network."""

NIR <> lava-dl support #218

Are you sure you want to change the base?

NIR <> lava-dl support #218

Conversation

stevenabreu7 commented Aug 9, 2023 • edited Loading

Pull request checklist

Pull request type

What is the current behavior?

What is the new behavior?

Does this introduce a breaking change?

Supplemental information

bamsumit commented Aug 9, 2023 • edited Loading

stevenabreu7 commented Aug 9, 2023

bamsumit commented Aug 9, 2023

bamsumit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevenabreu7 commented Aug 25, 2023

bamsumit commented Sep 25, 2023

stevenabreu7 commented Aug 9, 2023 •

edited

Loading

bamsumit commented Aug 9, 2023 •

edited

Loading