This repository contains the PyTorch implementation of the methods described in the paper "What Should a Neuron Aim For? Designing Local Objective Functions Based on Information Theory.". The code introduces a novel PyTorch layer type that leverages Partial Information Decomposition (PID) to optimize local information processing of each neuron.
The Infomorphic_Network package was developed by Valentin Neuhaus in cooperation with Andreas C. Schneider and David A. Ehrlich and is actively maintained by the aforementioned.
All packages needed to run the code are listed in env.yaml
. To create a conda environment with all the necessary packages and dependencies, and to activate the environment, run the following commands:
conda env create -f env.yaml
conda activate infomorphic_env
To run a specific infomorpic network model, you can use the following command:
python paper_trivariate/base_model.py +models=model_name +goals/model_name=goal_name
Where model_name
is the name of the model you want to run and goal_name
is the name of the goal you want to run. The available models are:
infomorphic_context
infomorphic_lateral
infomorphic_context_lateral
infomorphic_context_lateral_feedback
infomorphic_random_projection
*
The available goal functions for each of the models can be found in the goals
folder.
The models marked with * don't have a specific goal function.
To run a single layer model, you can use the following command:
python paper_trivariate/base_model_singlelayer.py +models=infomorphic_readout
All results, like performance and atom sizes of the neurons, as well as the parameters used, are saved to the experiments
folder. The results are saved in a subfolder with the structure dataset_name/model_name/goal_name/singleruns/day_month_year/hour_minute_second
.
To perform multiple runs at once, you can list the parameters that are different in the runs in /conf/hydra_base_config.yaml
. The parameters that are different in the runs are listed in the multirun
section of the configuration file. To perform multiple runs, you can use the following command:
python paper_trivariate/base_model.py --multirun +models=model_name +goals/model_name=goal_name
The results are saved in a subfolder with the structure dataset_name/model_name/goal_name/multiruns/day_month_year/hour_minute_second/job_idx
.
To perform an optimization, you can use the following command:
python paper_trivariate/base_model.py --multirun +models=model_name +goals/model_name=optuna_sweep.yaml hydra=hydra_optuna_config.yaml
Before performing the optimization, check the hydra_optuna_config
file to set the parameters of the optimization. The optimization runs are saved in a subfolder with the structure
dataset_name/model_name/goal_name/optuna_runs/day_month_year/hour_minute_second
and the optuna study is saved in a database in dataset_name/model_name/goal_name/optuna_runs/optuna_results.sqlite3
. The study name can be changed in the hydra_optuna_config
file.
To modify the parameters of the model, you can use the hydra
configuration file. The configuration files are located in the conf
folder. The configuration files are written in YAML format. To modify the parameters of the model, you can change the values in the configuration file. Additionally, one can add overrides to the command line to change the parameters of the model. For example, to change the number of epochs of the model, you can use the following command:
python paper_trivariate/base_model.py +models=model_name +goals/model_name=goal_name exp_params.epochs=500
To reproduce the figures of the paper, look at the juptyer notebooks in the notebooks
folder. The notebooks list the runs that are necessary to reproduce the figure as well as the code to generate the figure. The figures of the paper are saved in the figures
folder.