dashboard and tutorials data, models and labels locations (issue #667)

[elboyran]

solves issue #667

[elboyran]

Conflicting files seems to be only about clear output from notebooks. Not sure how to ignore this.

[elboyran]

Old and proposed folder structures of dianna/dianna and dianna/tutorials :

main branch

dianna (6 directories, 56 files)

├── init.py
├── ...
├── dashboard
│   ├── ...
├── data
│   ├── bee_2.png
│   ├── binary-mnist.npz
│   ├── digit0.jpeg
│   ├── digit0.jpg
│   ├── digit0.png
│   ├── digit1.png
│   ├── labels_mnist.txt
│   ├── labels_resnet50.txt
│   ├── labels_text.txt
│   ├── leafsnap_model.onnx
│   ├── logo.png
│   ├── mnist_model_tf.onnx
│   ├── movie_review_model.onnx
│   ├── movie_reviews_word_vectors.txt
│   ├── season_prediction_model_temp_max_binary.onnx
│   ├── weather_data.npy
│   └── weather_data_labels.txt
├── methods
│   ├── ...
├── utils
│   ├── ...
└── visualization
│   ├── ...

tutorials (4 directories, 38 files)

├── README.md
├── conversion_onnx
│   ├── ...
├── data
│   ├── binary-mnist.npz
│   ├── coffee_test.csv
│   ├── coffee_train.csv
│   ├── leafsnap_classes.csv
│   ├── leafsnap_example_acer_rubrum.jpg
│   ├── movie_reviews_word_vectors.txt
│   └── shapes.npz
├── demo.ipynb
├── img
│   ├── bee.jpg
│   └── digit0.jpg
├── kernelshap_geometric_shapes.ipynb
├── kernelshap_mnist.ipynb
├── kernelshap_tabular_penguin.ipynb
├── kernelshap_tabular_weather.ipynb
├── lime_images.ipynb
├── lime_tabular_penguin.ipynb
├── lime_tabular_weather.ipynb
├── lime_text.ipynb
├── lime_timeseries_coffee.ipynb
├── lime_timeseries_weather.ipynb
├── models
│   ├── coffee.onnx
│   ├── geometric_shapes_model.onnx
│   ├── leafsnap_model.onnx
│   ├── mnist_model.onnx
│   ├── mnist_model_tf.onnx
│   ├── movie_review_model.onnx
│   ├── penguin_model.onnx
│   ├── season_prediction_model_temp_max_binary.onnx
│   └── sunshine_hours_regression_model.onnx
├── rise_imagenet.ipynb
├── rise_mnist.ipynb
├── rise_text.ipynb
└── rise_timeseries_weather.ipynb

---

667-reorganize branch

dianna (8 directories, 66 files)

├── init.py
├── ...
├── dashboard
│   ├── ...
├── data
│   ├── bee.jpg
│   ├── binary-mnist.npz
│   ├── digit0.jpeg
│   ├── digit0.jpg
│   ├── digit0.png
│   ├── digit1.png
│   ├── leafsnap_example_acer_rubrum.jpg
│   ├── logo.png
│   ├── shapes.npz
│   └── weather_data.npy
├── labels
│   ├── coffee_test.csv
│   ├── coffee_train.csv
│   ├── labels_mnist.txt
│   ├── labels_resnet50.txt
│   ├── labels_text.txt
│   ├── leafsnap_classes.csv
│   ├── movie_reviews_word_vectors.txt
│   └── weather_data_labels.txt
├── methods
│   ├── ...
├── models
│   ├── coffee.onnx
│   ├── geometric_shapes_model.onnx
│   ├── leafsnap_model.onnx
│   ├── mnist_model.onnx
│   ├── mnist_model_tf.onnx
│   ├── movie_review_model.onnx
│   ├── penguin_model.onnx
│   ├── season_prediction_model_temp_max_binary.onnx
│   └── sunshine_hours_regression_model.onnx
├── utils
│   ├── ...
└── visualization
├── ...

tutorials (5 directories, 20 files)

├── README.md
├── conversion_onnx
│   ├── ...
├── demo.ipynb
└── explainers
├── KernelSHAP
│   ├── kernelshap_geometric_shapes.ipynb
│   ├── kernelshap_mnist.ipynb
│   ├── kernelshap_tabular_penguin.ipynb
│   └── kernelshap_tabular_weather.ipynb
├── LIME
│   ├── lime_images.ipynb
│   ├── lime_tabular_penguin.ipynb
│   ├── lime_tabular_weather.ipynb
│   ├── lime_text.ipynb
│   ├── lime_timeseries_coffee.ipynb
│   └── lime_timeseries_weather.ipynb
└── RISE
├── rise_imagenet.ipynb
├── rise_mnist.ipynb
├── rise_text.ipynb
└── rise_timeseries_weather.ipynb

[loostrum]

Very nice, thanks for reorganizing this! I very much like that all data files are in one place now. I do have one thing to discuss: we now include everything under dianna/data in the pip package, so everyone that install dianna gets a pretty big download as it includes all those files. However, the files are only needed for the tutorials and dashboard. Would it make sense to not include the bigger files in the pip package, and only download them when needed in the notebooks/dashboard code?

[elboyran]

Very nice, thanks for reorganizing this! I very much like that all data files are in one place now. I do have one thing to discuss: we now include everything under dianna/data in the pip package, so everyone that install dianna gets a pretty big download as it includes all those files. However, the files are only needed for the tutorials and dashboard. Would it make sense to not include the bigger files in the pip package, and only download them when needed in the notebooks/dashboard code?

You are right. Maybe after addressing #171 it will not be an issue.

[elboyran]

@cwmeijer do you mind merging this after the new release?

[loostrum]

Very nice, thanks for reorganizing this! I very much like that all data files are in one place now. I do have one thing to discuss: we now include everything under dianna/data in the pip package, so everyone that install dianna gets a pretty big download as it includes all those files. However, the files are only needed for the tutorials and dashboard. Would it make sense to not include the bigger files in the pip package, and only download them when needed in the notebooks/dashboard code?

You are right. Maybe after addressing #171 it will not be an issue.

Let's leave it for now indeed. For reference: it can be done by editing the MANIFEST.in file. The pypi package default is to not include any non-python files.

[elboyran]

Very nice, thanks for reorganizing this! I very much like that all data files are in one place now. I do have one thing to discuss: we now include everything under dianna/data in the pip package, so everyone that install dianna gets a pretty big download as it includes all those files. However, the files are only needed for the tutorials and dashboard. Would it make sense to not include the bigger files in the pip package, and only download them when needed in the notebooks/dashboard code?

You are right. Maybe after addressing #171 it will not be an issue.

Let's leave it for now indeed. For reference: it can be done by editing the MANIFEST.in file. The pypi package default is to not include any non-python files.

Is your review for this PR positive then? ;-)

[loostrum]

Good point, it is! :)

There are just 2 merge conflicts to resolve first in the notebooks, but I assume that will work out

[elboyran]

Good point, it is! :)

There are just 2 merge conflicts to resolve first in the notebooks, but I assume that will work out

Indeed, as I mentioned earlier I think these are just 2 notebooks that differ in with or without cell output. I'm not fluent enough in git to resolve manually the conflicts (cannot be done anutomatically). Can someone who knows better, merge the branch 667-reorganise to main? @loostrum, @cwmeijer ?

[loostrum]

I'll try fixing it now

[elboyran]

[heart] Elena Ranguelova reacted to your message:

…

________________________________ From: Leon Oostrum ***@***.***> Sent: Wednesday, March 20, 2024 11:46:47 AM To: dianna-ai/dianna ***@***.***> Cc: Elena Ranguelova ***@***.***>; Assign ***@***.***> Subject: Re: [dianna-ai/dianna] dashboard and tutorials data, models and labels locations (issue #667) (PR #699) I'll try fixing it now — Reply to this email directly, view it on GitHub<#699 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAYYBWJC3H4IN26XZX2E3TTYZFZKPAVCNFSM6AAAAABCMLSU2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBZGM4DCNRTGM>. You are receiving this because you were assigned.Message ID: ***@***.***>