New biobb_pytorch Molecular dynamics autoencoder wrapper

[PauAndrio]

Wrappers for the new collection of blocks to train and apply autoencoder models to molecular dynamics data.

[bgruening]

Please have a look at the linting logs: https://github.com/galaxycomputationalchemistry/galaxy-tools-compchem/actions/runs/9906049158/job/27368207101?pr=173

planemo can also test this with planemo lint

[bgruening]

If this output is optional you can use a filter here to only define this output when your boolean is true.

[bgruening]

Suggested change

	`#Required Outputs using file extension`
	touch '$output_reconstructed_data_npy_path' &&
	ln -s '$output_reconstructed_data_npy_path' ./output_reconstructed_data_npy_path.$output_reconstructed_data_npy_path.ext &&
	`#Optional Outputs using file extension`
	#if $output_latent_space_npy_path:
	touch '$output_latent_space_npy_path' &&
	ln -s '$output_latent_space_npy_path' ./output_latent_space_npy_path.$output_latent_space_npy_path.ext &&
	#end if

This looks very complicated, I guess you can remove this and try something like below

[PauAndrio]

Thank you for your feedback. I removed the part you mentioned, used a literal in the command, and utilized the "from_work_dir" attribute. This has significantly simplified things.

[bgruening]

Cool glad you found that useful. And sorry this PR takes so long, my hope is that if we do this one properly the rest will be a piece of cake :-)

[PauAndrio]

Thank you. I have also removed the comments inside the command section to avoid potential issues with quotes or other special characters in the future. Is there anything else that needs to be changed before we merge?

[bgruening]

You could also, and that is the simplest way possible, just write in the command section: --output ./foo.npz.

And than in your outputs do <output name="blabla" from_work_dir="foo.npz"/>. The from_work_dir will then pick up the file and you don't need a mv or ln

[bgruening]

https://docs.galaxyproject.org/en/latest/dev/schema.html#id82

[bgruening]

Just an idea. Is that something that is interesting?

[bgruening]

pickled objects are a bit dangerous, would it be possible to use ONNX https://pytorch.org/docs/stable/onnx.html

[bgruening]

ping, any comments here?
Pickles contain arbitrary code and are a bad exchange format, as you need to trust them.

[PauAndrio]

While ONNX is a great choice for model export, as you suggested, and exporting models to ONNX is straightforward, there is currently no built-in functionality in PyTorch for importing ONNX models back into PyTorch. This has been a long-standing request from the PyTorch community since 2019, but as of now, there's no reliable or officially supported method for importing ONNX models back into PyTorch for further training or fine-tuning.

Given this limitation, switching entirely to ONNX would mean losing the ability to easily retrain, modify, or extend our models within the PyTorch ecosystem. For these reasons, despite the risks associated with pickling, we still favor the use of PyTorch’s torch.save() and torch.load() methods for model serialization. We're aware that these methods are not ideal from a security perspective, but they provide the necessary flexibility for ongoing model development.

Our intent developing this tools is to provide a way to create or improve very specialized models not to deploy them.

[bgruening]

Yeah, its a bit of a pain, I had hoped that https://github.com/Talmaj/onnx2pytorch is working for you.

[bgruening]

Are users supposed to create such JSON files?

How many parameters do such files have in average? Is that 1-2 or more like 20-30?

[PauAndrio]

The biobb_object has 38 configurable properties, and in addition, train_mdae introduces 16 specific properties, bringing the total to 54 potential parameters that users could modify in the JSON files. However, all of these properties come with default values, so users only need to tweak those that are relevant to their specific use case. We’ve put considerable effort into documenting these parameters and guiding users to focus on the most impactful ones, ensuring they can make adjustments as needed without being overwhelmed.

[bgruening]

Suggested change

	<configfiles>
	<configfile name="train_config">
	{
	"properties": {
	"num_epochs": $num_epoch,
	"seed": $seed
	}
	}
	</configfile>
	</configfiles>

This way you can create those configfiles on the fly and ask your users for the inputs

[PauAndrio]

That’s something we’re planning for a future release, where the most relevant properties will be integrated into Galaxy's UI through sliders, multi-select options, number validators, filters, etc., making the configuration process more user-friendly. However, for now, I’d like to keep things as simple as possible and focus on getting my first tool published in the Galaxy Toolshed.

[bgruening]

Suggested change

	#if $config_json:
	ln -s '$config_json' ./config_json.$config_json.ext &&
	#end if

[PauAndrio]

That’s something we’re planning for a future release, where the most relevant properties will be integrated into Galaxy's UI through sliders, multi-select options, number validators, filters, etc., making the configuration process more user-friendly. However, for now, I’d like to keep things as simple as possible and focus on getting my first tool published in the Galaxy Toolshed.

[bgruening]

Suggested change

	--config ./config_json.$config_json.ext
	--config ./$train_config

[PauAndrio]

That’s something we’re planning for a future release, where the most relevant properties will be integrated into Galaxy's UI through sliders, multi-select options, number validators, filters, etc., making the configuration process more user-friendly. However, for now, I’d like to keep things as simple as possible and focus on getting my first tool published in the Galaxy Toolshed.

[bgruening]

@PauAndrio looks good to me. Just this one security issue with the python pickels.

[PauAndrio]

@PauAndrio looks good to me. Just this one security issue with the python pickels.

Sorry I thought this was already answered:

While ONNX is a great choice for model export, as you suggested, and exporting models to ONNX is straightforward, there is currently no built-in functionality in PyTorch for importing ONNX models back into PyTorch. This has been a long-standing request from the PyTorch community since 2019, but as of now, there's no reliable or officially supported method for importing ONNX models back into PyTorch for further training or fine-tuning.

Given this limitation, switching entirely to ONNX would mean losing the ability to easily retrain, modify, or extend our models within the PyTorch ecosystem. For these reasons, despite the risks associated with pickling, we still favor the use of PyTorch’s torch.save() and torch.load() methods for model serialization. We're aware that these methods are not ideal from a security perspective, but they provide the necessary flexibility for ongoing model development.

Our intent developing this tools is to provide a way to create or improve very specialized models not to deploy them.

[bgruening]

Thanks @PauAndrio and sorry for the long turn-around.

Please change the shed.yml and then we merge.

[bgruening]

Yeah, its a bit of a pain, I had hoped that https://github.com/Talmaj/onnx2pytorch is working for you.

[bgruening]

can you convert this file into a suite-based file? https://github.com/galaxyproject/tools-iuc/blob/main/tools/semibin/.shed.yml

[bgruening]

Suggested change

;

[PauAndrio]

Hi, @bgruening,

Please, let me know if there is anything else that I have to do to resolve and merge this pull request?

Regards,
Pau

[bgruening]

Let's get this in. Thanks a lot @PauAndrio!