[REVIEW]: NiaAML: AutoML framework based on stochastic population-based nature-inspired algorithms

[whedon]

Submitting author: @lukapecnik (Luka Pečnik)
Repository: https://github.com/lukapecnik/NiaAML
Version: 1.1.0
Editor: @arfon
Reviewer: @adi3, @sara-02
Archive: 10.5281/zenodo.4782445

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/273ea8b67562eab528ec57c0458449cb"><img src="https://joss.theoj.org/papers/273ea8b67562eab528ec57c0458449cb/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/273ea8b67562eab528ec57c0458449cb/status.svg)](https://joss.theoj.org/papers/273ea8b67562eab528ec57c0458449cb)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@adi3 & @sara-02, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @arfon know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @adi3

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@lukapecnik) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @sara-02

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@lukapecnik) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @adi3, @sara-02 it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1007/978-981-15-2133-1_13 is OK
- 10.21105/joss.00613 is OK
- 10.1109/IJCNN.2015.7280767 is OK
- 10.1007/978-3-030-05318-5 is OK

MISSING DOIs

- 10.1007/978-3-030-05318-5_8 may be a valid DOI for title: TPOT: A tree-based pipeline optimization tool for automating machine learning

INVALID DOIs

- None

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[arfon]

@adi3, @sara-02 - This is the review thread for the paper. All of our communications will happen here from now on.

Please read the "Reviewer instructions & questions" in the first comment above.

Both reviewers have checklists at the top of this thread (in that first comment) with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention #2949 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for the review process to be completed within about 4-6 weeks but please make a start well ahead of this as JOSS reviews are by their nature iterative and any early feedback you may be able to provide to the author will be very helpful in meeting this schedule.

[adi3]

@lukapecnik thank you for your submission to JOSS. Your paper is a great start, but can do with a few modifications. Here are a few recommendations:

1. The summary section gets a little repetitive. For instance:

• "The framework incorporates stochastic population-based nature-inspired algorithms to search for the best classification pipeline"
• "NiaAML is slightly different, regarding the search for a perfect pipeline with the use of stochastic population-based nature-inspired algorithms"
• "In conclusion, NiaAML is an AutoML framework based on stochastic population-based nature-inspired algorithms."

Please edit the summary to avoid such repetitions.

2. The first paragraph of your Summary should to belong to the Statement of Need section.

3. I would like to see some content, if only a couple of lines, on what this "nature-inspired algorithm" is. How is it nature-inspired and why is that a good approach? It seems like that is the critical scholarly effort here, and without any explanation around this concept, there is nothing that clearly distinguishes NiaAML from other AutoML libraries.

[lukapecnik]

The authors would like to thank the Reviewer @adi3 sincerely for the
time spent on reading and commenting on the paper.

The given remarks are very thorough, and extremely helpful to overcome
the shortcomings and ambiguities in our paper.

In what follows, a point-by-point reply to the Reviewer’s remarks,
comments and suggestions is given.

---

1. Repetitive sentences are now omitted from the manuscript.

2. As per your request, the first paragraph in the Summary Section has
   been moved to the Statement of Need Section.

3. Good point! We have added a few lines of nature-inspired algorithms`
   description. Moreover, we have also added some
   literature about nature-inspired algorithms where the readers can gain
   more information about these concepts.
   Arguments that distinguish NiaAML from the other similar solutions in a
   little bit more detail have now been added to
   our paper.

[adi3]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[adi3]

@lukapecnik Thanks for the changes. I will be making some editorial changes myself to the draft and issuing a pull request. Please review it and merge it into your master branch. These edits will mostly be to correct syntactic usage or word choices. The aim is to keep the article crisp and to the point. It is faster for me to make these changes and issue a pull request instead of leading you through them. Let me know if you have any questions. Once you have merged the pull request, please notify me here so we can proceed.

[lukapecnik]

@adi3 Thank you for the pull request. I have reviewed and merged it.

[adi3]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[adi3]

@lukapecnik Thank you. While I go through the remainder of the checklist, can you please figure out why one of your references is showing up as having a missing DOI?

MISSING DOIs

- 10.1007/978-3-030-05318-5_8 may be a valid DOI for title: TPOT: A tree-based pipeline optimization tool for automating machine learning

[lukapecnik]

@adi3 We have added the missing DOI after the submission. I think this should be fixed now.

[adi3]

@whedon check references

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1007/978-981-15-2133-1_13 is OK
- 10.21105/joss.00613 is OK
- 10.1109/IJCNN.2015.7280767 is OK
- 10.1007/978-3-030-05318-5_8 is OK
- 10.1007/978-3-030-05318-5 is OK
- 10.1002/9780470512517 is OK
- 10.1016/b978-0-12-416743-8.00017-8 is OK
- 10.1016/j.dib.2020.105792 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

👋 @adi3, please update us on how your review is going.

[whedon]

👋 @sara-02, please update us on how your review is going.

[adi3]

I'll make further progress on my checklist this week.

[adi3]

@lukapecnik I'm reviewing your installation instructions.

• I recommend you specify supported OS that your software has been tested on and is confirmed to work. For instance, I was able to do pip installation just fine on Ubuntu, but ran into hurdles installing it on RHEL.
• Suggest you switch pip to pip3 in installation instructions since python2 is deprecated.
• The conda installation is failing at my end. I'm opening an issue on your repo so you can debug and look into it.

Please let me know when these points have been fixed and I'll proceed. Thanks!

[lukapecnik]

@adi3 Thank you.

• We have specified the OS we have tested the NiaAML package on.
• Good point, we have switched pip to pip3 as you suggested.
• Thank you for pointing out the conda failure. We surely will fix this option in the near future, but the investigation may take some time as after a few days we still have not found the reason why it is not working. That is why we decided to temporarily disable the installation using conda so the process of reviewing the package may proceed for now if you agree. Of course I am leaving the issue open.

Thank you.

[lukapecnik]

Hello @adi3 and @sara-02, I hope you are both doing well. How are things going forward with this review? Is there anything we can do so far to improve this framework and make it more suitable for publishing the paper in JOSS?

We are looking forward to hearing from you.
Thank you in advance.

[adi3]

Hi @lukapecnik - I could not find any automated tests in your repo. Can you add some for the key functionalities, or if you already have them, document them in the readme?

[arfon]

Hi @lukapecnik - you have exactly the same information as me at this point sorry. I'd like to give a little more time to @sara-02 before seeking out a second reviewer.

[sara-02]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[sara-02]

Hey @lukapecnik I was trying to run the following example post installing niaaml and Niapy but when I copy pasted the example code to a python file and ran it, I am getting this error:

Traceback (most recent call last):
  File "niml.py", line 1, in <module>
    from niaaml import PipelineOptimizer, Pipeline
  File "/home/sarah/.local/lib/python3.8/site-packages/niaaml/__init__.py", line 3, in <module>
    from niaaml import preprocessing
  File "/home/sarah/.local/lib/python3.8/site-packages/niaaml/preprocessing/__init__.py", line 2, in <module>
    from niaaml.preprocessing import feature_selection
  File "/home/sarah/.local/lib/python3.8/site-packages/niaaml/preprocessing/feature_selection/__init__.py", line 3, in <module>
    from niaaml.preprocessing.feature_selection.jDEFSTH import jDEFSTH
  File "/home/sarah/.local/lib/python3.8/site-packages/niaaml/preprocessing/feature_selection/jDEFSTH.py", line 2, in <module>
    from NiaPy.algorithms.modified import SelfAdaptiveDifferentialEvolution
ModuleNotFoundError: No module named 'NiaPy'

Python: 3.8.5
Ubuntu: 20.04.2 LTS

[lukapecnik]

Hi @sara-02! Thank you for issue report. It seems that there's been a bit of an API change for the niapy package in the meantime. I created a new release of the niaaml (version 1.1.2) that should fix this problem. Please update the niaaml package and then the example should run smoothly.

Best regards

[sara-02]

@arfon @lukapecnik Sorry for the delay from my side, I have completed the review now.

[lukapecnik]

@sara-02 great news! Thank you very much for your effort and reported issue. You helped with making our software better. I wish you all the best 🎉 🙌

@arfon is there anything else that needs to be done from our side now? Thank you

[arfon]

@lukapecnik – At this point could you make a new release of this software that includes the changes that have resulted from this review. Then, please make an archive of the software in Zenodo/figshare/other service and update this thread with the DOI of the archive? For the Zenodo/figshare archive, please make sure that:

• The title of the archive is the same as the JOSS paper title
• That the authors of the archive are the same as the JOSS paper authors

I can then move forward with accepting the submission.

[lukapecnik]

Hello @arfon! I've archived the project using Zenodo. The DOI is 10.5281/zenodo.4782445 and the link to the archive is https://zenodo.org/record/4782445#.YKqPs6gzaUl. I hope this is what you had in mind with updating this thread, otherwise please let me know.

[arfon]

@whedon set 10.5281/zenodo.4782445 as archive

[whedon]

OK. 10.5281/zenodo.4782445 is the archive.

[arfon]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1007/978-981-15-2133-1_13 is OK
- 10.21105/joss.00613 is OK
- 10.1109/IJCNN.2015.7280767 is OK
- 10.1007/978-3-030-05318-5_8 is OK
- 10.1007/978-3-030-05318-5 is OK
- 10.1002/9780470512517 is OK
- 10.1016/b978-0-12-416743-8.00017-8 is OK
- 10.1016/j.dib.2020.105792 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#2335

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#2335, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[arfon]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.02949 joss-papers#2336
1. Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.02949
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[arfon]

@adi3, @sara-02 – many thanks for your reviews here! JOSS relies upon the volunteer efforts of folks like you, and we simply wouldn't be able to do this without you ✨

@lukapecnik – your paper is now accepted and published in JOSS ⚡🚀💥

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.02949/status.svg)](https://doi.org/10.21105/joss.02949)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.02949">
  <img src="https://joss.theoj.org/papers/10.21105/joss.02949/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.02949/status.svg
   :target: https://doi.org/10.21105/joss.02949

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

[lukapecnik]

@arfon thank you very much! 🎉

@adi3 and @sara-02 I thank you again in the name of both authors for your reviews and making our project better.

Best regards