JOSS Review

[ranocha]

JOSS Review

Xref openjournals/joss-reviews#5772

• Contribution and authorship: Does the full list of paper authors seem appropriate and complete?
  - There is a description how to cite the package at https://github.com/worlddynamics/WorldDynamics.jl#how-to-cite-this-work. However, the author list there does not coincide with the JOSS author list.
  - Moreover, it looks like an additional person (@universmile) has made non-trivial contributions - at least based on the GitHub commit list and compared to other authors.
• There are no installation instructions in the documentation. Since there is something in the README.md, I suggest to include it in the online docs, too.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers.
  I can only partially reproduce the results. First, it needs some experimentation to find out that "figure 7-7" or "Fig. 7.7" is reproduced by World3.fig_7(). This could be stated more explicitly. Next, I can't overlay the original plot - but I also don't have access to the book and putting it only may be complicated due to copyright restrictions.
• The docs of many functions such as https://worlddynamics.github.io/WorldDynamics.jl/stable/source/#WorldDynamics.World1.fig_1-Tuple{} do not tell me how to call these functions - there is no function/method signature.
• There seem to be some tests that can be run locally. However, there is no description of this in the docs. Moreover, there are no automated CI tests.
• There are no community guidelines.
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)?
  - Julia itself should be cited.
  - The paper mentions official implementations on Vensim and Stella without any references.

[aurorarossi]

We thank the reviewer for his useful comments on our work.

• Contribution and authorship: Does the full list of paper authors seem appropriate and complete?
  - There is a description how to cite the package at https://github.com/worlddynamics/WorldDynamics.jl#how-to-cite-this-work. However, the author list there does not coincide with the JOSS author list.
  - Moreover, it looks like an additional person (@universmile) has made non-trivial contributions - at least based on the GitHub commit list and compared to other authors.

Regarding the contribution, the one showed in the README is outdated and we were aiming at replacing it with a direct reference to a publication on our work (e.g. JOSS, if accepted).
The author @aurorarossi has been included as an author both for her involvement in the development (past and present) of the overall software produced by the WorldDynamics organization and for having been the main author of the submitted paper. While not evident from the line counts of the commit history, @universmile's has contributed less substantially to the software and, after having co-authored the extended abstract presented at ROADEF 2023 (without proceedings), he quit the project development group at the end of October 2022.

• There are no installation instructions in the documentation. Since there is something in the README.md, I suggest to includet it in the online docs, too.

Thank you for the remark. We added a Getting Started section to the Documentation as well.

• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers.
  I can only partially reproduce the results. First, it needs some experimentation to find out that "figure 7-7" or "Fig. 7.7" is reproduced by World3.fig_7(). This could be stated more explicitly. Next, I can't overlay the original plot - but I also don't have access to the book and putting it only may be complicated due to copyright restrictions.

• The docs of many functions such as https://worlddynamics.github.io/WorldDynamics.jl/stable/source/#WorldDynamics.World1.fig_1-Tuple{} do not tell me how to call these functions - there is no function/method signature.

We provided specific pointers to the original figures corresponding to each function in the docstrings of the function itself, which gets exported in the documentation.
Indeed we did not emphasize this. In the Getting Started section that we added (answer to previous comment) we make this more explicit.
We indeed refrained from providing all original pictures because of copyrights. The book is publicly accessible (with mild limitations) at https://archive.org/details/dynamicsofgrowth0000unse . By referring to this version we were able to verify the correspondence of all plots with the original figures.
We also added the explicit signature to the docstring of each function producing the figures.

• There seem to be some tests that can be run locally. However, there is no description of this in the docs. Moreover, there are no automated CI tests.

The tests currently present are only mocked: they only check whether some figure is produced at the end. Because of the nature of the package, we did not manage, at least as for now, to design meaningful tests. For this reason, we did not deem it worth to document them and automate them. To avoid confusion, we propose to remove the test folder althogether.

• There are no community guidelines.

We do not provide specific guidelines. In the README, we added a statement saying that we adhere to the general standards of the Julia community.

• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)?
  - Julia itself should be cited.
  - The paper mentions official implementations on Vensim and Stella without any references.

We thank the reviewer for the remark. We added references to Julia, and to the Vensim and Stella references. In fact, after carefully checking, the Vensim implementation is not an official one. The only official implementation which should be available is the Stella implementation of World3-03 which should ship in a CD-ROM together with the book by Meadows et al. 2004. We corrected our text accordingly.
We remark that we were not able to retrieve such CD (even by asking to the publisher of the book), and that the Stella software is proprietary and not available on Linux.

[ranocha]

Regarding the contribution, the one showed in the README is outdated and we were aiming at replacing it with a direct reference to a publication on our work (e.g. JOSS, if accepted).

I can only review the current state of the repo. Could you please update the reference in the README.md file? Then, I can check the box.

[ranocha]

We added a Getting Started section to the Documentation as well.

Please release a new version so that it shows up in the stable docs.

[ranocha]

The tests currently present are only mocked: they only check whether some figure is produced at the end. Because of the nature of the package, we did not manage, at least as for now, to design meaningful tests. For this reason, we did not deem it worth to document them and automate them. To avoid confusion, we propose to remove the test folder althogether.

Please consider this carefully. The required JOSS checklist contains the item

Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?

[ranocha]

We do not provide specific guidelines. In the README, we added a statement saying that we adhere to the general standards of the Julia community.

The JOSS review checklist contains the item

Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

I need something to be able to check this box. You could for example consider setting up a CONTRIBUTING.md file with the relevant information.

[aurorarossi]

We have made a new release that includes the new docs with updated Getting Started information, Automatic Test Actions, and Contributing Guidelines in the CONTRIBUTING.md file. We have also updated the README.md accordingly.
Thank you

[ranocha]

Thank you