[REVIEW]: Multiple Inference: A Python package for comparing multiple parameters

[editorialbot]

Submitting author: @dsbowen (DILLON BOWEN)
Repository: https://gitlab.com/dsbowen/conditional-inference
Branch with paper.md (empty if default branch):
Version: 1.1.0
Editor: @vissarion
Reviewers: @blakeaw, @mattpitkin, @nhejazi
Archive: 10.5281/zenodo.6859614

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/7a2a4af277c0ad6ad6f41897f4489888"><img src="https://joss.theoj.org/papers/7a2a4af277c0ad6ad6f41897f4489888/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/7a2a4af277c0ad6ad6f41897f4489888/status.svg)](https://joss.theoj.org/papers/7a2a4af277c0ad6ad6f41897f4489888)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@blakeaw & @mattpitkin & @nhejazi, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @vissarion know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @blakeaw

📝 Checklist for @mattpitkin

📝 Checklist for @nhejazi

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.11 s (714.7 files/s, 110912.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          34            964           1320           3077
Jupyter Notebook                10              0           3211           1938
TeX                              1             27              0            259
Markdown                         6             60              0            139
YAML                             4             15             10            122
reStructuredText                13            264            136             88
make                             2             10             12             51
DOS Batch                        1              8              1             26
Bourne Shell                     3              7              6             20
INI                              1              2              0             15
TOML                             1              0              0              6
-------------------------------------------------------------------------------
SUM:                            76           1357           4696           5741
-------------------------------------------------------------------------------


gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 1196

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.3386/w22193 is OK
- 10.3386/w12338 is OK
- 10.3386/w28726 is OK
- 10.2139/ssrn.3689456 is OK
- 10.3386/w25147 is OK
- 10.3386/w23002 is OK
- 10.3386/w19843 is OK
- 10.3386/w25456 is OK
- 10.1257/pandp.20221065 is OK
- 10.1111/j.1468-0262.2005.00615.x is OK
- 10.3386/w26883 is OK
- 10.1525/9780520313880-018 is OK
- 10.1145/3292500.3330771 is OK
- 10.1214/08-aos630 is OK
- 10.1596/27528 is OK
- 10.25080/majora-92bf1922-011 is OK
- 10.1038/s41592-019-0686-2 is OK
- 10.1111/rssb.12162 is OK
- 10.1109/wsc.2015.7408180 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[mattpitkin]

Review checklist for @mattpitkin

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://gitlab.com/dsbowen/conditional-inference?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@dsbowen) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[blakeaw]

Review checklist for @blakeaw

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://gitlab.com/dsbowen/conditional-inference?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@dsbowen) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[nhejazi]

@vissarion it looks like you assigned me as a reviewer but i'm missing a checklist?

[mattpitkin]

@nhejazi if you put "@editorialbot generate my checklist" (without the quotation marks) in a comment it will generate a checklist for you.

[nhejazi]

Review checklist for @nhejazi

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://gitlab.com/dsbowen/conditional-inference?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@dsbowen) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[nhejazi]

@dsbowen and @vissarion, I've now completed my review. the package is very well documented and the methods implemented appear both interesting and useful. In three issues opened at the GitLab repository, I've included two changes/updates required as part of the JOSS review as well as some recommendations to improve the notebooks/paper for the intended audience. I'll be able to check off the two items in my checklist above once the two required issues have been resolved.

[dsbowen]

@nhejazi, thank you so much for the review! I think your changes will greatly improve the repo, and I'll be happy to implement them next week.

@blakeaw and @mattpitkin, thank you for your reviews as well! I'm eager to understand what I can do to improve the repo. Can you please tell me why my work doesn't constitute a "substantial scholarly effort?" I've been working on this project for a year, the source code is over 3,000 lines long, and the package implements techniques from more than a dozen papers. I'd very much appreciate it if you can give me guidance on your expectations. Thank you in advance for your feedback, and I look forward to improving the repo!

[mattpitkin]

@dsbowen don't worry, I'm sure the work is "substantial scholarly effort", I just hadn't got round to checking that off yet. I'll try and complete my review by the end of this week or early next week.

[vissarion]

@dsbowen it is expected that reviewers take some time to complete the reviews so if you see an incomplete list this does not mean something is missing or there is a problem with the repo. Usually when a reviewer finishes with the review they write a message in this thread that the review is complete by potentially raising some issues.

[blakeaw]

Hi, @dsbowen, thanks for checking in. I'll just confirm that I also haven't completed my review yet.

As @vissarion suggested, I will post a message here when I have completed it as well as raise any issues during the process at the source code repo.

I'll also try to complete my review this week or early next week.

[dsbowen]

@mattpitkin @blakeaw @vissarion Thanks, I didn't mean to rush you. This is my first submission to JOSS, so I'm not sure what the norms are. I thought that, since you checked all the boxes up to "substantial scholarly effort," that meant it was the point at which the paper failed. I didn't realize it might also be that your review was still in progress. Looking forward to your comments!

[mattpitkin]

@dsbowen After looking through the paper and the software I have a few minor suggestions:

• could you spell out the acronyms OLS and IV at their first use in the paper summary?
• it would be useful to use, e.g., nbsphinx to render the example notebooks as pages within the docs website as well as linking to the binders.
• when describing the empirical Bayes estimator method in the bayes_primer.inpyb notebook, it probably worth linking to, .e.g., the wikipedia entry on it and noting how it is an approximation to a hierarchical Bayesian model (this is mainly for my benefit - I knew about the latter, but not about the former). It's probably also worth mentioning in the notebook that by plotting the messages in a ranked order of increasing "increase in vaccination rate" it can fool you into thinking that there some sort of correlation or cumulative effect that leads to the top message being best, whereas they are (probably) uncorrelated. Plotting them in a random order would essentially show off what the empirical Bayes method is finding, i.e., that the results are just uncorrelated points drawn from an underlying Gaussian distribution. It might be interesting to look at two text messages versus one text message ... (I'm not suggesting you need to add this for the example!)
• in the "State of the field" section of the paper, when describing Bayesian estimators, you could also mention the PosteriorStacker package, which implements simple Gaussian and non-parametric models - it does the full hierarchical analysis of sampling the posterior distribution of the prior's hyperparameters, so is not as quick as a maximised empirical Bayes method.
• unrelated to the paper, it would be useful for the package version to be exposed within the package. I've created an issue about this.

[mattpitkin]

DOIs to add:

• the DOI for Lai et al is https://doi.org/10.1037/a0036260
• the DOI for Milkman et al (2022) is https://doi.org/10.1073/pnas.2115126119
• the DOI for Milkman et al (2021), "Megastudies improve the impact of applied behavioural sciences" is https://doi.org/10.1038/s41586-021-04128-4
• the DOI for Milkman et al (2021), "A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor’s appointment" is https://doi.org/10.1073/pnas.2101165118

[blakeaw]

Hi, @dsbowen and @vissarion, I have completed my initial review. I don't have too much to add beyond what has already been raised by @nhejazi and @mattpitkin, but I went ahead and noted everything in the following issues:

• Documentation: https://gitlab.com/dsbowen/conditional-inference/-/issues/5
• Software paper: https://gitlab.com/dsbowen/conditional-inference/-/issues/6

[dsbowen]

@nhejazi, @mattpitkin, and @blakeaw, thank you very much for your reviews! I changed the package and documentation based on your reviews, and I think they're much better for it. Please let me know if you have more changes you'd like to see.

[dsbowen]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[vissarion]

Thanks @dsbowen for addressing the issues, thanks @nhejazi, @mattpitkin, and @blakeaw for the reviews.
To reviewers: if you find some issue covered please close it and also please check out the remaining boxes in your list if you find that the requirements are met now otherwise please comment on this.

[blakeaw]

Hi, @vissarion, @dsbowen has addressed my initial comments. However, when going back over the references in the software paper I noticed one more minor style issue that I commented on at https://gitlab.com/dsbowen/conditional-inference/-/issues/6.

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[vissarion]

Thanks a lot for all the work and the explanations!

We can now move on to the creation of a tagged release and archive (e.g. zenodo). Please report the version number and archive DOI here. The archive metadata (title, author list, affiliations) should be exactly the same as the ones of the paper.

[dsbowen]

Thank you! Here is the archive DOI: 10.5281/zenodo.6859614, version number 1.1.0. Here's the link.

[vissarion]

@editorialbot set 10.5281/zenodo.6859614 as archive

[editorialbot]

Done! Archive is now [ 10.5281/zenodo.6859614](https://doi.org/ 10.5281/zenodo.6859614)

[vissarion]

@editorialbot set 1.1.0 as version

[editorialbot]

Done! version is now 1.1.0

[vissarion]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[vissarion]

@editorialbot recommend-accept

[editorialbot]

Attempting dry run of processing paper acceptance...

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.4324/9780429318405-4 is OK
- 10.1073/pnas.2101165118 is OK
- 10.1073/pnas.2115126119 is OK
- 10.1038/s41586-021-04128-4 is OK
- 10.3386/w22193 is OK
- 10.1037/a0036260 is OK
- 10.3386/w12338 is OK
- 10.3386/w28726 is OK
- 10.2139/ssrn.3689456 is OK
- 10.3386/w25147 is OK
- 10.3386/w23002 is OK
- 10.3386/w19843 is OK
- 10.3386/w25456 is OK
- 10.1257/pandp.20221065 is OK
- 10.1111/j.1468-0262.2005.00615.x is OK
- 10.1073/pnas.1530509100 is OK
- 10.3386/w26883 is OK
- 10.1525/9780520313880-018 is OK
- 10.1007/978-1-4612-0919-5_30 is OK
- 10.1145/3292500.3330771 is OK
- 10.1093/mnras/staa2684 is OK
- 10.48550/arXiv.2106.08881 is OK
- 10.1214/08-aos630 is OK
- 10.48550/arXiv.2004.03448 is OK
- 10.1596/27528 is OK
- 10.25080/majora-92bf1922-011 is OK
- 10.7717/peerj-cs.55 is OK
- 10.1038/s41592-019-0686-2 is OK
- 10.1111/rssb.12162 is OK
- 10.1109/wsc.2015.7408180 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉📄 Download article

If the paper PDF and the deposit XML files look good in openjournals/joss-papers#3380, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[dsbowen]

When it says, "you can now move forward... with the command @editorbot accept", is that referring to me?

[danielskatz]

no, it's directed to the JOSS associate-editor-in-chief on duty (@openjournals/joss-eics)

[arfon]

@editorialbot accept

[editorialbot]

Doing it live! Attempting automated processing of paper acceptance...

[editorialbot]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[editorialbot]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.04492 joss-papers#3389
1. Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.04492
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[arfon]

@blakeaw, @mattpitkin, @nhejazi – many thanks for your reviews here and to @vissarion for editing this submission! JOSS relies upon the volunteer effort of people like you and we simply wouldn't be able to do this without you ✨

@dsbowen – your paper is now accepted and published in JOSS ⚡🚀💥

[editorialbot]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.04492/status.svg)](https://doi.org/10.21105/joss.04492)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.04492">
  <img src="https://joss.theoj.org/papers/10.21105/joss.04492/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.04492/status.svg
   :target: https://doi.org/10.21105/joss.04492

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss