[REVIEW]: MUQ: The MIT Uncertainty Quantification Library

[whedon]

Submitting author: @mparno (Matthew Parno)
Repository: https://bitbucket.org/mituq/muq2.git
Version: 0.4.1
Editor: @pdebuyl
Reviewers: @martinmodrak, @georgiastuart
Archive: 10.5281/zenodo.5770267

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/b83db89b376c0956ec05d547675aac42"><img src="https://joss.theoj.org/papers/b83db89b376c0956ec05d547675aac42/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/b83db89b376c0956ec05d547675aac42/status.svg)](https://joss.theoj.org/papers/b83db89b376c0956ec05d547675aac42)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@martinmodrak, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @pdebuyl know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @georgiastuart

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@mparno) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @martinmodrak

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@mparno) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @martinmodrak it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[pdebuyl]

@whedon add @georgiastuart as reviewer

[whedon]

OK, @georgiastuart is now a reviewer

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=6.26 s (159.0 files/s, 27439.9 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
C/C++ Header                    366          12654          19594          49204
C++                             353          11682           6172          37366
Python                           76           2494           1609           7464
CMake                           108           1206            813           4382
reStructuredText                 32           2450           2778           4153
SVG                               3              0              0           1186
D                                 1              0              0            961
Jupyter Notebook                  8              0           2901            807
YAML                             23             41             30            557
TeX                               3             45              0            304
Markdown                          8             73              0            277
XML                               1              5              7            170
Dockerfile                        4             28             14             91
Bourne Shell                      3             32             19             69
HTML                              1              3             16             46
INI                               1              0              0             19
CSS                               1              0              0             11
TOML                              2              0              0              6
Bourne Again Shell                1              0              0              2
--------------------------------------------------------------------------------
SUM:                            995          30713          33953         107075
--------------------------------------------------------------------------------


Statistical information for the repository '8116a45d3372a921a3269c25' was
gathered on 2021/03/02.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Alexandra Datz                   3            77             20            0.03
Andrew                          85         13805           8030            7.40
Andrew Davis                   108         19392           8923            9.59
Andy Davis                      15          1536            430            0.67
Arnold Song                     15          1984           2350            1.47
Brendan West                    15          1699            165            0.63
Devin O'Connor                  13           963            254            0.41
Hodgdon, Taylor S ER             8           828             97            0.31
Josephine Westermann             5           179             84            0.09
Ki-Tae Kim                       2             8              2            0.00
Linus Seelinger                116         26050          17873           14.88
Matt                            25          3778            568            1.47
Matt Parno                      11          2169            443            0.89
Matthew Parno                  383         97268          32459           43.96
Max Liu                          1             1              0            0.00
andyddavis                       1            53              6            0.02
cassielumbrazo                   3            92             42            0.05
davisad                         66         49707           3611           18.07
vagrant                          1           156              0            0.05

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Alexandra Datz               69           89.6          4.3                0.00
Andrew                     6323           45.8         45.0               24.92
Andrew Davis               8626           44.5         36.4               17.12
Arnold Song                1003           50.6         31.6               34.20
Brendan West                370           21.8         35.5               51.62
Devin O'Connor              628           65.2         33.6                7.32
Hodgdon, Taylor S ER        533           64.4         32.6                1.13
Josephine Westermann        179          100.0          3.4               29.61
Ki-Tae Kim                    8          100.0          8.2                0.00
Linus Seelinger            6133           23.5         18.0               10.19
Matt                       3740           99.0         15.1               22.19
Matt Parno                  293           13.5         47.6               14.33
Matthew Parno             71297           73.3         31.1               16.22
Max Liu                       1          100.0         18.7                0.00
andyddavis                   53          100.0         11.1               16.98
cassielumbrazo               63           68.5         42.4               22.22
davisad                   48920           98.4         49.7               23.33

[pdebuyl]

@georgiastuart @martinmodrak make sure to accept the invitation to the reviewers group and to have a look at the reviewer guidelines linked to at the top of this review page.

The review process will happen in this issue page, so questions to the author or to me can be added as comments here.

In a JOSS review, you can query the authors directly here without the delay of the editorial office. Given that the codebase is large, it is good to keep it in mind as you go.

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1137/130915005 is OK
- 10.1137/19M126966X is OK
- 10.1287/opre.1070.0496 is OK

MISSING DOIs

- 10.1016/j.jcp.2015.10.008 may be a valid DOI for title: Dimension-independent likelihood-informed MCMC
- 10.1214/13-sts421 may be a valid DOI for title: MCMC methods for functions: modifying old algorithms to make them faster
- 10.1615/int.j.uncertaintyquantification.2019027384 may be a valid DOI for title: Embedded model error representation for Bayesian model calibration
- 10.1137/140964023 may be a valid DOI for title: Randomize-then-optimize: A method for sampling from posterior distributions in nonlinear inverse problems
- 10.1137/120890715 may be a valid DOI for title: Adaptive Smolyak pseudospectral approximations
- 10.1137/16m1084080 may be a valid DOI for title: Parallel local approximation MCMC for expensive models
- 10.1137/17m1134640 may be a valid DOI for title: Transport map accelerated Markov chain Monte Carlo

INVALID DOIs

- None

[pdebuyl]

@georgiastuart I forgot to assign you in the pre-review, sorry. If there is any issue let me know (especially if you don't receive an invitation to JOSS' reviewer group which is necessary to proceed).

[pdebuyl]

@mparno can you fix all the DOIs already ? This will be necessary anyway before acceptance. Also, some high-level information such as the use of pybind for the C++ / Python binding is welcome. You can also list some examples from the extensive list that is provided with the software (no code illustration but simple mention that the capabilities are well demonstrated is useful).

[georgiastuart]

@pdebuyl I got the invite and I'm good to go. As you mentioned in the pre-review issue, I'm fine with focusing on the python interface and @martinmodrak can focus on the C++ interface.

[mparno]

@mparno can you fix all the DOIs already ? This will be necessary anyway before acceptance. Also, some high-level information such as the use of pybind for the C++ / Python binding is welcome. You can also list some examples from the extensive list that is provided with the software (no code illustration but simple mention that the capabilities are well demonstrated is useful).

@pdebuyl Of course. I'll work on this today. Thank you for your feedback.

[mparno]

@pdebuyl Reference DOIs have been updated and additional information about our examples and python bindings have been added the summary section of the paper. Everything is up to date in the master branch of MUQ's repository.

[georgiastuart]

@mparno Is Boost necessary for installation (by building the software), or is it just using Eigen? If just using Eigen, would it be possible to remove the boost dependency as it seems unnecessary and boost is hefty. I ask because boost is listed in the same line as Eigen in your dependency table.

Edit: nevermind, answered my own question by looking at the code and I see you're using other boost functionality. I would recommend splitting boost and eigen onto two lines if you think that makes sense for your documentation.

[mparno]

@georgiastuart Thanks for pointing this out. Yes, we require Boost (as you figured out). There must be a bug in the doxygen for producing that table of dependencies. Boost and Eigen should be separated. I'll work on fixing that later today. Thanks!

Update: The dependency table has been fixed.

[pdebuyl]

@whedon check references

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.jcp.2015.10.008 is OK
- 10.1214/13-STS421 is OK
- 10.1615/Int.J.UncertaintyQuantification.2019027384 is OK
- 10.1137/140964023 is OK
- 10.1137/130915005 is OK
- 10.1137/19M126966X is OK
- 10.1137/120890715 is OK
- 10.1137/16M1084080 is OK
- 10.1137/17M1134640 is OK
- 10.1287/opre.1070.0496 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[pdebuyl]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[pdebuyl]

@georgiastuart @martinmodrak gentle reminder

[whedon]

👋 @martinmodrak, please update us on how your review is going (this is an automated reminder).

[whedon]

OK. 10.5281/zenodo.5770267 is the archive.

[pdebuyl]

Thanks all :-)

[pdebuyl]

@whedon recommend-accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#2804

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#2804, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.7717/peerj-cs.55 is OK
- 10.1016/j.jcp.2015.10.008 is OK
- 10.1145/1089014.1089020 is OK
- 10.18637/jss.v076.i01 is OK
- 10.1002/sim.3680 is OK
- 10.1214/13-STS421 is OK
- 10.1111/1467-9868.00294 is OK
- 10.1615/Int.J.UncertaintyQuantification.2019027384 is OK
- 10.1137/140964023 is OK
- 10.1137/130915005 is OK
- 10.1137/19M126966X is OK
- 10.1137/120890715 is OK
- 10.1137/16M1084080 is OK
- 10.1137/17M1134640 is OK
- 10.1287/opre.1070.0496 is OK
- 10.1145/3458817.3476150 is OK

MISSING DOIs

- 10.1007/978-1-4419-7719-9_6 may be a valid DOI for title: Boost C++ Libraries

INVALID DOIs

- None

[danielskatz]

👋 @mparno - I'm the AEiC who will handle the rest of the publication process.

• Can you please remove the abstract fields from your bib entries? For those that do not have DOIs, this is part of the XML record, and it might confuse the indexing service.
• Please also check cases of words in the bib file, and use {} to protect letters or words that should be upper case, e.g. {C}++, {S}tan {M}ath library, {S}molyak, {M}arkov, {B}ayesian, {P}ython, etc.
• In the first paragraph of the paper, please change "c++" to "C++"

Let me know when this is done, and we can proceed.

[linusseelinger]

@danielskatz Thanks! I updated the paper and references in our repo accordingly.

[danielskatz]

@whedon recommend-accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#2805

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#2805, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.7717/peerj-cs.55 is OK
- 10.1016/j.jcp.2015.10.008 is OK
- 10.1145/1089014.1089020 is OK
- 10.18637/jss.v076.i01 is OK
- 10.1002/sim.3680 is OK
- 10.1214/13-STS421 is OK
- 10.1111/1467-9868.00294 is OK
- 10.1615/Int.J.UncertaintyQuantification.2019027384 is OK
- 10.1137/140964023 is OK
- 10.1137/130915005 is OK
- 10.1137/19M126966X is OK
- 10.1137/120890715 is OK
- 10.1137/16M1084080 is OK
- 10.1137/17M1134640 is OK
- 10.1287/opre.1070.0496 is OK
- 10.1145/3458817.3476150 is OK

MISSING DOIs

- 10.1007/978-1-4419-7719-9_6 may be a valid DOI for title: Boost C++ Libraries

INVALID DOIs

- None

[danielskatz]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.03076 joss-papers#2806
1. Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.03076
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[danielskatz]

Congratulations to @mparno (Matthew Parno) and co-authors!!

And thanks to @martinmodrak and @georgiastuart for reviewing, and @pdebuyl for editing!
We couldn't do this without you

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.03076/status.svg)](https://doi.org/10.21105/joss.03076)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.03076">
  <img src="https://joss.theoj.org/papers/10.21105/joss.03076/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.03076/status.svg
   :target: https://doi.org/10.21105/joss.03076

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

[pdebuyl]

Thanks Daniel :-)

[mparno]

Thanks @pdebuyl @danielskatz @georgiastuart and @martinmodrak . We really appreciate all of your time. You've definitely helped us make MUQ a better package.