[REVIEW]: Open OnDemand: A web-based client portal for HPC centers

[whedon]

Submitting author: @ericfranz (Eric Franz)
Repository: https://github.com/OSC/ondemand
Version: v1.2
Editor: @danielskatz
Reviewer: @marpierc, @smgallo
Archive: 10.6084/m9.figshare.6265514.v1

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/32245eb2f375c5628701a1ce9ec8298a"><img src="http://joss.theoj.org/papers/32245eb2f375c5628701a1ce9ec8298a/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/32245eb2f375c5628701a1ce9ec8298a/status.svg)](http://joss.theoj.org/papers/32245eb2f375c5628701a1ce9ec8298a)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer1 instructions & questions

@marpierc, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @danielskatz know.

Conflict of interest

• As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: Does the release version given match the GitHub release (v1.2)?
• Authorship: Has the submitting author (@ericfranz) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Reviewer2 instructions & questions

@smgallo, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @danielskatz know.

Conflict of interest

• As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: Does the release version given match the GitHub release (v1.2)?
• Authorship: Has the submitting author (@ericfranz) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @marpierc it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

--> Check article proof 📄 <--

[danielskatz]

@arfon - can you invite @smgallo so that he can be a reviewer?
@marpierc - you automatically be invited as described in the review instructions.

[danielskatz]

My one initial comment is that the paper doesn't have any references - the reviewers might think about any things (papers, systems) that should be referenced...

[arfon]

@arfon - can you invite @smgallo so that he can be a reviewer?

Done.

[danielskatz]

discussing the references in https://github.com/OSC/Open-OnDemand/issues/30

[danielskatz]

For the record, the reviews from @smgallo and @marpierc are expected by the end of March and April 5, respectively.

[danielskatz]

@whedon commands

[whedon]

Here are some things you can ask me to do:

# List all of Whedon's capabilities
@whedon commands

# Assign a GitHub user as the sole reviewer of this submission
@whedon assign @username as reviewer

# Add a GitHub user to the reviewers of this submission
@whedon add @username as reviewer

# Remove a GitHub user from the reviewers of this submission
@whedon remove @username as reviewer

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

# Change editorial assignment
@whedon assign @username as editor

# Set the software archive DOI at the top of the issue e.g.
@whedon set 10.0000/zenodo.00000 as archive

# Open the review issue
@whedon start review

🚧 🚧 🚧 Experimental Whedon features 🚧 🚧 🚧

# Compile the paper
@whedon generate pdf

[danielskatz]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

--> Check article proof 📄 <--

[smgallo]

@danielskatz

OOD is an excellent package and fills a need for HPC users by simplifying and improving access to complicated HPC resources. HPC centers are able to provide their users with simplified access to resources while making it easier to submit and manage both jobs and data. Single sign-on is supported making it easier to integrate with center or campus-wide authentication systems. Because OOD runs in a web-browser, users do not need to install additional software on their computers to access HPC resources.

Our center recently installed OOD to serve novice and light HPC users and has found that even experienced users have said that using OOD is "so much easier" than logging in via the command line to manage their compute jobs and data. In addition, faculty who use our HPC resources to teach courses are entheuiastic about OOD. Over the years, we had implemented a subset of the functionality that OOD provides to support the needs of our users, but having all of this functionality in a single integrated framework is great. There is an active mailing list which is available for general questions. Multiple authentication methods are supported via Apache modules including shibboleth, LDAP, and CiLogin. We have implemented CiLogin backed by FreeIPA for access to our resources.

OOD supports a number of useful applications including a job status viewer, file manager and editor, shell tool, and an interactive desktop. In addition, there are interactive desktop applications that are supported such as MATLAb, Jupyter, and R Studio, all of which are desired by end users. Development of custom applications is documented (with examples). The interactive desktop application has proven especially useful and allowed us to deprecate a licensed application that was previously provided to our users. A very useful feature of OOD are the Per User NGIX Processes (PUN). This allows users to run their applications via the web server in a process that they own (rather than the web server owning the process) making system and data security far simpler.

General Comments

1. An RPM install is not available but it would be great to have this (along with an upgrade path from a source install). Is an RPM (or other package manager) version planned?
2. Individual apps are contained in their own git repos. While this makes extra work when installing the apps, it is nicely compartmentalized.

Versioning

The versioning is a bit confusing and could use clarification in the documentation. OOD is made up of several components including infrastructure and applications. The infrastructure component versions do not appear to be to match the 1.2 version of OOD as a whole and the only application that I could find with a recent 1.23 version number was the dashboard. Other applications seem to have their own version tags but are now far ahead of version 1.2. For example, the myjobs application is now at version 2.8.2 with 1.2 released almost 2 years ago.

Thanks to CCR staff members @aebruno, @dsajdak, and Cynthia Cornelius at CCR for their input. CCR has installed Open on Demand and made it available to our users.

[danielskatz]

Thanks @smgallo - it looks like there are some issues you raised for @ericfranz to respond to or resolve. Additionally, I see you did not check the automated tests item, or discuss testing in your comment above. Can you explain?

[smgallo]

@danielskatz Apologies, I forgot to add that in.

@ericfranz While the documentation for the various components does explain how to use or access the component, there does not appear to be an automated test suite or manual process specified for each component to verify that the component is installed correctly (note that some components do instruct the installer to manually verify). This is a complex software and infrastructure package and it would be useful for each component to have a simple verification script that could perhaps test for the existence of a file or test the data returned from a REST endpoint. I can certainly appreciate the effort involved in creating and maintaining tests of this nature so it would be good to see plans for incremental steps in this direction.

[danielskatz]

@marpierc - just a reminder that your review is expected by April 5.

[danielskatz]

@ericfranz - Do you have a response to @smgallo's comments? You can certainly address them before @marpierc's review

[ericfranz]

Thank you for the comments! I opened three issues in https://github.com/OSC/Open-OnDemand for further discussion.

1. We have an rpm we are using at OSC now and will be the default installation for OnDemand 1.3 released in April. I opened https://github.com/OSC/Open-OnDemand/issues/31 to address the problem of not advertising our development roadmap.
2. For each OnDemand release we document all the corresponding versions in the release notes (see 1.2 example). I opened https://github.com/OSC/Open-OnDemand/issues/32 to track the need to put this list in a more prominent location. We are also looking to combine some repos into one in the future to help with this.
3. We have some unit test coverage for components which target developers, but have no system tests for installation validation which would target admins. I opened an issue https://github.com/OSC/Open-OnDemand/issues/33 to discuss that further.

[smgallo]

Thanks @ericfranz, I'll follow up on those issues to provide further input as needed. @danielskatz that addresses any comments that I had.

[danielskatz]

@marpierc - just a reminder that your review is expected by April 5.

[danielskatz]

@marpierc - I think I saw a comment from you about the paper and what it should contain, though I no longer see it, so perhaps it was deleted? But in any case, see http://joss.theoj.org/about#author_guidelines (what should my paper contain)

[marpierc]

Yes, I deleted. I found the answer to my question. Thanks, Marlon From: ajinkya-dhamnaskar <[email protected]> Reply-To: openjournals/joss-reviews <[email protected]> Date: Wednesday, April 4, 2018 at 11:54 AM To: openjournals/joss-reviews <[email protected]> Cc: "Pierce, Marlon" <[email protected]>, Mention <[email protected]> Subject: Re: [openjournals/joss-reviews] [REVIEW]: Open OnDemand: A web-based client portal for HPC centers (#622) @marpierc - I think I saw a comment from you about the paper and what it should contain, though I no longer see it, so perhaps it was deleted? But in any case, see http://joss.theoj.org/about#author_guidelines (what should my paper contain) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[marpierc]

I have made comments here: https://github.com/OSC/Open-OnDemand/issues.

[danielskatz]

Thanks @marpierc - please update this issue as those issues get resolved.

[danielskatz]

👋 @ericfranz - please let us know when you have addressed these issues.

[ericfranz]

I believe all issues have been addressed:

1. The 1.3 release next week will include deployment via rpm to address the complicated installation process
2. We added a set of auto-generated Rake tasks to test and verify job submission from the web node for each configured cluster (https://github.com/OSC/Open-OnDemand/issues/33 and Add Rake simple task to test job submission on clusters OSC/ood-dashboard#360) which will be included in the 1.3 release. This should address concerns for tests to help admins verify the installation.
3. We added a CONTRIBUTING.md file to the root of the OnDemand repo to address third party contributions.

Assuming everything goes well this week, OnDemand 1.3 will be released next Monday. Is there a need to wait for this release or can we proceed with the JOSS publication?

[danielskatz]

If the current software meets what reviewers think is needed, we can proceed.

At this point, we need @marpierc and @smgallo to examine this again and see if they can check off all the items in their checklists.

[smgallo]

@danielskatz The recent/pending updates to OOD are sufficient for me.

[danielskatz]

The recent/pending updates to OOD are sufficient for me.

@smgallo - can you check off the remaining item on your list in this case?

[smgallo]

@danielskatz Done!

[danielskatz]

@marpierc - can you verify that the remaining point has been addressed and check off the remaining item on your list?

[marpierc]

Done

[danielskatz]

@marpierc and @smgallo - thanks very much for your reviews

[danielskatz]

👋 Arfon - over to you to continue the acceptance process

[arfon]

@ericfranz - At this point could you make an archive of the reviewed software in Zenodo/figshare/other service and update this thread with the DOI of the archive? I can then move forward with accepting the submission.

[ericfranz]

@arfon would it be sufficient to upload a zip or a tarball of the related rpms at https://yum.osc.edu/ondemand/1.3/ to figshare? (this is the latest release of OnDemand just released this week, which include suggested modifications). If not, what is your recommendation for how to package in a single tarball software that is spread across several repos and whose installation results in files being placed in the appropriate locations following the Filesystem Hierarchy Standard?

Other approaches I am considering include:

1. unpack the 1.3 rpms, adding a README at the root for how to proceed with the installation, and tar the parent directory
2. get tarballs of each git repo, placing them in the appropriate location in the filesystem that an rpm installation would place them, adding a README at the root of how to proceed with installation

[arfon]

@arfon would it be sufficient to upload a zip or a tarball of the related rpms at https://yum.osc.edu/ondemand/1.3/ to figshare?

The archive needs to include the source code which I don't think would be true in this case right?

If so, option #2 above is probably the best option...

[ericfranz]

@arfon see https://doi.org/10.6084/m9.figshare.6265514.v1

This tarball is also attached to the packaging repo https://github.com/OSC/ondemand/releases/tag/v1.3.6 and is all of the source for the core infrastructure and apps. The README.md in the tarball is markdown formatted and provides instructions for building from source, where to get help when stuck, and recommends using the RPM based installation.

[arfon]

@whedon set 10.6084/m9.figshare.6265514.v1 as archive

[whedon]

OK. 10.6084/m9.figshare.6265514.v1 is the archive.

[arfon]

@marpierc, @smgallo many thanks for your reviews here and to @danielskatz for editing this submission ✨

@ericfranz - your paper is now accepted into JOSS and your DOI is https://doi.org/10.21105/joss.00622 ⚡ 🚀💥

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippet:

[![DOI](http://joss.theoj.org/papers/10.21105/joss.00622/status.svg)](https://doi.org/10.21105/joss.00622)

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://www.flipcause.com/secure/cause_pdetails/Mjk3Nzk=

[ericfranz]

NSF would like us to add an acknowledgement to the paper. After updating the paper.md file to contain an Acknowledgements section similar to what I see in other JOSS publications, what is the next step to get the PDF to be regenerated? https://github.com/OSC/Open-OnDemand/issues/44

[arfon]

Please add the language to your paper.md file in the master branch of your repository and we can regenerate it.

[ericfranz]

@arfon ok the paper.md has been updated

[arfon]

@arfon ok the paper.md has been updated

I've updated the PDF too. This may take a few hours to show up on the live site due to the caching we do.

[ericfranz]

@arfon Thank you!