[REVIEW]: Caliban: Docker-based job manager for reproducible workflows

[whedon]

Submitting author: @sritchie (Sam Ritchie)
Repository: https://github.com/google/caliban
Version: 0.4.1
Editor: @diehlpk
Reviewer: @lukasheinrich, @arokem
Archive: 10.5281/zenodo.4026612

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/c33c8b464103b2fb3b641878722bf8f3"><img src="https://joss.theoj.org/papers/c33c8b464103b2fb3b641878722bf8f3/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/c33c8b464103b2fb3b641878722bf8f3/status.svg)](https://joss.theoj.org/papers/c33c8b464103b2fb3b641878722bf8f3)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@lukasheinrich & @arokem, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @diehlpk know.

✨ Please try and complete your review in the next six weeks ✨

Review checklist for @lukasheinrich

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@sritchie) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @arokem

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@sritchie) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @lukasheinrich, @arokem it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Reference check summary:

OK DOIs

- 10.1145/2889160.2891057 is OK

MISSING DOIs

- https://doi.org/10.1109/cvpr.2009.5206848 may be missing for title: Imagenet: A large-scale hierarchical image database

INVALID DOIs

- None

[whedon]

👉 Check article proof 📄 👈

[diehlpk]

@sritchie Could you please check the missing DOI reported above?

[sritchie]

@whedon check references

[whedon]

Reference check summary:

OK DOIs

- 10.1145/2889160.2891057 is OK
- 10.1109/cvpr.2009.5206848 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[sritchie]

@diehlpk looks like we're good to go!

[lukasheinrich]

hi @sritchie -- part of the described functionality requires Google Cloud Access. Will you provide credits or similar to allow us to test that functionality (I don't have any free credits left)?

[sritchie]

@lukasheinrich and @arokem , of course - I have a Cloud project I can add you both to, and can send you a service account key that you can install on your machine to submit jobs and push containers. Do you want to send me your email address at [email protected]? I'll go ahead and add you right away.

[sritchie]

@lukasheinrich , if you go ahead and send me an email I can get you set up. @arokem got in touch, so I have a little guide written up for you both.

Cheers!
Sam

[sritchie]

Hey @arokem, @lukasheinrich and @diehlpk - I responded a week ago with details here, but haven't heard anything back. Let me know if I can assist with anything!

[diehlpk]

Hey @sritchie I think you are fine and the reviewers will proceed with their review. Normally, the review time is about 6 weeks, since we asked the reviewers to finish their review within 6 weeks.

[diehlpk]

Hey @arokem and @lukasheinrich how is your review going?

[arokem]

@diehlpk : I am making slow progress through the checklist. My main remaining items are about testing the various functionality of the library. So far, it's been an enjoyable experience 😄

Regarding community guidelines, I've posted an issue in the software repo.

I also have a small comment for this item:

Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?

The authors tend to make liberal use of superlatives (the word "tremendous" appears more than once). I think that it would be good to use these sparingly.

[arokem]

One more comment, regarding the state of the art. There are two strands of research not currently cited, that I believe could be connected to this software and would better embed the software in context:

The first regards automated docker deployment for scientific software. I am thinking primarily of repo2docker, which implements functionality similar to caliban notebook and binder. This paper could be cited: https://openreview.net/forum?id=BJlR6KTE3X

The second regards automated deployment of scientific computing to the cloud. One set of work there is pywren and related papers (e.g., https://shivaram.org/publications/pywren-socc17.pdf). The other is our own work on cloudknot. If you feel so compelled, you could cite our scipy paper on that: http://conference.scipy.org/proceedings/scipy2018/adam_richie-halford.html.

[arokem]

Relatedly, but definitely not required for the review, some food for thought here: google/caliban#49

[arokem]

I am trying to run caliban cloud and ran into google/caliban#51.

Setting the environment variable seems to resolve that, but I then run into:

$caliban cloud --nogpu mnist.py
I0723 08:10:11.261835 4394732864 core.py:497] Generating Docker image with parameters:
I0723 08:10:11.262293 4394732864 core.py:498] {'adc_path': None,
 'build_path': '/Users/arokem/tmp/calibam/demo',
 'caliban_config': {},
 'conda_env_path': None,
 'credentials_path': None,
 'extra_dirs': None,
 'job_mode': <JobMode.CPU: 'CPU'>,
 'no_cache': False,
 'package': Package(executable=['python'], package_path='.', script_path='mnist.py', main_module=None),
 'requirements_path': 'requirements.txt',
 'setup_extras': None}
I0723 08:10:11.262627 4394732864 docker.py:614] Running command: docker build --rm -f- /Users/arokem/tmp/calibam/demo
Sending build context to Docker daemon  9.775kB
Step 1/11 : FROM gcr.io/blueshift-playground/blueshift:cpu
 ---> fafdb20241ad
Step 2/11 : RUN [ $(getent group 20) ] || groupadd --gid 20 20
 ---> Using cache
 ---> 0fb794a9d15b
Step 3/11 : RUN useradd --no-log-init --no-create-home -u 501 -g 20 --shell /bin/bash arokem
 ---> Using cache
 ---> 18bce73381b9
Step 4/11 : RUN mkdir -m 777 /usr/app /.creds /home/arokem
 ---> Using cache
 ---> 18c96e44e68c
Step 5/11 : ENV HOME=/home/arokem
 ---> Using cache
 ---> 7a4bb135f05e
Step 6/11 : WORKDIR /usr/app
 ---> Using cache
 ---> 266bb6485852
Step 7/11 : USER 501:20
 ---> Using cache
 ---> e3954e929d29
Step 8/11 : COPY --chown=501:20 requirements.txt /usr/app
 ---> Using cache
 ---> 08d9467b6ea9
Step 9/11 : RUN /bin/bash -c "pip install --no-cache-dir -r requirements.txt"
 ---> Using cache
 ---> d9eaf5a76590
Step 10/11 : COPY --chown=501:20 . /usr/app/.
 ---> Using cache
 ---> fdd5a32ee33f
Step 11/11 : ENTRYPOINT ["python", "mnist.py"]
 ---> Using cache
 ---> bbbce372b9f5
Successfully built bbbce372b9f5
The push refers to repository [gcr.io/landscape-238422/bbbce372b9f5]
45fcaa2b93ad: Preparing 
3c1176c86461: Preparing 
50e8c557ccbf: Preparing 
bd07837419cb: Preparing 
42d270e25e53: Preparing 
c83c21629dcf: Waiting 
a8c372c103ab: Waiting 
4e84efa3db52: Waiting 
25f9bffed627: Waiting 
d4b51844f98b: Waiting 
ed1d1d4a83ac: Waiting 
df7bab7a7925: Waiting 
28ba7458d04b: Waiting 
838a37a24627: Waiting 
a6ebef4a95c3: Waiting 
b7f7d2967507: Waiting 
denied: Token exchange failed for project 'landscape-238422'. Caller does not have permission 'storage.buckets.create'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control
Traceback (most recent call last):
  File "/Users/arokem/.virtualenvs/caliban/bin/caliban", line 8, in <module>
    sys.exit(main())
  File "/Users/arokem/.virtualenvs/caliban/lib/python3.7/site-packages/caliban/main.py", line 155, in main
    app.run(run_app, flags_parser=cli.parse_flags)
  File "/Users/arokem/.virtualenvs/caliban/lib/python3.7/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/Users/arokem/.virtualenvs/caliban/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/Users/arokem/.virtualenvs/caliban/lib/python3.7/site-packages/caliban/main.py", line 145, in run_app
    xgroup=xgroup,
  File "/Users/arokem/.virtualenvs/caliban/lib/python3.7/site-packages/caliban/cloud/core.py", line 637, in submit_ml_job
    image_tag = generate_image_tag(project_id, docker_args, dry_run=dry_run)
  File "/Users/arokem/.virtualenvs/caliban/lib/python3.7/site-packages/caliban/cloud/core.py", line 505, in generate_image_tag
    image_tag = d.push_uuid_tag(project_id, image_id)
  File "/Users/arokem/.virtualenvs/caliban/lib/python3.7/site-packages/caliban/docker.py", line 655, in push_uuid_tag
    subprocess.run(["docker", "push", image_tag], check=True)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 468, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker', 'push', 'gcr.io/landscape-238422/bbbce372b9f5:latest']' returned non-zero exit status 1.

Do the permissions on the project need to be adjusted somehow?

[diehlpk]

@sritchie Can you please add the country (no abbreviation) to the affiliation?

[sritchie]

@arokem , yes, sorry about that... this was an older project, and I hadn't quite synced up permissions yet for the service account. I thought I'd done this for you, but my mistake! This would not come up on your own project.

Give it a try now!

[sritchie]

@whedon generate pdf

[whedon]

👉 Check article proof 📄 👈

[sritchie]

@diehlpk whoops, sorry, I just realized I did exactly what you asked me not to do and added "USA", copying the other papers I'd seen. What would you like me to add? "United States", or "United States of America"? Let me know and I'll be happy to make the change.

[diehlpk]

@sritchie I think we like to have United States of America.

[sritchie]

@diehlpk okay, done. let me regenerate

[sritchie]

@whedon generate pdf

[diehlpk]

@sritchie Congratulations to the accepted paper. @lukasheinrich and @arokem thanks for your review.

[sritchie]

Thanks all!!

[kyleniemeyer]

Hi @sritchie, I'm doing some final checks on your submission. In the paper, it looks like the Forde et al. and Zaharia et al. references may be missing some details. Also, please look at the other references, and add curly braces {} around words to enforce capitalization where appropriate (e.g., "IEEE", "Linux").

[sritchie]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[sritchie]

@kyleniemeyer , just took another pass and I think we're set here.

[diehlpk]

@sritchie For the second reference ieee should be IEEE.

[sritchie]

@whedon generate pdf

[sritchie]

@diehlpk I'll figure this out someday... nice catch, fixed!

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[kyleniemeyer]

Hey @sritchie, looks like the Forde et al. reference is still missing something. Where was that published—a journal, or on a website, or something?

[sritchie]

@kyleniemeyer , I took the bibtex entry from here: https://scholar.google.com/scholar?q=Reproducing+%7BMachine+%7BLearning+%7BResearch+on+%7BBinder&hl=en&as_sdt=0&as_vis=1&oi=scholart:

@article{forde2018reproducing,
  title={Reproducing Machine Learning Research on Binder},
  author={Forde, Jessica and Bussonnier, Matthias and Fortin, F{\'e}lix-Antoine and Granger, Brian and Head, Tim and Holdgraf, Chris and Ivanov, Paul and Kelley, Kyle and Pacer, M and Panda, Yuvi and others},
  year={2018}
}

It lives on OpenReview.net: https://openreview.net/forum?id=BJlR6KTE3X

and was published at NIPS 2018 Workshop MLOSS; you're right that even though that isn't in the bibtex entry Scholar generates it needs to be here. Adding now.

[sritchie]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[kyleniemeyer]

@sritchie yeah, unfortunately Google Scholar's BiBTeX generation only seems to work consistently for regular journal articles.

[kyleniemeyer]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1145/2889160.2891057 is OK
- 10.1109/cvpr.2009.5206848 is OK
-  10.25080/Majora-4af1f417-001  is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#1734

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#1734, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[kyleniemeyer]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.02403 joss-papers#1735
1. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.02403
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[kyleniemeyer]

Congrats @sritchie on your article's publication in JOSS!

Many thanks to @lukasheinrich and @arokem for reviewing this, and @diehlpk for editing it.

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.02403/status.svg)](https://doi.org/10.21105/joss.02403)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.02403">
  <img src="https://joss.theoj.org/papers/10.21105/joss.02403/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.02403/status.svg
   :target: https://doi.org/10.21105/joss.02403

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss