Not all output files have the right permissions

[mfdeakin-sandia]

This is a recent issue which is causing issues for the coupled group sharing results.
@PeterCaldwell, @vanroekel can you clarify on what files are being created with the wrong permissions and provide reproduction steps (mainly, is there a resubmit)? From the conversation, it looks like all output files from year 31+ are created with incorrect permissions:

[PeterCaldwell]

I'd like to ask that figuring out the exact conditions under which problems occur be part of your task. To make that tractable, how about: use run_acme to create a new case and run for 2 months. Check whether permissions are ok. If so, use case_scripts/case.submit to continue the run and complete another 2 months. Check again whether permissions are ok. If so, continue the run again using case.run and see if permissions are ok. If so, re-run run_acme with model_start_type=continue to continue the simulation for another 2 months and check permissions. If permissions are always fine, just report that back to me and consider yourself done. If permissions are ever not ok, figure out why and fix it.

And by "permissions are ok", I mean that someone who's a member of the acme group should be able to read everything in the $case_root_dir directory.

Does this seem reasonable?

[mfdeakin-sandia]

Yes, that's reasonable. I had just hoped someone already had the reproduction steps

[mfdeakin-sandia]

CIME doesn't seem to be affecting the permissions at all; so if you don't change your umask all of the output files generated will have the permissions you specify.

@rljacob @jedwards4b @jgfouca Is this how CIME is supposed to work? Would adding setting the default permissions of the output files be a reasonable feature to add?

[jedwards4b]

Note that cray DOE machines have a tool that constantly changes permissions on files to discourage sharing - are you sure this isn't a feature of your HPC platform?

[PeterCaldwell]

Good point. The issue we are seeing is that new files have wrong permissions. I've seen DOE machines (e.g. at LLNL) crawl through directories and remove read permissions, but I haven't seen machines arbitrarily changing umask settings (which is the symptom here).

[mfdeakin-sandia]

@PeterCaldwell Could it instead be that someone ran the run_acme script, then resubmitted with case.submit without changing their own umask settings?

[PeterCaldwell]

Hi Michael - your hypothesis is likely. I was assuming that each job submission would inherit the mask of previous submissions, but I don't really have a good reason for thinking that. Is it possible for case.run and case.submit to be changed to make files world readable by default?

[mfdeakin-sandia]

I think you'd want the permissions to be a bit more global; so that the directories created by create_newcase and case.setup are accessible.
This could be added as an option for create_newcase and adding an xml variable; or maybe just inheriting the permissions of the case directory.

[PeterCaldwell]

Yes, I agree that having all directories be world-readable would be much better. I was just trying to make life easy and the run directory is the one people want to see most often. I particularly like the idea of having submissions inherit the permissions of existing directories.

[rljacob]

does CIME try to set the umask anywhere? I think it did for the timing archive but that was it.

[jgfouca]

@rljacob see utils.SharedArea.

@PeterCaldwell unfortunately, it's not easy in Linux to have subdirectories inherit permissions of their parent directory. I have looked into this several times. The best approach is to use metagroups and subdirectories can atleast inherit group ownership of the parent with chmod g+s. If all users of the area set their umask to 002, then things will work.

[jedwards4b]

I would like to put forward the suggestion that this is an HPC system / science team issue and the solution is probably not a change in cime.

[mfdeakin-sandia]

@PeterCaldwell I'm playing around with ACLs, they seem to work on Edison, and allow enabling access for multiple groups without enabling access for all: https://unix.stackexchange.com/questions/115631/getting-new-files-to-inherit-group-permissions-on-linux
If you want to test it, try reading /global/homes/m/mdeakin/test/file on Edison

@jedwards4b I guess I feel like this is something that we could add as a feature to help support the science teams without much trouble. Perhaps we could add ACL group setting options to create_newcase, and have case.setup ensure the build, run, and archive directories have the same ones? The ACLs seem to be inherited by files created later, so this should be sufficient for the science teams purposes.

[jedwards4b]

If there is a solution that can be implemented in cime without too much pain I don't have any fundamental problem with it.

[PeterCaldwell]

@mfdeakin-sandia - I tried to read /global/homes/m/mdeakin/test/file but got permission denied errors at ~mdeakin and every subdirectory under it.

[mfdeakin-sandia]

Ah, sorry, I thought I checked that but only checked the symlink. Try again?

[PeterCaldwell]

I can see your home directory now, but still can't see the contents of the test directory.

I'm confused why we need to use ACLs instead of just changing the default group for all our files to acme. What are the advantages you're going for?

[mfdeakin-sandia]

ACLs allows multiple groups to be specified, solving the issue of sharing with people from other projects. I was also having issues getting the sticky bit to stick with the stackexchange instructions; though I've since resolved that, so maybe they are the way to go if ACLs aren't easy to get working. Admittedly, I've never used them before, so this is likely a user error, and I think I need to add the execute permission to the directory
EDIT: Nope, I had the execute bit set

[PeterCaldwell]

Ok. My past experience with people not in the acme group is that these people are usually just joining the project and don't have access to any particular group yet. I don't think ACLs will help that. I guess another use case would be someone funded by CMDV but not ACME might not belong in the acme group but could need to view ACME files. I'm kind of surprised I haven't run into this before... Maybe everyone is getting added to the acme group on nersc but not given any allocation unless they are core ACME developers?

I'm slightly opposed to ACLs because I've found them very counter-intuitive and confusing in the past. But if you can make them work seamlessly for us, I could be convinced.

[mfdeakin-sandia]

@PeterCaldwell I've been talking to NERSC consulting to try and determine the issue with sharing with ACLs, and we think it should work with the permissions I have now. Would you mind trying to access '/global/u2/m/mdeakin/acl-ftest' and '/global/u2/m/mdeakin/acl-test/acltest'? If it doesn't work; I'll give up on this path and add a sticky bit setting to the run-acme script instead of in CIME.

[PeterCaldwell]

/global/u2/m/mdeakin/acl-ftest seems to be an empty file. /global/u2/m/mdeakin/acl-test/acltest is a file that says "ACL working". Is this what you expect?

[mfdeakin-sandia]

Yes, that's correct. I'll see about adding an option to CIME for this after fixing to the branch run and hooks for run_acme

[rljacob]

Closing. Not sure we can overcome DOE system policies.