CTSM / CESM documentation does not mention LFS

[gold2718]

CTSM seems to be configured to use the LFS functionality to store binary files. However, neither the CTSM Recommended git setup nor the CESM Software requirements for installing, building and running CESM mention this required functionality.

manage_externals reports a failure on systems that do not have git-lfs installed.

I believe that this tool should be documented as required and instructions provided for installing it on different systems (e.g., macOS, Linux).

[billsacks]

@gold2718 I'd like to better understand the scenario where you ran into this. Git LFS is needed in limited circumstances for CTSM, but shouldn't be needed in general. You are right that we should have better documentation of this need; thank you for pointing that out. I have just added some documentation here: https://github.com/ESCOMP/CTSM/wiki/Recommended-git-setup#24installing-git-lfs.

The scenario where I have run into this issue is the one given in the note near the bottom of that section. Is that possibly what is going on for you? If so, I'm not sure what the best way is to document this: it's not so much that we have a requirement for git lfs, but it's more like "If you have your environment setup to use git lfs, then you need to be sure to have a working Git LFS tool installed"... and I'm not sure whether or where that should be documented. I'm open to advice here.

[gold2718]

I'd like to better understand the scenario where you ran into this.

While this did not happen in earlier tags (at least up through ctsm5.1.dev067), it does happen for ctsm5.1.dev069 and ctsm5.1.dev070.

I am not sure what other information to provide (git --version just reports git version 2.32.0).

I see the item you wrote about installing git-LFS, however, if as you said, I do not need this, please add instructions for turning it off so that running manage_externals does not try to load these files. I do not have any [filter "lfs"] section in my .gitconfig.

[billsacks]

I'm trying to reproduce this but am unable to. I do know the change in recent versions of CTSM that is probably responsible for this: we now instruct git lfs to fetch some files by default (in the .lfsconfig file at the root of the CTSM repository), but from my testing, this gets ignored if you don't have lfs installed. So I'm trying to figure out why it isn't getting ignored for you.

Can you please give me more details to help me reproduce this so that I can figure out what's going on and how to resolve it? For example:

• what system is this on?
• what are steps to reproduce it? (e.g., does this happen when updating an existing repo, checking out a fresh copy, etc.?)
• what is the error message that you get?

[ekluzek]

I think this one relates to #1611. As such I think it probably refers to an existing clone, where the user has not done "git lfs install" nor added lfs handling in .gitconfig.

[billsacks]

I think this one relates to #1611. As such I think it probably refers to an existing clone, where the user has not done "git lfs install" nor added lfs handling in .gitconfig.

I'm not sure about that myself. I would still like to hear from @gold2718 what his exact conditions were, but my sense is that these are two different issues:

• Master/Main branch Python make tests fail at the end #1611 describes an issue where you want to get the files from git lfs but git lfs didn't automatically get them for you (e.g., because you hadn't run 'git lfs install' at the time when you did your CTSM clone), so the python testing fails
• This issue seems to describe a different issue where git itself is giving an error for reasons I still haven't been able to determine, but I'm guessing may be due to some incompatibility between the available git / git lfs version and what's in your .gitconfig file?

So, while they both relate to git lfs, my guess is that the underlying issue differs. But I look forward to seeing a reproducer for both of them so that we can figure this out for sure.

[gold2718]

Sorry for the delay but it seems that I can no longer reproduce this. Also, I believe that one time, the problem went away and then came back.
This leads me to suspect that what I was seeing was maybe some glitch in the GitHub LFS system or server. I do not have git lfs installed but now can check out different versions of CTSM, either an update to an existing checkout or to an empty directory.

Sorry for not posting the original error message. Maybe close this for now unless I run into this again and can submit more complete information?

[ekluzek]

Thanks for getting back to us @gold2718. Note that this issue asks for documentation and both @billsacks and I worked on the recommended git setup page. @billsacks look at what I did there and feel free to change it. Also I'm closing this, but everyone feel free to reopen if that makes sense.

[billsacks]

Okay, thanks for the reply @gold2718 . As you suggest, I will close this for now, but please do let us know if you run into this again. My impression is that the GitHub LFS system shouldn't be hit at all if you haven't done a "git lfs install" (and so don't have anything related to lfs in your git configuration), but I could be wrong on that.

[billsacks]

Sorry, @ekluzek , my message crossed with yours. Yes, I'm happy with your changes to the wiki page (it looks to me like your main change was to move the lfs section up and add a note pointing to that section; I'm happy with both of those changes).