Problematic SVN certificate with repo #2

cponder · 2023-03-09T13:54:30Z

This setup command

./manage_externals/checkout_externals -v

gives these failure messages

    svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04 /lustre/fsw/devtech/hpc-devtech/cponder/EarthWorks/2023-03-08/CESM/components/cam/chem_proc
ERROR:root:Command '['svn', 'checkout', '--quiet', 'https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04', '/lustre/fsw/devtech/hpc-devtech/cponder/EarthWorks/2023-03-08/CESM/components/cam/chem_proc']' returned non-zero exit status 1.
ERROR:root:Failed with output:
    svn: E170013: Unable to connect to a repository at URL 'https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04'
    svn: E230001: Server SSL certificate verification failed: issuer is not trusted

ERROR: In directory
    /lustre/fsw/devtech/hpc-devtech/cponder/EarthWorks/2023-03-08/CESM
Process did not run successfully; returned status 1:
    svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04 /lustre/fsw/devtech/hpc-devtech/cponder/EarthWorks/2023-03-08/CESM/components/cam/chem_proc
See above for output from failed command.

The text was updated successfully, but these errors were encountered:

cponder · 2023-03-09T13:54:55Z

I can reproduce the failure if I run the command standalone. It goes away if I add these flags

svn checkout --trust-server-cert-failures=unknown-ca --non-interactive --quiet ...

cponder · 2023-03-09T13:58:08Z

If I have push permission on the repository, I can try to fix this.
The svn operations are not in the top-level script so I can't bypass the problem just by rewriting one file.

gold2718 · 2023-03-09T14:30:54Z

Have you considered updating that external to the GitHub version?

[chem_proc]
local_path = chem_proc
protocol = git
repo_url = https://github.com/ESCOMP/CHEM_PREPROCESSOR.git
tag = chem_proc5_0_05
required = True

cponder · 2023-03-09T14:48:51Z

Whatever settings are in place, came from the repo here. What are you telling me that I should change?
The checkout_externals script is calling other things that invoke svn commands.

cponder · 2023-03-09T14:51:53Z

I'd prefer that everything be git-based, rather than mixing revision-control methods.
I'm seeing svn being discouraged in some work environments.
If the same files are accessible from git repos instead, this codebase ought to be updated to use it.

BTW, I've been looking to see if the --trust-server-cert-failures=unknown-ca --non-interactive --quiet can be set as environment variables, which would be a workaround in this case.
I get the impression that no one has documented the list of SVN environment variables, though.

briandobbins · 2023-03-09T14:55:05Z

I think this'll depend on where Earthworks's CAM diverges from our CAM - the EW one currently pulls from Dylan's repo, so maybe he can chime in? For CAM, this change happened in cam6_3_067, but it seems EW is based on an older version using the old SVN repo. If it's a drop-in replacement, you just need to update the Externals_CAM.cfg (in https://github.com/gdicker1/CAM/blob/earthworks/Externals_CAM.cfg ) from: [chem_proc] local_path = chem_proc protocol = svn repo_url = https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags tag = chem_proc5_0_04 required = True To: [chem_proc] local_path = chem_proc protocol = git repo_url = https://github.com/ESCOMP/CHEM_PREPROCESSOR.git tag = chem_proc5_0_05 required = True Note that there *is* a minor bump in chem_proc version, but I don't know the details of that and whether it impacts you. I suspect not, but again, understanding where Earthworks needs CAM to be would be helpful.

…

On Thu, Mar 9, 2023 at 7:49 AM Carl Ponder ***@***.***> wrote: Whatever settings are in place, came from the repo here. What are you telling me that I should change? — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACL2HPPURXQYMN4CMB7O7KDW3HUV7ANCNFSM6AAAAAAVVEYESE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

gold2718 · 2023-03-09T14:58:15Z

Whatever settings are in place, came from the repo here. What are you telling me that I should change? The checkout_externals script is calling other things that invoke svn commands.

If you look at the file, Externals_CAM.cfg (https://github.com/gdicker1/CAM/blob/earthworks/Externals_CAM.cfg), you can see that the EarthWorks version of CAM (currently based on the cam6_3_059 version of CAM from May, 2022) still points at the SVN version of the chem_proc repository. CAM moved to a GitHub version back in July, 2022 (cam6_3_067, partially because of the issues with the NCAR svn server). To avoid these problems, simply update your CAM externals file (as noted by @briandobbins above).

cponder · 2023-03-09T15:03:56Z

There are others too:

[chem_proc]
local_path = chem_proc
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags
tag = chem_proc5_0_04
required = True

cosp2]
local_path = src/physics/cosp2/src
protocol = svn
repo_url = https://github.com/CFMIP/COSPv2.0/tags/
tag = v2.1.4cesm/src
required = True

[clubb]
local_path = src/physics/clubb
protocol = svn
repo_url = https://github.com/larson-group/clubb_release/tags/
tag = clubb_4ncar_20220311_f51de38/src/CLUBB_core
required = True

[silhs]
local_path = src/physics/silhs
protocol = svn
repo_url = https://github.com/larson-group/clubb_release/tags/
tag = clubb_4ncar_20220311_f51de38/src/SILHS
required = True

In the GitHub cases, can I just change svn to git?

briandobbins · 2023-03-09T15:06:24Z

The other ones are pointing to Github, no? The only problematic one is the 'https://svn-ccsm-models.cgd.ucar.edu' server, which doesn't have a valid certificate. - Brian

…

On Thu, Mar 9, 2023 at 8:04 AM Carl Ponder ***@***.***> wrote: There are others too: [chem_proc] local_path = chem_proc protocol = svn repo_url = https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags tag = chem_proc5_0_04 required = True cosp2] local_path = src/physics/cosp2/src protocol = svn repo_url = https://github.com/CFMIP/COSPv2.0/tags/ tag = v2.1.4cesm/src required = True [clubb] local_path = src/physics/clubb protocol = svn repo_url = https://github.com/larson-group/clubb_release/tags/ tag = clubb_4ncar_20220311_f51de38/src/CLUBB_core required = True [silhs] local_path = src/physics/silhs protocol = svn repo_url = https://github.com/larson-group/clubb_release/tags/ tag = clubb_4ncar_20220311_f51de38/src/SILHS required = True If they all have corresponding Git repos, can you give me the parameters? — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACL2HPJ6SPQMUFA32ZDUH3DW3HWOPANCNFSM6AAAAAAVVEYESE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

cponder · 2023-03-09T15:17:55Z

The config-file gets pulled-in by the checkout_externals script, which does a partial build.
When I run that script, it leaves some wreckage in the directory, so if I modify the config file I can't resume the build because of the broken pieces:

The external repositories labeled with 'M' above are not in a clean state.

This file needs to be fixed in the repo.
@gdicker1 Can you at least create a branch that I can try using?

gold2718 · 2023-03-09T15:20:04Z

There are others too:

These are using the GitHub svn bridge in order to pull a subdirectory from a very large repository since CAM only needs part of the code. Until recently, git versions that can do this natively have not been universally available.

cponder · 2023-03-09T15:22:37Z

I think the SVN ops worked ok in the other cases. Can you at least change the chem_proc case?
You could just put it on a branch for now.

gold2718 · 2023-03-09T15:25:38Z

The config-file gets pulled-in by the checkout_externals script, which does a partial build.

checkout_externals does not do any building

When I run that script, it leaves some wreckage in the directory, so if I modify the config file I can't resume the build because of the broken pieces:

When switching from SVN to git, there is a one-time issue where you need to remove that directory (rm -r <root>/components/cam/chem_proc) and rerun checkout_externals.

gdicker1 · 2023-03-09T15:30:05Z

Not the correct repo please use https://github.com/EarthWorksOrg/EarthWorks for development

gold2718 · 2023-03-09T15:34:22Z

@gdicker1, are you going to open a corresponding issue on https://github.com/EarthWorksOrg/CAM?

gdicker1 · 2023-03-09T15:46:28Z

You can continue your discussion here: EarthWorksOrg/EarthWorks#10

This may be something CAM developers have already addressed. EarthWorksOrg/CAM is based on cam6_3_059 but we plan to upgrade it and other CESM-based externals in the coming weeks. At least to whatever versions are referenced ESCOM/CESM cesm2_3_beta11 tag

cponder · 2023-03-09T15:52:49Z

Removing the <root>/components/cam/chem_proc and re-trying just causes the same problem:

M  ./components/cam
        modified sandbox, on release-cam-ew1.0
e-  ./components/cam/chem_proc
        -, not checked out --> chem_proc5_0_05
The external repositories labeled with 'M' above are not in a clean state.

gold2718 · 2023-03-09T17:00:43Z

Removing the /components/cam/chem_proc and re-trying just causes the same problem:

Oops, sorry about that. Given the comments about this repo being deprecated, I'm not sure what your best move is. One easy thing is to just remove the chem_proc external from Externals_CAM.cfg. This will lead to other problems if you still need to run checkout_externals but this is really an EarthWorks code management issue so I can't really help more than this. You could always make your own branch and update the top-level Externals.cfg but that is a significant escalation in the amount of work you need to do. Hopefully, the EarthWorks externals will be update soon to get you past this.

BTW, if you do remove the chem_proc entry from your Externals_CAM.cfg, you might be able to:
From the top level:

./manage_externals/checkout_externals --exclude CAM

From the CAM directory:

./manage_externals/checkout_externals

I haven't tested this but it might work.

gdicker1 closed this as not planned Won't fix, can't repro, duplicate, stale Mar 9, 2023

gdicker1 mentioned this issue Mar 9, 2023

NVIDIA Problematic SVN certificate with repo EarthWorksOrg/EarthWorks#10

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problematic SVN certificate with repo #2

Problematic SVN certificate with repo #2

cponder commented Mar 9, 2023 •

edited

Loading

cponder commented Mar 9, 2023

cponder commented Mar 9, 2023 •

edited

Loading

gold2718 commented Mar 9, 2023

cponder commented Mar 9, 2023 •

edited

Loading

cponder commented Mar 9, 2023 •

edited

Loading

briandobbins commented Mar 9, 2023 via email

gold2718 commented Mar 9, 2023

cponder commented Mar 9, 2023 •

edited

Loading

briandobbins commented Mar 9, 2023 via email

cponder commented Mar 9, 2023 •

edited

Loading

gold2718 commented Mar 9, 2023

cponder commented Mar 9, 2023

gold2718 commented Mar 9, 2023

gdicker1 commented Mar 9, 2023

gold2718 commented Mar 9, 2023

gdicker1 commented Mar 9, 2023

cponder commented Mar 9, 2023

gold2718 commented Mar 9, 2023

Problematic SVN certificate with repo #2

Problematic SVN certificate with repo #2

Comments

cponder commented Mar 9, 2023 • edited Loading

cponder commented Mar 9, 2023

cponder commented Mar 9, 2023 • edited Loading

gold2718 commented Mar 9, 2023

cponder commented Mar 9, 2023 • edited Loading

cponder commented Mar 9, 2023 • edited Loading

briandobbins commented Mar 9, 2023 via email

gold2718 commented Mar 9, 2023

cponder commented Mar 9, 2023 • edited Loading

briandobbins commented Mar 9, 2023 via email

cponder commented Mar 9, 2023 • edited Loading

gold2718 commented Mar 9, 2023

cponder commented Mar 9, 2023

gold2718 commented Mar 9, 2023

gdicker1 commented Mar 9, 2023

gold2718 commented Mar 9, 2023

gdicker1 commented Mar 9, 2023

cponder commented Mar 9, 2023

gold2718 commented Mar 9, 2023

cponder commented Mar 9, 2023 •

edited

Loading

cponder commented Mar 9, 2023 •

edited

Loading

cponder commented Mar 9, 2023 •

edited

Loading

cponder commented Mar 9, 2023 •

edited

Loading

cponder commented Mar 9, 2023 •

edited

Loading

cponder commented Mar 9, 2023 •

edited

Loading