Squashed 'manage_externals/' changes from 5fc8709..fde04e4

fde04e4 Merge pull request ESCOMP#138 from billsacks/add_python38_tests
37e4c4a Do not update dictionary in-place in loop
7e8474b Remove testing on mac os
7f41c56 Fix pylint issue
3065b0d Add travis-ci tests with python3.7 and python3.8
34fbf55 Add support for git sparse checkout
6c6ef9f Fix pylint errors
6a659ad Added test for sparse checkout and updated documentation
1443243 Support for git sparsecheckout via read-tree.
a48558d Merge pull request ESCOMP#119 from gold2718/submodules
f72ffe7 Do not try git submodule update if no .gitmodules file (git bug)
804e0af Fix a pylint error
45aef95 Addressed review concerns
7da5031 New capability to use git submodule information to checkout externals
1926530 Merge pull request ESCOMP#118 from mnlevy1981/svn_switch
b1b028d Updates after testing
9ea73e6 Add --svn-ignore-ancestry argument
fc5acda Merge pull request ESCOMP#114 from billsacks/fix_large_output_hang
aa2eb71 Try getting travis-ci working on MacOS
96842b4 Fix pylint errors
813fe3c pylint: disable useless-object-inheritance
c49d878 Rework execute_subprocess timeout handling
8fc0e5f Cleanup from 'make style'
b0b23a6 Merge pull request ESCOMP#110 from gold2718/help_fix
3cbcd16 Fixed and clarified help documentation
025e6cb Merge pull request ESCOMP#107 from jedwards4b/ignore_empty_git_dir
489842b if you encounter an empty directory clone into it
0c5a2f6 Merge pull request ESCOMP#106 from billsacks/remove_logfile_message
7799e99 Remove message about checking the log file for more details
0427305 Merge pull request ESCOMP#103 from billsacks/no_logging
9bb46aa Make no-logging be the default
9af6b02 Merge pull request ESCOMP#102 from billsacks/explain_qmark
7f973ae Run through make style
d077a57 Add message describing meaning of '?'
60fc03b Merge pull request ESCOMP#101 from ESMCI/catch_svn_error
28073ec add exception class
4fb7e47 catch errors from svn status --xml
bfa4831 Merge pull request ESCOMP#98 from billsacks/quieter
7d12650 make style
afb4f11 Make more git and svn commands quieter
a465b4f add --quiet argument to improve performance
b2f3ae8 Merge pull request ESCOMP#83 from jedwards4b/jedwards/components_arg
3f4c88f fix comment
c1b5b09 remove unneeded logic
4fdf180 one more test
f78d60f another test
bf52ac6 add a test
91d4851 fix pylint issue
987df5a only use components if populated
98a810d add a components arg to checkout only select components
6923119 Merge pull request ESCOMP#90 from ESMCI/issue-86-detached-sync-status
b11ad61 Merge branch 'master' into issue-86-detached-sync-status
3b624cf Merge pull request ESCOMP#93 from billsacks/work_on_coverage
2562830 Run a single coverage command rather than two separate commands
d1de5f8 Return to starting directory after each test
144f7d9 Merge pull request ESCOMP#92 from billsacks/point_to_esmci
58b8d3e Point to location of repository
0b46d81 Point to correct location for build/coverage status
a385070 fix pylint problems
dcf17b6 make style cleanup
92d342c Rewrite _current_ref to use plumbing rather than parsing porcelain
ca0a5d3 Rework some git repository functions, and major rework of unit tests
719383e Remove commented-out pdb.set_trace() call
376c780 Bugfix: detect and report 'detached from' correctly
21813e9 Add system test demonstrating failure to detect out of sync status.
1a7c59d Merge documentation update into master.
f1e9e99 Merge schema support for git hashes into master.
247fee1 Document return values of checkout.py: main
195c1d0 Implement explicit use of a hash for git repositories.
12dd743 Refactor: schema validation output
fdbc720 Bugfix: incorrect order of operations validing user input
d6423c6 Bugfix: timeout limit for subprocesses
7998f60 Update readme and help output
00b6fb2 Bugfix: add explicit schema version checking
0527869 Update readme
1ae8c84 Merge bugfix branch for stale subexternals into master.
b0c16d7 Bugfix: stale sub-externals after checkout.
30a4e44 Finish implementing system test for mixed-use externals
ac7ff96 Update mixed-use test repo.
bfda7b9 Bugfix: regexp for determining git tracking branches

git-subtree-dir: manage_externals
git-subtree-split: fde04e4d9a758b3aa277aa5fa44a59f5153f2958

.travis.yml

@@ -1,24 +1,12 @@
-# NOTE(bja, 2017-11) travis-ci dosen't support python language builds
-#  on mac os. As a work around, we use built-in python on linux, and
-#  declare osx a 'generic' language, and create our own python env.
-
 language: python
 os: linux
 python: 
   - "2.7"
   - "3.4"
   - "3.5"
   - "3.6"
-matrix:
-  include:
-  - os: osx
-    language: generic
-    before_install:
-      # NOTE(bja, 2017-11) update is slow, 2.7.12 installed by default, good enough!
-      # - brew update
-      # - brew outdated python2 || brew upgrade python2
-      - virtualenv env -p python2
-      - source env/bin/activate
+  - "3.7"
+  - "3.8"
 install:
   - pip install -r test/requirements.txt
 before_script:
@@ -29,4 +17,3 @@ script:
 after_success:
   - cd test; make coverage
   - cd test; coveralls
-

README.md

@@ -1,20 +1,24 @@
 -- AUTOMATICALLY GENERATED FILE. DO NOT EDIT --
 
--n [![Build Status](https://travis-ci.org/NCAR/manage_externals.svg?branch=master)](https://travis-ci.org/NCAR/manage_externals)
-[![Coverage Status](https://coveralls.io/repos/github/NCAR/manage_externals/badge.svg?branch=master)](https://coveralls.io/github/NCAR/manage_externals?branch=master)
+[![Build Status](https://travis-ci.org/ESMCI/manage_externals.svg?branch=master)](https://travis-ci.org/ESMCI/manage_externals)[![Coverage Status](https://coveralls.io/repos/github/ESMCI/manage_externals/badge.svg?branch=master)](https://coveralls.io/github/ESMCI/manage_externals?branch=master)
 ```
-
 usage: checkout_externals [-h] [-e [EXTERNALS]] [-o] [-S] [-v] [--backtrace]
                           [-d] [--no-logging]
 
-checkout_externals manages checking out CESM externals from revision control
-based on a externals description file. By default only the required
-externals are checkout out.
+checkout_externals manages checking out groups of externals from revision
+control based on a externals description file. By default only the
+required externals are checkout out.
+
+Operations performed by manage_externals utilities are explicit and
+data driven. checkout_externals will always make the working copy *exactly*
+match what is in the externals file when modifying the working copy of
+a repository.
 
-NOTE: checkout_externals *MUST* be run from the root of the source tree.
+If checkout_externals isn't doing what you expected, double check the contents
+of the externals description file.
 
 Running checkout_externals without the '--status' option will always attempt to
-synchronize the working copy with the externals description.
+synchronize the working copy to exactly match the externals description.
 
 optional arguments:
   -h, --help            show this help message and exit
@@ -39,18 +43,18 @@ optional arguments:
 
 ```
 NOTE: checkout_externals *MUST* be run from the root of the source tree it
-is managing. For example, if you cloned CLM with:
+is managing. For example, if you cloned a repository with:
 
-    $ git clone [email protected]/ncar/clm clm-dev
+    $ git clone [email protected]/{SOME_ORG}/some-project some-project-dev
 
-Then the root of the source tree is /path/to/clm-dev. If you obtained
-CLM via a checkout of CESM:
+Then the root of the source tree is /path/to/some-project-dev. If you
+obtained a sub-project via a checkout of another project:
 
-    $ git clone [email protected]/escomp/cesm cesm-dev
+    $ git clone [email protected]/{SOME_ORG}/some-project some-project-dev
 
-and you need to checkout the CLM externals, then the root of the
-source tree is /path/to/cesm-dev. Do *NOT* run checkout_externals
-from within /path/to/cesm-dev/components/clm.
+and you need to checkout the sub-project externals, then the root of the
+source tree is /path/to/some-project-dev. Do *NOT* run checkout_externals
+from within /path/to/some-project-dev/sub-project
 
 The root of the source tree will be referred to as `${SRC_ROOT}` below.
 
@@ -74,11 +78,14 @@ The root of the source tree will be referred to as `${SRC_ROOT}` below.
     include: modified files, added files, removed files, or missing
     files.
 
+    To avoid this safety check, edit the externals description file
+    and comment out the modified external block.
+
   * Checkout all required components from a user specified externals
     description file:
 
         $ cd ${SRC_ROOT}
-        $ ./manage_externals/checkout_externals --excernals myCESM.xml
+        $ ./manage_externals/checkout_externals --externals my-externals.cfg
 
   * Status summary of the repositories managed by checkout_externals:
 
@@ -121,12 +128,17 @@ The root of the source tree will be referred to as `${SRC_ROOT}` below.
 # Externals description file
 
   The externals description contains a list of the external
-  repositories that are used and their version control locations. Each
-  external has:
+  repositories that are used and their version control locations. The
+  file format is the standard ini/cfg configuration file format. Each
+  external is defined by a section containing the component name in
+  square brackets:
 
-  * name (string) : component name, e.g. cime, cism, clm, cam, etc.
+  * name (string) : component name, e.g. [cime], [cism], etc.
 
-  * required (boolean) : whether the component is a required checkout
+  Each section has the following keyword-value pairs:
+
+  * required (boolean) : whether the component is a required checkout,
+    'true' or 'false'.
 
   * local_path (string) : component path *relative* to where
     checkout_externals is called.
@@ -135,38 +147,80 @@ The root of the source tree will be referred to as `${SRC_ROOT}` below.
     manage the component.  Valid values are 'git', 'svn',
     'externals_only'.
 
+    Switching an external between different protocols is not
+    supported, e.g. from svn to git. To switch protocols, you need to
+    manually move the old working copy to a new location.
+
     Note: 'externals_only' will only process the external's own
     external description file without trying to manage a repository
     for the component. This is used for retreiving externals for
-    standalone components like cam and clm.
+    standalone components like cam and clm. If the source root of the
+    externals_only component is the same as the main source root, then
+    the local path must be set to '.', the unix current working
+    directory, e. g. 'local_path = .'
 
   * repo_url (string) : URL for the repository location, examples:
     * https://svn-ccsm-models.cgd.ucar.edu/glc
     * [email protected]:esmci/cime.git
     * /path/to/local/repository
+    * .
+
+    NOTE: To operate on only the local clone and and ignore remote
+    repositories, set the url to '.' (the unix current path),
+    i.e. 'repo_url = .' . This can be used to checkout a local branch
+    instead of the upstream branch.
 
     If a repo url is determined to be a local path (not a network url)
     then user expansion, e.g. ~/, and environment variable expansion,
     e.g. $HOME or $REPO_ROOT, will be performed.
 
     Relative paths are difficult to get correct, especially for mixed
-    use repos like clm. It is advised that local paths expand to
-    absolute paths. If relative paths are used, they should be
-    relative to one level above local_path. If local path is
-    'src/foo', the the relative url should be relative to
-    'src'.
+    use repos. It is advised that local paths expand to absolute paths.
+    If relative paths are used, they should be relative to one level
+    above local_path. If local path is 'src/foo', the the relative url
+    should be relative to 'src'.
 
   * tag (string) : tag to checkout
 
-    This can also be a git SHA-1
-
-  * branch (string) : branch to checkout
-
-    Note: either tag or branch must be supplied, but not both.
-
-  * externals (string) : relative path to the external's own external
-    description file that should also be used. It is *relative* to the
-    component local_path. For example, the CESM externals description
-    will load clm. CLM has additional externals that must be
-    downloaded to be complete. Those additional externals are managed
-    from the clm source root by the file pointed to by 'externals'.
+  * hash (string) : the git hash to checkout. Only applies to git
+    repositories.
+
+  * branch (string) : branch to checkout from the specified
+    repository. Specifying a branch on a remote repository means that
+    checkout_externals will checkout the version of the branch in the remote,
+    not the the version in the local repository (if it exists).
+
+    Note: one and only one of tag, branch hash must be supplied.
+
+  * externals (string) : used to make manage_externals aware of
+    sub-externals required by an external. This is a relative path to
+    the external's root directory. For example, the main externals
+    description has an external checkout out at 'src/useful_library'.
+    useful_library requires additional externals to be complete.
+    Those additional externals are managed from the source root by the
+    externals description file pointed 'useful_library/sub-xternals.cfg',
+    Then the main 'externals' field in the top level repo should point to
+    'sub-externals.cfg'.
+
+  * from_submodule (True / False) : used to pull the repo_url, local_path,
+    and hash properties for this external from the .gitmodules file in
+    this repository. Note that the section name (the entry in square
+    brackets) must match the name in the .gitmodules file.
+    If from_submodule is True, the protocol must be git and no repo_url,
+    local_path, hash, branch, or tag entries are allowed.
+    Default: False
+
+  * sparse (string) : used to control a sparse checkout. This optional
+    entry should point to a filename (path relative to local_path) that
+    contains instructions on which repository paths to include (or
+    exclude) from the working tree.
+    See the "SPARSE CHECKOUT" section of https://git-scm.com/docs/git-read-tree
+    Default: sparse checkout is disabled
+
+  * Lines begining with '#' or ';' are comments and will be ignored.
+
+# Obtaining this tool, reporting issues, etc.
+
+  The master repository for manage_externals is
+  https://github.com/ESMCI/manage_externals. Any issues with this tool
+  should be reported there.

manic/__init__.py

@@ -1,7 +1,7 @@
 """Public API for the manage_externals library
 """
 
-import manic.checkout as checkout
+from manic import checkout
 from manic.utils import printlog
 
 __all__ = [