clone_extra_repos.py fails on Windows #98

lefebvre · 2015-11-23T17:36:55Z

clone_extra_repos.py fails attempting to import gitdist.

jap@warroom4 MINGW64 ~/projects/tutela
$ TriBITS/tribits/ci_support/clone_extra_repos.py
Traceback (most recent call last):
  File "TriBITS/tribits/ci_support/clone_extra_repos.py", line 45, in <module>
    import gitdist
  File "C:\Users\jap\projects\tutela\TriBITS\tribits\python_utils\gitdist.py", line 1, in <module>
    gitdist
NameError: name 'gitdist' is not defined

The text was updated successfully, but these errors were encountered:

bartlettroscoe · 2015-11-23T17:43:39Z

I know what the problem is. The problem is that gitdist.py is a symlink to gitdist. To fix this on Windows, we just need to make gidist.py the real file and gitdist the symlink. Then, on windows, if you want to use gitdist, just use gitdsit.py (or copy and paste gitdist.py to gitdist where you want to install it).

I will go ahead and make this change.

@lefebvre, What repo are you pulling TriBITS from?

lefebvre · 2015-11-23T17:53:37Z

For this new project I am working on I have a fork of TriBITSPub/TriBITS.

bartlettroscoe · 2015-11-23T17:55:57Z

Okay, then I will push the the Github 'master' branch first. Then you can just merge from this main TriBITS 'master' branch to your fork.

If you are making changes to TriBITS, let's try to get then back into the main version so they can be maintained long term. For that, take note of Contributing to TriBITS.

The clone_extra_repos.py script imports the gitdist.py module to get some of its functionality. On windows this fails because gitdist.py was just a symlink to gitdist. So, to support Windows, I just made the real file gitdist.py and the symlink is now gitdist. This should not impact Linux usage at all. And if you copy/install gitdist into another location, it will follow the symlink and copy the file in its entirety. With this change, on Windows, if you want to use gitdist, you will need to use gitdist.py. Fixes #9

bartlettroscoe · 2015-11-23T18:02:11Z

Jordan,

Please merge the updated 'master' branch (which I forced pushed to fix the GitHub Issue ID) into your fork and see if that does the trick.

Please give the results one way or the other so that I can confirm.

lefebvre · 2015-11-24T13:22:50Z

Ross,
It appears I'm running into an new issue.

$ TriBITS/tribits/ci_support/clone_extra_repos.py

**************************************************************************
Script: clone_extra_repos.py \
  --extra-repos='' \
  --not-extra-repos='' \
  --extra-repos-file='cmake/ExtraRepositoriesList.cmake' \
  --extra-repos-type='Nightly' \
  --gitolite-root='' \
  --with-cmake='cmake' \
  --verbosity='more' \
  --do-clone \
  --do-op \
  --create-gitdist-file='' \

Traceback (most recent call last):
  File "TriBITS/tribits/ci_support/clone_extra_repos.py", line 553, in <module>
    cloneExtraRepos(inOptions)
  File "TriBITS/tribits/ci_support/clone_extra_repos.py", line 457, in cloneExtraRepos
    verbose=isVerbosityLevel(inOptions, "most")
  File "TriBITS/tribits/ci_support/clone_extra_repos.py", line 297, in getExtraReposDictListFromCmakefile
    (headerOutput, extraReposPytonDictList) = getHeaderOutputAndExtraReposDictList(rawOutput)
  File "TriBITS/tribits/ci_support/clone_extra_repos.py", line 278, in getHeaderOutputAndExtraReposDictList
    pythonDictList = eval(pythonDictListStr)
  File "<string>", line 0

    ^
SyntaxError: unexpected EOF while parsing

bartlettroscoe · 2015-11-24T13:40:08Z

@lefebvre, can you provide more info on the context for this so I can try to reproduce?

lefebvre · 2015-11-24T15:33:01Z

Ross,
It looks like you are getting more then expected. Here is what Windows is getting on STDOUT.

$ TriBITS/tribits/ci_support/clone_extra_repos.py 2> error.txt

**************************************************************************
Script: clone_extra_repos.py \
  --extra-repos='' \
  --not-extra-repos='' \
  --extra-repos-file='cmake/ExtraRepositoriesList.cmake' \
  --extra-repos-type='Nightly' \
  --gitolite-root='' \
  --with-cmake='cmake' \
  --verbosity='more' \
  --do-clone \
  --do-op \
  --create-gitdist-file='' \

raw output:  *** Generate a Python datastructure containing TriBITS/git repos ...
PROJECT_SOURCE_DIR = C:\Users\jap\projects\tutela
TRIBITS_BASE_DIR = C:\Users\jap\projects\tutela\TriBITS\tribits
EXTRA_REPOS_FILE = cmake/ExtraRepositoriesList.cmake
EXTRA_REPOS =
EXTRA_REPOS_PYTHON_OUT_FILE =
ENABLE_KNOWN_EXTERNAL_REPOS_TYPE = Nightly
IGNORE_MISSING_EXTRA_REPOSITORIES =
CHECK_EXTRAREPOS_EXIST = FALSE
PROJECT_NAME = Tutela

Reading the list of extra repositories from cmake/ExtraRepositoriesList.cmake

NOTICE: the repo HYSPLIT is a Mercurial repo: these are tolerated, but not fully supported.
NOTICE: the repo APT is a Mercurial repo: these are tolerated, but not fully supported.

Selecting the set of 'Nightly' extra repos (asserting all selected repos exist) ...

-- Adding POST extra Continuous repository DELFIC ...
-- Adding POST extra Continuous repository HYSPLIT ...
-- Adding POST extra Continuous repository DELFIC-HYSPLIT ...
-- Adding POST extra Continuous repository APT ...

*** Extra Repositories Python Dictionary
[
{'NAME' : 'DELFIC', 'DIR' : 'DELFIC', 'REPOTYPE' : 'GIT', 'REPOURL' : 'http://ci.ornl.gov:8000/scm/git/delfic', 'HASPKGS' : 'HASPACKAGES', 'PREPOST' : 'POST', 'CATEGORY' : 'Continuous', },
{'NAME' : 'HYSPLIT', 'DIR' : 'HYSPLIT', 'REPOTYPE' : 'HG', 'REPOURL' : 'https://fogbugz.ornl.gov/kiln/code/Fallout/group/hysplit', 'HASPKGS' : 'HASPACKAGES', 'PREPOST' : 'POST', 'CATEGORY' : 'Continuous', },
{'NAME' : 'DELFIC-HYSPLIT', 'DIR' : 'DELFIC-HYSPLIT', 'REPOTYPE' : 'GIT', 'REPOURL' : 'https://code.ornl.gov/RNSD-NSM/DELFIC-HYSPLIT.git', 'HASPKGS' : 'HASPACKAGES', 'PREPOST' : 'POST', 'CATEGORY' : 'Continuous', },
{'NAME' : 'APT', 'DIR' : 'APT', 'REPOTYPE' : 'HG', 'REPOURL' : 'https://fogbugz.ornl.gov/kiln/code/Fallout/group/APT', 'HASPKGS' : 'HASPACKAGES', 'PREPOST' : 'POST', 'CATEGORY' : 'Continuous', },
]

My guess is that you are just expecting the array structure.

bartlettroscoe · 2015-11-24T15:44:28Z

The clone_extra_repos.py script has not been tested with Hg repos. It would need to be extended for Hg repos. You can look at the automated tests for clone_extra_repos.py in the file:

 TriBITS/test/ci_support/clone_extra_repos_UnitTests.py

It should be much easier to add Hg support to the clone_extra_repos.py script than say the checkin-test.py script.

lefebvre · 2015-11-24T16:19:14Z

Ross,
I have already added the necessary mercurial support in clone_extra_repos.
You can see this activity in TriBITS #17.
I did a little more digging.
The issue is actually because of windows line carriage returns.
This resolved the issue with python 2.7.10. I don't know about backwards compatability:

diff --git a/tribits/ci_support/clone_extra_repos.py b/tribits/ci_support/clone_extra_repos.py
index a7f8111..a6f4c2f 100755
--- a/tribits/ci_support/clone_extra_repos.py
+++ b/tribits/ci_support/clone_extra_repos.py
@@ -265,7 +265,7 @@ def getHeaderOutputAndExtraReposDictList(rawOutputFromCmakefile):
   headerOuput = ""
   pythonDictListStr = ""
   processingPythonDict = False
-  for line in rawOutputFromCmakefile.split("\n"):
+  for line in rawOutputFromCmakefile.splitlines():
     if line == "*** Extra Repositories Python Dictionary":
       processingPythonDict=True
       continue

…latform agnostic splitlines().

bartlettroscoe · 2015-11-24T16:43:46Z

That change seems to work fine with python 2.6.6. That is what we have on the CASL machines. That is good enough for me.

I will go ahead and push this to the main TriBITS GitHub 'master' branch.

lefebvre · 2015-11-24T16:56:42Z

Ross,
I noticed this split('\n') in many different locations.
You may want to replace them all across *.py to ensure Windows support.

bartlettroscoe · 2015-11-24T17:18:30Z

It looks like splitlines() is not supported with Python 2.4.4 as shown here but is listed as supported in Python 2.6.9 as shown here. So, with this change, we loose backward compatibility to Python 2.4. As recently as 3/3/2015, there was an old Trilinos test machine that was running Python 2.4.3. But it looks like the remaining test machines are all 2.6.6 and above.

I am replacing all usages of split("\n") with splitlines() and upgrading the min version of Python required for the TriBITS python code from 2.4 to 2.6. I just know that some Trilinos developer is going to complain but Python 2.6.6 is an old version of Python released way back in 2010 (see here).

bartlettroscoe · 2015-11-24T17:30:05Z

I noticed this split('\n') in many different locations.

I am replacing those too.

You may want to replace them all across *.py to ensure Windows support.

That is what I am in the process of doing. Amazingly, there is slightly different behavior between split("\n") and splitlines(). I am having to fix up some code to get the tests to pass again.

bartlettroscoe · 2015-11-24T17:45:06Z

Looks like split("\n") returns an empty array element at the end while splitlines() does not. That is good, except I wrote the code to expect an empty element at the end. Now I am fixing that.

lefebvre · 2015-11-24T17:48:01Z

Devil in the details...

Changed from split("\n") to splitlines() to deal with newlines in MS Windows. Jordan reports that this fixes a problem on MS Windows with the clone_extra_repos.py script. This also seems to be supported with Python 2.6.6 (which is what is running on the current machine). This means that the min version of Python required for TriBITS python support is now Python 2.6.6. It actually took a good bit of work to adjust everything for this change as split("\n") will always return an empty element at the end while splitlines() will not. It was tricky updating all of the code and tests due this simple change.

Now that the TriBITS python code is using splitlines() which is not supported in Python 2.4, I have upgraded the min Python version to 2.6. Hopefully there are not still machines running Python 2.4. According to: https://www.python.org/doc/versions/ Python 2.6.1 was first released way back in Dec. 2008. I don't know of any other moderately recent machines that don't run 2.6.6 or above. Build/Test Cases Summary Enabled Packages: TriBITS Enabled all Forward Packages 0) MPI_DEBUG => passed: passed=199,notpassed=0 (0.93 min) 1) SERIAL_RELEASE => passed: passed=199,notpassed=0 (0.78 min) Other local commits for this build/test group: 2ce3206

bartlettroscoe · 2015-11-25T03:26:54Z

@lefebvre, I just pushed the commits. Let me know if that solves the problem or not so that I can close this Issue.

lefebvre · 2015-11-25T15:01:10Z

Windows works. Thanks.

bartlettroscoe added the type: bug label Nov 23, 2015

bartlettroscoe added this to the 3_in_progress milestone Nov 23, 2015

bartlettroscoe self-assigned this Nov 23, 2015

bartlettroscoe closed this as completed in 3605bfe Nov 23, 2015

lefebvre pushed a commit to lefebvre/TriBITS that referenced this issue Nov 24, 2015

Merging TriBITS TriBITSPub#98 fix into mercurial_support

3f6cc8d

lefebvre added a commit to lefebvre/TriBITS that referenced this issue Nov 24, 2015

TriBITS TriBITSPub#98: Fixing clone_extra_repos.py split('\n') with p…

5628206

…latform agnostic splitlines().

bartlettroscoe reopened this Nov 24, 2015

lefebvre closed this as completed Nov 25, 2015

bartlettroscoe added this to the 6_deployed milestone Jan 7, 2016

bartlettroscoe removed this from the 3_in_progress milestone Jan 7, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

clone_extra_repos.py fails on Windows #98

clone_extra_repos.py fails on Windows #98

lefebvre commented Nov 23, 2015

bartlettroscoe commented Nov 23, 2015

lefebvre commented Nov 23, 2015

bartlettroscoe commented Nov 23, 2015

bartlettroscoe commented Nov 23, 2015

lefebvre commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

lefebvre commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

lefebvre commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

lefebvre commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

lefebvre commented Nov 24, 2015

bartlettroscoe commented Nov 25, 2015

lefebvre commented Nov 25, 2015

clone_extra_repos.py fails on Windows #98

clone_extra_repos.py fails on Windows #98

Comments

lefebvre commented Nov 23, 2015

bartlettroscoe commented Nov 23, 2015

lefebvre commented Nov 23, 2015

bartlettroscoe commented Nov 23, 2015

bartlettroscoe commented Nov 23, 2015

lefebvre commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

lefebvre commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

lefebvre commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

lefebvre commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

bartlettroscoe commented Nov 24, 2015

lefebvre commented Nov 24, 2015

bartlettroscoe commented Nov 25, 2015

lefebvre commented Nov 25, 2015