set TMPDIR variable to soft link pointing to PWD #21419

perrozzi · 2017-11-21T16:05:09Z

No description provided.

cmsbuild · 2017-11-21T16:05:49Z

The code-checks are being triggered in jenkins.

cmsbuild · 2017-11-21T16:08:40Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/PR-21419/2126

Code check has found code style and quality issues which could be resolved by applying a patch in https://cmssdt.cern.ch/SDT/code-checks/PR-21419/2126/git-diff.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/PR-21419/2126/git-diff.patch | patch -p1

You can run scram build code-checks to apply code checks directly

cmsbuild · 2017-11-21T16:12:36Z

The code-checks are being triggered in jenkins.

cmsbuild · 2017-11-21T16:16:20Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/PR-21419/2127

perrozzi · 2017-11-21T16:16:33Z

please test

cmsbuild · 2017-11-21T16:16:38Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24589/console Started: 2017/11/21 18:43

cmsbuild · 2017-11-21T16:16:40Z

A new Pull Request was created by @perrozzi for master.

It involves the following packages:

GeneratorInterface/SherpaInterface

@cmsbuild, @efeyazgan, @perrozzi, @thuer, @govoni can you please review it and eventually sign? Thanks.
@alberto-sanchez, @agrohsje, @mkirsano, @thuer this is something you requested to watch as well.
@davidlange6, @slava77 you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2017-11-21T19:57:17Z

+1
Tested at: 7269afc
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21419/24589/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
0707ff5
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21419/24589/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21419/24589/git-merge-result

cmsbuild · 2017-11-21T19:57:20Z

Comparison job queued.

cmsbuild · 2017-11-21T21:53:20Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21419/24589/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 27
DQMHistoTests: Total histograms compared: 2833444
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2833265
DQMHistoTests: Total skipped: 178
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1.00999999999 KiB( 23 files compared)
Checked 111 log files, 8 edm output root files, 27 DQM output files

cmsbuild · 2017-11-23T21:15:24Z

Pull request #21419 was updated. @cmsbuild, @efeyazgan, @perrozzi, @thuer, @govoni can you please check and sign again.

perrozzi · 2017-11-23T21:15:29Z

please test

cmsbuild · 2017-11-23T21:16:02Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24662/console Started: 2017/11/23 22:16

cmsbuild · 2017-11-23T23:37:37Z

+1
Tested at: b932324
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21419/24662/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
63e59e6
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21419/24662/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21419/24662/git-merge-result

cmsbuild · 2017-11-23T23:37:40Z

Comparison job queued.

davidlange6 · 2017-11-24T18:02:29Z

+1

davidlange6 · 2017-11-24T18:02:33Z

merge

davidlange6 · 2017-11-24T18:02:55Z

hi @perrozzi - could you back port this to 93x?

cmsbuild · 2017-11-25T12:43:21Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21419/24662/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 27
DQMHistoTests: Total histograms compared: 2833444
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2833265
DQMHistoTests: Total skipped: 178
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1.42999999996 KiB( 23 files compared)
Checked 111 log files, 8 edm output root files, 27 DQM output files

mrodozov · 2017-11-27T16:23:22Z

Unfortunately this PR broke this relval https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc630/CMSSW_10_0_X_2017-11-26-2300/pyRelValMatrixLogs/run/534.0_sherpa_ZtoEE_0j_OpenLoops_13TeV_MASTER+sherpa_ZtoEE_0j_OpenLoops_13TeV_MASTER+HARVESTGEN/step1_sherpa_ZtoEE_0j_OpenLoops_13TeV_MASTER+sherpa_ZtoEE_0j_OpenLoops_13TeV_MASTER+HARVESTGEN.log
and if I remove the changes it works

perrozzi · 2017-11-28T02:12:17Z

can you please provide a cmsDriver command or cfg to reproduce the error?

perrozzi · 2017-11-28T02:42:51Z

I tried with CMSSW_10_0_X_2017-11-26-2300 after adding the commits of this PR

    Merge pull request #21419 from perrozzi/setenv
    set TMPDIR variable to soft link pointing to PWD
commit 565fdd170c8639a4382bc7df5fcaac85a044ee10
Merge: dc1e4ec5d67 e150ca74ccb
--
Author: perrozzi <[email protected]>
Date:   Thu Nov 23 22:12:03 2017 +0100

and using the command
cmsDriver.py sherpa_ZtoEE_0j_OpenLoops_13TeV_MASTER_cff -s GEN,VALIDATION:genvalid --relval 250000,5000 --eventcontent RAWSIM,DQM --datatier GEN,DQMIO --conditions auto:run2_mc_FULL
I don't observe any crash

mrodozov · 2017-11-28T09:11:39Z

Try again with CMSSW_10_0_X_2017-11-27-2300 , same command or
runTheMatrix -l 534.0 -i all. Fail both ways

perrozzi · 2017-11-28T15:18:23Z

ok, I can reproduce the error.
it doesn't directly have to do with the modification, that goes smoothly,
but it has consequences that I am trying to understand when the TMPDIR is actually modified, in my case to
$TMPDIR = /tmp/perrozzi/tmp

perrozzi · 2017-11-28T15:40:37Z

first thing to note: if /tmp/perrozzi/tmp/ is a directory, things work. if is a soft link, it crashes....

perrozzi · 2017-11-28T15:48:32Z

might be related to this discussion, but I don't have any clue...
open-mpi/ompi#2328
anyway, this PR indeed doesn't seem to solve the issue
@Dr15Jones @davidlange6

Dr15Jones · 2017-11-28T16:22:30Z

GeneratorInterface/SherpaInterface/src/SherpackUtilities.cc

+    std::string tmpdir = getenv("TMPDIR");
+    if (tmpdir.size() > 50 ){
+      std::string command = "ln -s $PWD "+tmpdir+"/tmp;";
+      system(command.c_str());


This does not check if system returns a failure value.

I put printouts and confirm that things get created properly

A few things to consider in addition to Chris's comment:

What happens if this path already exists (but is owned by someone else)?

What happens if $TMPDIR is not unique to the job (several jobs run in the same directory, potentially owned by different users)?

What happens if $PWD contains spaces or special characters in it?

Who cleans up this symlink when the job finishes?

As written, if there's a shared $TMPDIR with a long name, one job will run successfully and then all subsequent ones on the worker node will fail.

@Dr15Jones @bbockelm investigating a bit further I came across this
https://www.open-mpi.org/faq/?category=osx#startup-errors-with-open-mpi-2.0.x
and the conclusion is:
The workaround for the Open MPI 2.0.x and v2.1.x release series is to set the TMPDIR environment variable to /tmp or other short directory name.
as far as I see from the example, the directory created in the TMPDIR path should be unique and related to the MPI session.
Would it be an acceptable solution to assing /tmp/ or /tmp/$USER/ as TMPDIR value in SherpackUtilities.cc? I don't see how we could move forward otherwise. Thanks a lot

most important, the written data seems to be cleaned up by MPI after use (i.e. if I point TMPDIR to any directory, it is empty after the job has either succeeded or failed)

set TMPDIR variable to soft link pointing to PWD

32e9981

cmsbuild added this to the CMSSW_10_0_X milestone Nov 21, 2017

cmsbuild added code-checks-pending comparison-pending generators-pending orp-pending pending-signatures tests-pending labels Nov 21, 2017

cmsbuild added code-checks-rejected and removed code-checks-pending labels Nov 21, 2017

apply code-checks

7269afc

cmsbuild added code-checks-pending and removed code-checks-rejected labels Nov 21, 2017

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 21, 2017

cmsbuild added tests-started and removed tests-pending labels Nov 21, 2017

cmsbuild added tests-approved and removed tests-started labels Nov 21, 2017

cmsbuild removed the comparison-pending label Nov 21, 2017

cmsbuild added the code-checks-approved label Nov 23, 2017

cmsbuild added tests-started and removed tests-pending labels Nov 23, 2017

cmsbuild added tests-approved and removed tests-started labels Nov 23, 2017

cmsbuild added orp-approved and removed orp-pending labels Nov 24, 2017

cmsbuild merged commit f2e3b7f into cms-sw:master Nov 24, 2017

perrozzi mentioned this pull request Nov 24, 2017

set TMPDIR variable to soft link pointing to PWD 93x #21458

Merged

cmsbuild added comparison-available and removed comparison-pending labels Nov 25, 2017

Dr15Jones reviewed Nov 28, 2017

View reviewed changes

perrozzi mentioned this pull request Dec 11, 2017

impose TMPDIR=/tmp/$USER #21682

Merged

makortel mentioned this pull request Sep 19, 2024

setenv() is not thread safe #46002

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

set TMPDIR variable to soft link pointing to PWD #21419

set TMPDIR variable to soft link pointing to PWD #21419

perrozzi commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

perrozzi commented Nov 21, 2017

cmsbuild commented Nov 21, 2017 •

edited

Loading

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 23, 2017

perrozzi commented Nov 23, 2017

cmsbuild commented Nov 23, 2017 •

edited

Loading

cmsbuild commented Nov 23, 2017

cmsbuild commented Nov 23, 2017

davidlange6 commented Nov 24, 2017

davidlange6 commented Nov 24, 2017

davidlange6 commented Nov 24, 2017

cmsbuild commented Nov 25, 2017

mrodozov commented Nov 27, 2017

perrozzi commented Nov 28, 2017

perrozzi commented Nov 28, 2017 •

edited

Loading

mrodozov commented Nov 28, 2017 •

edited

Loading

perrozzi commented Nov 28, 2017

perrozzi commented Nov 28, 2017

perrozzi commented Nov 28, 2017

Dr15Jones Nov 28, 2017

perrozzi Nov 28, 2017

bbockelm Nov 28, 2017

perrozzi Dec 11, 2017

perrozzi Dec 11, 2017 •

edited

Loading

set TMPDIR variable to soft link pointing to PWD #21419

set TMPDIR variable to soft link pointing to PWD #21419

Conversation

perrozzi commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

perrozzi commented Nov 21, 2017

cmsbuild commented Nov 21, 2017 • edited Loading

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 21, 2017

cmsbuild commented Nov 23, 2017

perrozzi commented Nov 23, 2017

cmsbuild commented Nov 23, 2017 • edited Loading

cmsbuild commented Nov 23, 2017

cmsbuild commented Nov 23, 2017

davidlange6 commented Nov 24, 2017

davidlange6 commented Nov 24, 2017

davidlange6 commented Nov 24, 2017

cmsbuild commented Nov 25, 2017

mrodozov commented Nov 27, 2017

perrozzi commented Nov 28, 2017

perrozzi commented Nov 28, 2017 • edited Loading

mrodozov commented Nov 28, 2017 • edited Loading

perrozzi commented Nov 28, 2017

perrozzi commented Nov 28, 2017

perrozzi commented Nov 28, 2017

Dr15Jones Nov 28, 2017

Choose a reason for hiding this comment

perrozzi Nov 28, 2017

Choose a reason for hiding this comment

bbockelm Nov 28, 2017

Choose a reason for hiding this comment

perrozzi Dec 11, 2017

Choose a reason for hiding this comment

perrozzi Dec 11, 2017 • edited Loading

Choose a reason for hiding this comment

cmsbuild commented Nov 21, 2017 •

edited

Loading

cmsbuild commented Nov 23, 2017 •

edited

Loading

perrozzi commented Nov 28, 2017 •

edited

Loading

mrodozov commented Nov 28, 2017 •

edited

Loading

perrozzi Dec 11, 2017 •

edited

Loading