Protection Domain Tags (Cray Specific)

Note: If you are going to be using the "ugni" transport (RDMA over Gemini) you will need to run with either system (as root) or user (as user) ptags. While root CAN run using any ptag the fact that its use is unknown to ALPS could cause collisions with applications.

Viewing Current Ptags

To see current ptags (XE/XK):

 > apstat -P
 PDomainID         Type      Uid     PTag     Cookie
 LDMS              system      0       84 0xa9380000

To see current configured RDMA credentials (XC CLE6 or greater):

 > module load rdma-credentials
 > drccli list
 Credential Sys_Id       Name          uid      gid  wlmid     Cookie1     Cookie2   State       Refs
     1 	    29979        LDMS            0        0      0     0x759a0000        0    READY       1

Creating Ptags

To create a userspace ptag (XE/XK):

 > apmgr pdomain -c foo
 > apstat -P
 PDomainID         Type       Uid  PTag      Cookie
 LDMS              system       0    84  0xa9380000
 foo               user     12345   233  0xa1230000

Note: A system administrator will have to setup system ptags and/or enable users to set up ptags in the following way.

• In /etc/alps.conf file in sharedroot modify the following:

 # Variables to enable user/system protection domains
 # 
 # Max # of user protection domains allowed
        pDomainMax      10
 # List of system protection domains used by all applications
        pDomainIDs      sys1,sys2

To create userspace RDMA credentials (XC CLE6 or greater):

 A system administrator will have to do the following, possibly on an internal login node, depending on your system setup.
 > module load rdma-credentials
 > drccli acquire -n LDMS 
 > drccli list
 Credential Sys_Id       Name          uid      gid  wlmid     Cookie1     Cookie2   State       Refs
     1 	    29979        LDMS            0        0      0     0x759a0000        0    READY       1

Removing PTags

To remove a userspace ptag (XE/XK):

 apmgr pdomain -r <somenamehere>

Note: The userid of the ptag being removed must match that of the user running the command or root

To remove a userspace ptag (XC CLE6 or greater):

 > module load rdma-credentials
 > drccli release 1

Note: argument of release is the Credential number from drccli list Note: The userid of the ptag being removed must match that of the user running the command or root

LDMS Environment Variables Related to PTags

Set the following environment variables for either user or system ptags (example shows user ptag values) (XE/XK):

 export LDMS_UGNI_PTAG 233
 export LDMS_UGNI_COOKIE 0xa1230000

Set the following environment variables for either user or system ptags (example shows user ptag values) (XC CLE6 or greater):

 export LDMS_UGNI_PTAG 0
 export LDMS_UGNI_COOKIE 0x86bb0000

Running with PTags

When running with user space ptags you must specify the ptag name when using aprun (XE/XK/XC):

 aprun <usual aprun args here> -p foo ldmsd <usual ldmsd flags here>
 or
 aprun <usual aprun args here> -p foo ldms_ls <usual ldms_ls flags here>
 
 Note: On some systems you will run aprun after a qsub -I or 
 within a script specified in qsub or similar.