Prov: gni fi_endpoint returned 16 (Device or resource busy) when creating 70 endpoints

[tonyzinger]

While executing a test that is creating 70 endpoints on a node, the fi_endpoint returns a "Device or resource busy" error. This test creates 70 pthreads. Then each pthread creates its own endpoint. So this test has created about 70 endpoints (I believe that maximum number of endpoints that I can create is 63).

From the fi_getinfo(), the domain attribute ep_cnt has the value 4294967295. This value is unrealistic.

Here is the Libfabric trace information:
2: [ 214] MR: GNI_MemRegister: Mem reg of 4096 length at addr 0x7fff66aa7000
2: [ 214] MR: GNI_MemRegister: ioctl(GNI_IOC_MEM_REGISTER) returned error - No space left on device
2: libfabric:gni:ep_ctrl:__nic_setup_irq_cq():230 [38214:57] GNI_MemRegister returned GNI_RC_ERROR_RESOURCE
2: libfabric:gni:ep_ctrl:gnix_nic_alloc():1257 [38214:57] __nic_setup_irq_cq returned Device or resource busy
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: part_dev_res_info: device resource_type: 2 not valid for this architecture
2: [ 214] JOB: GNI_GetDevResInfo: device resource_type: 2 not valid for this architecture
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: part_dev_res_info: Unable to parse '/sys/class/gni/ghal0/partitions' (rc: 1, expected: 7)
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: part_dev_res_info: Unable to parse '/sys/class/gni/ghal0/partitions' (rc: 1, expected: 7)
2: libfabric:gni:fabric:_gnix_dump_gni_res():673 [38214:57] Device Resources:
2: dev res: MDD, avail: 3894 res: 409 held: 0 total: 4095
2: dev res: CQ, avail: 1654 res: 64 held: 0 total: 2047
2: dev res: FMA, avail: 125 res: 4 held: 0 total: 127
2: dev res: CE, avail: 4 res: 0 held: 0 total: 4
2: dev res: DLA, avail: 16384 res: 1024 held: 0 total: 16384
2: dev res: TCR, avail: 60584 res: 0 held: 0 total: 16
2: dev res: DVA, avail: 4398046511104 res: 1099511627776 held: 0 total: 4398046511104
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource: MDD (1) used: 192 limit: 1806
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource_type: MRT (2) not valid for this architecture
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource: IOMMU (3) used: 134217728 limit: 134217728
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource_type: GART (4) not valid for this architecture
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource: CQ (5) used: 384 limit: 495
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource: FMA (6) used: 0 limit: 123
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource: RDMA (7) used: 0 limit: 18446744073709551615
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource: CE (8) used: 0 limit: 1
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource: DLA (9) used: 0 limit: 15360
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource: SFMA (10) used: 0 limit: 123
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource: DIRECT (11) used: 0 limit: 18446744073709551615
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource: PCI-IOMMU (12) used: 0 limit: 18446744073709551615
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [ 214] JOB: GNI_GetJobResInfo: job resource: non-VMDH (13) used: 127 limit: 18446744073709551615
2: libfabric:gni:fabric:_gnix_dump_gni_res():689 [38214:57] Job Resources:
2: ptag[214] job res: MDD used: 192 limit: 1806
2: ptag[214] job res: IOMMU used: 134217728 limit: 134217728
2: ptag[214] job res: CQ used: 384 limit: 495
2: ptag[214] job res: FMA used: 0 limit: 123
2: ptag[214] job res: RDMA used: 0 limit: 18446744073709551615
2: ptag[214] job res: CE used: 0 limit: 1
2: ptag[214] job res: DLA used: 0 limit: 15360
2: ptag[214] job res: SFMA used: 0 limit: 123
2: ptag[214] job res: DIRECT used: 0 limit: 18446744073709551615
2: ptag[214] job res: PCI-IOMMU used: 0 limit: 18446744073709551615
2: ptag[214] job res: non-VMDH used: 127 limit: 18446744073709551615
2: libfabric:gni:ep_ctrl:_gnix_mbox_allocator_destroy():613 [38214:57]
2: libfabric:gni:ep_ctrl:__destroy_slab():390 [38214:57]
2: libfabric:gni:ep_ctrl:_gnix_cm_nic_alloc():649 [38214:57] gnix_nic_alloc returned Device or resource busy
2: libfabric:gni:ep_ctrl:_gnix_ep_nic_init():2133 [38214:57] _gnix_cm_nic_alloc returned Device or resource busy
2: libfabric:gni:ep_ctrl:gnix_ep_open():2350 [38214:57] _gnix_ep_nic_init returned -16
2: libfabric:gni:ep_ctrl:_gnix_xpmem_handle_destroy():310 [38214:57]
2: libfabric:gni:ep_ctrl:__xpmem_hndl_destruct():98 [38214:57]
2: [nid00010, Rank: 2, Pthread: 169] ltu_ep_init Line: 1149, ERROR: : fi_endpoint returned 16 (Device or resource busy)
2: [nid00010, Rank: 2, Pthread: 169] rma_execute Line: 521, ERROR: : ltu_ep_init returned 16 (Device or resource busy)
2: [ 214] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x2a6b
2: [nid00010, Rank: 2, Pthread: 169] RESULT: RMA test Aborted.
2: [ 214] FMA: GNI_CdmAttach: FMA window size: 32768
2: [ 214] FMA: GNI_CdmAttach: NOPRIV_ERR masked
2: [ 214] JOB: GNI_CdmAttach: ptag = 214 inst_id = 5529665 fma_window = 0x0000000000000000 fma_ctrl = 0x0000000000000000
2: [ 214] CQ: GNI_CqCreate: entry_count: 1361 reqs: 1361 adjusted entries: 1395 alloc_count: 1396
2: [ 214] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)
2: [ 214] CQ: cq_create: #1 cq created, kern_cq_descr = 65, mode = 2, rd_index_ptr = 0x7fff66bc7ba0, queue = 0x7fff66bc5000, intr_mask = (nil)
2: [ 214] CQ: GNI_CqCreate: entry_count: 1361 reqs: 1361 adjusted entries: 1395 alloc_count: 1396
2: [ 214] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)
2: [ 214] CQ: cq_create: #1 cq created, kern_cq_descr = 1375, mode = 20, rd_index_ptr = 0x7fff66bc0000, queue = 0x7fff66bc1000, intr_mask = (nil)
2: [ 214] FLBTE: GNII_FlbteInit: FLBTE: tx_counter 0x7fff673cf008, chan 2, max_len -1, total 511
2: [ 214] CQ: GNI_CqCreate: entry_count: 2048 reqs: 2048 adjusted entries: 2559 alloc_count: 2560
2: [ 214] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)
2: [ 214] CQ: cq_create: #1 cq created, kern_cq_descr = 1376, mode = 5, rd_index_ptr = 0x7fff66bb6000, queue = 0x7fff66bb7000, intr_mask = 0x7fff66bcc1f8
2: [ 214] CQ: GNI_CqCreate: entry_count: 2048 reqs: 2048 adjusted entries: 2559 alloc_count: 2560
2: [ 214] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)
2: [ 214] CQ: cq_create: #1 cq created, kern_cq_descr = 1377, mode = 4, rd_index_ptr = 0x7fff66bad000, queue = 0x7fff66bae000, intr_mask = (nil)
2: [ 214] CQ: GNI_CqCreate: entry_count: 16384 reqs: 16384 adjusted entries: 16895 alloc_count: 16896
2: [ 214] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE) returned error - Invalid argument
2: [ 214] CQ: GNI_CqCreate: GNI_IOC_CQ_CREATE with PHYS_MEM failed trying without PHYS_MEM
2: [ 214] MR: GNI_MemRegister: Mem reg of 135168 length at addr 0x7fff66b8c000
2: [ 214] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)
2: [ 214] CQ: cq_create: #1 cq created, kern_cq_descr = 1378, mode = 1, rd_index_ptr = 0x7fff66b8b000, queue = 0x7fff66b8c000, intr_mask = 0x7fff66bcc1fc
2: [ 214] CQ: GNI_CqCreate: entry_count: 16384 reqs: 16384 adjusted entries: 16895 alloc_count: 16896
2: [ 214] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE) returned error - Invalid argument
2: [ 214] CQ: GNI_CqCreate: GNI_IOC_CQ_CREATE with PHYS_MEM failed trying without PHYS_MEM
2: [ 214] MR: GNI_MemRegister: Mem reg of 135168 length at addr 0x7fff66b6a000
2: [ 214] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)
2: [ 214] CQ: cq_create: #1 cq created, kern_cq_descr = 1379, mode = 0, rd_index_ptr = 0x7fff66b69000, queue = 0x7fff66b6a000, intr_mask = (nil)
2: libfabric:gni:ep_ctrl:__gnix_nic_tx_freelist_init():661 [38214:70]
2: libfabric:gni:ep_ctrl:_gnix_mbox_allocator_create():555 [38214:70]
2: libfabric:gni:ep_ctrl:__open_huge_page():195 [38214:70]
2: libfabric:gni:ep_ctrl:__create_slab():300 [38214:70]

[hppritcha]

@tonyzinger what system were you running this test on? These are really messages from uGNI.

[jswaro]

@tonyzinger Can you attach a test case?

[jshimek]

I set ep_cnt value based on the assumption there is no limit in libfabrics based on input from the rest of the group. With Tony seeing these errors there is actually some sort of limit in ugni or we are exposing bug in gni provider.

[tonyzinger]

@hppritcha I was doing this testing on Jupiter using the daily build of Libfabrics.

[tonyzinger]

On Jupiter, in the directory:
/home/users/ajz/ofi-cray/INSTALL/bin
I executed the following test case:
srun -l --exclusive --ntasks=3 --ntasks-per-node=1 -t 30 ./rma -P gni -g 2 -a 0x8 -C 0 -S 0x20 -B 1 -N 2 --dbg-test 4 --dbg-ltu 3 -T 64 -L /home/users/ajz/libfabric-1351 --log

This test will create 3 ranks with each rank creating 64 (-T) pthreads.
If I created 63 pthreads, this test executes successfully.

Here is the debug output for one of the pthreads, that encountered the "fi_endpoint returned 16 (Device or resource busy)" error.

ajz@jupiter libfabric-1351 $ cat RMA_gni_rank_0_pth_48_tid_23112.log
[nid00008, Rank: 0, Pthread: 48] CONFIG: environment: date = 2017-08-04 08:27:46
[nid00008, Rank: 0, Pthread: 48] CONFIG: environment: command = RMA
[nid00008, Rank: 0, Pthread: 48] CONFIG: environment: number of nodes = 3
[nid00008, Rank: 0, Pthread: 48] CONFIG: environment: number of ranks = 3
[nid00008, Rank: 0, Pthread: 48] CONFIG: environment: number of ranks per node = 1

[nid00008, Rank: 0, Pthread: 48] CONFIG: Executing on CPU: 0, requested CPU: 0

[nid00008, Rank: 0, Pthread: 48] INFO: [prov: 0] fabric(gni), prov_name: 'gni', prov version: 0x00010001, api version: 0x00010004
[nid00008, Rank: 0, Pthread: 48] INFO: [prov: 0] caps, requested: 0x0100000000002204, returned: 0x0100000000312204
[nid00008, Rank: 0, Pthread: 48] INFO: [prov: 0] mode, requested: 0x0980000000000000, returned: 0x0980000000000000
[nid00008, Rank: 0, Pthread: 48] INFO: [prov: 0] domain(/sys/class/gni/kgni0), mr_mode, requested: 1, returned: 1, cntr_prog: 1, data_prog: 1, cq_cnt: 247, ep_cnt: -1, max_ep_rx_ctx: 128, max_ep_tx_ctx: 128, mr_iov_limit: 1
[nid00008, Rank: 0, Pthread: 48] INFO: [prov: 0] endpoint: type, requested: 3, returned: 3, max_msg_size: -1, rx_ctx_cnt: 1, tx_ctx_cnt: 1
[nid00008, Rank: 0, Pthread: 48] INFO: [prov: 0] transmit: caps: 0x100000000312204, mode: 0x980000000000000, op_flags: 0x0, inject_size: 64, size: 500, iov_limit: 8, rma_iov_limit: 1
[nid00008, Rank: 0, Pthread: 48] INFO: [prov: 0] receive: caps: 0x100000000312204, mode: 0x980000000000000, op_flags: 0x0, size: 500, iov_limit: 8
[nid00008, Rank: 0, Pthread: 48] INFO: number of returned fi_info structures: 1
[nid00008, Rank: 0, Pthread: 48] INFO: Using Provider: 'gni', Fabric Name: 'gni', Domain Name: '/sys/class/gni/kgni0', Endpoint Type: 3, index: 0
[nid00008, Rank: 0, Pthread: 48] DEBUG: fabric has been successfully initialized
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: apis = 0x0000000000000008 - FI_WRITE(3)
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: groups = 0x02 - Write(1)
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: msg-flag = 0 - None
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: api-version = 0x00010004
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: comm-pattern = 0 - Round-Robin
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: data-types = 0x0000000000000020 - UINT32(5)
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: data-patterns = 0x0000000100000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: elements = 1
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: endpoint type = 3 - FI_EP_RDM
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: validate-hints = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: iterations = 1
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: logfile_dir = /home/users/ajz/libfabric-1351
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: provider = gni
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: threads = 64
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: transfers = 2
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: use-sched-affin = 1
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: log-libfabric = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: logs = 1
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: no-output = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: no-verify = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: one_rank = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: output_file_name =
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: result-format = 2 - SFT Results
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: csv = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: xml = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: yaml = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: dbg-cntr-timeout = 1200
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: dbg-cq-delay = 100000
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: dbg-cq-retry = 500
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: dbg-ltu = 3
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: dbg-reset-cntr = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: parameter: dbg-test = 4

[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: caps = 0x0100000000312204
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: mode = 0x0980000000000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: addr_format = 0x00000006
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: src_addrlen = 48
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: dest_addrlen = 48
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: src_addr = 0x00007fffc0004b10
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: dest_addr = 0x0000000000000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: fabric: name = gni
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: fabric: prov_name = gni
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: fabric: prov_version = 0x00010001
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: fabric: api_version = 0x00010004
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: name = /sys/class/gni/kgni0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: threading = 1
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: control_progress = 0x1
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: data_progress = 0x1
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: resource_mgmt = 2
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: av_type = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: mr_mode = 1
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: mr_key_size = 8
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: cq_data_size = 8
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: cq_cnt = 247
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: ep_cnt = 4294967295
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: tx_ctx_cnt = 60
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: rx_ctx_cnt = 61
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: max_ep_tx_ctx = 128
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: max_ep_rx_ctx = 128
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: max_ep_stx_ctx = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: cntr_cnt = 247
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: mr_iov_limit = 1
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: caps = 0x10000000000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: mode = 0x00000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: auth_key = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: domain: max_err_data = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: type = 3
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: protocol = 10
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: protocol_version = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: max_msg_size = 4294967295
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: msg_prefix_size = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: max_order_raw_size = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: max_order_war_size = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: max_order_waw_size = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: mem_tag_format = 0x00000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: tx_ctx_cnt = 1
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: rx_ctx_cnt = 1
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: endpoint: auth_key = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: transmit: caps = 0x100000000312204
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: transmit: mode = 0x980000000000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: transmit: op_flags = 0x00000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: transmit: msg_order = 0x00000100
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: transmit: comp_order = 0x00000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: transmit: inject_size = 64
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: transmit: size = 500
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: transmit: iov_limit = 8
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: transmit: rma_iov_limit = 1
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: receive: caps = 0x100000000312204
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: receive: mode = 0x980000000000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: receive: op_flags = 0x00000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: receive: msg_order = 0x00000100
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: receive: comp_order = 0x00000000
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: receive: total_buffered_recv = 0
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: receive: size = 500
[nid00008, Rank: 0, Pthread: 48] CONFIG: fi_info: receive: iov_limit = 8
[nid00008, Rank: 0, Pthread: 48] ERROR: ltu_ep_init Line: 1149, fi_endpoint returned 16 (Device or resource busy)
[nid00008, Rank: 0, Pthread: 48] ERROR: ltu_ep_init Line: 1263, errors have been encountered during endpoint initialization: error returned 16 (Device or resource busy)
[nid00008, Rank: 0, Pthread: 48] ERROR: rma_execute Line: 553, ltu_ep_init returned 16 (Device or resource busy)
[nid00008, Rank: 0, Pthread: 48] RESULT: RMA test Aborted.
[nid00008, Rank: 0, Pthread: 48] DEBUG: Waiting for RMA test completed, all threads to complete.
[nid00008, Rank: 0, Pthread: 48] DEBUG: fabric has been successfully closed

[hppritcha]

could you add --cpus-per-task=32 to the srun command line and see if that helps?

[tonyzinger]

Sorry, I did not see this request.

Adding the --cpus-per-task=32 did NOT help.

This test was executed on tiger.
[nid00228, Rank: 0, Pthread: 27] ERROR: ltu_ep_init Line: 1150, fi_endpoint returned 16 (Device or resource busy)
[nid00228, Rank: 0, Pthread: 27] ERROR: ltu_ep_init Line: 1270, errors have been encountered during endpoint initialization: error returned 16 (Device or resource busy)
[nid00228, Rank: 0, Pthread: 27] ERROR: rma_execute Line: 441, ltu_ep_init returned 16 (Device or resource busy)