zfs_iput_taskq spinning again #2128
Comments
|
Interesting. Thanks for filing this detailed debugging for us! |
HardwareCPU: AMD E-350 Processor I think my issue is related or the same as this issue. The pool itself is only ~3 days old. Snapshots are enabled using zfs-auto-snapshot zpool iostatAnd I saw once the following kernel log message. Since the scrub was so slow I wanted to stop the scrub and export/import the pool. Most of the time I see the following stack for the process zfs_iput_taskq zpool statusUntil now I saw the issue 2 times. |
|
Hi, Similar problem here. Running rdiff-backup of another host. Perhaps 100GB total. Running on a Supermicro server w/ a single dual core Opteron CPU (details below) and 8GB of RAM. 16 2TB drives in two ZFS pools. Ubuntu 12.04.4 LTS (64bit) OUTPUT OF TOP: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND Strangely, htop does not show zfs_input_taskq VERSIONS OF ZFS, et. Al. Installed via package manager: CPU INFO FROM PROC Second core would repeat above. ZPOOL STATUS errors: No known data errors pool: fergus errors: No known data errors Is there other info I should forward to you? Thanks for all your hard work. |
|
This looks like a duplicate of #1469. It's a known issue which hasn't yet been resolved, using SA based xattrs should minimize the issue. |
|
Interesting. I'll have to wait until Monday to check out my system. I thought xattrs were not in use but I don't recall explicitly disabling them either. |
|
They're enabled by default. If you know they aren't needed explicitly disabling them should prevent the issue as well. |
# zpool history -i 2014-02-04.13:29:50 [txg:5] create pool version 5000; software version 5000/5; uts H264 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 ... (receive snapshot remotely) ... (make clones) ... 2014-02-04.13:37:00 [txg:385] set netlab (21) xattr=0 ... While the filesystem may have xattrs from the receive operation they do have the xattr mount option disabled. Since setting xattr=off was the last thing to happen maybe that makes sense? (If existing xattrs exist they would NOT have been 'sa' based.) (History requires -i due to the pool being affected by an earlier bug) |
This reverts commit 7973e46. That had been intended to workaround a deadlock issue involving zfs_zget(), which was fixed by 6f9548c. The workaround had the side effect of causing zfs_zinactive() to cause excessive cpu utilization in zfs_iput_taskq by queuing an iteration of all objects in a dataset on every unlink on a directory that had extended attributes. That resulted in many issue reports about iput_taskq spinning. Since the original rationale for the change is no longer valid, we can safely revert it to resolve all of those issue reports. Conflicts: module/zfs/zfs_dir.c Closes: openzfs#457 openzfs#2058 openzfs#2128 openzfs#2240
This reverts commit 7973e46. That had been intended to workaround a deadlock issue involving zfs_zget(), which was fixed by 6f9548c. The workaround had the side effect of causing zfs_zinactive() to cause excessive cpu utilization in zfs_iput_taskq by queuing an iteration of all objects in a dataset on every unlink on a directory that had extended attributes. That resulted in many issue reports about iput_taskq spinning. Since the original rationale for the change is no longer valid, we can safely revert it to resolve all of those issue reports. Conflicts: module/zfs/zfs_dir.c Closes: openzfs#457 openzfs#2058 openzfs#2128 openzfs#2240
This reverts commit 7973e46 which brings the basic flow of the code back inline with the other ZFS implementations. This was possible due to the work done in these in previous commits. e89260a Directory xattr znodes hold a reference on their parent 26cb948 Avoid 128K kmem allocations in mzap_upgrade() 4acaaf7 Add zfs_iput_async() interface Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#2408 Issue openzfs#457 Issue openzfs#2058 Issue openzfs#2128 Issue openzfs#2240
This reverts commit 7973e46 which brings the basic flow of the code back in line with the other ZFS implementations. This was possible due to the following related changes. e89260a Directory xattr znodes hold a reference on their parent 6f9548c Fix deadlock in zfs_zget() 26cb948 Avoid 128K kmem allocations in mzap_upgrade() 4acaaf7 Add zfs_iput_async() interface Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#457 Issue openzfs#2058 Issue openzfs#2128 Issue openzfs#2240
This reverts commit 7973e46 which brings the basic flow of the code back in line with the other ZFS implementations. This was possible due to the following related changes. e89260a Directory xattr znodes hold a reference on their parent 6f9548c Fix deadlock in zfs_zget() ca043ca Add zfs_iput_async() interface dd2a794 Avoid 128K kmem allocations in mzap_upgrade() Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#457 Issue openzfs#2058 Issue openzfs#2128 Issue openzfs#2240
This reverts commit 7973e46 which brings the basic flow of the code back in line with the other ZFS implementations. This was possible due to the following related changes. e89260a Directory xattr znodes hold a reference on their parent 6f9548c Fix deadlock in zfs_zget() ca043ca Add zfs_iput_async() interface dd2a794 Avoid 128K kmem allocations in mzap_upgrade() Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#457 Issue openzfs#2058 Issue openzfs#2128 Issue openzfs#2240
This reverts commit 7973e46 which brings the basic flow of the code back in line with the other ZFS implementations. This was possible due to the following related changes. e89260a Directory xattr znodes hold a reference on their parent 6f9548c Fix deadlock in zfs_zget() ca043ca Add zfs_iput_async() interface dd2a794 Avoid 128K kmem allocations in mzap_upgrade() Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#457 Issue openzfs#2058 Issue openzfs#2128 Issue openzfs#2240
This reverts commit 7973e46 which brings the basic flow of the code back in line with the other ZFS implementations. This was possible due to the following related changes. e89260a Directory xattr znodes hold a reference on their parent 6f9548c Fix deadlock in zfs_zget() 0a50679 Add zfs_iput_async() interface 4dd1893 Avoid 128K kmem allocations in mzap_upgrade() Signed-off-by: Brian Behlendorf <[email protected]> Signed-off-by: Richard Yao <[email protected]> Closes openzfs#457 Closes openzfs#2058 Closes openzfs#2128 Closes openzfs#2240
This reverts commit 7973e46 which brings the basic flow of the code back in line with the other ZFS implementations. This was possible due to the following related changes. e89260a Directory xattr znodes hold a reference on their parent 6f9548c Fix deadlock in zfs_zget() ca043ca Add zfs_iput_async() interface dd2a794 Avoid 128K kmem allocations in mzap_upgrade() Signed-off-by: Brian Behlendorf <[email protected]> Signed-off-by: Richard Yao <[email protected]> Closes openzfs#457 Closes openzfs#2058 Closes openzfs#2128 Closes openzfs#2240
Hardware
CPU: Intel(R) Core(TM) i5-3330 CPU @ 3.00GHz
RAM: 4 GB
OS: CentOS 6
Kernel: Linux H264 2.6.32-431.3.1.el6.x86_64 # 1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
SPL: spl-0.6.2-23-g4c99541
ZFS: zfs-0.6.2-171-g2e7b765
ZFS stats
# zpool status pool: netlab state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM netlab ONLINE 0 0 0 VolGroup00-zfs ONLINE 0 0 0 errors: No known data errors # zfs list -o name,size,used,mounted NAME REFER USED MOUNTED netlab 32.5K 974M yes netlab/lxc 30K 259M no netlab/lxc/clone1 692M 4.43M yes netlab/lxc/clone10 695M 7.39M yes netlab/lxc/clone100 688M 34K yes netlab/lxc/clone101 688M 809K yes netlab/lxc/clone102 688M 809K yes netlab/lxc/clone103 688M 804K yes netlab/lxc/clone104 688M 804K yes netlab/lxc/clone105 688M 802K yes ... netlab/lxc/clone2 695M 8.15M yes .... netlab/lxc/clone3 696M 8.47M yes netlab/lxc/clone30 688M 1.02M yes netlab/lxc/clone300 688M 34K yes netlab/lxc/clone31 688M 830K yes netlab/lxc/clone32 688M 830K yes netlab/lxc/clone33 688M 830K yes netlab/lxc/clone34 688M 829K yes netlab/lxc/clone35 688M 832K yes netlab/lxc/clone36 688M 834K yes netlab/lxc/clone37 688M 830K yes netlab/lxc/clone38 688M 833K yes netlab/lxc/clone39 688M 833K yes netlab/lxc/clone4 694M 6.30M yes netlab/lxc/clone40 688M 834K yes netlab/lxc/clone41 688M 833K yes netlab/lxc/clone42 688M 832K yes netlab/lxc/clone43 688M 833K yes netlab/lxc/clone44 688M 832K yes netlab/lxc/clone45 688M 833K yes netlab/lxc/clone46 688M 833K yes netlab/lxc/clone47 688M 834K yes netlab/lxc/clone48 688M 834K yes netlab/lxc/clone49 688M 834K yes netlab/lxc/clone5 695M 7.21M yes netlab/lxc/clone50 688M 831K yes netlab/lxc/clone51 688M 804K yes netlab/lxc/clone52 688M 806K yes netlab/lxc/clone53 688M 805K yes netlab/lxc/clone54 688M 803K yes netlab/lxc/clone55 688M 804K yes netlab/lxc/clone56 688M 814K yes netlab/lxc/clone57 688M 818K yes netlab/lxc/clone58 688M 817K yes netlab/lxc/clone59 688M 806K yes netlab/lxc/clone6 695M 7.55M yes netlab/lxc/clone60 688M 816K yes netlab/lxc/clone61 688M 818K yes netlab/lxc/clone62 688M 817K yes netlab/lxc/clone63 688M 814K yes netlab/lxc/clone64 688M 815K yes netlab/lxc/clone65 688M 817K yes netlab/lxc/clone66 688M 818K yes netlab/lxc/clone67 688M 802K yes netlab/lxc/clone68 688M 818K yes netlab/lxc/clone69 688M 816K yes netlab/lxc/clone7 695M 7.74M yes netlab/lxc/clone70 688M 813K yes netlab/lxc/clone71 688M 816K yes netlab/lxc/clone72 688M 813K yes netlab/lxc/clone73 688M 813K yes netlab/lxc/clone74 688M 817K yes netlab/lxc/clone75 688M 814K yes netlab/lxc/clone76 688M 813K yes netlab/lxc/clone77 688M 816K yes netlab/lxc/clone78 688M 816K yes netlab/lxc/clone79 688M 819K yes netlab/lxc/clone8 754M 66.4M yes netlab/lxc/clone80 688M 813K yes netlab/lxc/clone81 688M 813K yes netlab/lxc/clone82 688M 814K yes netlab/lxc/clone83 688M 816K yes netlab/lxc/clone84 688M 816K yes netlab/lxc/clone85 688M 814K yes netlab/lxc/clone86 688M 814K yes netlab/lxc/clone87 688M 814K yes netlab/lxc/clone88 688M 816K yes netlab/lxc/clone89 688M 816K yes netlab/lxc/clone9 697M 9.99M yes netlab/lxc/clone90 688M 814K yes netlab/lxc/clone91 688M 820K yes netlab/lxc/clone92 688M 813K yes netlab/lxc/clone93 688M 813K yes netlab/lxc/clone94 688M 816K yes netlab/lxc/clone95 688M 814K yes netlab/lxc/clone96 688M 820K yes netlab/lxc/clone97 688M 813K yes netlab/lxc/clone98 688M 814K yes netlab/lxc/clone99 688M 813K yes netlab/template 688M 688M yesLVM partition used for ZFS, total 294 filesystems (obviously I removed a lot for the sake of brevity), all mounted. Only ~10 of them are in use for LXC, the rest are sitting idle. Each container runs a very minimal CentOS install - sshd, rsyslog, 6 ttys, and a bunch of network sessions up.
Task info
top - 12:51:54 up 9 days, 23:25, 2 users, load average: 1.02, 1.13, 1.54 Tasks: 1947 total, 2 running, 1945 sleeping, 0 stopped, 0 zombie Cpu(s): 3.8%us, 34.6%sy, 0.0%ni, 61.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3833148k total, 2986820k used, 846328k free, 39148k buffers Swap: 2097144k total, 51868k used, 2045276k free, 506324k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11845 root 0 -20 0 0 0 R 94.0 0.0 1308:01 zfs_iput_taskq/ 4927 root 20 0 16520 2712 948 R 62.6 0.1 0:00.19 top 1 root 20 0 19364 528 316 S 0.0 0.0 0:01.36 initStack traces
Interesting supplemental information
# arcstat.py 2 time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c 12:22:15 1.1K 0 0 0 0 0 0 0 0 309M 309M 12:22:17 276K 0 0 0 0 0 0 0 0 309M 309M 12:22:19 276K 0 0 0 0 0 0 0 0 309M 309M 12:22:21 277K 0 0 0 0 0 0 0 0 309M 309M 12:22:23 275K 0 0 0 0 0 0 0 0 309M 309M 12:22:25 277K 0 0 0 0 0 0 0 0 309M 309M 12:22:27 276K 0 0 0 0 0 0 0 0 309M 309M 12:22:29 277K 0 0 0 0 0 0 0 0 309M 309M ... 12:50:44 277K 0 0 0 0 0 0 0 0 309M 309M 12:50:46 281K 0 0 0 0 0 0 0 0 309M 309M 12:50:48 280K 0 0 0 0 0 0 0 0 309M 309M 12:50:50 281K 0 0 0 0 0 0 0 0 309M 309M 12:50:52 282K 0 0 0 0 0 0 0 0 309M 309M 12:50:54 281K 0 0 0 0 0 0 0 0 309M 309M 12:50:56 281K 0 0 0 0 0 0 0 0 309M 309M... which is a lot, but IO on the system is pretty much settled.
While I was collecting the above information the thread finally went back to sleep. My ability to access the pool did not appear to be negatively impacted, but this system only uses its disk casually.
The text was updated successfully, but these errors were encountered: