High CPU usage of zfs_iput_taskq and slow writes #1953
Comments
|
If a kernel thread is spinning on a CPU you can usually get its stack trace with |
|
That gives me this: |
|
Do lots of the files have xattrs? If so, are you using xattr=sa? If you're not sure, you could try something like |
|
Running that shows many many xattrs.... Is there a way to nondestructively move xattrs over to system attributes? |
|
Unfortunately, the only way at the moment is to copy the files. WIth current master code, even removing all the xattrs won'd delete the hidden xattr directory. Just out of curiosity, what is the nature of those xattrs? The reason I ask is because there is an unrelated issue (forgot the number right now) on a filesystem used by netatalk and I'd love to get a handle on the types of xattr operations it's performing. A quick glance at netatalk's source code made it appear that it simply passes-through xattr operations from the client. |
|
This is a filesystem used by netatalk... Let me know if you want any further debug info. |
|
When I skimmed through the netatalk source code, I clearly didn't look closely enough. Apparently, as of version 3.1, it can store Mac Metadata in the org.netatalk.Metadata xattr. In your original report, you mentioned backing up 250 GB of data "replacing another 250 GB of data". Does this imply the backup process was deleting files on the destination as the backup was continuing? If so, there are definitely issues when it comes to deleting files with dir-style xattrs, especially when lots of files are being deleted. I've been working on various xattr-related issues and would like to know whether, after changing to xattr=sa, your system was updating or deleting xattrs regularly of whether it mainly left them alone once they were created. |
|
I'm not quite sure what was happening. Maybe there was a metadata mismatch and time machine decided to do the backup all over again, at which point I think it was attempting to overwrite all of the data. Let me know if there's any debugging you think I can enable to help you track down what you're looking for. |
|
I believe that #2408 will fix this. |
|
I agree, this was almost certainly fixed by #2408. Closing. |
I have a folder shared via netatalk to do time machine backups. I'm attempting to backup ~ 250Gb of data (replacing another 250gb of data). The backup slows to a crawl about ~30Gb in though. Top reveals that zfs_iput_taskq is at 99%. Writes to other zpools on other disks also slow to a near crawl at this point.
On an Ubuntu 13.10 system with a 3.12.3 kernel. (This is an issue I have seen before. I've usually been able to reboot once the zfs_iput_tasq is done, and then the issue goes away for a while, but it comes back.)
I've also now set xattr=sa since I've read a little bit on how this might be connected... but I assume this doesn't take effect immediately.
top:
zpool iostat 1:
zdb output:
module info:
zfs
spl:
The text was updated successfully, but these errors were encountered: