Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ZFS send sometimes increases sender's load average #3518

Closed
galindro opened this issue Jun 23, 2015 · 9 comments
Closed

ZFS send sometimes increases sender's load average #3518

galindro opened this issue Jun 23, 2015 · 9 comments

Comments

@galindro
Copy link

It is the second time that one zfs send operation that my script performs every night (starting at 01:00 am) stuck and the sender's load average go to incredible value: between 800 and 1000.

My script exec a sync of snapshots between two machines (I'll call them sender and receiver). It is executed on sender. The script gets the last snapshot from the receiver and compares it with the last sender's snapshot and them execute zfs send to send the difference. After 1 hour and 28 minutes of sync, the zfs send command hangs in this point (219M):

Jun 23 02:28:32 imap zfs_snapshot_migration.sh: 02:28:32    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:33 imap zfs_snapshot_migration.sh: 02:28:33    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:34 imap zfs_snapshot_migration.sh: 02:28:34    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:35 imap zfs_snapshot_migration.sh: 02:28:35    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:36 imap zfs_snapshot_migration.sh: 02:28:36    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:37 imap zfs_snapshot_migration.sh: 02:28:37    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:38 imap zfs_snapshot_migration.sh: 02:28:38    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:39 imap zfs_snapshot_migration.sh: 02:28:39    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:40 imap zfs_snapshot_migration.sh: 02:28:40    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:41 imap zfs_snapshot_migration.sh: 02:28:41    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:42 imap zfs_snapshot_migration.sh: 02:28:42    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:43 imap zfs_snapshot_migration.sh: 02:28:43    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:44 imap zfs_snapshot_migration.sh: 02:28:44    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:45 imap zfs_snapshot_migration.sh: 02:28:45    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:46 imap zfs_snapshot_migration.sh: 02:28:46    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:47 imap zfs_snapshot_migration.sh: 02:28:47    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:48 imap zfs_snapshot_migration.sh: 02:28:48    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:49 imap zfs_snapshot_migration.sh: 02:28:49    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:50 imap zfs_snapshot_migration.sh: 02:28:50    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:28:51 imap zfs_snapshot_migration.sh: 02:28:51    219M   datastore/mail@hourly_Mon_20150622_1300
.
.
Jun 23 02:53:07 imap zfs_snapshot_migration.sh: 02:53:07    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:53:08 imap zfs_snapshot_migration.sh: 02:53:08    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:53:09 imap zfs_snapshot_migration.sh: 02:53:09    219M   datastore/mail@hourly_Mon_20150622_1300
Jun 23 02:53:10 imap zfs_snapshot_migration.sh: 02:53:10    219M   datastore/mail@hourly_Mon_20150622_1300

After the last message above, none was logged in syslog. Is there any debug information that can I attach to the executed command to see what is happening?

The script is listed bellow:

#!/bin/bash

# binary search function
fn_binary_search() {
    SEARCH_TYPE="${1}"
    shift
    SEARCHED_ITEM=${1}
    shift
    ARRAY=("${@}")

    LOW_INDEX=0
    HEIGH_INDEX=$((${#ARRAY[@]}-1))

    while [ $LOW_INDEX -le $HEIGH_INDEX ]; do

        MIDDLE_INDEX=$(($LOW_INDEX+($HEIGH_INDEX-$LOW_INDEX)/2))
        MIDDLE_ELEMENT=`fn_parse_snap_date ${ARRAY[$MIDDLE_INDEX]}`

        if [ $MIDDLE_ELEMENT -eq $SEARCHED_ITEM ]; then
            if [ "$SEARCH_TYPE" == "next" ]; then
                NEXT_MIDDLE_INDEX=$(($MIDDLE_INDEX+1))
                echo $NEXT_MIDDLE_INDEX
            else
                echo $MIDDLE_INDEX
            fi

            return

        elif [ $SEARCHED_ITEM -lt $MIDDLE_ELEMENT ]; then
            HEIGH_INDEX=$(($MIDDLE_INDEX-1))
        else
            LOW_INDEX=$(($MIDDLE_INDEX+1))
        fi

    done
}

# parse snapshot date into YYYYmmddHHMM
fn_parse_snap_date() {
    sed -r 's/^(.*[a-z]_[0-1]*)([0-9]{8})_([0-9]{4})(.*$)/\2\3/' <<< $1
    return
}

# Remote host
HOST_DST='root@destiny'
# Local dataset name
DATASET_SOURCE='datastore/mail'
# Remote dataset name
DATASET_DESTINY='ds/mail/mail'
DATASTORE_DESTINY='ds/mail'

# Not run while another process is running
ANOTHER_PROC=`ps -ef | grep 'zfs recv' | grep ssh | wc -l`
if [ "$ANOTHER_PROC" != "0" ];then
    echo "Script already running. PID: $ANOTHER_PROC" 
    exit 1
fi

# Take the snapshot list from local server
mapfile -t ZFS_LIST_LOCAL < <(/sbin/zfs list -t snap | grep "$DATASET_SOURCE@" | sed -r "s/^(.*@)(.*[a-z]_[0-1]*)([0-9]{8})_([0-9]{4})(.*$)/\2\3_\4/")

# Take the snapshot list from remote server
mapfile -t ZFS_LIST_REMOTE < <(ssh ${HOST_DST} '/sbin/zfs list -t snap' | grep "$DATASET_DESTINY@" | sed -r "s/^(.*@)(.*[a-z]_[0-1]*)([0-9]{8})_([0-9]{4})(.*$)/\2\3_\4/")

# Get the last local snapshot
LAST_LOCAL="${ZFS_LIST_LOCAL[$((${#ZFS_LIST_LOCAL[@]} - 1))]}"
LAST_LOCAL_DATE=`fn_parse_snap_date ${ZFS_LIST_LOCAL[$((${#ZFS_LIST_LOCAL[@]} - 1))]}`

# Get the last remote snapshot
LAST_REMOTE=${ZFS_LIST_REMOTE[$((${#ZFS_LIST_REMOTE[@]} - 1))]}
LAST_REMOTE_DATE=`fn_parse_snap_date ${ZFS_LIST_REMOTE[$((${#ZFS_LIST_REMOTE[@]} - 1))]}`

# Check if Snapshots are synced
if [ $LAST_LOCAL_DATE -eq $LAST_REMOTE_DATE ]; then 
    echo "Snapshots synced - Last snapshot: ${LAST_LOCAL_FULL_NAME}"
    exit 0

# Check if Snapshots are synced
elif [ $LAST_LOCAL_DATE -lt $LAST_REMOTE_DATE ]; then 
    echo "Error! Remote host has a newer last snapshot" 
    echo "Last Local: ${LAST_LOCAL}"
    echo "Last Remote: ${LAST_REMOTE}"
    exit 1

else
    # Try to find the index of LAST_REMOTE snapshot value in ZFS_LIST_LOCAL array
    LAST_REMOTE_INDEX_IN_LOCAL=`fn_binary_search " " "${LAST_REMOTE_DATE}" "${ZFS_LIST_LOCAL[@]}"`

    # If LAST_REMOTE snapshot doesn't exists in ZFS_LIST_LOCAL
    if [ "${LAST_REMOTE_INDEX_IN_LOCAL}" == "" ]; then

        # Insert the last remote snapshot in local snapshot array.
        ZFS_LIST_LOCAL+=(${LAST_REMOTE})

        # Sort the local snapshot list array
        readarray -t ZFS_LIST_LOCAL < <(printf '%s\0' "${ZFS_LIST_LOCAL[@]}" | sort -z -k3 -k4 -t_ -n | xargs -0n1 | uniq)

        # Do a binary search to get the next snapshot index of local snapshot array 
        LAST_REMOTE_INDEX_IN_LOCAL=`fn_binary_search " " "${LAST_REMOTE_DATE}" "${ZFS_LIST_LOCAL[@]}" `

        SNAP_FROM="${ZFS_LIST_LOCAL[LAST_REMOTE_INDEX_IN_LOCAL-1]}"

    else
        SNAP_FROM="${ZFS_LIST_LOCAL[$((LAST_REMOTE_INDEX_IN_LOCAL))]}"
    fi

    # Print send statistics and exit
    if [ "${1}" == "dryrun" ]; then
        /sbin/zfs send -nvI $DATASET_SOURCE@$SNAP_FROM $DATASET_SOURCE@$LAST_LOCAL
        exit 0
    fi

    # Send snapshot
    /sbin/zfs send -veI $DATASET_SOURCE@$SNAP_FROM $DATASET_SOURCE@$LAST_LOCAL | ssh $HOST_DST zfs recv -vFd $DATASTORE_DESTINY

    # Get return code of previous command
    RETURN_CODE=$?
fi

exit $RETURN_CODE
@dweeezil
Copy link
Contributor

@galindro Does the sending system have 8e70975 applied? Some distros (Fedora) don't define CONFIG_DETECT_HUNG_TASK in their kernels so if a process blocks for a long time, you'll not see any kernel messages. I suspect you'll see a number of blocked processes ("D" state in ps output).

@galindro
Copy link
Author

Does the sending system have 8e70975 applied?

I think no @dweeezil , because I've installed ZFS from apt repository (http://ppa.launchpad.net/zfs-native/stable/ubuntu) and this change was authored 14 days ago by @behlendorf .

I suspect you'll see a number of blocked processes ("D" state in ps output).

I can't confirm this because I can't log into the VM because it has a load above 800. To recover the system, I need to force a reboot through VMware.

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.5 LTS
Release:    12.04
Codename:   precise
# dpkg -l |grep zfs
ii  dkms                                           2.2.0.3-1ubuntu3.2+zfs6~precise     Dynamic Kernel Module Support Framework
rc  libzfs1                                        0.6.2-1~precise                     Native ZFS filesystem library for Linux
ii  libzfs2                                        0.6.4.1-1~precise                   Native OpenZFS filesystem library for Linux
ii  mountall                                       2.36.4-zfs2                         filesystem mounting tool
ii  ubuntu-zfs                                     8~precise                           Native ZFS filesystem metapackage for Ubuntu.
ii  zfs-dkms                                       0.6.4.1-1~precise                   Native OpenZFS filesystem kernel modules for Linux
ii  zfs-doc                                        0.6.4.1-1~precise                   Native OpenZFS filesystem documentation and examples.
ii  zfsutils                                       0.6.4.1-1~precise                   Native OpenZFS management utilities for Linux

When this patch will be available for download through apt?

@galindro
Copy link
Author

@dweeezil , I've said before that none was logged in syslog, but this is not true. My fault, sorry...

Searching in logs, I've found the bellow messages. I think this is the same case as pointed by #3450 and #3402 , all referenced by 8e70975.

Jun 23 02:30:24 imap kernel: [1189298.208629] INFO: task kswapd0:51 blocked for more than 120 seconds.
Jun 23 02:30:24 imap kernel: [1189298.208949] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 02:30:24 imap kernel: [1189298.209029] kswapd0         D ffffffff81806200     0    51      2 0x00000000
Jun 23 02:30:24 imap kernel: [1189298.209034]  ffff880421b9f7e0 0000000000000046 ffff880425648000 ffff88043fdd2800
Jun 23 02:30:24 imap kernel: [1189298.209038]  ffff880421b9ffd8 ffff880421b9ffd8 ffff880421b9ffd8 0000000000012800
Jun 23 02:30:24 imap kernel: [1189298.209041]  ffff880425649700 ffff880421ba2e00 ffff880421b9f820 ffffffffa0251210
Jun 23 02:30:24 imap kernel: [1189298.209045] Call Trace:
Jun 23 02:30:24 imap kernel: [1189298.209068]  [<ffffffff81662d6f>] schedule+0x3f/0x60
Jun 23 02:30:24 imap kernel: [1189298.209072]  [<ffffffff81663b67>] __mutex_lock_slowpath+0xd7/0x150
Jun 23 02:30:24 imap kernel: [1189298.209076]  [<ffffffff8166377a>] mutex_lock+0x2a/0x50
Jun 23 02:30:24 imap kernel: [1189298.209129]  [<ffffffffa0123547>] arc_buf_remove_ref+0xa7/0x140 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209144]  [<ffffffffa0129b67>] dbuf_rele_and_unlock+0x137/0x3f0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209151]  [<ffffffff8116524a>] ? discard_slab+0x3a/0x50
Jun 23 02:30:24 imap kernel: [1189298.209155]  [<ffffffff811657ff>] ? unfreeze_partials+0x24f/0x2b0
Jun 23 02:30:24 imap kernel: [1189298.209159]  [<ffffffff81121a45>] ? free_one_page+0x155/0x330
Jun 23 02:30:24 imap kernel: [1189298.209173]  [<ffffffffa012a2af>] dbuf_rele+0x3f/0x50 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209194]  [<ffffffffa0145da0>] dnode_rele_and_unlock+0xa0/0xb0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209209]  [<ffffffffa0129da8>] dbuf_rele_and_unlock+0x378/0x3f0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209212]  [<ffffffff81165964>] ? kmem_cache_free+0x104/0x110
Jun 23 02:30:24 imap kernel: [1189298.209215]  [<ffffffff8166376d>] ? mutex_lock+0x1d/0x50
Jun 23 02:30:24 imap kernel: [1189298.209219]  [<ffffffff81664cce>] ? _raw_spin_lock+0xe/0x20
Jun 23 02:30:24 imap kernel: [1189298.209233]  [<ffffffffa012a2af>] dbuf_rele+0x3f/0x50 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209248]  [<ffffffffa012a4ae>] dmu_buf_rele+0xe/0x10 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209289]  [<ffffffffa0169c93>] sa_handle_destroy+0x73/0xc0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209320]  [<ffffffffa01c051d>] zfs_znode_dmu_fini+0x1d/0x30 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209345]  [<ffffffffa01c3bdd>] zfs_zinactive+0xad/0x190 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209371]  [<ffffffffa01bde0b>] zfs_inactive+0x7b/0x260 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209398]  [<ffffffffa01d525e>] zpl_evict_inode+0x4e/0x70 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209403]  [<ffffffff81196538>] evict+0xb8/0x1c0
Jun 23 02:30:24 imap kernel: [1189298.209405]  [<ffffffff81196c21>] dispose_list+0x41/0x50
Jun 23 02:30:24 imap kernel: [1189298.209408]  [<ffffffff81664c9f>] ? _raw_spin_trylock+0xf/0x30
Jun 23 02:30:24 imap kernel: [1189298.209411]  [<ffffffff81197668>] prune_icache_sb+0x168/0x330
Jun 23 02:30:24 imap kernel: [1189298.209416]  [<ffffffff8117ecd8>] prune_super+0xf8/0x1b0
Jun 23 02:30:24 imap kernel: [1189298.209421]  [<ffffffff8112c764>] shrink_slab+0x154/0x300
Jun 23 02:30:24 imap kernel: [1189298.209424]  [<ffffffff8112fa5a>] balance_pgdat+0x50a/0x6d0
Jun 23 02:30:24 imap kernel: [1189298.209427]  [<ffffffff8112fd49>] kswapd+0x129/0x230
Jun 23 02:30:24 imap kernel: [1189298.209429]  [<ffffffff8112fc20>] ? balance_pgdat+0x6d0/0x6d0
Jun 23 02:30:24 imap kernel: [1189298.209434]  [<ffffffff8108b9dc>] kthread+0x8c/0xa0
Jun 23 02:30:24 imap kernel: [1189298.209437]  [<ffffffff8166f574>] kernel_thread_helper+0x4/0x10
Jun 23 02:30:24 imap kernel: [1189298.209440]  [<ffffffff8108b950>] ? flush_kthread_worker+0xa0/0xa0
Jun 23 02:30:24 imap kernel: [1189298.209442]  [<ffffffff8166f570>] ? gs_change+0x13/0x13
Jun 23 02:30:24 imap kernel: [1189298.209451] INFO: task spl_system_task:2406 blocked for more than 120 seconds.
Jun 23 02:30:24 imap kernel: [1189298.209544] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 02:30:24 imap kernel: [1189298.209613] spl_system_task D ffffffff81806200     0  2406      2 0x00000000
Jun 23 02:30:24 imap kernel: [1189298.209616]  ffff88042363a9c0 0000000000000046 ffff88004c93dc00 0000000000000013
Jun 23 02:30:24 imap kernel: [1189298.209619]  ffff88042363bfd8 ffff88042363bfd8 ffff88042363bfd8 0000000000012800
Jun 23 02:30:24 imap kernel: [1189298.209623]  ffff880425589700 ffff880420f08000 ffff880420f08000 ffffffffa0251210
Jun 23 02:30:24 imap kernel: [1189298.209626] Call Trace:
Jun 23 02:30:24 imap kernel: [1189298.209630]  [<ffffffff81662d6f>] schedule+0x3f/0x60
Jun 23 02:30:24 imap kernel: [1189298.209633]  [<ffffffff81663b67>] __mutex_lock_slowpath+0xd7/0x150
Jun 23 02:30:24 imap kernel: [1189298.209636]  [<ffffffff8166377a>] mutex_lock+0x2a/0x50
Jun 23 02:30:24 imap kernel: [1189298.209650]  [<ffffffffa0123547>] arc_buf_remove_ref+0xa7/0x140 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209665]  [<ffffffffa0129b67>] dbuf_rele_and_unlock+0x137/0x3f0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209668]  [<ffffffff81664cce>] ? _raw_spin_lock+0xe/0x20
Jun 23 02:30:24 imap kernel: [1189298.209670]  [<ffffffff81664cce>] ? _raw_spin_lock+0xe/0x20
Jun 23 02:30:24 imap kernel: [1189298.209673]  [<ffffffff811657ce>] ? unfreeze_partials+0x21e/0x2b0
Jun 23 02:30:24 imap kernel: [1189298.209688]  [<ffffffffa012a2af>] dbuf_rele+0x3f/0x50 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209707]  [<ffffffffa0145da0>] dnode_rele_and_unlock+0xa0/0xb0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209721]  [<ffffffffa0129da8>] dbuf_rele_and_unlock+0x378/0x3f0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209725]  [<ffffffff81165964>] ? kmem_cache_free+0x104/0x110
Jun 23 02:30:24 imap kernel: [1189298.209727]  [<ffffffff8166376d>] ? mutex_lock+0x1d/0x50
Jun 23 02:30:24 imap kernel: [1189298.209730]  [<ffffffff81664cce>] ? _raw_spin_lock+0xe/0x20
Jun 23 02:30:24 imap kernel: [1189298.209744]  [<ffffffffa012a2af>] dbuf_rele+0x3f/0x50 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209758]  [<ffffffffa012a4ae>] dmu_buf_rele+0xe/0x10 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209781]  [<ffffffffa0169c93>] sa_handle_destroy+0x73/0xc0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209806]  [<ffffffffa01c051d>] zfs_znode_dmu_fini+0x1d/0x30 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209832]  [<ffffffffa01c3bdd>] zfs_zinactive+0xad/0x190 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209857]  [<ffffffffa01bde0b>] zfs_inactive+0x7b/0x260 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209882]  [<ffffffffa01d525e>] zpl_evict_inode+0x4e/0x70 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209885]  [<ffffffff81196538>] evict+0xb8/0x1c0
Jun 23 02:30:24 imap kernel: [1189298.209888]  [<ffffffff81196c21>] dispose_list+0x41/0x50
Jun 23 02:30:24 imap kernel: [1189298.209891]  [<ffffffff81664c9f>] ? _raw_spin_trylock+0xf/0x30
Jun 23 02:30:24 imap kernel: [1189298.209893]  [<ffffffff81197668>] prune_icache_sb+0x168/0x330
Jun 23 02:30:24 imap kernel: [1189298.209896]  [<ffffffff8117ecd8>] prune_super+0xf8/0x1b0
Jun 23 02:30:24 imap kernel: [1189298.209899]  [<ffffffff8112c764>] shrink_slab+0x154/0x300
Jun 23 02:30:24 imap kernel: [1189298.209902]  [<ffffffff8115f310>] ? compact_zone.part.14+0x1c0/0x260
Jun 23 02:30:24 imap kernel: [1189298.209905]  [<ffffffff8112eff8>] do_try_to_free_pages+0x268/0x2d0
Jun 23 02:30:24 imap kernel: [1189298.209909]  [<ffffffff81137449>] ? __mod_zone_page_state+0x49/0x50
Jun 23 02:30:24 imap kernel: [1189298.209912]  [<ffffffff8112f33b>] try_to_free_pages+0xab/0x170
Jun 23 02:30:24 imap kernel: [1189298.209916]  [<ffffffff8112358e>] __alloc_pages_nodemask+0x59e/0x8f0
Jun 23 02:30:24 imap kernel: [1189298.209923]  [<ffffffffa0049121>] ? mptscsih_qcmd+0x201/0x340 [mptscsih]
Jun 23 02:30:24 imap kernel: [1189298.209926]  [<ffffffff8115b1a6>] alloc_pages_current+0xb6/0x120
Jun 23 02:30:24 imap kernel: [1189298.209929]  [<ffffffff811642dd>] allocate_slab+0x13d/0x1a0
Jun 23 02:30:24 imap kernel: [1189298.209932]  [<ffffffff81164370>] new_slab+0x30/0x180
Jun 23 02:30:24 imap kernel: [1189298.209936]  [<ffffffff816511c6>] __slab_alloc+0x165/0x269
Jun 23 02:30:24 imap kernel: [1189298.209945]  [<ffffffffa00c460f>] ? spl_kmem_cache_alloc+0x7f/0x7f0 [spl]
Jun 23 02:30:24 imap kernel: [1189298.209950]  [<ffffffffa00c460f>] ? spl_kmem_cache_alloc+0x7f/0x7f0 [spl]
Jun 23 02:30:24 imap kernel: [1189298.209953]  [<ffffffff81167c36>] kmem_cache_alloc+0x136/0x140
Jun 23 02:30:24 imap kernel: [1189298.209958]  [<ffffffffa00c460f>] spl_kmem_cache_alloc+0x7f/0x7f0 [spl]
Jun 23 02:30:24 imap kernel: [1189298.209983]  [<ffffffffa01ca423>] zio_buf_alloc+0x23/0x30 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.209996]  [<ffffffffa011fded>] arc_get_data_buf.isra.23+0x48d/0x4e0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210009]  [<ffffffffa0123ad7>] arc_read+0x4f7/0x9e0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210016]  [<ffffffffa00c9fc3>] ? cv_wait_common+0xf3/0x130 [spl]
Jun 23 02:30:24 imap kernel: [1189298.210035]  [<ffffffffa013f6e8>] traverse_prefetcher+0x158/0x170 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210047]  [<ffffffffa011e3f7>] ? add_reference.isra.12+0x137/0x170 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210065]  [<ffffffffa013fb26>] traverse_visitbp+0x426/0x840 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210070]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:30:24 imap kernel: [1189298.210075]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:30:24 imap kernel: [1189298.210079]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:30:24 imap kernel: [1189298.210098]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210116]  [<ffffffffa0140551>] traverse_dnode+0x71/0xd0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210134]  [<ffffffffa013fdc6>] traverse_visitbp+0x6c6/0x840 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210153]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210171]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210190]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210208]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210226]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210245]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210263]  [<ffffffffa0140551>] traverse_dnode+0x71/0xd0 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210281]  [<ffffffffa013fe56>] traverse_visitbp+0x756/0x840 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210286]  [<ffffffff810136a9>] ? __switch_to+0xb9/0x370
Jun 23 02:30:24 imap kernel: [1189298.210305]  [<ffffffffa014063b>] traverse_prefetch_thread+0x8b/0x100 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210323]  [<ffffffffa013f590>] ? prefetch_needed.isra.3+0x50/0x50 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210328]  [<ffffffffa00c6466>] taskq_thread+0x1b6/0x3e0 [spl]
Jun 23 02:30:24 imap kernel: [1189298.210334]  [<ffffffff81060b00>] ? try_to_wake_up+0x200/0x200
Jun 23 02:30:24 imap kernel: [1189298.210339]  [<ffffffffa00c62b0>] ? task_done+0xb0/0xb0 [spl]
Jun 23 02:30:24 imap kernel: [1189298.210342]  [<ffffffff8108b9dc>] kthread+0x8c/0xa0
Jun 23 02:30:24 imap kernel: [1189298.210344]  [<ffffffff8166f574>] kernel_thread_helper+0x4/0x10
Jun 23 02:30:24 imap kernel: [1189298.210347]  [<ffffffff8108b950>] ? flush_kthread_worker+0xa0/0xa0
Jun 23 02:30:24 imap kernel: [1189298.210349]  [<ffffffff8166f570>] ? gs_change+0x13/0x13
Jun 23 02:30:24 imap kernel: [1189298.210389] INFO: task dsmc:6812 blocked for more than 120 seconds.
Jun 23 02:30:24 imap kernel: [1189298.210445] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 02:30:24 imap kernel: [1189298.210513] dsmc            D 0000000000000000     0  6812      1 0x00000000
Jun 23 02:30:24 imap kernel: [1189298.210517]  ffff88034823d8a8 0000000000000082 ffff88040f01ade0 0000000000000000
Jun 23 02:30:24 imap kernel: [1189298.210520]  ffff88034823dfd8 ffff88034823dfd8 ffff88034823dfd8 0000000000012800
Jun 23 02:30:24 imap kernel: [1189298.210523]  ffff880419021700 ffff88004c93dc00 0000000000000000 ffff8803ff196970
Jun 23 02:30:24 imap kernel: [1189298.210526] Call Trace:
Jun 23 02:30:24 imap kernel: [1189298.210529]  [<ffffffff81662d6f>] schedule+0x3f/0x60
Jun 23 02:30:24 imap kernel: [1189298.210532]  [<ffffffff81663b67>] __mutex_lock_slowpath+0xd7/0x150
Jun 23 02:30:24 imap kernel: [1189298.210535]  [<ffffffff8166377a>] mutex_lock+0x2a/0x50
Jun 23 02:30:24 imap kernel: [1189298.210561]  [<ffffffffa01c320a>] zfs_zget+0x18a/0x350 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210566]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:30:24 imap kernel: [1189298.210592]  [<ffffffffa01a1e34>] zfs_dirent_lock+0x524/0x630 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210617]  [<ffffffffa01a1fcb>] zfs_dirlook+0x8b/0x300 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210643]  [<ffffffffa019e51d>] ? zfs_zaccess+0x9d/0x430 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210668]  [<ffffffffa01bbeb1>] zfs_lookup+0x2e1/0x330 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210693]  [<ffffffffa01d41bb>] zpl_lookup+0x9b/0x120 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210696]  [<ffffffff81186875>] d_alloc_and_lookup+0x45/0x90
Jun 23 02:30:24 imap kernel: [1189298.210699]  [<ffffffff81194095>] ? d_lookup+0x35/0x60
Jun 23 02:30:24 imap kernel: [1189298.210702]  [<ffffffff81188e22>] do_lookup+0x202/0x310
Jun 23 02:30:24 imap kernel: [1189298.210704]  [<ffffffff81189f8c>] path_lookupat+0x11c/0x750
Jun 23 02:30:24 imap kernel: [1189298.210709]  [<ffffffffa00c2feb>] ? spl_kmem_alloc_debug+0xab/0x1e0 [spl]
Jun 23 02:30:24 imap kernel: [1189298.210714]  [<ffffffff8131f217>] ? __strncpy_from_user+0x27/0x60
Jun 23 02:30:24 imap kernel: [1189298.210717]  [<ffffffff8118a5f1>] do_path_lookup+0x31/0xc0
Jun 23 02:30:24 imap kernel: [1189298.210720]  [<ffffffff8118b909>] user_path_at_empty+0x59/0xa0
Jun 23 02:30:24 imap kernel: [1189298.210724]  [<ffffffff8103ec29>] ? default_spin_lock_flags+0x9/0x10
Jun 23 02:30:24 imap kernel: [1189298.210749]  [<ffffffffa01d65e4>] ? zpl_xattr_list+0x154/0x300 [zfs]
Jun 23 02:30:24 imap kernel: [1189298.210751]  [<ffffffff81186e55>] ? putname+0x35/0x50
Jun 23 02:30:24 imap kernel: [1189298.210754]  [<ffffffff8118b961>] user_path_at+0x11/0x20
Jun 23 02:30:24 imap kernel: [1189298.210757]  [<ffffffff8118084a>] vfs_fstatat+0x3a/0x70
Jun 23 02:30:24 imap kernel: [1189298.210760]  [<ffffffff8118089e>] vfs_lstat+0x1e/0x20
Jun 23 02:30:24 imap kernel: [1189298.210762]  [<ffffffff81180a3a>] sys_newlstat+0x1a/0x40
Jun 23 02:30:24 imap kernel: [1189298.210765]  [<ffffffff8119b3af>] ? mntput+0x1f/0x30
Jun 23 02:30:24 imap kernel: [1189298.210767]  [<ffffffff81186782>] ? path_put+0x22/0x30
Jun 23 02:30:24 imap kernel: [1189298.210772]  [<ffffffff811a0061>] ? sys_llistxattr+0x51/0x60
Jun 23 02:30:24 imap kernel: [1189298.210777]  [<ffffffff8166d402>] system_call_fastpath+0x16/0x1b
Jun 23 02:32:24 imap kernel: [1189418.169418] INFO: task kswapd0:51 blocked for more than 120 seconds.
Jun 23 02:32:24 imap kernel: [1189418.169492] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 02:32:24 imap kernel: [1189418.169561] kswapd0         D ffffffff81806200     0    51      2 0x00000000
Jun 23 02:32:24 imap kernel: [1189418.169566]  ffff880421b9f7e0 0000000000000046 ffff880425648000 ffff88043fdd2800
Jun 23 02:32:24 imap kernel: [1189418.169570]  ffff880421b9ffd8 ffff880421b9ffd8 ffff880421b9ffd8 0000000000012800
Jun 23 02:32:24 imap kernel: [1189418.169573]  ffff880425649700 ffff880421ba2e00 ffff880421b9f820 ffffffffa0251210
Jun 23 02:32:24 imap kernel: [1189418.169577] Call Trace:
Jun 23 02:32:24 imap kernel: [1189418.169600]  [<ffffffff81662d6f>] schedule+0x3f/0x60
Jun 23 02:32:24 imap kernel: [1189418.169603]  [<ffffffff81663b67>] __mutex_lock_slowpath+0xd7/0x150
Jun 23 02:32:24 imap kernel: [1189418.169607]  [<ffffffff8166377a>] mutex_lock+0x2a/0x50
Jun 23 02:32:24 imap kernel: [1189418.169653]  [<ffffffffa0123547>] arc_buf_remove_ref+0xa7/0x140 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169673]  [<ffffffffa0129b67>] dbuf_rele_and_unlock+0x137/0x3f0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169700]  [<ffffffff8116524a>] ? discard_slab+0x3a/0x50
Jun 23 02:32:24 imap kernel: [1189418.169705]  [<ffffffff811657ff>] ? unfreeze_partials+0x24f/0x2b0
Jun 23 02:32:24 imap kernel: [1189418.169711]  [<ffffffff81121a45>] ? free_one_page+0x155/0x330
Jun 23 02:32:24 imap kernel: [1189418.169733]  [<ffffffffa012a2af>] dbuf_rele+0x3f/0x50 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169759]  [<ffffffffa0145da0>] dnode_rele_and_unlock+0xa0/0xb0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169774]  [<ffffffffa0129da8>] dbuf_rele_and_unlock+0x378/0x3f0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169778]  [<ffffffff81165964>] ? kmem_cache_free+0x104/0x110
Jun 23 02:32:24 imap kernel: [1189418.169782]  [<ffffffff8166376d>] ? mutex_lock+0x1d/0x50
Jun 23 02:32:24 imap kernel: [1189418.169785]  [<ffffffff81664cce>] ? _raw_spin_lock+0xe/0x20
Jun 23 02:32:24 imap kernel: [1189418.169800]  [<ffffffffa012a2af>] dbuf_rele+0x3f/0x50 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169814]  [<ffffffffa012a4ae>] dmu_buf_rele+0xe/0x10 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169839]  [<ffffffffa0169c93>] sa_handle_destroy+0x73/0xc0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169884]  [<ffffffffa01c051d>] zfs_znode_dmu_fini+0x1d/0x30 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169911]  [<ffffffffa01c3bdd>] zfs_zinactive+0xad/0x190 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169937]  [<ffffffffa01bde0b>] zfs_inactive+0x7b/0x260 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169963]  [<ffffffffa01d525e>] zpl_evict_inode+0x4e/0x70 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.169968]  [<ffffffff81196538>] evict+0xb8/0x1c0
Jun 23 02:32:24 imap kernel: [1189418.169970]  [<ffffffff81196c21>] dispose_list+0x41/0x50
Jun 23 02:32:24 imap kernel: [1189418.169973]  [<ffffffff81664c9f>] ? _raw_spin_trylock+0xf/0x30
Jun 23 02:32:24 imap kernel: [1189418.169976]  [<ffffffff81197668>] prune_icache_sb+0x168/0x330
Jun 23 02:32:24 imap kernel: [1189418.169980]  [<ffffffff8117ecd8>] prune_super+0xf8/0x1b0
Jun 23 02:32:24 imap kernel: [1189418.169985]  [<ffffffff8112c764>] shrink_slab+0x154/0x300
Jun 23 02:32:24 imap kernel: [1189418.169988]  [<ffffffff8112fa5a>] balance_pgdat+0x50a/0x6d0
Jun 23 02:32:24 imap kernel: [1189418.169991]  [<ffffffff8112fd49>] kswapd+0x129/0x230
Jun 23 02:32:24 imap kernel: [1189418.169994]  [<ffffffff8112fc20>] ? balance_pgdat+0x6d0/0x6d0
Jun 23 02:32:24 imap kernel: [1189418.169998]  [<ffffffff8108b9dc>] kthread+0x8c/0xa0
Jun 23 02:32:24 imap kernel: [1189418.170002]  [<ffffffff8166f574>] kernel_thread_helper+0x4/0x10
Jun 23 02:32:24 imap kernel: [1189418.170004]  [<ffffffff8108b950>] ? flush_kthread_worker+0xa0/0xa0
Jun 23 02:32:24 imap kernel: [1189418.170007]  [<ffffffff8166f570>] ? gs_change+0x13/0x13
Jun 23 02:32:24 imap kernel: [1189418.170015] INFO: task spl_system_task:2406 blocked for more than 120 seconds.
Jun 23 02:32:24 imap kernel: [1189418.170081] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 02:32:24 imap kernel: [1189418.170149] spl_system_task D ffffffff81806200     0  2406      2 0x00000000
Jun 23 02:32:24 imap kernel: [1189418.170153]  ffff88042363a9c0 0000000000000046 ffff88004c93dc00 0000000000000013
Jun 23 02:32:24 imap kernel: [1189418.170156]  ffff88042363bfd8 ffff88042363bfd8 ffff88042363bfd8 0000000000012800
Jun 23 02:32:24 imap kernel: [1189418.170159]  ffff880425589700 ffff880420f08000 ffff880420f08000 ffffffffa0251210
Jun 23 02:32:24 imap kernel: [1189418.170162] Call Trace:
Jun 23 02:32:24 imap kernel: [1189418.170166]  [<ffffffff81662d6f>] schedule+0x3f/0x60
Jun 23 02:32:24 imap kernel: [1189418.170169]  [<ffffffff81663b67>] __mutex_lock_slowpath+0xd7/0x150
Jun 23 02:32:24 imap kernel: [1189418.170173]  [<ffffffff8166377a>] mutex_lock+0x2a/0x50
Jun 23 02:32:24 imap kernel: [1189418.170186]  [<ffffffffa0123547>] arc_buf_remove_ref+0xa7/0x140 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170201]  [<ffffffffa0129b67>] dbuf_rele_and_unlock+0x137/0x3f0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170204]  [<ffffffff81664cce>] ? _raw_spin_lock+0xe/0x20
Jun 23 02:32:24 imap kernel: [1189418.170206]  [<ffffffff81664cce>] ? _raw_spin_lock+0xe/0x20
Jun 23 02:32:24 imap kernel: [1189418.170210]  [<ffffffff811657ce>] ? unfreeze_partials+0x21e/0x2b0
Jun 23 02:32:24 imap kernel: [1189418.170224]  [<ffffffffa012a2af>] dbuf_rele+0x3f/0x50 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170243]  [<ffffffffa0145da0>] dnode_rele_and_unlock+0xa0/0xb0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170258]  [<ffffffffa0129da8>] dbuf_rele_and_unlock+0x378/0x3f0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170261]  [<ffffffff81165964>] ? kmem_cache_free+0x104/0x110
Jun 23 02:32:24 imap kernel: [1189418.170264]  [<ffffffff8166376d>] ? mutex_lock+0x1d/0x50
Jun 23 02:32:24 imap kernel: [1189418.170266]  [<ffffffff81664cce>] ? _raw_spin_lock+0xe/0x20
Jun 23 02:32:24 imap kernel: [1189418.170280]  [<ffffffffa012a2af>] dbuf_rele+0x3f/0x50 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170294]  [<ffffffffa012a4ae>] dmu_buf_rele+0xe/0x10 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170318]  [<ffffffffa0169c93>] sa_handle_destroy+0x73/0xc0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170343]  [<ffffffffa01c051d>] zfs_znode_dmu_fini+0x1d/0x30 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170368]  [<ffffffffa01c3bdd>] zfs_zinactive+0xad/0x190 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170394]  [<ffffffffa01bde0b>] zfs_inactive+0x7b/0x260 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170418]  [<ffffffffa01d525e>] zpl_evict_inode+0x4e/0x70 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170422]  [<ffffffff81196538>] evict+0xb8/0x1c0
Jun 23 02:32:24 imap kernel: [1189418.170424]  [<ffffffff81196c21>] dispose_list+0x41/0x50
Jun 23 02:32:24 imap kernel: [1189418.170427]  [<ffffffff81664c9f>] ? _raw_spin_trylock+0xf/0x30
Jun 23 02:32:24 imap kernel: [1189418.170429]  [<ffffffff81197668>] prune_icache_sb+0x168/0x330
Jun 23 02:32:24 imap kernel: [1189418.170432]  [<ffffffff8117ecd8>] prune_super+0xf8/0x1b0
Jun 23 02:32:24 imap kernel: [1189418.170435]  [<ffffffff8112c764>] shrink_slab+0x154/0x300
Jun 23 02:32:24 imap kernel: [1189418.170438]  [<ffffffff8115f310>] ? compact_zone.part.14+0x1c0/0x260
Jun 23 02:32:24 imap kernel: [1189418.170441]  [<ffffffff8112eff8>] do_try_to_free_pages+0x268/0x2d0
Jun 23 02:32:24 imap kernel: [1189418.170445]  [<ffffffff81137449>] ? __mod_zone_page_state+0x49/0x50
Jun 23 02:32:24 imap kernel: [1189418.170448]  [<ffffffff8112f33b>] try_to_free_pages+0xab/0x170
Jun 23 02:32:24 imap kernel: [1189418.170451]  [<ffffffff8112358e>] __alloc_pages_nodemask+0x59e/0x8f0
Jun 23 02:32:24 imap kernel: [1189418.170458]  [<ffffffffa0049121>] ? mptscsih_qcmd+0x201/0x340 [mptscsih]
Jun 23 02:32:24 imap kernel: [1189418.170462]  [<ffffffff8115b1a6>] alloc_pages_current+0xb6/0x120
Jun 23 02:32:24 imap kernel: [1189418.170465]  [<ffffffff811642dd>] allocate_slab+0x13d/0x1a0
Jun 23 02:32:24 imap kernel: [1189418.170467]  [<ffffffff81164370>] new_slab+0x30/0x180
Jun 23 02:32:24 imap kernel: [1189418.170471]  [<ffffffff816511c6>] __slab_alloc+0x165/0x269
Jun 23 02:32:24 imap kernel: [1189418.170480]  [<ffffffffa00c460f>] ? spl_kmem_cache_alloc+0x7f/0x7f0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.170486]  [<ffffffffa00c460f>] ? spl_kmem_cache_alloc+0x7f/0x7f0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.170488]  [<ffffffff81167c36>] kmem_cache_alloc+0x136/0x140
Jun 23 02:32:24 imap kernel: [1189418.170493]  [<ffffffffa00c460f>] spl_kmem_cache_alloc+0x7f/0x7f0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.170519]  [<ffffffffa01ca423>] zio_buf_alloc+0x23/0x30 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170532]  [<ffffffffa011fded>] arc_get_data_buf.isra.23+0x48d/0x4e0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170545]  [<ffffffffa0123ad7>] arc_read+0x4f7/0x9e0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170552]  [<ffffffffa00c9fc3>] ? cv_wait_common+0xf3/0x130 [spl]
Jun 23 02:32:24 imap kernel: [1189418.170570]  [<ffffffffa013f6e8>] traverse_prefetcher+0x158/0x170 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170583]  [<ffffffffa011e3f7>] ? add_reference.isra.12+0x137/0x170 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170608]  [<ffffffffa013fb26>] traverse_visitbp+0x426/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170614]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.170618]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.170623]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.170641]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170660]  [<ffffffffa0140551>] traverse_dnode+0x71/0xd0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170678]  [<ffffffffa013fdc6>] traverse_visitbp+0x6c6/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170696]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170715]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170733]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170751]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170770]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170788]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170807]  [<ffffffffa0140551>] traverse_dnode+0x71/0xd0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170825]  [<ffffffffa013fe56>] traverse_visitbp+0x756/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170830]  [<ffffffff810136a9>] ? __switch_to+0xb9/0x370
Jun 23 02:32:24 imap kernel: [1189418.170848]  [<ffffffffa014063b>] traverse_prefetch_thread+0x8b/0x100 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170866]  [<ffffffffa013f590>] ? prefetch_needed.isra.3+0x50/0x50 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.170872]  [<ffffffffa00c6466>] taskq_thread+0x1b6/0x3e0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.170878]  [<ffffffff81060b00>] ? try_to_wake_up+0x200/0x200
Jun 23 02:32:24 imap kernel: [1189418.170883]  [<ffffffffa00c62b0>] ? task_done+0xb0/0xb0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.170886]  [<ffffffff8108b9dc>] kthread+0x8c/0xa0
Jun 23 02:32:24 imap kernel: [1189418.170888]  [<ffffffff8166f574>] kernel_thread_helper+0x4/0x10
Jun 23 02:32:24 imap kernel: [1189418.170891]  [<ffffffff8108b950>] ? flush_kthread_worker+0xa0/0xa0
Jun 23 02:32:24 imap kernel: [1189418.170893]  [<ffffffff8166f570>] ? gs_change+0x13/0x13
Jun 23 02:32:24 imap kernel: [1189418.170909] INFO: task z_wr_int/14:2999 blocked for more than 120 seconds.
Jun 23 02:32:24 imap kernel: [1189418.170971] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 02:32:24 imap kernel: [1189418.171149] z_wr_int/14     D ffffffff81806200     0  2999      2 0x00000000
Jun 23 02:32:24 imap kernel: [1189418.171152]  ffff880418a41ba0 0000000000000046 0000000000000000 0000000000000000
Jun 23 02:32:24 imap kernel: [1189418.171156]  ffff880418a41fd8 ffff880418a41fd8 ffff880418a41fd8 0000000000012800
Jun 23 02:32:24 imap kernel: [1189418.171159]  ffff8804255d1700 ffff880420b60000 ffff880418a41be0 ffffffffa0251210
Jun 23 02:32:24 imap kernel: [1189418.171162] Call Trace:
Jun 23 02:32:24 imap kernel: [1189418.171168]  [<ffffffff81662d6f>] schedule+0x3f/0x60
Jun 23 02:32:24 imap kernel: [1189418.171171]  [<ffffffff81663b67>] __mutex_lock_slowpath+0xd7/0x150
Jun 23 02:32:24 imap kernel: [1189418.171174]  [<ffffffff8166377a>] mutex_lock+0x2a/0x50
Jun 23 02:32:24 imap kernel: [1189418.171188]  [<ffffffffa011d310>] buf_hash_insert+0x80/0x1b0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171201]  [<ffffffffa0120e82>] arc_write_done+0xc2/0x430 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171226]  [<ffffffffa01cf069>] zio_done+0x349/0xe60 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171231]  [<ffffffffa00c2892>] ? spl_kmem_free+0x32/0x50 [spl]
Jun 23 02:32:24 imap kernel: [1189418.171256]  [<ffffffffa018e4f0>] ? vdev_mirror_map_free+0x20/0x30 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171281]  [<ffffffffa01cf572>] zio_done+0x852/0xe60 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171307]  [<ffffffffa01caef9>] zio_execute+0xc9/0x180 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171312]  [<ffffffffa00c6466>] taskq_thread+0x1b6/0x3e0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.171316]  [<ffffffff81060b00>] ? try_to_wake_up+0x200/0x200
Jun 23 02:32:24 imap kernel: [1189418.171321]  [<ffffffffa00c62b0>] ? task_done+0xb0/0xb0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.171323]  [<ffffffff8108b9dc>] kthread+0x8c/0xa0
Jun 23 02:32:24 imap kernel: [1189418.171326]  [<ffffffff8166f574>] kernel_thread_helper+0x4/0x10
Jun 23 02:32:24 imap kernel: [1189418.171329]  [<ffffffff8108b950>] ? flush_kthread_worker+0xa0/0xa0
Jun 23 02:32:24 imap kernel: [1189418.171331]  [<ffffffff8166f570>] ? gs_change+0x13/0x13
Jun 23 02:32:24 imap kernel: [1189418.171347] INFO: task txg_sync:3171 blocked for more than 120 seconds.
Jun 23 02:32:24 imap kernel: [1189418.171433] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 02:32:24 imap kernel: [1189418.171553] txg_sync        D 0000000000000001     0  3171      2 0x00000000
Jun 23 02:32:24 imap kernel: [1189418.171556]  ffff880415fcdb50 0000000000000046 ffff880415fcdaf0 ffff880420ddc7a8
Jun 23 02:32:24 imap kernel: [1189418.171560]  ffff880415fcdfd8 ffff880415fcdfd8 ffff880415fcdfd8 0000000000012800
Jun 23 02:32:24 imap kernel: [1189418.171563]  ffff880417d29700 ffff880417ff2e00 ffff880415fcdb40 ffff88043fc530c0
Jun 23 02:32:24 imap kernel: [1189418.171566] Call Trace:
Jun 23 02:32:24 imap kernel: [1189418.171570]  [<ffffffff81662d6f>] schedule+0x3f/0x60
Jun 23 02:32:24 imap kernel: [1189418.171572]  [<ffffffff81662e1f>] io_schedule+0x8f/0xd0
Jun 23 02:32:24 imap kernel: [1189418.171580]  [<ffffffffa00c9f7a>] cv_wait_common+0xaa/0x130 [spl]
Jun 23 02:32:24 imap kernel: [1189418.171582]  [<ffffffff8108c480>] ? add_wait_queue+0x60/0x60
Jun 23 02:32:24 imap kernel: [1189418.171588]  [<ffffffffa00ca018>] __cv_wait_io+0x18/0x20 [spl]
Jun 23 02:32:24 imap kernel: [1189418.171614]  [<ffffffffa01cd0ab>] zio_wait+0x10b/0x1f0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171636]  [<ffffffffa015846d>] dsl_pool_sync+0x2ad/0x410 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171660]  [<ffffffffa017265c>] spa_sync+0x44c/0xba0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171663]  [<ffffffff8108c496>] ? autoremove_wake_function+0x16/0x40
Jun 23 02:32:24 imap kernel: [1189418.171667]  [<ffffffff8101bc09>] ? read_tsc+0x9/0x20
Jun 23 02:32:24 imap kernel: [1189418.171692]  [<ffffffffa01837a1>] txg_sync_thread+0x351/0x5a0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171717]  [<ffffffffa0183450>] ? txg_fini+0x280/0x280 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171742]  [<ffffffffa0183450>] ? txg_fini+0x280/0x280 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171747]  [<ffffffffa00c5698>] thread_generic_wrapper+0x78/0x90 [spl]
Jun 23 02:32:24 imap kernel: [1189418.171752]  [<ffffffffa00c5620>] ? __thread_create+0x160/0x160 [spl]
Jun 23 02:32:24 imap kernel: [1189418.171755]  [<ffffffff8108b9dc>] kthread+0x8c/0xa0
Jun 23 02:32:24 imap kernel: [1189418.171757]  [<ffffffff8166f574>] kernel_thread_helper+0x4/0x10
Jun 23 02:32:24 imap kernel: [1189418.171760]  [<ffffffff8108b950>] ? flush_kthread_worker+0xa0/0xa0
Jun 23 02:32:24 imap kernel: [1189418.171762]  [<ffffffff8166f570>] ? gs_change+0x13/0x13
Jun 23 02:32:24 imap kernel: [1189418.171774] INFO: task dsmc:6812 blocked for more than 120 seconds.
Jun 23 02:32:24 imap kernel: [1189418.171845] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 02:32:24 imap kernel: [1189418.171943] dsmc            D 0000000000000000     0  6812      1 0x00000000
Jun 23 02:32:24 imap kernel: [1189418.171947]  ffff88034823d8a8 0000000000000082 ffff88040f01ade0 0000000000000000
Jun 23 02:32:24 imap kernel: [1189418.171950]  ffff88034823dfd8 ffff88034823dfd8 ffff88034823dfd8 0000000000012800
Jun 23 02:32:24 imap kernel: [1189418.171953]  ffff880419021700 ffff88004c93dc00 0000000000000000 ffff8803ff196970
Jun 23 02:32:24 imap kernel: [1189418.171956] Call Trace:
Jun 23 02:32:24 imap kernel: [1189418.171960]  [<ffffffff81662d6f>] schedule+0x3f/0x60
Jun 23 02:32:24 imap kernel: [1189418.171962]  [<ffffffff81663b67>] __mutex_lock_slowpath+0xd7/0x150
Jun 23 02:32:24 imap kernel: [1189418.171965]  [<ffffffff8166377a>] mutex_lock+0x2a/0x50
Jun 23 02:32:24 imap kernel: [1189418.171991]  [<ffffffffa01c320a>] zfs_zget+0x18a/0x350 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.171996]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.172022]  [<ffffffffa01a1e34>] zfs_dirent_lock+0x524/0x630 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172047]  [<ffffffffa01a1fcb>] zfs_dirlook+0x8b/0x300 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172072]  [<ffffffffa019e51d>] ? zfs_zaccess+0x9d/0x430 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172098]  [<ffffffffa01bbeb1>] zfs_lookup+0x2e1/0x330 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172123]  [<ffffffffa01d41bb>] zpl_lookup+0x9b/0x120 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172126]  [<ffffffff81186875>] d_alloc_and_lookup+0x45/0x90
Jun 23 02:32:24 imap kernel: [1189418.172129]  [<ffffffff81194095>] ? d_lookup+0x35/0x60
Jun 23 02:32:24 imap kernel: [1189418.172132]  [<ffffffff81188e22>] do_lookup+0x202/0x310
Jun 23 02:32:24 imap kernel: [1189418.172134]  [<ffffffff81189f8c>] path_lookupat+0x11c/0x750
Jun 23 02:32:24 imap kernel: [1189418.172139]  [<ffffffffa00c2feb>] ? spl_kmem_alloc_debug+0xab/0x1e0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.172145]  [<ffffffff8131f217>] ? __strncpy_from_user+0x27/0x60
Jun 23 02:32:24 imap kernel: [1189418.172147]  [<ffffffff8118a5f1>] do_path_lookup+0x31/0xc0
Jun 23 02:32:24 imap kernel: [1189418.172150]  [<ffffffff8118b909>] user_path_at_empty+0x59/0xa0
Jun 23 02:32:24 imap kernel: [1189418.172155]  [<ffffffff8103ec29>] ? default_spin_lock_flags+0x9/0x10
Jun 23 02:32:24 imap kernel: [1189418.172180]  [<ffffffffa01d65e4>] ? zpl_xattr_list+0x154/0x300 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172183]  [<ffffffff81186e55>] ? putname+0x35/0x50
Jun 23 02:32:24 imap kernel: [1189418.172185]  [<ffffffff8118b961>] user_path_at+0x11/0x20
Jun 23 02:32:24 imap kernel: [1189418.172188]  [<ffffffff8118084a>] vfs_fstatat+0x3a/0x70
Jun 23 02:32:24 imap kernel: [1189418.172191]  [<ffffffff8118089e>] vfs_lstat+0x1e/0x20
Jun 23 02:32:24 imap kernel: [1189418.172194]  [<ffffffff81180a3a>] sys_newlstat+0x1a/0x40
Jun 23 02:32:24 imap kernel: [1189418.172196]  [<ffffffff8119b3af>] ? mntput+0x1f/0x30
Jun 23 02:32:24 imap kernel: [1189418.172199]  [<ffffffff81186782>] ? path_put+0x22/0x30
Jun 23 02:32:24 imap kernel: [1189418.172202]  [<ffffffff811a0061>] ? sys_llistxattr+0x51/0x60
Jun 23 02:32:24 imap kernel: [1189418.172207]  [<ffffffff8166d402>] system_call_fastpath+0x16/0x1b
Jun 23 02:32:24 imap kernel: [1189418.172304] INFO: task imap:13647 blocked for more than 120 seconds.
Jun 23 02:32:24 imap kernel: [1189418.172375] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 02:32:24 imap kernel: [1189418.172472] imap            D ffffffff81806200     0 13647  29012 0x00000000
Jun 23 02:32:24 imap kernel: [1189418.172475]  ffff8801959617d8 0000000000000086 ffff88022de91720 0000000000000000
Jun 23 02:32:24 imap kernel: [1189418.172479]  ffff880195961fd8 ffff880195961fd8 ffff880195961fd8 0000000000012800
Jun 23 02:32:24 imap kernel: [1189418.172482]  ffff880425631700 ffff8802be6e5c00 0000000000000000 ffff8803ff196970
Jun 23 02:32:24 imap kernel: [1189418.172485] Call Trace:
Jun 23 02:32:24 imap kernel: [1189418.172488]  [<ffffffff81662d6f>] schedule+0x3f/0x60
Jun 23 02:32:24 imap kernel: [1189418.172491]  [<ffffffff81663b67>] __mutex_lock_slowpath+0xd7/0x150
Jun 23 02:32:24 imap kernel: [1189418.172493]  [<ffffffff8166377a>] mutex_lock+0x2a/0x50
Jun 23 02:32:24 imap kernel: [1189418.172519]  [<ffffffffa01c320a>] zfs_zget+0x18a/0x350 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172525]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.172550]  [<ffffffffa01a1e34>] zfs_dirent_lock+0x524/0x630 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172575]  [<ffffffffa01a1fcb>] zfs_dirlook+0x8b/0x300 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172601]  [<ffffffffa019e51d>] ? zfs_zaccess+0x9d/0x430 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172626]  [<ffffffffa01bbeb1>] zfs_lookup+0x2e1/0x330 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172651]  [<ffffffffa01d41bb>] zpl_lookup+0x9b/0x120 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172654]  [<ffffffff81186875>] d_alloc_and_lookup+0x45/0x90
Jun 23 02:32:24 imap kernel: [1189418.172656]  [<ffffffff81194095>] ? d_lookup+0x35/0x60
Jun 23 02:32:24 imap kernel: [1189418.172659]  [<ffffffff81188e22>] do_lookup+0x202/0x310
Jun 23 02:32:24 imap kernel: [1189418.172664]  [<ffffffff812a222c>] ? security_inode_permission+0x1c/0x30
Jun 23 02:32:24 imap kernel: [1189418.172667]  [<ffffffff81189738>] link_path_walk+0x138/0x870
Jun 23 02:32:24 imap kernel: [1189418.172673]  [<ffffffffa00cb018>] ? tsd_hash_search.isra.1+0xc8/0xe0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.172677]  [<ffffffff8166376d>] ? mutex_lock+0x1d/0x50
Jun 23 02:32:24 imap kernel: [1189418.172679]  [<ffffffff81188677>] ? path_init+0x1d7/0x420
Jun 23 02:32:24 imap kernel: [1189418.172682]  [<ffffffff81189ec8>] path_lookupat+0x58/0x750
Jun 23 02:32:24 imap kernel: [1189418.172684]  [<ffffffff8131f217>] ? __strncpy_from_user+0x27/0x60
Jun 23 02:32:24 imap kernel: [1189418.172687]  [<ffffffff8118a5f1>] do_path_lookup+0x31/0xc0
Jun 23 02:32:24 imap kernel: [1189418.172690]  [<ffffffff8118b909>] user_path_at_empty+0x59/0xa0
Jun 23 02:32:24 imap kernel: [1189418.172693]  [<ffffffff8118b9b2>] ? do_filp_open+0x42/0xa0
Jun 23 02:32:24 imap kernel: [1189418.172695]  [<ffffffff81165a4b>] ? kfree+0x3b/0x140
Jun 23 02:32:24 imap kernel: [1189418.172698]  [<ffffffff8118b961>] user_path_at+0x11/0x20
Jun 23 02:32:24 imap kernel: [1189418.172701]  [<ffffffff8118084a>] vfs_fstatat+0x3a/0x70
Jun 23 02:32:24 imap kernel: [1189418.172703]  [<ffffffff8119b3af>] ? mntput+0x1f/0x30
Jun 23 02:32:24 imap kernel: [1189418.172706]  [<ffffffff8117cfd3>] ? __fput+0x153/0x210
Jun 23 02:32:24 imap kernel: [1189418.172709]  [<ffffffff811808bb>] vfs_stat+0x1b/0x20
Jun 23 02:32:24 imap kernel: [1189418.172711]  [<ffffffff811809fa>] sys_newstat+0x1a/0x40
Jun 23 02:32:24 imap kernel: [1189418.172714]  [<ffffffff8117d0b5>] ? fput+0x25/0x30
Jun 23 02:32:24 imap kernel: [1189418.172716]  [<ffffffff81179a96>] ? filp_close+0x66/0x90
Jun 23 02:32:24 imap kernel: [1189418.172719]  [<ffffffff81179b72>] ? sys_close+0xb2/0x120
Jun 23 02:32:24 imap kernel: [1189418.172722]  [<ffffffff8166d402>] system_call_fastpath+0x16/0x1b
Jun 23 02:32:24 imap kernel: [1189418.172736] INFO: task zfs:14977 blocked for more than 120 seconds.
Jun 23 02:32:24 imap kernel: [1189418.172806] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 02:32:24 imap kernel: [1189418.172903] zfs             D 0000000000000006     0 14977  14102 0x00000000
Jun 23 02:32:24 imap kernel: [1189418.172906]  ffff880264319238 0000000000000082 ffff8802643191d8 ffffffff81664cce
Jun 23 02:32:24 imap kernel: [1189418.172910]  ffff880264319fd8 ffff880264319fd8 ffff880264319fd8 0000000000012800
Jun 23 02:32:24 imap kernel: [1189418.172913]  ffff8800a406c500 ffff88031326c500 ffffffffa028f3b8 ffffffffa0251210
Jun 23 02:32:24 imap kernel: [1189418.172916] Call Trace:
Jun 23 02:32:24 imap kernel: [1189418.172919]  [<ffffffff81664cce>] ? _raw_spin_lock+0xe/0x20
Jun 23 02:32:24 imap kernel: [1189418.172923]  [<ffffffff81662d6f>] schedule+0x3f/0x60
Jun 23 02:32:24 imap kernel: [1189418.172925]  [<ffffffff81663b67>] __mutex_lock_slowpath+0xd7/0x150
Jun 23 02:32:24 imap kernel: [1189418.172928]  [<ffffffff8166377a>] mutex_lock+0x2a/0x50
Jun 23 02:32:24 imap kernel: [1189418.172942]  [<ffffffffa011d4de>] buf_hash_find+0x9e/0x160 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172955]  [<ffffffffa01236c1>] arc_read+0xe1/0x9e0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172960]  [<ffffffffa00c460f>] ? spl_kmem_cache_alloc+0x7f/0x7f0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.172974]  [<ffffffffa0123fc0>] ? arc_read+0x9e0/0x9e0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.172992]  [<ffffffffa013d39a>] backup_cb+0x18a/0x9e0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173004]  [<ffffffffa011fdfd>] ? arc_get_data_buf.isra.23+0x49d/0x4e0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173023]  [<ffffffffa013fb26>] traverse_visitbp+0x426/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173028]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.173032]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.173037]  [<ffffffffa00c2c2e>] ? spl_kmem_alloc+0xae/0x1e0 [spl]
Jun 23 02:32:24 imap kernel: [1189418.173055]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173073]  [<ffffffffa0140551>] traverse_dnode+0x71/0xd0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173091]  [<ffffffffa013fdc6>] traverse_visitbp+0x6c6/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173110]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173128]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173147]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173165]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173183]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173202]  [<ffffffffa013f9f7>] traverse_visitbp+0x2f7/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173220]  [<ffffffffa0140551>] traverse_dnode+0x71/0xd0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173238]  [<ffffffffa013fe56>] traverse_visitbp+0x756/0x840 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173243]  [<ffffffff8104f163>] ? __wake_up+0x53/0x70
Jun 23 02:32:24 imap kernel: [1189418.173261]  [<ffffffffa014009b>] traverse_impl+0x15b/0x380 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173279]  [<ffffffffa013c880>] ? dmu_send_impl+0x550/0x550 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173297]  [<ffffffffa0140304>] traverse_dataset+0x44/0x50 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173315]  [<ffffffffa013d210>] ? dmu_recv_begin_check+0x4c0/0x4c0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173353]  [<ffffffffa013c6fb>] dmu_send_impl+0x3cb/0x550 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173372]  [<ffffffffa013dd21>] dmu_send_obj+0x131/0x1e0 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173397]  [<ffffffffa01a73f2>] zfs_ioc_send+0xd2/0x280 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173402]  [<ffffffffa00c284f>] ? strdup+0x5f/0x70 [spl]
Jun 23 02:32:24 imap kernel: [1189418.173427]  [<ffffffffa01adbaa>] zfsdev_ioctl+0x4da/0x520 [zfs]
Jun 23 02:32:24 imap kernel: [1189418.173431]  [<ffffffff8118ddfa>] do_vfs_ioctl+0x8a/0x340
Jun 23 02:32:24 imap kernel: [1189418.173436]  [<ffffffff810a25b7>] ? sys_futex+0x147/0x1a0
Jun 23 02:32:24 imap kernel: [1189418.173439]  [<ffffffff8118e141>] sys_ioctl+0x91/0xa0
Jun 23 02:32:24 imap kernel: [1189418.173442]  [<ffffffff8166d402>] system_call_fastpath+0x16/0x1b

@dweeezil
Copy link
Contributor

@galindro Then it sure sounds like 8e70975 will fix your problem.

Somewhat unrelated, I'm interested in the absurd load averages. Could you please post the output of dmesg | grep nr_cpu from the system. You indicated it was a VMware guest. It sounds like your guest kernel was built with a huge NR_CPUS and that it is not trimming nr_cpu_ids. If so, a patch which should address issues caused by giant nr_cpu_ids should be coming soon.

@galindro
Copy link
Author

root@imap:~# dmesg |grep nr_cpu
[ 0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:8 nr_node_ids:1

When this patch will be available for download through apt?

@dweeezil
Copy link
Contributor

@galindro Your nr_cpu_ids is being clamped at 8. I've got no idea why you're seeing the absurdly high load averages. You'll have to somehow run a ps to see what's happening. The patch I had in mind will enable dynamic taskqs and will dramatically cut down the number of kernel threads on high core count systems (which yours is not).

@galindro
Copy link
Author

Right now I've installed atop to monitor process each 5 minutes. So, when this happens again, I could analyse what is causing the high load average.

Could you please, answer my question bellow?

When the patch 8e70975 will be compiled and available for download through apt?

@behlendorf
Copy link
Contributor

@galindro we expect to tag 0.6.4.2 this week which will include 8e70975. After which it may take a few days for the PPA to be updated with the latest packages.

@galindro
Copy link
Author

Thank you very much for this great work guys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@behlendorf @dweeezil @galindro