can't import / mount pool #2861
Comments
|
Since the machine locks up when trying to import the pool, there is no way to get stacktraces. None appear on the screen either, other than the mentioned rcu_sched message. |
|
@mvernimmen I doubt this is related to your system update but you could always trying try importing the pool from a live CD running your distribution of choice. Also when you say you're pool is full, how full is it? 90%, 95%, 99%? If it's close to completely full you may start seeing significant performance problems. Unfortunately, the only way to do debug this will be to get a backtrace from the system. If you can get anything from the console or dmesg prior to a complete lockup the would be very helpful. |
|
Only a small part of the backtrace is logged to disk when I redirect all rsyslog messages to a file. Much more is printed on screen. Here is the output that was logged to file: The pool is 2TB in size, I think there is about 600GB free. I tried to mount the pool with a livecd (gentoo based sysresccd) and had the same problem, but this time i was able to catch the full stacktraces: [ 47.537271] SPL: Loaded module v0.6.3-r0-gentoo [ 209.602005] [] arch_trigger_all_cpu_backtrace+0x55/0x87 [ 209.602173] [] ? tick_broadcast_oneshot_control+0x159/0x164 [ 389.703327] [] arch_trigger_all_cpu_backtrace+0x55/0x87 [ 389.703494] [] ? tick_broadcast_oneshot_control+0x159/0x164 |
|
@behlendorf Is there anything else I can do or provide to help debug this? |
|
@mvernimmen Can you check what CONFIG_PREEMPT is set to for your new kernel. We're had reports of similar rcu issues depended on preempt. |
|
@behlendorf sure, here's the output: % grep CONFIG_PREEMPT /boot/config-3.16.6-203.fc20.x86_64 # CONFIG_PREEMPT_RCU is not set CONFIG_PREEMPT_NOTIFIERS=y # CONFIG_PREEMPT_NONE is not set CONFIG_PREEMPT_VOLUNTARY=y # CONFIG_PREEMPT is not set |
|
OK, well that's not it. Thanks for the quick turn around. |
|
@behlendorf Anything else I can do or provide to help this along? |
|
@mvernimmen could you try the latest testing version. We've fixed several issues in the area of the code where you're having problems. Since you're running fedora all you need to do is enable the zfs-testing repository, update the packages, and reboot. |
|
@behlendorf indeed it works fine with the testing release of the following composition: Why didn't I think of trying that? Thanks for the help. If there is anything you need or anything I can do to make sure the fix gets improved or included in the 0.6.4 release, let me know. Otherwise I'm very happy with this solution and the issue can be closed. |
|
Sounds good. Thanks for following up. |
|
@behlendorf Sorry to continue on this but it turns out one of the subvolumes remains unmountable and continues to display the symptoms of this issue. All volumes and subvolumes except one mount and work fine. I get these stacktraces when I try to mount the non-working subvolume: Nov 24 20:05:16 server kernel: [ 93.094477] SPL: using hostid 0x00000000 etc. This was with the testing repo enabled and installed on fc20 with kernel 3.17.2-200.fc20.x86_64 |
|
@behlendorf Can this issue be re-opened or should I create a new one for this sub-volume issue? |
|
@mvernimmen I've reopened it. Can you set the |
|
Hi @behlendorf , I updated to the latest testing version (gc944be5) and latest fedora kernel (3.17.4-200), then loaded zfs with the zil_replay_disable=1 option. I verified the module was loaded correctly with the setting by doing Maybe I could force stacktraces through echo 'stacktrace' > .../tracing/events/somesys/someevent/trigger or with ftrace, but I would need some help in knowing what event and sys to trigger on or how to set up ftrace for this specifically. I'd love a pointer ;) |
|
@mvernimmen If the mount hangs you could get a backtrace from it by look at /proc/mount-pid/stack. |
|
@behlendorf Here is the trace: |
|
And in case it helps, here is an strace: |
|
Hi @behlendorf , anything I can do to help this problem analysed and fixed? |
|
@mvernimmen I may have a workaround for you. Try setting the property |
|
Hi @behlendorf , |
|
Ahh, I see. Then would it be possible for you to grab the latest code from github and then recreate the problem. But this time if at all possible dump the stacks from all the processes on your system using |
|
@behlendorf |
|
@behlendorf could it maybe have something to do with this? openzfsonosx/zfs#255 |
|
@mvernimmen Thanks pointing this out, openzfsonosx/zfs#255 does look very similar. If somehow there are damaged entries it could manifest itself like this. |
|
@behlendorf Anything that I can do to help at this point? Anything we can try as a workaround or possible fix ? |
We need truncate and remove be in the same tx when doing zfs_rmnode on xattr dir. Otherwise, if we truncate and crash, we'll end up with inconsistent zap object on the delete queue. We do this by skipping dmu_free_long_range and let zfs_znode_delete to do the work. Signed-off-by: Chunwei Chen <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#4114 Issue openzfs#4052 Issue openzfs#4006 Issue openzfs#3018 Issue openzfs#2861
During zfs_rmnode on a xattr dir, if the system crash just after dmu_free_long_range, we would get empty xattr dir in delete queue. This would cause blkid=0 be passed into zap_get_leaf_byblk when doing zfs_purgedir during mount, and would try to do rw_enter on a wrong structure and cause system lockup. We fix this by returning ENOENT when blkid is zero in zap_get_leaf_byblk. Signed-off-by: Chunwei Chen <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #4114 Closes #4052 Closes #4006 Closes #3018 Closes #2861
We need truncate and remove be in the same tx when doing zfs_rmnode on xattr dir. Otherwise, if we truncate and crash, we'll end up with inconsistent zap object on the delete queue. We do this by skipping dmu_free_long_range and let zfs_znode_delete to do the work. Signed-off-by: Chunwei Chen <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Issue #4114 Issue #4052 Issue #4006 Issue #3018 Issue #2861
During zfs_rmnode on a xattr dir, if the system crash just after dmu_free_long_range, we would get empty xattr dir in delete queue. This would cause blkid=0 be passed into zap_get_leaf_byblk when doing zfs_purgedir during mount, and would try to do rw_enter on a wrong structure and cause system lockup. We fix this by returning ENOENT when blkid is zero in zap_get_leaf_byblk. Signed-off-by: Chunwei Chen <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes openzfs#4114 Closes openzfs#4052 Closes openzfs#4006 Closes openzfs#3018 Closes openzfs#2861
We need truncate and remove be in the same tx when doing zfs_rmnode on xattr dir. Otherwise, if we truncate and crash, we'll end up with inconsistent zap object on the delete queue. We do this by skipping dmu_free_long_range and let zfs_znode_delete to do the work. Signed-off-by: Chunwei Chen <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#4114 Issue openzfs#4052 Issue openzfs#4006 Issue openzfs#3018 Issue openzfs#2861
During zfs_rmnode on a xattr dir, if the system crash just after dmu_free_long_range, we would get empty xattr dir in delete queue. This would cause blkid=0 be passed into zap_get_leaf_byblk when doing zfs_purgedir during mount, and would try to do rw_enter on a wrong structure and cause system lockup. We fix this by returning ENOENT when blkid is zero in zap_get_leaf_byblk. Signed-off-by: Chunwei Chen <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes openzfs#4114 Closes openzfs#4052 Closes openzfs#4006 Closes openzfs#3018 Closes openzfs#2861
We need truncate and remove be in the same tx when doing zfs_rmnode on xattr dir. Otherwise, if we truncate and crash, we'll end up with inconsistent zap object on the delete queue. We do this by skipping dmu_free_long_range and let zfs_znode_delete to do the work. Signed-off-by: Chunwei Chen <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#4114 Issue openzfs#4052 Issue openzfs#4006 Issue openzfs#3018 Issue openzfs#2861
During zfs_rmnode on a xattr dir, if the system crash just after dmu_free_long_range, we would get empty xattr dir in delete queue. This would cause blkid=0 be passed into zap_get_leaf_byblk when doing zfs_purgedir during mount, and would try to do rw_enter on a wrong structure and cause system lockup. We fix this by returning ENOENT when blkid is zero in zap_get_leaf_byblk. Signed-off-by: Chunwei Chen <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes openzfs#4114 Closes openzfs#4052 Closes openzfs#4006 Closes openzfs#3018 Closes openzfs#2861
We need truncate and remove be in the same tx when doing zfs_rmnode on xattr dir. Otherwise, if we truncate and crash, we'll end up with inconsistent zap object on the delete queue. We do this by skipping dmu_free_long_range and let zfs_znode_delete to do the work. Signed-off-by: Chunwei Chen <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#4114 Issue openzfs#4052 Issue openzfs#4006 Issue openzfs#3018 Issue openzfs#2861
During zfs_rmnode on a xattr dir, if the system crash just after dmu_free_long_range, we would get empty xattr dir in delete queue. This would cause blkid=0 be passed into zap_get_leaf_byblk when doing zfs_purgedir during mount, and would try to do rw_enter on a wrong structure and cause system lockup. We fix this by returning ENOENT when blkid is zero in zap_get_leaf_byblk. Signed-off-by: Chunwei Chen <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes openzfs#4114 Closes openzfs#4052 Closes openzfs#4006 Closes openzfs#3018 Closes openzfs#2861
We need truncate and remove be in the same tx when doing zfs_rmnode on xattr dir. Otherwise, if we truncate and crash, we'll end up with inconsistent zap object on the delete queue. We do this by skipping dmu_free_long_range and let zfs_znode_delete to do the work. Signed-off-by: Chunwei Chen <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#4114 Issue openzfs#4052 Issue openzfs#4006 Issue openzfs#3018 Issue openzfs#2861
I'm not sure when this started, whether it was when I upgraded from fedora 19 to 20, when I upgraded zfsonlinux to 0.6.3 or when the volume became almost full but I am currently unable to mount one of my pools in write mode (readonly works fine). When trying to do so the system seems to hang and periodically prints out a message that rcu_sched found hanging cpu/tasks. The pool contains my family photo albums and some other things. Although I have backups I would prefer to get the pool back in normal working order. Here follows information as listed at http://zfsonlinux.org/faq.html#HowDoIReportProblems
machine:
1 socket, amd athlon II X2 250
4GB ECC ram
fedora 20 runs directly on the hardware, no hypervisors.
dmesg:
[ 16.272825] SPL: Loaded module v0.6.3-1.1
[ 16.660578] ZFS: Loaded module v0.6.3-1.1, ZFS pool version 5000, ZFS filesystem version 5
[ 23.501920] SPL: using hostid 0x00000000
uname:
Linux server.local 3.16.6-203.fc20.x86_64 #1 SMP Sat Oct 25 12:44:32 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
rpm versions:
zfs-0.6.3-1.1.fc20.x86_64
libzfs2-0.6.3-1.1.fc20.x86_64
spl-0.6.3-1.1.fc20.x86_64
zfs-dkms-0.6.3-1.1.fc20.noarch
spl-dkms-0.6.3-1.1.fc20.noarch
zfs-release-1-3.fc20.noarch
zdb:
mir-2tb:
version: 23
name: 'mir-2tb'
state: 0
txg: 2535963
pool_guid: 12859994781307727703
errata: 0
hostname: 'server.local'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 12859994781307727703
children[0]:
type: 'mirror'
id: 0
guid: 5092114349905149760
whole_disk: 0
metaslab_array: 23
metaslab_shift: 31
ashift: 9
asize: 2000394125312
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 7233940709623163225
path: '/dev/disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30SHWWD'
whole_disk: 0
DTL: 282
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 12314288060364456657
path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05894'
whole_disk: 0
DTL: 1017
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 18210654718933789590
path: '/dev/disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30U4RKD'
whole_disk: 0
DTL: 1023
create_txg: 4
features_for_read:
mir-2tb2:
version: 23
name: 'mir-2tb2'
state: 0
txg: 6386512
pool_guid: 107074157304257492
errata: 0
hostname: 'server.local'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 107074157304257492
children[0]:
type: 'mirror'
id: 0
guid: 5600193728825606093
whole_disk: 0
metaslab_array: 23
metaslab_shift: 31
ashift: 9
asize: 2000394125312
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 8074817977783627925
path: '/dev/disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30TG22D'
whole_disk: 0
DTL: 184
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 1126741303624888001
path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05884'
whole_disk: 0
DTL: 780
create_txg: 4
features_for_read:
params:
spl_debug_mask 188
spl_debug_mb -1
spl_debug_panic_on_bug 0
spl_debug_printk 60
spl_debug_subsys 18446744073709551615
spl_hostid 0
spl_hostid_path /etc/hostid
spl_kmem_cache_expire 2
spl_kmem_cache_kmem_limit 1024
spl_kmem_cache_max_size 32
spl_kmem_cache_obj_per_slab 16
spl_kmem_cache_obj_per_slab_min 8
spl_kmem_cache_reclaim 1
spl_kmem_cache_slab_limit 0
spl_taskq_thread_bind 0
l2arc_feed_again 1
l2arc_feed_min_ms 200
l2arc_feed_secs 1
l2arc_headroom 2
l2arc_headroom_boost 200
l2arc_nocompress 0
l2arc_noprefetch 1
l2arc_norw 0
l2arc_write_boost 8388608
l2arc_write_max 8388608
metaslab_debug_load 0
metaslab_debug_unload 0
spa_asize_inflation 24
spa_config_path /etc/zfs/zpool.cache
zfetch_array_rd_sz 1048576
zfetch_block_cap 256
zfetch_max_streams 8
zfetch_min_sec_reap 2
zfs_arc_grow_retry 5
zfs_arc_max 0
zfs_arc_memory_throttle_disable 1
zfs_arc_meta_limit 0
zfs_arc_meta_prune 1048576
zfs_arc_min 0
zfs_arc_min_prefetch_lifespan 1000
zfs_arc_p_aggressive_disable 1
zfs_arc_p_dampener_disable 1
zfs_arc_shrink_shift 5
zfs_autoimport_disable 0
zfs_dbuf_state_index 0
zfs_deadman_enabled 1
zfs_deadman_synctime_ms 1000000
zfs_dedup_prefetch 1
zfs_delay_min_dirty_percent 60
zfs_delay_scale 500000
zfs_dirty_data_max 388050534
zfs_dirty_data_max_max 970126336
zfs_dirty_data_max_max_percent 25
zfs_dirty_data_max_percent 10
zfs_dirty_data_sync 67108864
zfs_disable_dup_eviction 0
zfs_expire_snapshot 300
zfs_flags 1
zfs_free_min_time_ms 1000
zfs_immediate_write_sz 32768
zfs_mdcomp_disable 0
zfs_nocacheflush 0
zfs_nopwrite_enabled 1
zfs_no_scrub_io 0
zfs_no_scrub_prefetch 0
zfs_pd_blks_max 100
zfs_prefetch_disable 0
zfs_read_chunk_size 1048576
zfs_read_history 0
zfs_read_history_hits 0
zfs_recover 0
zfs_resilver_delay 2
zfs_resilver_min_time_ms 3000
zfs_scan_idle 50
zfs_scan_min_time_ms 1000
zfs_scrub_delay 4
zfs_send_corrupt_data 0
zfs_sync_pass_deferred_free 2
zfs_sync_pass_dont_compress 5
zfs_sync_pass_rewrite 2
zfs_top_maxinflight 32
zfs_txg_history 0
zfs_txg_timeout 5
zfs_vdev_aggregation_limit 131072
zfs_vdev_async_read_max_active 3
zfs_vdev_async_read_min_active 1
zfs_vdev_async_write_active_max_dirty_percent 60
zfs_vdev_async_write_active_min_dirty_percent 30
zfs_vdev_async_write_max_active 10
zfs_vdev_async_write_min_active 1
zfs_vdev_cache_bshift 16
zfs_vdev_cache_max 16384
zfs_vdev_cache_size 0
zfs_vdev_max_active 1000
zfs_vdev_mirror_switch_us 10000
zfs_vdev_read_gap_limit 32768
zfs_vdev_scheduler noop
zfs_vdev_scrub_max_active 2
zfs_vdev_scrub_min_active 1
zfs_vdev_sync_read_max_active 10
zfs_vdev_sync_read_min_active 10
zfs_vdev_sync_write_max_active 10
zfs_vdev_sync_write_min_active 10
zfs_vdev_write_gap_limit 4096
zfs_zevent_cols 80
zfs_zevent_console 0
zfs_zevent_len_max 96
zil_replay_disable 0
zil_slog_limit 1048576
zio_bulk_flags 0
zio_delay_max 30000
zio_injection_enabled 0
zio_requeue_io_start_cut_in_line 1
zvol_inhibit_dev 0
zvol_major 230
zvol_max_discard_blocks 16384
zvol_threads 32
The /proc/spl folder does not exist.
I'll try and get the stacktraces.
The text was updated successfully, but these errors were encountered: