Can't set autoexpand=on with recent git snapshot on 3.13.5

[satmandu]

$ sudo zpool get all tank2
NAME  PROPERTY               VALUE                  SOURCE
tank2  feature@async_destroy  disabled               local
tank2  feature@empty_bpobj    disabled               local
tank2  feature@lz4_compress   disabled               local
$ sudo zpool set autoexpand=on tank2
internal error: Cannot allocate memory

Version info:

$ /sbin/modinfo zfs
filename:       /lib/modules/3.13.5-031305-generic/updates/dkms/zfs.ko
version:        0.6.2-1
license:        CDDL
author:         Sun Microsystems/Oracle, Lawrence Livermore National Laboratory
description:    ZFS
srcversion:     063366A6FE1081181A1D7B8
depends:        spl,znvpair,zcommon,zunicode,zavl
vermagic:       3.13.5-031305-generic SMP mod_unload modversions 
parm:           zvol_inhibit_dev:Do not create zvol device nodes (uint)
parm:           zvol_major:Major number for zvol device (uint)
parm:           zvol_threads:Number of threads for zvol device (uint)
parm:           zvol_max_discard_blocks:Max number of blocks to discard (ulong)
parm:           zio_injection_enabled:Enable fault injection (int)
parm:           zio_bulk_flags:Additional flags to pass to bulk buffers (int)
parm:           zio_delay_max:Max zio millisec delay before posting event (int)
parm:           zio_requeue_io_start_cut_in_line:Prioritize requeued I/O (int)
parm:           zfs_sync_pass_deferred_free:Defer frees starting in this pass (int)
parm:           zfs_sync_pass_dont_compress:Don't compress starting in this pass (int)
parm:           zfs_sync_pass_rewrite:Rewrite new bps starting in this pass (int)
parm:           zil_replay_disable:Disable intent logging replay (int)
parm:           zfs_nocacheflush:Disable cache flushes (int)
parm:           zil_slog_limit:Max commit bytes to separate log device (ulong)
parm:           zfs_read_chunk_size:Bytes to read per chunk (long)
parm:           zfs_immediate_write_sz:Largest data block to write to zil (long)
parm:           zfs_flags:Set additional debugging flags (int)
parm:           zfs_recover:Set to attempt to recover from fatal errors (int)
parm:           zfs_expire_snapshot:Seconds to expire .zfs/snapshot (int)
parm:           zfs_vdev_aggregation_limit:Max vdev I/O aggregation size (int)
parm:           zfs_vdev_read_gap_limit:Aggregate read I/O over gap (int)
parm:           zfs_vdev_write_gap_limit:Aggregate write I/O over gap (int)
parm:           zfs_vdev_max_active:Maximum number of active I/Os per vdev (int)
parm:           zfs_vdev_async_write_active_max_dirty_percent:Async write concurrency max threshold (int)
parm:           zfs_vdev_async_write_active_min_dirty_percent:Async write concurrency min threshold (int)
parm:           zfs_vdev_async_read_max_active:Max active async read I/Os per vdev (int)
parm:           zfs_vdev_async_read_min_active:Min active async read I/Os per vdev (int)
parm:           zfs_vdev_async_write_max_active:Max active async write I/Os per vdev (int)
parm:           zfs_vdev_async_write_min_active:Min active async write I/Os per vdev (int)
parm:           zfs_vdev_scrub_max_active:Max active scrub I/Os per vdev (int)
parm:           zfs_vdev_scrub_min_active:Min active scrub I/Os per vdev (int)
parm:           zfs_vdev_sync_read_max_active:Max active sync read I/Os per vdev (int)
parm:           zfs_vdev_sync_read_min_active:Min active sync read I/Os per vdev (int)
parm:           zfs_vdev_sync_write_max_active:Max active sync write I/Os per vdev (int)
parm:           zfs_vdev_sync_write_min_active:Min active sync write I/Osper vdev (int)
parm:           zfs_vdev_mirror_switch_us:Switch mirrors every N usecs (int)
parm:           zfs_vdev_scheduler:I/O scheduler (charp)
parm:           zfs_vdev_cache_max:Inflate reads small than max (int)
parm:           zfs_vdev_cache_size:Total size of the per-disk cache (int)
parm:           zfs_vdev_cache_bshift:Shift size to inflate reads too (int)
parm:           zfs_txg_timeout:Max seconds worth of delta per txg (int)
parm:           zfs_read_history:Historic statistics for the last N reads (int)
parm:           zfs_read_history_hits:Include cache hits in read history (int)
parm:           zfs_txg_history:Historic statistics for the last N txgs (int)
parm:           zfs_deadman_synctime_ms:Expiration time in milliseconds (ulong)
parm:           zfs_deadman_enabled:Enable deadman timer (int)
parm:           spa_asize_inflation:SPA size estimate multiplication factor (int)
parm:           spa_config_path:SPA config file (/etc/zfs/zpool.cache) (charp)
parm:           zfs_autoimport_disable:Disable pool import at module load (int)
parm:           metaslab_debug:keep space maps in core to verify frees (int)
parm:           zfs_zevent_len_max:Max event queue length (int)
parm:           zfs_zevent_cols:Max event column width (int)
parm:           zfs_zevent_console:Log events to the console (int)
parm:           zfs_top_maxinflight:Max I/Os per top-level (int)
parm:           zfs_resilver_delay:Number of ticks to delay resilver (int)
parm:           zfs_scrub_delay:Number of ticks to delay scrub (int)
parm:           zfs_scan_idle:Idle window in clock ticks (int)
parm:           zfs_scan_min_time_ms:Min millisecs to scrub per txg (int)
parm:           zfs_free_min_time_ms:Min millisecs to free per txg (int)
parm:           zfs_resilver_min_time_ms:Min millisecs to resilver per txg (int)
parm:           zfs_no_scrub_io:Set to disable scrub I/O (int)
parm:           zfs_no_scrub_prefetch:Set to disable scrub prefetching (int)
parm:           zfs_dirty_data_max_percent:percent of ram can be dirty (int)
parm:           zfs_dirty_data_max_max_percent:zfs_dirty_data_max upper bound as % of RAM (int)
parm:           zfs_delay_min_dirty_percent:transaction delay threshold (int)
parm:           zfs_dirty_data_max:determines the dirty space limit (ulong)
parm:           zfs_dirty_data_max_max:zfs_dirty_data_max upper bound in bytes (ulong)
parm:           zfs_dirty_data_sync:sync txg when this much dirty data (ulong)
parm:           zfs_delay_scale:how quickly delay approaches infinity (ulong)
parm:           zfs_prefetch_disable:Disable all ZFS prefetching (int)
parm:           zfetch_max_streams:Max number of streams per zfetch (uint)
parm:           zfetch_min_sec_reap:Min time before stream reclaim (uint)
parm:           zfetch_block_cap:Max number of blocks to fetch at a time (uint)
parm:           zfetch_array_rd_sz:Number of bytes in a array_read (ulong)
parm:           zfs_pd_blks_max:Max number of blocks to prefetch (int)
parm:           zfs_send_corrupt_data:Allow sending corrupt data (int)
parm:           zfs_mdcomp_disable:Disable meta data compression (int)
parm:           zfs_nopwrite_enabled:Enable NOP writes (int)
parm:           zfs_dedup_prefetch:Enable prefetching dedup-ed blks (int)
parm:           zfs_dbuf_state_index:Calculate arc header index (int)
parm:           zfs_arc_min:Min arc size (ulong)
parm:           zfs_arc_max:Max arc size (ulong)
parm:           zfs_arc_meta_limit:Meta limit for arc size (ulong)
parm:           zfs_arc_meta_prune:Bytes of meta data to prune (int)
parm:           zfs_arc_grow_retry:Seconds before growing arc size (int)
parm:           zfs_arc_shrink_shift:log2(fraction of arc to reclaim) (int)
parm:           zfs_arc_p_min_shift:arc_c shift to calc min/max arc_p (int)
parm:           zfs_disable_dup_eviction:disable duplicate buffer eviction (int)
parm:           zfs_arc_memory_throttle_disable:disable memory throttle (int)
parm:           zfs_arc_min_prefetch_lifespan:Min life of prefetch block (int)
parm:           l2arc_write_max:Max write bytes per interval (ulong)
parm:           l2arc_write_boost:Extra write bytes during device warmup (ulong)
parm:           l2arc_headroom:Number of max device writes to precache (ulong)
parm:           l2arc_headroom_boost:Compressed l2arc_headroom multiplier (ulong)
parm:           l2arc_feed_secs:Seconds between L2ARC writing (ulong)
parm:           l2arc_feed_min_ms:Min feed interval in milliseconds (ulong)
parm:           l2arc_noprefetch:Skip caching prefetched buffers (int)
parm:           l2arc_nocompress:Skip compressing L2ARC buffers (int)
parm:           l2arc_feed_again:Turbo L2ARC warmup (int)
parm:           l2arc_norw:No reads during writes (int)
$ /sbin/modinfo spl
filename:       /lib/modules/3.13.5-031305-generic/updates/dkms/spl.ko
version:        0.6.2-1
license:        GPL
description:    Solaris Porting Layer
author:         Lawrence Livermore National Labs
srcversion:     9FBFCB3CE7F99B543ACB568
depends:        
vermagic:       3.13.5-031305-generic SMP mod_unload modversions 
parm:           spl_hostid:The system hostid. (ulong)
parm:           spl_hostid_path:The system hostid file (/etc/hostid) (charp)
parm:           spl_kmem_cache_expire:By age (0x1) or low memory (0x2) (uint)
parm:           spl_debug_subsys:Subsystem debugging level mask. (ulong)
parm:           spl_debug_mask:Debugging level mask. (ulong)
parm:           spl_debug_printk:Console printk level mask. (ulong)
parm:           spl_debug_mb:Total debug buffer size. (int)
parm:           spl_debug_panic_on_bug:Panic on BUG (uint)

[behlendorf]

I'm unable to reproduce this with the latest code. However, the error indicates that an allocation failed presumably due to memory being low on the node. I'd expect this to be a transient issue but I'm happy to leave it open until we can at least identify the offending allocation.

[satmandu]

How much memory do I need free to allow this operation to work?

             total       used       free     shared    buffers     cached
Mem:       6094984    1041664    5053320       1232       2020     350608
-/+ buffers/cache:     689036    5405948
Swap:       975868          0     975868

I haven't built a build with the latest code since last month, but I will if you think that might help.

[behlendorf]

It shouldn't take much at all and you clearly have more than enough free. Are you able to consistently cause this?

[satmandu]

The same error message comes up consistently. I'll recompile with master once 3.13.7 comes out and see if I get the same error again.

[satmandu]

Same issue building with master on kernel 3.13.7.

 free
             total       used       free     shared    buffers     cached
Mem:       6094984    2297396    3797588       1288       2020     595956
-/+ buffers/cache:    1699420    4395564
Swap:       975868          0     975868
:~$ sudo zpool get all tank2
NAME  PROPERTY               VALUE                  SOURCE
tank2  feature@async_destroy  disabled               local
tank2  feature@empty_bpobj    disabled               local
tank2  feature@lz4_compress   disabled               local
:~$ sudo zpool set autoexpand=on tank2
internal error: Cannot allocate memory

Build info:

:~$ /sbin/modinfo zfs ; /sbin/modinfo spl
filename:       /lib/modules/3.13.7-031307-generic/updates/dkms/zfs.ko
version:        0.6.2-1
license:        CDDL
author:         Sun Microsystems/Oracle, Lawrence Livermore National Laboratory
description:    ZFS
srcversion:     E3F39F9EC62C6A8048DA997
depends:        spl,znvpair,zcommon,zunicode,zavl
vermagic:       3.13.7-031307-generic SMP mod_unload modversions 
parm:           zvol_inhibit_dev:Do not create zvol device nodes (uint)
parm:           zvol_major:Major number for zvol device (uint)
parm:           zvol_threads:Number of threads for zvol device (uint)
parm:           zvol_max_discard_blocks:Max number of blocks to discard (ulong)
parm:           zio_injection_enabled:Enable fault injection (int)
parm:           zio_bulk_flags:Additional flags to pass to bulk buffers (int)
parm:           zio_delay_max:Max zio millisec delay before posting event (int)
parm:           zio_requeue_io_start_cut_in_line:Prioritize requeued I/O (int)
parm:           zfs_sync_pass_deferred_free:Defer frees starting in this pass (int)
parm:           zfs_sync_pass_dont_compress:Don't compress starting in this pass (int)
parm:           zfs_sync_pass_rewrite:Rewrite new bps starting in this pass (int)
parm:           zil_replay_disable:Disable intent logging replay (int)
parm:           zfs_nocacheflush:Disable cache flushes (int)
parm:           zil_slog_limit:Max commit bytes to separate log device (ulong)
parm:           zfs_read_chunk_size:Bytes to read per chunk (long)
parm:           zfs_immediate_write_sz:Largest data block to write to zil (long)
parm:           zfs_flags:Set additional debugging flags (int)
parm:           zfs_recover:Set to attempt to recover from fatal errors (int)
parm:           zfs_expire_snapshot:Seconds to expire .zfs/snapshot (int)
parm:           zfs_vdev_aggregation_limit:Max vdev I/O aggregation size (int)
parm:           zfs_vdev_read_gap_limit:Aggregate read I/O over gap (int)
parm:           zfs_vdev_write_gap_limit:Aggregate write I/O over gap (int)
parm:           zfs_vdev_max_active:Maximum number of active I/Os per vdev (int)
parm:           zfs_vdev_async_write_active_max_dirty_percent:Async write concurrency max threshold (int)
parm:           zfs_vdev_async_write_active_min_dirty_percent:Async write concurrency min threshold (int)
parm:           zfs_vdev_async_read_max_active:Max active async read I/Os per vdev (int)
parm:           zfs_vdev_async_read_min_active:Min active async read I/Os per vdev (int)
parm:           zfs_vdev_async_write_max_active:Max active async write I/Os per vdev (int)
parm:           zfs_vdev_async_write_min_active:Min active async write I/Os per vdev (int)
parm:           zfs_vdev_scrub_max_active:Max active scrub I/Os per vdev (int)
parm:           zfs_vdev_scrub_min_active:Min active scrub I/Os per vdev (int)
parm:           zfs_vdev_sync_read_max_active:Max active sync read I/Os per vdev (int)
parm:           zfs_vdev_sync_read_min_active:Min active sync read I/Os per vdev (int)
parm:           zfs_vdev_sync_write_max_active:Max active sync write I/Os per vdev (int)
parm:           zfs_vdev_sync_write_min_active:Min active sync write I/Osper vdev (int)
parm:           zfs_vdev_mirror_switch_us:Switch mirrors every N usecs (int)
parm:           zfs_vdev_scheduler:I/O scheduler (charp)
parm:           zfs_vdev_cache_max:Inflate reads small than max (int)
parm:           zfs_vdev_cache_size:Total size of the per-disk cache (int)
parm:           zfs_vdev_cache_bshift:Shift size to inflate reads too (int)
parm:           zfs_txg_timeout:Max seconds worth of delta per txg (int)
parm:           zfs_read_history:Historic statistics for the last N reads (int)
parm:           zfs_read_history_hits:Include cache hits in read history (int)
parm:           zfs_txg_history:Historic statistics for the last N txgs (int)
parm:           zfs_deadman_synctime_ms:Expiration time in milliseconds (ulong)
parm:           zfs_deadman_enabled:Enable deadman timer (int)
parm:           spa_asize_inflation:SPA size estimate multiplication factor (int)
parm:           spa_config_path:SPA config file (/etc/zfs/zpool.cache) (charp)
parm:           zfs_autoimport_disable:Disable pool import at module load (int)
parm:           metaslab_debug:keep space maps in core to verify frees (int)
parm:           zfs_zevent_len_max:Max event queue length (int)
parm:           zfs_zevent_cols:Max event column width (int)
parm:           zfs_zevent_console:Log events to the console (int)
parm:           zfs_top_maxinflight:Max I/Os per top-level (int)
parm:           zfs_resilver_delay:Number of ticks to delay resilver (int)
parm:           zfs_scrub_delay:Number of ticks to delay scrub (int)
parm:           zfs_scan_idle:Idle window in clock ticks (int)
parm:           zfs_scan_min_time_ms:Min millisecs to scrub per txg (int)
parm:           zfs_free_min_time_ms:Min millisecs to free per txg (int)
parm:           zfs_resilver_min_time_ms:Min millisecs to resilver per txg (int)
parm:           zfs_no_scrub_io:Set to disable scrub I/O (int)
parm:           zfs_no_scrub_prefetch:Set to disable scrub prefetching (int)
parm:           zfs_dirty_data_max_percent:percent of ram can be dirty (int)
parm:           zfs_dirty_data_max_max_percent:zfs_dirty_data_max upper bound as % of RAM (int)
parm:           zfs_delay_min_dirty_percent:transaction delay threshold (int)
parm:           zfs_dirty_data_max:determines the dirty space limit (ulong)
parm:           zfs_dirty_data_max_max:zfs_dirty_data_max upper bound in bytes (ulong)
parm:           zfs_dirty_data_sync:sync txg when this much dirty data (ulong)
parm:           zfs_delay_scale:how quickly delay approaches infinity (ulong)
parm:           zfs_prefetch_disable:Disable all ZFS prefetching (int)
parm:           zfetch_max_streams:Max number of streams per zfetch (uint)
parm:           zfetch_min_sec_reap:Min time before stream reclaim (uint)
parm:           zfetch_block_cap:Max number of blocks to fetch at a time (uint)
parm:           zfetch_array_rd_sz:Number of bytes in a array_read (ulong)
parm:           zfs_pd_blks_max:Max number of blocks to prefetch (int)
parm:           zfs_send_corrupt_data:Allow sending corrupt data (int)
parm:           zfs_mdcomp_disable:Disable meta data compression (int)
parm:           zfs_nopwrite_enabled:Enable NOP writes (int)
parm:           zfs_dedup_prefetch:Enable prefetching dedup-ed blks (int)
parm:           zfs_dbuf_state_index:Calculate arc header index (int)
parm:           zfs_arc_min:Min arc size (ulong)
parm:           zfs_arc_max:Max arc size (ulong)
parm:           zfs_arc_meta_limit:Meta limit for arc size (ulong)
parm:           zfs_arc_meta_prune:Bytes of meta data to prune (int)
parm:           zfs_arc_grow_retry:Seconds before growing arc size (int)
parm:           zfs_arc_p_aggressive_disable:disable aggressive arc_p grow (int)
parm:           zfs_arc_p_dampener_disable:disable arc_p adapt dampener (int)
parm:           zfs_arc_shrink_shift:log2(fraction of arc to reclaim) (int)
parm:           zfs_disable_dup_eviction:disable duplicate buffer eviction (int)
parm:           zfs_arc_memory_throttle_disable:disable memory throttle (int)
parm:           zfs_arc_min_prefetch_lifespan:Min life of prefetch block (int)
parm:           l2arc_write_max:Max write bytes per interval (ulong)
parm:           l2arc_write_boost:Extra write bytes during device warmup (ulong)
parm:           l2arc_headroom:Number of max device writes to precache (ulong)
parm:           l2arc_headroom_boost:Compressed l2arc_headroom multiplier (ulong)
parm:           l2arc_feed_secs:Seconds between L2ARC writing (ulong)
parm:           l2arc_feed_min_ms:Min feed interval in milliseconds (ulong)
parm:           l2arc_noprefetch:Skip caching prefetched buffers (int)
parm:           l2arc_nocompress:Skip compressing L2ARC buffers (int)
parm:           l2arc_feed_again:Turbo L2ARC warmup (int)
parm:           l2arc_norw:No reads during writes (int)
filename:       /lib/modules/3.13.7-031307-generic/updates/dkms/spl.ko
version:        0.6.2-1
license:        GPL
description:    Solaris Porting Layer
author:         Lawrence Livermore National Labs
srcversion:     9FBFCB3CE7F99B543ACB568
depends:        
vermagic:       3.13.7-031307-generic SMP mod_unload modversions 
parm:           spl_hostid:The system hostid. (ulong)
parm:           spl_hostid_path:The system hostid file (/etc/hostid) (charp)
parm:           spl_kmem_cache_expire:By age (0x1) or low memory (0x2) (uint)
parm:           spl_debug_subsys:Subsystem debugging level mask. (ulong)
parm:           spl_debug_mask:Debugging level mask. (ulong)
parm:           spl_debug_printk:Console printk level mask. (ulong)
parm:           spl_debug_mb:Total debug buffer size. (int)
parm:           spl_debug_panic_on_bug:Panic on BUG (uint)

[behlendorf]

@satmandu Is this still a reproducible issue with 0.6.3?

[satmandu]

I'm both using 3.16.x now and also haven't expanded disks in a while (and
have no plans to in the next 6-9 months). I think I was able to expand
disks at some point in the last six months though, so you can probably
close this for now.

On Thu, Oct 30, 2014 at 6:37 PM, Brian Behlendorf [email protected]
wrote:

@satmandu https://github.com/satmandu Is this still a reproducible
issue with 0.6.3?

—
Reply to this email directly or view it on GitHub
#2143 (comment).

[behlendorf]

Ok. Thanks.