Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

inputs.zfs misses pools without objset-* files #14993

Closed
NateCrawford opened this issue Mar 14, 2024 · 7 comments
Closed

inputs.zfs misses pools without objset-* files #14993

NateCrawford opened this issue Mar 14, 2024 · 7 comments
Labels
bug unexpected problem or unintended behavior waiting for response waiting for response from contributor

Comments

@NateCrawford
Copy link

Relevant telegraf.conf

[[inputs.zfs]]
   poolMetrics = true

Logs from Telegraf

# telegraf --test --input-filter zfs --debug
2024-03-14T18:44:46Z I! Loading config: /etc/telegraf/telegraf.conf
2024-03-14T18:44:46Z I! Loading config: /etc/telegraf/telegraf.d/node.conf
2024-03-14T18:44:46Z I! Starting Telegraf 1.29.5 brought to you by InfluxData the makers of InfluxDB
2024-03-14T18:44:46Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
2024-03-14T18:44:46Z I! Loaded inputs: zfs
2024-03-14T18:44:46Z I! Loaded aggregators: 
2024-03-14T18:44:46Z I! Loaded processors: 
2024-03-14T18:44:46Z I! Loaded secretstores: 
2024-03-14T18:44:46Z W! Outputs are not used in testing mode!
2024-03-14T18:44:46Z I! Tags enabled: host=xxx-data-2
2024-03-14T18:44:46Z D! [agent] Initializing plugins
2024-03-14T18:44:46Z D! [agent] Starting service inputs
> zfs_pool,dataset=state/var,host=xxx-data-2,pool=state nread=0i,nunlinked=0i,nunlinks=0i,nwritten=7585026i,reads=0i,writes=73745i 1710441886000000000
> zfs_pool,dataset=state,host=xxx-data-2,pool=state nread=0i,nunlinked=0i,nunlinks=0i,nwritten=0i,reads=0i,writes=0i 1710441886000000000
> zfs_pool,dataset=state/pools,host=xxx-data-2,pool=state nread=0i,nunlinked=0i,nunlinks=0i,nwritten=0i,reads=0i,writes=0i 1710441886000000000
> zfs_pool,dataset=xd2_local,host=xxx-data-2,pool=xd2_local nread=0i,nunlinked=0i,nunlinks=0i,nwritten=0i,reads=0i,writes=0i 1710441886000000000
2024-03-14T18:44:46Z D! [agent] Stopping service inputs
2024-03-14T18:44:46Z D! [agent] Input channel closed
2024-03-14T18:44:46Z D! [agent] Stopped Successfully
> zfs,host=xxx-data-2,pools=state::xd2_local abdstats_linear_cnt=239618i,abdstats_linear_data_size=414491648i,abdstats_scatter_chunk_waste=23040i,abdstats_scatter_cnt=90141i,abdstats_scatter_data_size=111183033856i,abdstats_scatter_order_0=17308i,abdstats_scatter_order_1=58698i,abdstats_scatter_order_10=11751i,abdstats_scatter_order_2=12365i,abdstats_scatter_order_3=12353i,abdstats_scatter_order_4=10698i,abdstats_scatter_order_5=10951i,abdstats_scatter_order_6=11660i,abdstats_scatter_order_7=7739i,abdstats_scatter_order_8=11091i,abdstats_scatter_order_9=19005i,abdstats_scatter_page_alloc_retry=0i,abdstats_scatter_page_multi_chunk=23215i,abdstats_scatter_page_multi_zone=13i,abdstats_scatter_sg_table_retry=0i,abdstats_struct_size=42326400i,arcstats_abd_chunk_waste_size=23040i,arcstats_access_skip=121i,arcstats_anon_evictable_data=0i,arcstats_anon_evictable_metadata=0i,arcstats_anon_size=697617408i,arcstats_arc_dnode_limit=12321187430i,arcstats_arc_loaned_bytes=0i,arcstats_arc_meta_limit=123211874304i,arcstats_arc_meta_max=11674731048i,arcstats_arc_meta_min=16777216i,arcstats_arc_meta_used=1409980024i,arcstats_arc_need_free=0i,arcstats_arc_no_grow=1i,arcstats_arc_prune=0i,arcstats_arc_raw_size=0i,arcstats_arc_sys_free=6769696384i,arcstats_arc_tempreserve=0i,arcstats_async_upgrade_sync=619602i,arcstats_bonus_size=5367040i,arcstats_c=117026034904i,arcstats_c_max=164282499072i,arcstats_c_min=6337404544i,arcstats_cached_only_in_progress=0i,arcstats_compressed_size=111305320960i,arcstats_data_size=114532958208i,arcstats_dbuf_size=9629184i,arcstats_deleted=53479113i,arcstats_demand_data_hits=95226221i,arcstats_demand_data_misses=9070243i,arcstats_demand_hit_predictive_prefetch=2309468i,arcstats_demand_hit_prescient_prefetch=0i,arcstats_demand_metadata_hits=788264277i,arcstats_demand_metadata_misses=12972253i,arcstats_dnode_size=19738000i,arcstats_evict_l2_cached=0i,arcstats_evict_l2_eligible=28279483152384i,arcstats_evict_l2_ineligible=696868043264i,arcstats_evict_l2_skip=0i,arcstats_evict_not_enough=27988i,arcstats_evict_skip=108771i,arcstats_hash_chain_max=7i,arcstats_hash_chains=33750i,arcstats_hash_collisions=28735326i,arcstats_hash_elements=1527785i,arcstats_hash_elements_max=8313137i,arcstats_hdr_size=514445800i,arcstats_hits=904555672i,arcstats_l2_abort_lowmem=0i,arcstats_l2_asize=0i,arcstats_l2_cksum_bad=0i,arcstats_l2_data_to_meta_ratio=0i,arcstats_l2_evict_l1cached=0i,arcstats_l2_evict_lock_retry=0i,arcstats_l2_evict_reading=0i,arcstats_l2_feeds=0i,arcstats_l2_free_on_write=0i,arcstats_l2_hdr_size=0i,arcstats_l2_hits=0i,arcstats_l2_io_error=0i,arcstats_l2_log_blk_asize=0i,arcstats_l2_log_blk_avg_asize=0i,arcstats_l2_log_blk_count=0i,arcstats_l2_log_blk_writes=0i,arcstats_l2_misses=0i,arcstats_l2_read_bytes=0i,arcstats_l2_rebuild_asize=0i,arcstats_l2_rebuild_bufs=0i,arcstats_l2_rebuild_bufs_precached=0i,arcstats_l2_rebuild_cksum_lb_errors=0i,arcstats_l2_rebuild_dh_errors=0i,arcstats_l2_rebuild_io_errors=0i,arcstats_l2_rebuild_log_blks=0i,arcstats_l2_rebuild_lowmem=0i,arcstats_l2_rebuild_size=0i,arcstats_l2_rebuild_success=0i,arcstats_l2_rebuild_unsupported=0i,arcstats_l2_rw_clash=0i,arcstats_l2_size=0i,arcstats_l2_write_bytes=0i,arcstats_l2_writes_done=0i,arcstats_l2_writes_error=0i,arcstats_l2_writes_lock_retry=0i,arcstats_l2_writes_sent=0i,arcstats_memory_all_bytes=202796945408i,arcstats_memory_available_bytes=63334015360i,arcstats_memory_direct_count=10i,arcstats_memory_free_bytes=70103711744i,arcstats_memory_indirect_count=5753342i,arcstats_memory_throttle_count=0i,arcstats_metadata_size=860800000i,arcstats_mfu_evictable_data=107622535168i,arcstats_mfu_evictable_metadata=561137152i,arcstats_mfu_ghost_evictable_data=2631335936i,arcstats_mfu_ghost_evictable_metadata=1729860608i,arcstats_mfu_ghost_hits=9082186i,arcstats_mfu_ghost_size=4361196544i,arcstats_mfu_hits=718516190i,arcstats_mfu_size=109088888320i,arcstats_misses=25412534i,arcstats_mru_evictable_data=1674977280i,arcstats_mru_evictable_metadata=161280i,arcstats_mru_ghost_evictable_data=105814822912i,arcstats_mru_ghost_evictable_metadata=5876673536i,arcstats_mru_ghost_hits=2664914i,arcstats_mru_ghost_size=111691496448i,arcstats_mru_hits=180435888i,arcstats_mru_size=5607252480i,arcstats_mutex_miss=4996i,arcstats_overhead_size=4088437248i,arcstats_p=7314358605i,arcstats_prefetch_data_hits=20407081i,arcstats_prefetch_data_misses=3039508i,arcstats_prefetch_metadata_hits=658093i,arcstats_prefetch_metadata_misses=330530i,arcstats_size=115942961272i,arcstats_uncompressed_size=142937707008i,dmu_tx_assigned=157695117i,dmu_tx_delay=0i,dmu_tx_dirty_delay=4613720i,dmu_tx_dirty_frees_delay=38i,dmu_tx_dirty_over_max=0i,dmu_tx_dirty_throttle=0i,dmu_tx_error=0i,dmu_tx_group=3081i,dmu_tx_memory_reclaim=0i,dmu_tx_memory_reserve=0i,dmu_tx_quota=1i,dmu_tx_suspended=0i,dnode_alloc_next_block=2i,dnode_alloc_next_chunk=4969i,dnode_alloc_race=0i,dnode_allocate=586035i,dnode_buf_evict=1250079i,dnode_free_interior_lock_retry=0i,dnode_hold_alloc_hits=551405139i,dnode_hold_alloc_interior=12474i,dnode_hold_alloc_lock_misses=7i,dnode_hold_alloc_lock_retry=1415i,dnode_hold_alloc_misses=5271246i,dnode_hold_alloc_type_none=758i,dnode_hold_dbuf_hold=0i,dnode_hold_dbuf_read=0i,dnode_hold_free_hits=586035i,dnode_hold_free_lock_misses=0i,dnode_hold_free_lock_retry=0i,dnode_hold_free_misses=8770i,dnode_hold_free_overflow=0i,dnode_hold_free_refcount=0i,dnode_move_active=0i,dnode_move_handle=0i,dnode_move_invalid=0i,dnode_move_recheck1=0i,dnode_move_recheck2=0i,dnode_move_rwlock=0i,dnode_move_special=0i,dnode_reallocate=0i,fm_erpt-dropped=0i,fm_erpt-duplicates=0i,fm_erpt-set-failed=0i,fm_fmri-set-failed=0i,fm_payload-set-failed=0i,vdev_mirror_stats_non_rotating_linear=0i,vdev_mirror_stats_non_rotating_seek=0i,vdev_mirror_stats_preferred_found=12140200i,vdev_mirror_stats_preferred_not_found=13435943i,vdev_mirror_stats_rotating_linear=552i,vdev_mirror_stats_rotating_offset=1527i,vdev_mirror_stats_rotating_seek=18317i,zfetchstats_hits=23819333i,zfetchstats_max_streams=351170710i,zfetchstats_misses=379671774i,zil_commit_count=140i,zil_commit_writer_count=140i,zil_itx_copied_bytes=0i,zil_itx_copied_count=0i,zil_itx_count=318i,zil_itx_indirect_bytes=0i,zil_itx_indirect_count=0i,zil_itx_metaslab_normal_bytes=97280i,zil_itx_metaslab_normal_count=37i,zil_itx_metaslab_slog_bytes=0i,zil_itx_metaslab_slog_count=0i,zil_itx_needcopy_bytes=29416i,zil_itx_needcopy_count=318i 1710441886000000000

System info

Telegraf 1.29.5, CentOS 7.9, ZFS 2.0.7

Docker

No response

Steps to reproduce

1.Create zpool "datapool"
2.Create a Lustre OST on datapool
3.Start Telegraf
...

Expected behavior

Telegraf should detect the zpool and gather metrics

Actual behavior

Telegraf does not detect the zpool as there are no /proc/spl/kstat/zfs/datapool/objset-* files.

Additional info

There are 9 zpools on this host:

ls -al /proc/spl/kstat/zfs
total 0
-rw-r--r-- 1 root root 0 Mar 14 11:51 abdstats
-rw-r--r-- 1 root root 0 Mar 14 11:51 arcstats
-rw------- 1 root root 0 Mar 14 11:51 dbgmsg
-rw------- 1 root root 0 Mar 14 11:51 dbufs
-rw-r--r-- 1 root root 0 Mar 14 11:51 dbufstats
-rw-r--r-- 1 root root 0 Mar 14 11:51 dmu_tx
-rw-r--r-- 1 root root 0 Mar 14 11:51 dnodestats
-rw-r--r-- 1 root root 0 Mar 14 11:51 fletcher_4_bench
-rw-r--r-- 1 root root 0 Mar 14 11:51 fm
-rw-r--r-- 1 root root 0 Mar 14 11:51 import_progress
dr-xr-xr-x 2 root root 0 Mar 14 11:51 state
-rw-r--r-- 1 root root 0 Mar 14 11:51 vdev_cache_stats
-rw-r--r-- 1 root root 0 Mar 14 11:51 vdev_mirror_stats
-rw-r--r-- 1 root root 0 Mar 14 11:51 vdev_raidz_bench
dr-xr-xr-x 2 root root 0 Mar 14 11:51 xd2_local
dr-xr-xr-x 2 root root 0 Mar 14 11:51 xdatapool-10
dr-xr-xr-x 2 root root 0 Mar 14 11:51 xdatapool-4
dr-xr-xr-x 2 root root 0 Mar 14 11:51 xdatapool-5
dr-xr-xr-x 2 root root 0 Mar 14 11:51 xdatapool-6
dr-xr-xr-x 2 root root 0 Mar 14 11:51 xdatapool-7
dr-xr-xr-x 2 root root 0 Mar 14 11:51 xdatapool-8
dr-xr-xr-x 2 root root 0 Mar 14 11:51 xdatapool-9
-rw-r--r-- 1 root root 0 Mar 14 11:51 zfetchstats
-rw-r--r-- 1 root root 0 Mar 14 11:51 zil
-rw-r--r-- 1 root root 0 Mar 14 11:51 zstd

Compare contents of xd2_local and xdatapool-10:

ls -al /proc/spl/kstat/zfs/xd2_local/
total 0
-rw-r--r-- 1 root root 0 Mar 14 11:52 dmu_tx_assign
-rw-r--r-- 1 root root 0 Mar 14 11:52 io
-rw-r--r-- 1 root root 0 Mar 14 11:52 iostats
-rw-r--r-- 1 root root 0 Mar 14 11:52 multihost
-rw-r--r-- 1 root root 0 Mar 14 11:52 objset-0x36
-rw------- 1 root root 0 Mar 14 11:52 reads
-rw-r--r-- 1 root root 0 Mar 14 11:52 state
-rw-r--r-- 1 root root 0 Mar 14 11:52 txgs
ls -al /proc/spl/kstat/zfs/xdatapool-10/
total 0
-rw-r--r-- 1 root root 0 Mar 14 11:52 dmu_tx_assign
-rw-r--r-- 1 root root 0 Mar 14 11:52 io
-rw-r--r-- 1 root root 0 Mar 14 11:52 iostats
-rw-r--r-- 1 root root 0 Mar 14 11:52 multihost
-rw------- 1 root root 0 Mar 14 11:52 reads
-rw-r--r-- 1 root root 0 Mar 14 11:52 state
-rw-r--r-- 1 root root 0 Mar 14 11:52 txgs

The objset files contain data for datasets created in a zpool, but the actual pool-level data (that I actually expect) appears to be in the "io" file:

cat /proc/spl/kstat/zfs/xd2_local/objset-0x36 
42 1 0x01 7 2160 93785836211 503972742241224
name                            type data
dataset_name                    7    xd2_local
writes                          4    0
nwritten                        4    0
reads                           4    0
nread                           4    0
nunlinks                        4    0
nunlinked                       4    0
cat /proc/spl/kstat/zfs/xd2_local/io
36 3 0x00 1 80 92397323513 503979162802010
nread    nwritten reads    writes   wtime    wlentime wupdate  rtime    rlentime rupdate  wcnt     rcnt    
11411456 13176832 467      950      627146472 49573101201 98866985045 1454011882 16227255491 98889148419 0        0    
cat /proc/spl/kstat/zfs/xdatapool-10/io
160 3 0x00 1 80 573385083392 504110985159262
nread    nwritten reads    writes   wtime    wlentime wupdate  rtime    rlentime rupdate  wcnt     rcnt    
1589067370496 2200713056256 3535887  115970674 64322053319642 8231076869550696 504110907933290 115546377460569 4017615674111110 504110956877959 0        0

It is likely that the dataset created by Lustre lacks some feature that would require an objset file.

@NateCrawford NateCrawford added the bug unexpected problem or unintended behavior label Mar 14, 2024
@powersj
Copy link
Contributor

powersj commented Mar 14, 2024

Hi,

Looking at the zfs_linux.go, it will fallback to io, but only if nothing is in objset-*. However, am I understanding correctly, that in your case you have both?

Is there some heuristic we can use to determine which one to read from other than one has all zeros? As the plugin treats the files under objset-* with higher priority currently.

@NateCrawford
Copy link
Author

In my case, the normal-use pools contain an io file for the entire pool, and objset files for each contained dataset, while the lustre-use pools only have the io file.

To complicate matters, the io file data is measuring different things than the objset- files. It is also removed in zfs >= 2.1 ( #openzfs/zfs#13810 better describes the problem). This also affects Prometheus: #prometheus/node_exporter#2068.

I'll poke around the lustre and zfs groups to see why there is no objset created, as having one would likely solve this problem. Otherwise, periodically parsing the output of zpool iostat would work, but is annoying.

@powersj
Copy link
Contributor

powersj commented Mar 14, 2024

It is also removed in zfs >= 2.1

Yeah the comments in the code and that thread seem to indicate that the io method is the old/deprecated, hence why it is the fallback and not default.

Happy to see if we can figure something out, but it does seem like the lustre pool creation (?) is doing something that is perhaps the "old" or legacy way?

let us know what you find out or if you think of a way to determine which way to parse.

@NateCrawford
Copy link
Author

I think this is what's happening: the objset file for a dataset is only created when that dataset is mounted ( #openzfs/zfs#10928), but Lustre doesn't mount the object store dataset; it accesses it at a lower level.

One big difference between the old io file vs new per-dataset objset files is that the new way will not recognize non-dataset pool io. I really want to see the effects of pool scrubs and resilvering operations, which should not appear in any dataset stats. It may be simplest to grab the stats directly from the disks that make up each pool, then aggregate. Still, eek.

@NateCrawford
Copy link
Author

As a work-around to help aggregate, the device_tags option in diskio is quite handy. The pool name is included as "ID_FS_LABEL"

[[inputs.diskio]]
  devices = ["sd*"]
  device_tags = ["ID_FS_LABEL"]
  name_templates = ["$DM_NAME"]

The other options are just there as the disks are all multipath and aliased with descriptive DM names, and double-counting is an issue.

This captures the zpool scrub reading, but not reads from the ARC or effects of compression, so the pool read speed from inputs.zfs can be much higher than the raw read from disk (i think).

One caveat is that disks that are in a zpool that hasn't been imported, or disks that were once in a pool, will still have the ID_FS_LABEL set.

@powersj
Copy link
Contributor

powersj commented Mar 21, 2024

Hi @NateCrawford,

Is your workaround using diskio sufficient? Or is there anything we could do in the zfs plugin for this case?

@powersj powersj added the waiting for response waiting for response from contributor label Mar 21, 2024
@telegraf-tiger
Copy link
Contributor

telegraf-tiger bot commented Apr 4, 2024

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Forums or provide additional details in this issue and reqeust that it be re-opened. Thank you!

@telegraf-tiger telegraf-tiger bot closed this as completed Apr 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unexpected problem or unintended behavior waiting for response waiting for response from contributor
Projects
None yet
Development

No branches or pull requests

2 participants