Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ceph]Collection of logs present in /var/log/ceph folder #3809

Open
AmarnatReddy opened this issue Oct 17, 2024 · 18 comments
Open

[Ceph]Collection of logs present in /var/log/ceph folder #3809

AmarnatReddy opened this issue Oct 17, 2024 · 18 comments

Comments

@AmarnatReddy
Copy link

AmarnatReddy commented Oct 17, 2024

Collection of logs present in /var/log/ceph folder

I see below commands getting executed and outputs getting collected with respect to ceph.
Is there a option to collect ceph logs present in /var/log/ceph folder

[root@ceph1-amk-mirror-9tdhoy-node1-installer sos_commands]# ls -lrt | grep ceph
drwx------. 2 root root   21 Oct 15 04:16 ceph_common
drwx------. 2 root root 8192 Oct 15 04:16 ceph_mds
drwx------. 3 root root 4096 Oct 15 04:17 ceph_mgr
drwx------. 3 root root 4096 Oct 15 04:18 ceph_mon
[root@ceph1-amk-mirror-9tdhoy-node1-installer sos_commands]# s -lrt ceph_common/
-bash: s: command not found
[root@ceph1-amk-mirror-9tdhoy-node1-installer sos_commands]# ls -lrt ceph_common/
total 4
-rw-r--r--. 1 root root 86 Oct 15 04:10 ceph_-v
[root@ceph1-amk-mirror-9tdhoy-node1-installer sos_commands]# ls -lrt ceph_mds
total 96
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_cache_status
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_client_ls
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_config_diff
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_config_show
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_counter_dump
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_counter_schema
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_damage_ls
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_dump_loads
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_dump_tree
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_dump_blocked_ops
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_dump_historic_ops
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_dump_historic_ops_by_duration
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_dump_mempools
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_dump_ops_in_flight
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_get_subtrees
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_objecter_requests
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_ops
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_perf_dump
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_perf_histogram_dump
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_perf_histogram_schema
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_perf_schema
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_session_ls
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_status
-rw-r--r--. 1 root root 76 Oct 15 04:10 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mds.cephfs_upgrade_mirror.ceph1-amk-mirror-9tdhoy-node1-installer.dehcgu.asok_version
[root@ceph1-amk-mirror-9tdhoy-node1-installer sos_commands]# ls -lrt ceph_mgr
total 616
-rw-r--r--. 1 root root    333 Oct 15 04:10 ceph_balancer_status
-rw-r--r--. 1 root root    158 Oct 15 04:10 ceph_healthcheck_history_ls
-rw-r--r--. 1 root root   4520 Oct 15 04:10 ceph_log_last_cephadm
-rw-r--r--. 1 root root 494613 Oct 15 04:10 ceph_mgr_dump
-rw-r--r--. 1 root root   2115 Oct 15 04:10 ceph_mgr_metadata
-rw-r--r--. 1 root root   1332 Oct 15 04:10 ceph_mgr_module_ls
-rw-r--r--. 1 root root    134 Oct 15 04:10 ceph_mgr_stat
-rw-r--r--. 1 root root     99 Oct 15 04:10 ceph_mgr_versions
-rw-r--r--. 1 root root    670 Oct 15 04:10 ceph_orch_host_ls
-rw-r--r--. 1 root root   3458 Oct 15 04:11 ceph_orch_device_ls
-rw-r--r--. 1 root root    973 Oct 15 04:11 ceph_orch_ls
-rw-r--r--. 1 root root    850 Oct 15 04:11 ceph_orch_ls_--export
-rw-r--r--. 1 root root   5304 Oct 15 04:11 ceph_orch_ps
-rw-r--r--. 1 root root     64 Oct 15 04:11 ceph_orch_status_--detail
-rw-r--r--. 1 root root    175 Oct 15 04:11 ceph_orch_upgrade_status
drwx------. 2 root root   4096 Oct 15 04:11 json_output
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_config_diff
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_config_show
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_counter_dump
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_counter_schema
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_dump_cache
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_dump_mempools
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_dump_osd_network
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_mds_requests
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_mds_sessions
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_objecter_requests
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_perf_dump
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_perf_histogram_dump
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_perf_histogram_schema
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_perf_schema
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_status
-rw-r--r--. 1 root root     76 Oct 15 04:11 ceph_daemon_.var.run.ceph.935cb2b8-87e5-11ef-8bd3-fa163e370153.ceph-mgr.ceph1-amk-mirror-9tdhoy-node1-installer.kzebfr.asok_version
[root@ceph1-amk-mirror-9tdhoy-node1-installer sos_commands]# ls -lrt ceph_mon
total 5408
-rw-r--r--. 1 root root       0 Oct 15 04:10 ceph_crash_ls
-rw-r--r--. 1 root root  150365 Oct 15 04:10 ceph_report
-rw-r--r--. 1 root root    2743 Oct 15 04:10 ceph_config_dump
-rw-r--r--. 1 root root     259 Oct 15 04:10 ceph_config_generate-minimal-conf
-rw-r--r--. 1 root root    2043 Oct 15 04:10 ceph_config_log
-rw-r--r--. 1 root root      19 Oct 15 04:10 ceph_crash_stat
-rw-r--r--. 1 root root     949 Oct 15 04:10 ceph_features
-rw-r--r--. 1 root root      10 Oct 15 04:10 ceph_health_detail
-rw-r--r--. 1 root root     140 Oct 15 04:10 ceph_insights
-rw-r--r--. 1 root root 2444622 Oct 15 04:11 ceph_log_last_10000_debug_audit
-rw-r--r--. 1 root root 2090582 Oct 15 04:11 ceph_log_last_10000_debug_cluster
-rw-r--r--. 1 root root  494613 Oct 15 04:11 ceph_mgr_dump
-rw-r--r--. 1 root root    2115 Oct 15 04:11 ceph_mgr_metadata
-rw-r--r--. 1 root root    1332 Oct 15 04:11 ceph_mgr_module_ls
-rw-r--r--. 1 root root      48 Oct 15 04:11 ceph_mgr_services
-rw-r--r--. 1 root root      99 Oct 15 04:11 ceph_mgr_versions
-rw-r--r--. 1 root root     472 Oct 15 04:11 ceph_mon_stat
-rw-r--r--. 1 root root    9646 Oct 15 04:11 ceph_osd_crush_dump
-rw-r--r--. 1 root root     687 Oct 15 04:11 ceph_osd_crush_show-tunables
-rw-r--r--. 1 root root    2079 Oct 15 04:11 ceph_osd_crush_tree_--show-shadow
-rw-r--r--. 1 root root       8 Oct 15 04:11 ceph_osd_erasure-code-profile_ls
-rw-r--r--. 1 root root   32943 Oct 15 04:11 ceph_osd_metadata
-rw-r--r--. 1 root root    1763 Oct 15 04:11 ceph_quorum_status
-rw-r--r--. 1 root root     733 Oct 15 04:11 ceph_versions
-rw-r--r--. 1 root root      74 Oct 15 04:11 ceph-disk_list
-rw-r--r--. 1 root root    1144 Oct 15 04:11 ceph_device_ls
-rw-r--r--. 1 root root    1302 Oct 15 04:11 ceph_df_detail
-rw-r--r--. 1 root root     726 Oct 15 04:11 ceph_df
-rw-r--r--. 1 root root    3195 Oct 15 04:11 ceph_fs_dump
-rw-r--r--. 1 root root     211 Oct 15 04:11 ceph_fs_ls
-rw-r--r--. 1 root root     214 Oct 15 04:11 ceph_mds_stat
-rw-r--r--. 1 root root     458 Oct 15 04:11 ceph_mon_dump
-rw-r--r--. 1 root root      17 Oct 15 04:11 ceph_osd_blocked-by
-rw-r--r--. 1 root root    2683 Oct 15 04:11 ceph_osd_df_tree
-rw-r--r--. 1 root root    1535 Oct 15 04:11 ceph_osd_df
-rw-r--r--. 1 root root    4801 Oct 15 04:11 ceph_osd_dump
-rw-r--r--. 1 root root     897 Oct 15 04:11 ceph_osd_numa-status
-rw-r--r--. 1 root root     559 Oct 15 04:11 ceph_osd_perf
-rw-r--r--. 1 root root     960 Oct 15 04:11 ceph_osd_pool_autoscale-status
-rw-r--r--. 1 root root    1297 Oct 15 04:11 ceph_osd_pool_ls_detail
-rw-r--r--. 1 root root      57 Oct 15 04:11 ceph_osd_stat
-rw-r--r--. 1 root root   48518 Oct 15 04:11 ceph_pg_dump
-rw-r--r--. 1 root root      77 Oct 15 04:11 ceph_pg_stat
-rw-r--r--. 1 root root     708 Oct 15 04:11 ceph_status
-rw-r--r--. 1 root root     626 Oct 15 04:11 ceph_time-sync-status
-rw-r--r--. 1 root root   94037 Oct 15 04:12 ceph_config-key_dump
drwx------. 2 root root    4096 Oct 15 04:18 json_output
[root@ceph1-amk-mirror-9tdhoy-node1-installer sos_commands]# cat ceph_mon/ceph_health_detail 
HEALTH_OK
[root@ceph1-amk-mirror-9tdhoy-node1-installer sos_commands]# cat ceph_mon/ceph_df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd    180 GiB  178 GiB  2.3 GiB   2.3 GiB       1.30
TOTAL  180 GiB  178 GiB  2.3 GiB   2.3 GiB       1.30
 
--- POOLS ---
POOL                               ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr                                1    1  449 KiB        2  1.3 MiB      0     56 GiB
cephfs.cephfs.meta                  2   16   10 KiB       22  118 KiB      0     56 GiB
cephfs.cephfs.data                  3   32      0 B        0      0 B      0     56 GiB
cephfs.cephfs_upgrade_mirror.meta   4   16   22 MiB       31   67 MiB   0.04     56 GiB
cephfs.cephfs_upgrade_mirror.data   5   32  500 MiB      138  1.5 GiB   0.87     56 GiB
[root@ceph1-amk-mirror-9tdhoy-node1-installer sos_commands]# 
@jcastill
Copy link
Member

Hi @AmarnatReddy

Most of the ceph_ plugins, if not all, attempt to capture files in /var/log/ceph. It looks like you are checking in the plugin captures inside sos_commands, but the /var/log/ceph dir should be in the parent directory. Can you check if it was written there in your case?

@AmarnatReddy
Copy link
Author

AmarnatReddy commented Oct 17, 2024

Thanks Jose Castillo
Logs are getting collected

total 24
drwxr-xr-x.   4 root root   30 Aug 27 07:57 usr
drwxr-xr-x.   5 root root   41 Oct 17 08:49 var
lrwxrwxrwx.   1 root root    7 Oct 17 09:02 lib -> usr/lib
dr-xr-xr-x. 137 root root 4096 Oct 17 09:05 proc
dr-xr-xr-x.  10 root root  112 Oct 17 09:05 sys
dr-xr-xr-x.   5 root root   87 Oct 17 09:05 boot
drwxr-xr-x.  55 root root 4096 Oct 17 10:17 etc
lrwxrwxrwx.   1 root root   29 Oct 17 10:19 mount -> sos_commands/filesys/mount_-l
lrwxrwxrwx.   1 root root   37 Oct 17 10:19 df -> sos_commands/filesys/df_-al_-x_autofs
lrwxrwxrwx.   1 root root   31 Oct 17 10:19 dmidecode -> sos_commands/hardware/dmidecode
lrwxrwxrwx.   1 root root   26 Oct 17 10:19 hostname -> sos_commands/host/hostname
lrwxrwxrwx.   1 root root   24 Oct 17 10:19 uptime -> sos_commands/host/uptime
lrwxrwxrwx.   1 root root   48 Oct 17 10:19 root-symlinks -> sos_commands/host/find_._-maxdepth_2_-type_l_-ls
lrwxrwxrwx.   1 root root   28 Oct 17 10:20 uname -> sos_commands/kernel/uname_-a
lrwxrwxrwx.   1 root root   25 Oct 17 10:20 lsmod -> sos_commands/kernel/lsmod
lrwxrwxrwx.   1 root root   23 Oct 17 10:20 last -> sos_commands/login/last
lrwxrwxrwx.   1 root root   90 Oct 17 10:20 vgdisplay -> sos_commands/lvm2/vgdisplay_-vv_--config_global_metadata_read_only_1_--nolocking_--foreign
lrwxrwxrwx.   1 root root   24 Oct 17 10:20 free -> sos_commands/memory/free
lrwxrwxrwx.   1 root root   34 Oct 17 10:20 ip_addr -> sos_commands/networking/ip_-o_addr
lrwxrwxrwx.   1 root root   47 Oct 17 10:20 ip_route -> sos_commands/networking/ip_route_show_table_all
lrwxrwxrwx.   1 root root   41 Oct 17 10:20 netstat -> sos_commands/networking/netstat_-W_-neopa
lrwxrwxrwx.   1 root root   28 Oct 17 10:20 lspci -> sos_commands/pci/lspci_-nnvv
lrwxrwxrwx.   1 root root   31 Oct 17 10:20 ps -> sos_commands/process/ps_auxwwwm
lrwxrwxrwx.   1 root root   31 Oct 17 10:20 pstree -> sos_commands/process/pstree_-lp
drwxr-xr-x.  10 root root  126 Oct 17 10:20 run
lrwxrwxrwx.   1 root root   80 Oct 17 10:20 installed-rpms -> sos_commands/rpm/sh_-c_rpm_--nodigest_-qa_--qf_-59_NVRA_INSTALLTIME_date_sort_-V
lrwxrwxrwx.   1 root root   32 Oct 17 10:20 date -> sos_commands/systemd/timedatectl
drwxr-xr-x.  78 root root 4096 Oct 17 10:20 sos_commands
lrwxrwxrwx.   1 root root   24 Oct 17 10:20 rpm-Va -> sos_commands/rpm/rpm_-Va
-rw-r--r--.   1 root root  203 Oct 17 10:20 environment
-rw-r--r--.   1 root root   16 Oct 17 10:21 version.txt
drwxr-xr-x.   2 root root   35 Oct 17 10:21 sos_logs
drwxr-xr-x.   2 root root   74 Oct 17 10:21 sos_reports
[root@ceph1-mirror-up-w4tqx4-node1-installer sosreport-ceph1-mirror-up-w4tqx4-node1-installer-test-2024-10-17-skfzcab]# cd /var/
.updated  adm/      cache/    crash/    db/       empty/    ftp/      games/    kerberos/ lib/      local/    lock/     log/      mail/     nis/      opt/      preserve/ run/      spool/    tmp/      yp/       
[root@ceph1-mirror-up-w4tqx4-node1-installer sosreport-ceph1-mirror-up-w4tqx4-node1-installer-test-2024-10-17-skfzcab]# cd /var/lo
local/ lock/  log/   
[root@ceph1-mirror-up-w4tqx4-node1-installer sosreport-ceph1-mirror-up-w4tqx4-node1-installer-test-2024-10-17-skfzcab]# cd /var/log/
README                 ceph/                  cloud-init.log         dnf.log                insights-client/       maillog                qemu-ga/               spooler                tuned/
audit/                 chrony/                cron                   dnf.rpm.log            kdump.log              messages               rhsm/                  sssd/                  wtmp
btmp                   cloud-init-output.log  dnf.librepo.log        hawkey.log             lastlog                private/               secure                 tallylog               
[root@ceph1-mirror-up-w4tqx4-node1-installer sosreport-ceph1-mirror-up-w4tqx4-node1-installer-test-2024-10-17-skfzcab]# cd /var/log/ceph/
5d3c2e12-8c8a-11ef-b799-fa163e147a61/ cephadm.log                           
[root@ceph1-mirror-up-w4tqx4-node1-installer sosreport-ceph1-mirror-up-w4tqx4-node1-installer-test-2024-10-17-skfzcab]# 

@AmarnatReddy
Copy link
Author

I have used this command and it is saying the ceph plugin is not supported without that it is not collecting /var/log/ceph logs

[root@ceph1-amk-weekly-kvjk8q-node1-installer tmp]# sudo sosreport -a --all-logs -e ceph --batch
Please note the 'sosreport' command has been deprecated in favor of the new 'sos' command, E.G. 'sos report'.
Redirecting to 'sos report -a --all-logs -e ceph --batch'

sosreport (version 4.7.2)

a non-existing plugin (ceph) was specified in the command line.
[root@ceph1-amk-weekly-kvjk8q-node1-installer tmp]# 

I removed and collected all-logs. In Previous comment I checked in /var/log instead of checking inside the sosreport

[root@ceph1-amk-weekly-kvjk8q-node1-installer tmp]# cd sosreport-ceph1-amk-weekly-kvjk8q-node1-installer-2024-11-27-khiobcl
[root@ceph1-amk-weekly-kvjk8q-node1-installer sosreport-ceph1-amk-weekly-kvjk8q-node1-installer-2024-11-27-khiobcl]# ls -lrt
total 24
drwxr-xr-x.   4 root root   30 Nov 14 16:26 usr
lrwxrwxrwx.   1 root root    7 Nov 14 16:26 lib -> usr/lib
drwxr-xr-x.   5 root root   41 Nov 26 10:12 var
dr-xr-xr-x. 158 root root 4096 Nov 26 10:28 proc
dr-xr-xr-x.  10 root root  112 Nov 26 10:29 sys
dr-xr-xr-x.   5 root root   87 Nov 26 10:29 boot
drwxr-xr-x.  55 root root 4096 Nov 27 12:04 etc
lrwxrwxrwx.   1 root root   29 Nov 27 15:09 mount -> sos_commands/filesys/mount_-l
lrwxrwxrwx.   1 root root   37 Nov 27 15:09 df -> sos_commands/filesys/df_-al_-x_autofs
lrwxrwxrwx.   1 root root   31 Nov 27 15:09 dmidecode -> sos_commands/hardware/dmidecode
lrwxrwxrwx.   1 root root   26 Nov 27 15:09 hostname -> sos_commands/host/hostname
lrwxrwxrwx.   1 root root   24 Nov 27 15:09 uptime -> sos_commands/host/uptime
lrwxrwxrwx.   1 root root   48 Nov 27 15:09 root-symlinks -> sos_commands/host/find_._-maxdepth_2_-type_l_-ls
lrwxrwxrwx.   1 root root   28 Nov 27 15:09 uname -> sos_commands/kernel/uname_-a
lrwxrwxrwx.   1 root root   25 Nov 27 15:09 lsmod -> sos_commands/kernel/lsmod
lrwxrwxrwx.   1 root root   23 Nov 27 15:09 last -> sos_commands/login/last
lrwxrwxrwx.   1 root root   90 Nov 27 15:09 vgdisplay -> sos_commands/lvm2/vgdisplay_-vv_--config_global_metadata_read_only_1_--nolocking_--foreign
lrwxrwxrwx.   1 root root   24 Nov 27 15:09 free -> sos_commands/memory/free
lrwxrwxrwx.   1 root root   34 Nov 27 15:09 ip_addr -> sos_commands/networking/ip_-o_addr
lrwxrwxrwx.   1 root root   47 Nov 27 15:09 ip_route -> sos_commands/networking/ip_route_show_table_all
lrwxrwxrwx.   1 root root   41 Nov 27 15:09 netstat -> sos_commands/networking/netstat_-W_-neopa
lrwxrwxrwx.   1 root root   28 Nov 27 15:09 lspci -> sos_commands/pci/lspci_-nnvv
lrwxrwxrwx.   1 root root   31 Nov 27 15:09 ps -> sos_commands/process/ps_auxwwwm
lrwxrwxrwx.   1 root root   31 Nov 27 15:09 pstree -> sos_commands/process/pstree_-lp
lrwxrwxrwx.   1 root root   80 Nov 27 15:09 installed-rpms -> sos_commands/rpm/sh_-c_rpm_--nodigest_-qa_--qf_-59_NVRA_INSTALLTIME_date_sort_-V
drwxr-xr-x.  10 root root  126 Nov 27 15:09 run
lrwxrwxrwx.   1 root root   24 Nov 27 15:09 rpm-Va -> sos_commands/rpm/rpm_-Va
lrwxrwxrwx.   1 root root   32 Nov 27 15:09 date -> sos_commands/systemd/timedatectl
drwxr-xr-x.  78 root root 4096 Nov 27 15:09 sos_commands
-rw-r--r--.   1 root root  203 Nov 27 15:10 environment
-rw-r--r--.   1 root root   16 Nov 27 15:10 version.txt
drwxr-xr-x.   2 root root   35 Nov 27 15:10 sos_logs
drwxr-xr-x.   2 root root   74 Nov 27 15:10 sos_reports
[root@ceph1-amk-weekly-kvjk8q-node1-installer sosreport-ceph1-amk-weekly-kvjk8q-node1-installer-2024-11-27-khiobcl]# pwd
/var/tmp/sosreport-ceph1-amk-weekly-kvjk8q-node1-installer-2024-11-27-khiobcl
[root@ceph1-amk-weekly-kvjk8q-node1-installer sosreport-ceph1-amk-weekly-kvjk8q-node1-installer-2024-11-27-khiobcl]# cd var/log/
audit/                 cloud-init.log         dnf.librepo.log        dnf.rpm.log            kdump.log              messages               secure                 tuned/                 
cloud-init-output.log  cron                   dnf.log                insights-client/       maillog                rhsm/                  spooler                
[root@ceph1-amk-weekly-kvjk8q-node1-installer sosreport-ceph1-amk-weekly-kvjk8q-node1-installer-2024-11-27-khiobcl]# cd var/log/
[root@ceph1-amk-weekly-kvjk8q-node1-installer log]# ls -lrt
total 34304
-rw-------. 1 root root        0 Nov 14 16:26 maillog
-rw-------. 1 root root        0 Nov 14 16:26 spooler
drwxr-xr-x. 2 root root       23 Nov 26 10:12 tuned
drwxr-xr-x. 2 root root       43 Nov 26 10:12 rhsm
-rw-r-----. 1 root adm      9259 Nov 26 10:29 cloud-init-output.log
-rw-r-----. 1 root root   202451 Nov 26 10:29 cloud-init.log
-rw-------. 1 root root     1366 Nov 26 10:29 kdump.log
drwx------. 2 root root       33 Nov 27 06:45 insights-client
drwx------. 2 root root       42 Nov 27 11:00 audit
-rw-------. 1 root root    26423 Nov 27 15:07 secure
-rw-------. 1 root root    16833 Nov 27 15:09 cron
-rw-r--r--. 1 root root    16986 Nov 27 15:09 dnf.rpm.log
-rw-r--r--. 1 root root    54117 Nov 27 15:09 dnf.librepo.log
-rw-r--r--. 1 root root   219109 Nov 27 15:09 dnf.log
-rw-------. 1 root root 34556086 Nov 27 15:11 messages
[root@ceph1-amk-weekly-kvjk8q-node1-installer log]# 

@AmarnatReddy AmarnatReddy reopened this Nov 27, 2024
@jcastill
Copy link
Member

@AmarnatReddy you are trying to activate the ceph plugin, but it was divided into the following ones:

ceph_ansible
ceph_common
ceph_iscsi
ceph_mds
ceph_mgr
ceph_mon
ceph_osd
ceph_rgw

Can you try any of these directly, or just run 'sos report' without explicitly mentioning the plugin and see if you get the files?

@AmarnatReddy
Copy link
Author

@jcastill
I tried with this command. I am not seeing the /var/log/ceph logs getting collected

[root@ceph1-amk-weekly-kvjk8q-node1-installer log]# sosreport -a --all-logs --batch
Please note the 'sosreport' command has been deprecated in favor of the new 'sos' command, E.G. 'sos report'.
Redirecting to 'sos report -a --all-logs --batch'

sosreport (version 4.7.2)

on the server /var/log/ceph contents

[root@ceph1-amk-weekly-kvjk8q-node1-installer log]# ls -lrt /var/log/ceph/
total 1984
drwxrwx---. 2 ceph ceph      29 Nov 26 10:45 5a65c3d8-abe3-11ef-aec0-fa163ee915ac
-rw-r--r--. 1 root root 1980588 Nov 27 16:35 cephadm.log

In sosreport /var/log contents

[root@ceph1-amk-weekly-kvjk8q-node1-installer log]# ls -lrt /var/tmp/sosreport-ceph1-amk-weekly-kvjk8q-node1-installer-2024-11-27-khiobcl/var/log 
total 34304
-rw-------. 1 root root        0 Nov 14 16:26 maillog
-rw-------. 1 root root        0 Nov 14 16:26 spooler
drwxr-xr-x. 2 root root       23 Nov 26 10:12 tuned
drwxr-xr-x. 2 root root       43 Nov 26 10:12 rhsm
-rw-r-----. 1 root adm      9259 Nov 26 10:29 cloud-init-output.log
-rw-r-----. 1 root root   202451 Nov 26 10:29 cloud-init.log
-rw-------. 1 root root     1366 Nov 26 10:29 kdump.log
drwx------. 2 root root       33 Nov 27 06:45 insights-client
drwx------. 2 root root       42 Nov 27 11:00 audit
-rw-------. 1 root root    26423 Nov 27 15:07 secure
-rw-------. 1 root root    16833 Nov 27 15:09 cron
-rw-r--r--. 1 root root    16986 Nov 27 15:09 dnf.rpm.log
-rw-r--r--. 1 root root    54117 Nov 27 15:09 dnf.librepo.log
-rw-r--r--. 1 root root   219109 Nov 27 15:09 dnf.log
-rw-------. 1 root root 34556086 Nov 27 15:11 messages

@jcastill
Copy link
Member

That's really strange. What do you get if you run:

grep ceph /var/tmp/sosreport-ceph1-amk-weekly-kvjk8q-node1-installer-2024-11-27-khiobcl/sos_logs/sos.log

@AmarnatReddy
Copy link
Author

AmarnatReddy commented Nov 27, 2024

It's big output. I trimmed for quick reference. Attaching complete log

sos.log

Log Reference:

[root@ceph1-amk-weekly-kvjk8q-node1-installer log]# grep ceph /var/tmp/sosreport-ceph1-amk-weekly-kvjk8q-node1-installer-2024-11-27-khiobcl/sos_logs/sos.log
2024-11-27 15:09:03,682 INFO: [archive:TarFileArchive] initialised empty FileCacheArchive at '/var/tmp/sos.lhl3hc4u/sosreport-ceph1-amk-weekly-kvjk8q-node1-installer-2024-11-27-khiobcl'
2024-11-27 15:09:03,731 INFO: [plugin:ceph_common] adding forbidden path '/etc/ceph/*keyring*'
2024-11-27 15:09:03,731 INFO: [plugin:ceph_common] adding forbidden path '/var/lib/ceph/*keyring*'
2024-11-27 15:09:03,731 INFO: [plugin:ceph_common] adding forbidden path '/var/lib/ceph/*/*keyring*'
2024-11-27 15:09:03,731 INFO: [plugin:ceph_common] adding forbidden path '/var/lib/ceph/*/*/*keyring*'
2024-11-27 15:09:03,731 INFO: [plugin:ceph_common] adding forbidden path '/var/lib/ceph/osd'
2024-11-27 15:09:03,731 INFO: [plugin:ceph_common] adding forbidden path '/var/lib/ceph/mon'
2024-11-27 15:09:03,731 INFO: [plugin:ceph_common] adding forbidden path '/var/lib/ceph/tmp/*mnt*'
2024-11-27 15:09:03,731 INFO: [plugin:ceph_common] adding forbidden path '/etc/ceph/*bindpass*'
2024-11-27 15:09:03,732 INFO: [plugin:ceph_common] added cmd output 'ceph -v'
2024-11-27 15:09:03,733 INFO: [plugin:ceph_mds] adding forbidden path '/etc/ceph/*keyring*'
2024-11-27 15:09:03,733 INFO: [plugin:ceph_mds] adding forbidden path '/var/lib/ceph/*keyring*'
2024-11-27 15:09:03,733 INFO: [plugin:ceph_mds] adding forbidden path '/var/lib/ceph/*/*keyring*'
2024-11-27 15:09:03,733 INFO: [plugin:ceph_mds] adding forbidden path '/var/lib/ceph/*/*/*keyring*'
2024-11-27 15:09:03,733 INFO: [plugin:ceph_mds] adding forbidden path '/var/lib/ceph/osd'
2024-11-27 15:09:03,733 INFO: [plugin:ceph_mds] adding forbidden path '/var/lib/ceph/mon'
2024-11-27 15:09:03,733 INFO: [plugin:ceph_mds] adding forbidden path '/var/lib/ceph/tmp/*mnt*'
2024-11-27 15:09:03,733 INFO: [plugin:ceph_mds] adding forbidden path '/etc/ceph/*bindpass*'
2024-11-27 15:09:03,801 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok cache status'
2024-11-27 15:09:03,801 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok client ls'
2024-11-27 15:09:03,801 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok config diff'
2024-11-27 15:09:03,801 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok config show'
2024-11-27 15:09:03,801 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok counter dump'
2024-11-27 15:09:03,801 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok counter schema'
2024-11-27 15:09:03,801 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok damage ls'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok dump loads'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok dump tree'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok dump_blocked_ops'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok dump_historic_ops'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok dump_historic_ops_by_duration'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok dump_mempools'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok dump_ops_in_flight'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok get subtrees'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok objecter_requests'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok ops'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok perf dump'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok perf histogram dump'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok perf histogram schema'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok perf schema'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok session ls'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok status'
2024-11-27 15:09:03,802 INFO: [plugin:ceph_mds] added cmd output 'ceph daemon /var/run/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ceph-mds.cephfs-spec.ceph1-amk-weekly-kvjk8q-node1-installer.cimfaj.asok version'
2024-11-27 15:09:04,172 INFO: [plugin:ceph_mgr] adding forbidden path '/etc/ceph/*keyring*'
2024-11-27 15:09:04,172 INFO: [plugin:ceph_mgr] adding forbidden path '/var/lib/ceph/**/*keyring*'
2024-11-27 15:09:04,172 INFO: [plugin:ceph_mgr] adding forbidden path '/var/lib/ceph/**/osd*'
2024-11-27 15:09:04,172 INFO: [plugin:ceph_mgr] adding forbidden path '/var/lib/ceph/**/mon*'
2024-11-27 15:09:04,172 INFO: [plugin:ceph_mgr] adding forbidden path '/var/lib/ceph/**/tmp/*mnt*'
2024-11-27 15:09:04,172 INFO: [plugin:ceph_mgr] adding forbidden path '/etc/ceph/*bindpass*'

@AmarnatReddy
Copy link
Author

Hi @jcastill

Is there anything I am missing

Regards,
amarnath

@jcastill
Copy link
Member

jcastill commented Dec 2, 2024

@AmarnatReddy I'm checking the logs now, and I'll let you know what I find soon.

@jcastill
Copy link
Member

jcastill commented Dec 2, 2024

If I recall correctly my Ceph knowledge, you have one fsid, 5a65c3d8-abe3-11ef-aec0-fa163ee915ac. So SoS should have gathered /var/log/ceph/5a65c3d8-abe3-11ef-aec0-fa163ee915ac/ and then inside that, the following files:

"/var/log/ceph/**/*ceph-mon*.log*"
"/var/log/ceph/**/ceph-mgr*.log*", 
"/var/log/ceph/ceph-mds*.log*"
"/var/log/ceph/**/ceph.log*",

Because the plugins ceph_mon, ceph_mgr, ceph_mds, and ceph_common were ran. So it's really interesting that it didn't capture that specific folder at all. Other plugins did manage to capture things from /var/log, i.e.:

2024-11-27 15:09:14,608 INFO: [plugin:auditd] collecting path '/var/log/audit/audit.log.1'
2024-11-27 15:09:14,626 INFO: [plugin:auditd] collecting path '/var/log/audit/audit.log'
2024-11-27 15:09:18,221 INFO: [plugin:cloud_init] collecting path '/var/log/cloud-init.log'
2024-11-27 15:09:18,223 INFO: [plugin:cloud_init] collecting path '/var/log/cloud-init-output.log'
2024-11-27 15:09:19,627 INFO: [plugin:cron] collecting path '/var/log/cron'
2024-11-27 15:09:20,252 INFO: [plugin:dnf] collecting path '/var/log/dnf.rpm.log'
2024-11-27 15:09:20,253 INFO: [plugin:dnf] collecting path '/var/log/dnf.log'
2024-11-27 15:09:20,254 INFO: [plugin:dnf] collecting path '/var/log/dnf.librepo.log'
2024-11-27 15:09:31,690 INFO: [plugin:insights] collecting path '/var/log/insights-client/insights-client.log'
2024-11-27 15:09:32,912 INFO: [plugin:kdump] collecting path '/var/log/kdump.log'
2024-11-27 15:09:34,870 INFO: [plugin:logs] collecting path '/var/log/spooler'
2024-11-27 15:09:34,870 INFO: [plugin:logs] collecting path '/var/log/secure'
2024-11-27 15:09:34,871 INFO: [plugin:logs] collecting path '/var/log/messages'
2024-11-27 15:09:34,902 INFO: [plugin:logs] collecting path '/var/log/maillog'
2024-11-27 15:09:34,902 INFO: [plugin:logs] collecting path '/var/log/cron'
2024-11-27 15:09:35,100 INFO: [plugin:logs] collecting output of 'ls -alRh /var/log/'
2024-11-27 15:09:44,688 INFO: [plugin:subscription_manager] collecting path '/var/log/rhsm/rhsmcertd.log'
2024-11-27 15:09:44,688 INFO: [plugin:subscription_manager] collecting path '/var/log/rhsm/rhsm.log'
2024-11-27 15:09:50,520 INFO: [plugin:tuned] collecting path '/var/log/tuned/tuned.log'

I can see that the 'logs' plugin ran and captured the following:

2024-11-27 15:09:14,002 INFO: [plugin:logs] added cmd output 'ls -alRh /var/log/'

Could you attach that file to this issue?

@AmarnatReddy
Copy link
Author

Hi @jcastill
Please find the output of the file

[root@ceph-amk-weekly-mefi8y-node1-installer logs]# cat ls_-alZR_.var.log
/var/log:
total 48760
drwxr-xr-x. 11 root   root   system_u:object_r:var_log_t:s0                     4096 Dec  2 14:25 .
drwxr-xr-x. 19 root   root   system_u:object_r:var_t:s0                         4096 Dec  2 14:11 ..
lrwxrwxrwx.  1 root   root   system_u:object_r:var_log_t:s0                       39 Nov 26 04:41 README -> ../../usr/share/doc/systemd/README.logs
drwx------.  2 root   root   system_u:object_r:auditd_log_t:s0                    23 Dec  2 14:11 audit
-rw-rw----.  1 root   utmp   system_u:object_r:faillog_t:s0                        0 Nov 26 04:41 btmp
drwxr-xr-x.  3 root   root   unconfined_u:object_r:var_log_t:s0                   69 Dec  2 14:26 ceph
drwxr-x---.  2 chrony chrony system_u:object_r:chronyd_var_log_t:s0                6 Jan 23  2024 chrony
-rw-r-----.  1 root   adm    system_u:object_r:cloud_log_t:s0                  18828 Dec  2 16:07 cloud-init-output.log
-rw-r-----.  1 root   root   system_u:object_r:cloud_log_t:s0                 437266 Dec  2 16:07 cloud-init.log
-rw-------.  1 root   root   system_u:object_r:cron_log_t:s0                   14979 Dec  3 13:36 cron
-rw-r--r--.  1 root   root   unconfined_u:object_r:rpm_log_t:s0                32434 Dec  3 13:36 dnf.librepo.log
-rw-r--r--.  1 root   root   unconfined_u:object_r:rpm_log_t:s0               101322 Dec  3 13:36 dnf.log
-rw-r--r--.  1 root   root   unconfined_u:object_r:rpm_log_t:s0                11086 Dec  3 13:36 dnf.rpm.log
-rw-r--r--.  1 root   root   system_u:object_r:rpm_log_t:s0                     1920 Dec  3 13:36 hawkey.log
drwx------.  2 root   root   system_u:object_r:insights_client_var_log_t:s0       33 Dec  3 13:36 insights-client
-rw-------.  1 root   root   system_u:object_r:var_log_t:s0                     3415 Dec  2 16:07 kdump.log
-rw-rw-r--.  1 root   utmp   system_u:object_r:lastlog_t:s0                   292292 Dec  3 13:35 lastlog
-rw-------.  1 root   root   system_u:object_r:var_log_t:s0                        0 Nov 26 04:41 maillog
-rw-------.  1 root   root   system_u:object_r:var_log_t:s0                 49206247 Dec  3 13:36 messages
drwx------.  2 root   root   system_u:object_r:var_log_t:s0                        6 Nov 26 04:41 private
drwxr-xr-x.  2 root   root   system_u:object_r:virt_qemu_ga_log_t:s0               6 Oct  1 07:42 qemu-ga
drwxr-xr-x.  2 root   root   system_u:object_r:rhsmcertd_log_t:s0                 43 Dec  2 14:11 rhsm
-rw-------.  1 root   root   system_u:object_r:var_log_t:s0                    23783 Dec  3 13:35 secure
-rw-------.  1 root   root   system_u:object_r:var_log_t:s0                        0 Nov 26 04:41 spooler
drwxr-x---.  2 sssd   sssd   system_u:object_r:sssd_var_log_t:s0                   6 Nov 13 06:23 sssd
-rw-------.  1 root   root   system_u:object_r:faillog_t:s0                        0 Nov 26 04:41 tallylog
drwxr-xr-x.  2 root   root   system_u:object_r:tuned_log_t:s0                     23 Dec  2 14:11 tuned
-rw-rw-r--.  1 root   utmp   system_u:object_r:wtmp_t:s0                       13440 Dec  3 13:35 wtmp

/var/log/audit:
total 7888
drwx------.  2 root root system_u:object_r:auditd_log_t:s0      23 Dec  2 14:11 .
drwxr-xr-x. 11 root root system_u:object_r:var_log_t:s0       4096 Dec  2 14:25 ..
-rw-------.  1 root root system_u:object_r:auditd_log_t:s0 8067293 Dec  3 13:36 audit.log

/var/log/ceph:
total 1604
drwxr-xr-x.  3 root root unconfined_u:object_r:var_log_t:s0        69 Dec  2 14:26 .
drwxr-xr-x. 11 root root system_u:object_r:var_log_t:s0          4096 Dec  2 14:25 ..
drwxrwx---.  2  167  167 system_u:object_r:container_file_t:s0     29 Dec  2 14:26 5a271a4a-b0b9-11ef-9e13-fa163e79593f
-rw-r--r--.  1 root root unconfined_u:object_r:var_log_t:s0    627496 Dec  3 13:36 cephadm.log

/var/log/ceph/5a271a4a-b0b9-11ef-9e13-fa163e79593f:
total 1220
drwxrwx---. 2  167  167 system_u:object_r:container_file_t:s0          29 Dec  2 14:26 .
drwxr-xr-x. 3 root root unconfined_u:object_r:var_log_t:s0             69 Dec  2 14:26 ..
-rw-r--r--. 1 root root unconfined_u:object_r:container_file_t:s0 1247709 Dec  3 13:28 ceph-volume.log

/var/log/chrony:
total 4
drwxr-x---.  2 chrony chrony system_u:object_r:chronyd_var_log_t:s0    6 Jan 23  2024 .
drwxr-xr-x. 11 root   root   system_u:object_r:var_log_t:s0         4096 Dec  2 14:25 ..

/var/log/insights-client:
total 12
drwx------.  2 root root system_u:object_r:insights_client_var_log_t:s0       33 Dec  3 13:36 .
drwxr-xr-x. 11 root root system_u:object_r:var_log_t:s0                     4096 Dec  2 14:25 ..
-rw-------.  1 root root unconfined_u:object_r:insights_client_var_log_t:s0 4140 Dec  3 13:36 insights-client.log

/var/log/private:
total 4
drwx------.  2 root root system_u:object_r:var_log_t:s0    6 Nov 26 04:41 .
drwxr-xr-x. 11 root root system_u:object_r:var_log_t:s0 4096 Dec  2 14:25 ..

/var/log/qemu-ga:
total 4
drwxr-xr-x.  2 root root system_u:object_r:virt_qemu_ga_log_t:s0    6 Oct  1 07:42 .
drwxr-xr-x. 11 root root system_u:object_r:var_log_t:s0          4096 Dec  2 14:25 ..

/var/log/rhsm:
total 12
drwxr-xr-x.  2 root root system_u:object_r:rhsmcertd_log_t:s0   43 Dec  2 14:11 .
drwxr-xr-x. 11 root root system_u:object_r:var_log_t:s0       4096 Dec  2 14:25 ..
-rw-r--r--.  1 root root system_u:object_r:rhsmcertd_log_t:s0 3671 Dec  3 10:57 rhsm.log
-rw-r--r--.  1 root root system_u:object_r:rhsmcertd_log_t:s0 3192 Dec  3 10:57 rhsmcertd.log

/var/log/sssd:
total 4
drwxr-x---.  2 sssd sssd system_u:object_r:sssd_var_log_t:s0    6 Nov 13 06:23 .
drwxr-xr-x. 11 root root system_u:object_r:var_log_t:s0      4096 Dec  2 14:25 ..

/var/log/tuned:
total 16
drwxr-xr-x.  2 root root system_u:object_r:tuned_log_t:s0   23 Dec  2 14:11 .
drwxr-xr-x. 11 root root system_u:object_r:var_log_t:s0   4096 Dec  2 14:25 ..
-rw-r--r--.  1 root root system_u:object_r:tuned_log_t:s0 8510 Dec  2 16:07 tuned.log
[root@ceph-amk-weekly-mefi8y-node1-installer logs]# pwd
/var/tmp/sosreport-ceph-amk-weekly-mefi8y-node1-installer-2024-12-03-cnryxro/sos_commands/logs

@jcastill
Copy link
Member

jcastill commented Dec 4, 2024

@AmarnatReddy I think I can see now what the problem is. See the file that we have inside the fsid's /var/log/ dir:

/var/log/ceph/5a271a4a-b0b9-11ef-9e13-fa163e79593f:
total 1220
drwxrwx---. 2  167  167 system_u:object_r:container_file_t:s0          29 Dec  2 14:26 .
drwxr-xr-x. 3 root root unconfined_u:object_r:var_log_t:s0             69 Dec  2 14:26 ..
-rw-r--r--. 1 root root unconfined_u:object_r:container_file_t:s0 1247709 Dec  3 13:28 ceph-volume.log

That specific log file is only gathered in the ceph_osd plugin:

sos/report/plugins/ceph_osd.py: "/var/log/ceph/**/ceph-volume*.log",

But the ceph_osd plugin didn't run. These are the conditions that should activate the plugin:

    containers = ('ceph-(.*-)?osd.*',)
    files = ('/var/lib/ceph/osd/*', '/var/lib/ceph/*/osd*',
             '/var/snap/microceph/common/data/osd/*')

Are there any of these files present, or are there any container running with the format described in the line containers? I imagine there's an OSD container running because it's a container file according to SELinux context, and also because it seems that the file has been updated very recently.

@AmarnatReddy
Copy link
Author

AmarnatReddy commented Dec 4, 2024

Hi @jcastill ,

You are correct. I executed the sosreport -a --all-logs --batch command on the node where the OSD is running, and the logs are being collected successfully.

However, we are still facing an issue where the cephadm.log from the /var/log/ceph folder is not being included in the collected logs.

on the node

[root@ceph-amk-weekly-mefi8y-node3 sosreport-ceph-amk-weekly-mefi8y-node3-2024-12-04-wyzyfob]# ls -lrt /var/log/ceph/
total 268
-rw-r--r--. 1 root root  47224 Dec  3 23:59 cephadm.log-20241204.gz
drwxrwx---. 2  167  167   4096 Dec  4 00:00 5a271a4a-b0b9-11ef-9e13-fa163e79593f
-rw-r--r--. 1 root root 220617 Dec  4 13:11 cephadm.log

In Sos report :

[root@ceph-amk-weekly-mefi8y-node3 sosreport-ceph-amk-weekly-mefi8y-node3-2024-12-04-wyzyfob]# ls -lrt /var/tmp/sosreport-ceph-amk-weekly-mefi8y-node3-2024-12-04-wyzyfob/var/log/ceph/
total 4
drwxrwx---. 2 root root 4096 Dec  4 00:00 5a271a4a-b0b9-11ef-9e13-fa163e79593f

We are properly see other logs

[root@ceph-amk-weekly-mefi8y-node3 sosreport-ceph-amk-weekly-mefi8y-node3-2024-12-04-wyzyfob]# ls -lrt /var/tmp/sosreport-ceph-amk-weekly-mefi8y-node3-2024-12-04-wyzyfob/var/log/ceph/5a271a4a-b0b9-11ef-9e13-fa163e79593f/
total 2612
-rw-r--r--. 1  167  167   29894 Dec  2 14:34 ceph-osd.0.log-20241204.gz
-rw-r--r--. 1  167  167   30150 Dec  2 14:34 ceph-osd.3.log-20241204.gz
-rw-r--r--. 1  167  167   30548 Dec  2 14:34 ceph-osd.6.log-20241204.gz
-rw-r--r--. 1  167  167   35018 Dec  2 14:34 ceph-osd.9.log-20241204.gz
-rw-r--r--. 1 root root  484778 Dec  3 23:42 ceph-volume.log-20241204.gz
-rw-r--r--. 1  167  167       0 Dec  4 00:00 ceph-osd.9.log
-rw-r--r--. 1  167  167       0 Dec  4 00:00 ceph-osd.6.log
-rw-r--r--. 1  167  167       0 Dec  4 00:00 ceph-osd.3.log
-rw-r--r--. 1  167  167       0 Dec  4 00:00 ceph-osd.0.log
-rw-r--r--. 1 root root 2050958 Dec  4 12:58 ceph-volume.log
[root@ceph-amk-weekly-mefi8y-node3 sosreport-ceph-amk-weekly-mefi8y-node3-2024-12-04-wyzyfob]# ls -lrt /var/log/ceph/5a271a4a-b0b9-11ef-9e13-fa163e79593f/
total 2612
-rw-r--r--. 1  167  167   29894 Dec  2 14:34 ceph-osd.0.log-20241204.gz
-rw-r--r--. 1  167  167   30150 Dec  2 14:34 ceph-osd.3.log-20241204.gz
-rw-r--r--. 1  167  167   30548 Dec  2 14:34 ceph-osd.6.log-20241204.gz
-rw-r--r--. 1  167  167   35018 Dec  2 14:34 ceph-osd.9.log-20241204.gz
-rw-r--r--. 1 root root  484778 Dec  3 23:42 ceph-volume.log-20241204.gz
-rw-r--r--. 1  167  167       0 Dec  4 00:00 ceph-osd.9.log
-rw-r--r--. 1  167  167       0 Dec  4 00:00 ceph-osd.6.log
-rw-r--r--. 1  167  167       0 Dec  4 00:00 ceph-osd.3.log
-rw-r--r--. 1  167  167       0 Dec  4 00:00 ceph-osd.0.log
-rw-r--r--. 1 root root 2050958 Dec  4 12:58 ceph-volume.log

@jcastill
Copy link
Member

jcastill commented Dec 4, 2024

@AmarnatReddy you are right about /var/log/ceph/cephadm.log . I have to check with the ceph engineering team for another issue, so I'll bring this up, and I'll prepare a PR to collect this file with their feedback.

@AmarnatReddy
Copy link
Author

Thanks @jcastill for looking into the issue

jcastill added a commit to jcastill/sos that referenced this issue Dec 5, 2024
Capture log files /var/log/ceph/cephadm.log
via the ceph_common plugin.

Related: sosreport#3809

Signed-off-by: Jose Castillo <[email protected]>
@jcastill
Copy link
Member

jcastill commented Dec 5, 2024

@AmarnatReddy thank you for reporting the issue and being so patient while we investigated!
I've raised a PR after speaking with engineering about what's the best plugin to gather this information.

@AmarnatReddy
Copy link
Author

Hi @jcastill,

Thank you for your support.
I hope the PR also addresses collecting logs from nodes that do not have OSD daemons running. In Ceph, it is not mandatory for cluster nodes to have OSD daemons running.

Best regards,
Amarnath

@jcastill
Copy link
Member

jcastill commented Dec 5, 2024

@AmarnatReddy the plugin ceph_common should run in all ceph nodes, as far as I know, so the case of no OSD daemons should be covered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants