removing a disk

[JeremyMarshall]

Issue report

What version of MicroCeph are you using ?

ceph-version: 18.2.4-0ubuntu1~cloud0; microceph-git: c9f2b08
ubuntu running on raspberry pi cluster (2x4, 1x5)

Use this section to describe the channel/revision which produces the unexpected behaviour.
This information can be fetched from the installed: section of sudo snap info microceph output.

from snap: installed: 18.2.4+snapc9f2b08f92 (1140) 93MB held

What are the steps to reproduce this issue ?

1. trying to remove an osd

What happens (observed behaviour) ?

root@pi4:/etc/systemd/system# systemctl restart snap.microceph.osd.service
root@pi4:/etc/systemd/system# microceph disk remove osd.4

What were you expecting to happen ?

remove osd.4

Relevant logs, error output, etc.

Removing osd.4, timeout 1800s
Error: failed to get disks: failed listing disks: Get "http://control.socket/1.0/disks": dial unix /var/snap/microceph/common/state/control.socket: connect: no such file or directory
root@pi4:/etc/systemd/system# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME      STATUS  REWEIGHT  PRI-AFF
-1         0.45177  root default                            
-3         0.11189      host pi4                            
 2    hdd        0          osd.2      up   1.00000  1.00000
 4    hdd  0.11189          osd.4      up   1.00000  1.00000
-4         0.11249      host pi44                           
 6         0.11249          osd.6      up   1.00000  1.00000
-2         0.22739      host pi5                            
 1    hdd  0.22739          osd.1      up   1.00000  1.00000

If it’s considerably long, please paste to https://gist.github.com/ and insert the link here.

Additional comments.

I'm running with USB drives 3x128, 1x256
I wanted to just use 128Gb and take the 256Gb out and add 1 on each host

[ivoruetsche]

We have the same problem:

root@sc13:~# ceph osd tree
ID  CLASS  WEIGHT    TYPE NAME                 STATUS  REWEIGHT  PRI-AFF
-1         38.42397  root default                                       
-2         10.47926      host sc11                           
 1    ssd   3.49309          osd.1                 up   1.00000  1.00000
 2    ssd   3.49309          osd.2                 up   1.00000  1.00000
 3    ssd   3.49309          osd.3                 up   1.00000  1.00000
-3         10.47926      host sc12                           
 4    ssd   3.49309          osd.4                 up   1.00000  1.00000
 5    ssd   3.49309          osd.5                 up   1.00000  1.00000
 6    ssd   3.49309          osd.6                 up   1.00000  1.00000
-4          6.98618      host sc13                           
 7    ssd   3.49309          osd.7                 up   1.00000  1.00000
 8    ssd   3.49309          osd.8                 up   1.00000  1.00000
 9    ssd         0          osd.9               down         0  1.00000
-5         10.47926      host sc14                           
10    ssd   3.49309          osd.10                up   1.00000  1.00000
11    ssd   3.49309          osd.11                up   1.00000  1.00000
12    ssd   3.49309          osd.12                up   1.00000  1.00000
root@sc13:~# 

root@sc13:~# microceph disk remove osd.9 --timeout 300
Removing osd.9, timeout 300s
Error: failed to remove disk, timeout (300s) reached - abort
root@sc13:~# 

osd.9 is down and we try to remove them, but it goes in a timeout.

[ivoruetsche]

okay, some more things:

dmesg says:

[ 2817.394959] audit: type=1400 audit(1732637891.773:78): apparmor="DENIED" operation="capable" profile="/usr/lib/snapd/snap-confine" pid=32285 comm="snap-confine" capability=12  capname="net_admin"
[ 2817.395011] audit: type=1400 audit(1732637891.773:79): apparmor="DENIED" operation="capable" profile="/usr/lib/snapd/snap-confine" pid=32285 comm="snap-confine" capability=38  capname="perfmon"

I did the way via ceph:

ceph osd out osd.9     # Still out
ceph osd down osd.9    # Still down
ceph osd rm osd.9
ceph osd crush rm osd.9

And added the replaced disk: microceph disk add /dev/disk/by-path/pci-0000:06:00.0-sas-phy2-lun-0

New tree:

ID  CLASS  WEIGHT    TYPE NAME                 STATUS  REWEIGHT  PRI-AFF
-1         41.91705  root default                                       
-2         10.47926      host sc11                           
 1    ssd   3.49309          osd.1                 up   1.00000  1.00000
 2    ssd   3.49309          osd.2                 up   1.00000  1.00000
 3    ssd   3.49309          osd.3                 up   1.00000  1.00000
-3         10.47926      host sc12                           
 4    ssd   3.49309          osd.4                 up   1.00000  1.00000
 5    ssd   3.49309          osd.5                 up   1.00000  1.00000
 6    ssd   3.49309          osd.6                 up   1.00000  1.00000
-4         10.47926      host sc13                           
13          3.49309          osd.13                up   1.00000  1.00000
 7    ssd   3.49309          osd.7                 up   1.00000  1.00000
 8    ssd   3.49309          osd.8                 up   1.00000  1.00000
-5         10.47926      host sc14                           
10    ssd   3.49309          osd.10                up   1.00000  1.00000
11    ssd   3.49309          osd.11                up   1.00000  1.00000
12    ssd   3.49309          osd.12                up   1.00000  1.00000