Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

df -h reports wrong size #3953

Closed
galindro opened this issue Oct 26, 2015 · 3 comments
Closed

df -h reports wrong size #3953

galindro opened this issue Oct 26, 2015 · 3 comments

Comments

@galindro
Copy link

On this morning, I wake up from my bed, turn on my computer, and see this error message after tried to access my monitoring system:

Error connecting to database: Can't connect to MySQL server on (....)

After connect to the database server, I've see that MySQL partition is out of space. But the strage part of this is: df -h is reporting a size for my ZFS partitions and zfs list is reporting another size for them, which is correct.

root@zabbix-mysql:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root   18G  2.9G   14G  18% /
none                         4.0K     0  4.0K   0% /sys/fs/cgroup
udev                         7.9G  404K  7.9G   1% /dev
datastore/tmp                1.0M   16K 1008K   2% /tmp
tmpfs                        1.6G  632K  1.6G   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                         7.9G     0  7.9G   0% /run/shm
none                         100M     0  100M   0% /run/user
datastore/var_log             21M   21M     0 100% /var/log
datastore/mysql               47G   47G     0 100% /var/lib/mysql
overflow                     1.0M   16K 1008K   2% /tmp
/dev/sda1                    236M  157M   68M  70% /boot

root@zabbix-mysql:~# zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
datastore           496G      0    30K  legacy
datastore/mysql     496G      0  46.8G  /var/lib/mysql
datastore/tmp        22K      0    22K  /tmp
datastore/var_log  20.3M      0  20.3M  legacy

root@zabbix-mysql:~# zpool status
  pool: datastore
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: resilvered 78.5K in 0h0m with 0 errors on Mon Dec 15 13:56:11 2014
config:

    NAME                              STATE     READ WRITE CKSUM
    datastore                         ONLINE       0     0     0
      pci-0000:00:10.0-scsi-0:0:2:0   ONLINE       0     0     0
      pci-0000:00:10.0-scsi-0:0:1:0   ONLINE       0     0     0
      pci-0000:00:10.0-scsi-0:0:5:0   ONLINE       0     0     0
      pci-0000:00:10.0-scsi-0:0:3:0   ONLINE       0     0     0
      pci-0000:00:10.0-scsi-0:0:4:0   ONLINE       0     0     0
      pci-0000:00:10.0-scsi-0:0:6:0   ONLINE       0     0     0
      pci-0000:00:10.0-scsi-0:0:8:0   ONLINE       0     0     0
      pci-0000:00:10.0-scsi-0:0:9:0   ONLINE       0     0     0
      pci-0000:00:10.0-scsi-0:0:10:0  ONLINE       0     0     0
      pci-0000:00:10.0-scsi-0:0:11:0  ONLINE       0     0     0

errors: No known data errors

Unfortunatelly I don't have any logs from my system because the /var/log partition is full too. It seems that this problem ocurred right after the creation of this snapshot: datastore/mysql@daily_Fri_20151023_2000

root@zabbix-mysql:~# zfs list -t snap
NAME                                       USED  AVAIL  REFER  MOUNTPOINT
datastore@hourly_Tue_20151020_0659            0      -    30K  -
datastore/mysql@daily_Tue_20151020_2000   27.1G      -   331G  -
datastore/mysql@daily_Wed_20151021_2000   16.7G      -   332G  -
datastore/mysql@daily_Thu_20151022_2000   7.14M      -   334G  -
datastore/mysql@hourly_Fri_20151023_0720  7.51M      -   334G  -
datastore/mysql@daily_Fri_20151023_2000   23.1G      -  40.1G  -
datastore/mysql@daily_Sat_20151024_2000   15.8G      -  43.5G  -
datastore/mysql@daily_Sun_20151025_2000   8.69G      -  46.1G  -

After this, I've did a zpool upgrade, them a zpool export and zpool import and nothing solved the problem.

What is wrong??
@galindro
Copy link
Author

Right now I've excluded all of the zfs snapshots. For each exclusion, the disk size was changed:

root@zabbix-mysql:~# zfs destroy datastore/mysql@daily_Sun_20151025_2000
root@zabbix-mysql:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root   18G  2.9G   14G  18% /
none                         4.0K     0  4.0K   0% /sys/fs/cgroup
udev                         7.9G  404K  7.9G   1% /dev
tmpfs                        1.6G  628K  1.6G   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                         7.9G     0  7.9G   0% /run/shm
none                         100M     0  100M   0% /run/user
/dev/sda1                    236M  157M   68M  70% /boot
datastore/tmp                5.0G  128K  5.0G   1% /tmp
datastore/mysql               55G   47G  7.8G  86% /var/lib/mysql
root@zabbix-mysql:~# zfs destroy datastore/mysql@daily_Sat_20151024_2000
root@zabbix-mysql:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root   18G  2.9G   14G  18% /
none                         4.0K     0  4.0K   0% /sys/fs/cgroup
udev                         7.9G  404K  7.9G   1% /dev
tmpfs                        1.6G  628K  1.6G   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                         7.9G     0  7.9G   0% /run/shm
none                         100M     0  100M   0% /run/user
/dev/sda1                    236M  157M   68M  70% /boot
datastore/tmp                5.0G  128K  5.0G   1% /tmp
datastore/mysql               73G   47G   26G  65% /var/lib/mysql
root@zabbix-mysql:~# zfs destroy datastore/mysql@daily_Fri_20151023_2000
root@zabbix-mysql:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root   18G  2.9G   14G  18% /
none                         4.0K     0  4.0K   0% /sys/fs/cgroup
udev                         7.9G  404K  7.9G   1% /dev
tmpfs                        1.6G  628K  1.6G   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                         7.9G     0  7.9G   0% /run/shm
none                         100M     0  100M   0% /run/user
/dev/sda1                    236M  157M   68M  70% /boot
datastore/tmp                5.0G  128K  5.0G   1% /tmp
datastore/mysql               83G   47G   36G  57% /var/lib/mysql
root@zabbix-mysql:~# zfs destroy datastore/mysql@hourly_Fri_20151023_0720
root@zabbix-mysql:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root   18G  2.9G   14G  18% /
none                         4.0K     0  4.0K   0% /sys/fs/cgroup
udev                         7.9G  404K  7.9G   1% /dev
tmpfs                        1.6G  628K  1.6G   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                         7.9G     0  7.9G   0% /run/shm
none                         100M     0  100M   0% /run/user
/dev/sda1                    236M  157M   68M  70% /boot
datastore/tmp                5.0G  128K  5.0G   1% /tmp
datastore/mysql              101G   47G   54G  47% /var/lib/mysql
root@zabbix-mysql:~# zfs destroy datastore/mysql@daily_Thu_20151022_2000
root@zabbix-mysql:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root   18G  2.9G   14G  18% /
none                         4.0K     0  4.0K   0% /sys/fs/cgroup
udev                         7.9G  404K  7.9G   1% /dev
tmpfs                        1.6G  628K  1.6G   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                         7.9G     0  7.9G   0% /run/shm
none                         100M     0  100M   0% /run/user
/dev/sda1                    236M  157M   68M  70% /boot
datastore/tmp                5.0G  128K  5.0G   1% /tmp
datastore/mysql              108G   47G   61G  44% /var/lib/mysql
root@zabbix-mysql:~# zfs destroy  datastore/mysql@daily_Wed_20151021_2000
root@zabbix-mysql:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root   18G  2.9G   14G  18% /
none                         4.0K     0  4.0K   0% /sys/fs/cgroup
udev                         7.9G  404K  7.9G   1% /dev
tmpfs                        1.6G  628K  1.6G   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                         7.9G     0  7.9G   0% /run/shm
none                         100M     0  100M   0% /run/user
/dev/sda1                    236M  157M   68M  70% /boot
datastore/tmp                5.0G  128K  5.0G   1% /tmp
datastore/mysql              125G   47G   78G  38% /var/lib/mysql
root@zabbix-mysql:~# zfs destroy datastore/mysql@daily_Tue_20151020_2000
root@zabbix-mysql:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root   18G  2.9G   14G  18% /
none                         4.0K     0  4.0K   0% /sys/fs/cgroup
udev                         7.9G  404K  7.9G   1% /dev
tmpfs                        1.6G  628K  1.6G   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                         7.9G     0  7.9G   0% /run/shm
none                         100M     0  100M   0% /run/user
/dev/sda1                    236M  157M   68M  70% /boot
datastore/tmp                5.0G  128K  5.0G   1% /tmp
datastore/mysql              157G   47G  111G  30% /var/lib/mysql
root@zabbix-mysql:~# zfs destroy datastore@hourly_Tue_20151020_0659
root@zabbix-mysql:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root   18G  2.9G   14G  18% /
none                         4.0K     0  4.0K   0% /sys/fs/cgroup
udev                         7.9G  404K  7.9G   1% /dev
tmpfs                        1.6G  628K  1.6G   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                         7.9G     0  7.9G   0% /run/shm
none                         100M     0  100M   0% /run/user
/dev/sda1                    236M  157M   68M  70% /boot
datastore/tmp                5.0G  128K  5.0G   1% /tmp
datastore/mysql              182G   47G  135G  26% /var/lib/mysql

Now there is no snapshots, but the partition size (/var/lib/mysql) is wrong

root@zabbix-mysql:~# zfs list -t snap
no datasets available

root@zabbix-mysql:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root   18G  2.9G   14G  18% /
none                         4.0K     0  4.0K   0% /sys/fs/cgroup
udev                         7.9G  404K  7.9G   1% /dev
tmpfs                        1.6G  628K  1.6G   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                         7.9G     0  7.9G   0% /run/shm
none                         100M     0  100M   0% /run/user
/dev/sda1                    236M  157M   68M  70% /boot
datastore/tmp                5.0G  128K  5.0G   1% /tmp
datastore/mysql              182G   47G  135G  26% /var/lib/mysql
datastore/var_log             10G   21M   10G   1% /var/log

root@zabbix-mysql:~# zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
datastore          47.0G   449G    30K  legacy
datastore/mysql    46.8G   449G  46.8G  /var/lib/mysql
datastore/tmp        22K  5.00G    22K  /tmp
datastore/var_log  20.3M  9.98G  20.3M  legacy

@galindro
Copy link
Author

Now I've fixed the problem. After the deletion of all snapshots, I've did this:

1 - umounted all datasets
2 - exported the pool with zpool export -a
3 - rebooted the VM
4 - imported the pool with zpool import && zpool import datastore

The good news is that there was no data loss and MySQL could recover itself. Now the question is: somebody has noticed this kind of issue before?

My environment is:

root@zabbix-mysql:~# dpkg -l |grep zfs
ii  dkms                                2.2.0.3-1.1ubuntu5.14.04.1+zfs10~trusty all          Dynamic Kernel Module Support Framework
ii  libzfs2                             0.6.5.2-1~trusty                        amd64        Native OpenZFS filesystem library for Linux
ii  mountall                            2.53-zfs1                               amd64        filesystem mounting tool
ii  ubuntu-zfs                          8~trusty                                amd64        Native ZFS filesystem metapackage for Ubuntu.
ii  zfs-dkms                            0.6.5.2-1~trusty                        amd64        Native OpenZFS filesystem kernel modules for Linux
ii  zfs-doc                             0.6.5.2-1~trusty                        amd64        Native OpenZFS filesystem documentation and examples.
ii  zfsutils                            0.6.5.2-1~trusty                        amd64        Native OpenZFS management utilities for Linux

root@zabbix-mysql:~# uname -a
Linux zabbix-mysql 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

root@zabbix-mysql:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.3 LTS
Release:    14.04
Codename:   trusty

@behlendorf
Copy link
Contributor

Closing as out of date. This was very likely caused by an open file handle preventing space from being freed. Unmounting and restarting everything would have resolved this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@behlendorf @galindro