Skip to content

Commit

Permalink
Use only name property for zfs list
Browse files Browse the repository at this point in the history 
zfs list is very slow if it needs to retrieve other properties than name. See openzfs/zfs#450 for reference.

This change uses only the name of the snapshot and uses awk to extract the snapshot date from the name and sort to sort it:

1. get all snapshot names, sorted by name
2. use grep to filter out any snapshot, that isn't correctly prefixed (maybe a check for the correct date format could also be added here)
3. use awk to extract the date and put the date before the snapshot
4. use sort to reverse sort this, first by date, then by snapshot name
5. use awk to remove the date, so that just the snapshot name is left

This significally speeds it up on my system (/root/zfs-auto-snapshot is the changed one, running with -n to get only the times for snapshot retrieval):

root@tatooine:/root# time /root/zfs-auto-snapshot -d -v // --label=frequent --keep=4 -n
Debug: Including rpool for regular snapshot.
Debug: Including rpool/DATA for regular snapshot.
Debug: Including rpool/DATA/ldap for recursive snapshot.
Debug: Including rpool/DATA/postgresql for recursive snapshot.
Debug: Including rpool/DATA/vm for regular snapshot.
Debug: Including rpool/DATA/vm/alderaan.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/bespin.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/coruscant.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/dagobah.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/dantooine.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/dev.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/monitor.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/office.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/test.example.net for recursive snapshot.
Debug: Including rpool/ROOT for recursive snapshot.
Debug: Excluding rpool/ROOT/ubuntu-1 because rpool/ROOT includes it recursively.
Doing regular snapshots of rpool rpool/DATA rpool/DATA/vm
Doing recursive snapshots of rpool/DATA/ldap rpool/DATA/postgresql rpool/DATA/vm/alderaan.example.net rpool/DATA/vm/bespin.example.net rpool/DATA/vm/coruscant.example.net rpool/DATA/vm/dagobah.example.net rpool/DATA/vm/dantooine.example.net rpool/DATA/vm/dev.example.net rpool/DATA/vm/monitor.example.net rpool/DATA/vm/office.example.net rpool/DATA/vm/test.example.net rpool/ROOT
Doing a dry run. Not running these commands...
zfs snapshot -o com.sun:auto-snapshot-desc='-'  'rpool@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy  'rpool@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-'  'rpool/DATA@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy  'rpool/DATA@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-'  'rpool/DATA/vm@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy  'rpool/DATA/vm@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/ldap@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/DATA/ldap@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/postgresql@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/DATA/postgresql@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/alderaan.example.net@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/DATA/vm/alderaan.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/bespin.example.net@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/DATA/vm/bespin.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/coruscant.example.net@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/DATA/vm/coruscant.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/dagobah.example.net@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/DATA/vm/dagobah.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/dantooine.example.net@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/DATA/vm/dantooine.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/dev.example.net@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/DATA/vm/dev.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/monitor.example.net@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/DATA/vm/monitor.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/office.example.net@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/DATA/vm/office.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/test.example.net@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/DATA/vm/test.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/ROOT@zfs-auto-snap_frequent-2013-04-16-2344'
zfs destroy -r 'rpool/ROOT@zfs-auto-snap_frequent-2013-04-16-2315'
@zfs-auto-snap_frequent-2013-04-16-2344, 15 created, 15 destroyed, 0 warnings.

real	0m6.936s
user	0m0.076s
sys	0m0.184s
root@tatooine:/root# time /sbin/zfs-auto-snapshot -d -v // --label=frequent --keep=4 -n
Debug: Including rpool for regular snapshot.
Debug: Including rpool/DATA for regular snapshot.
Debug: Including rpool/DATA/ldap for recursive snapshot.
Debug: Including rpool/DATA/postgresql for recursive snapshot.
Debug: Including rpool/DATA/vm for regular snapshot.
Debug: Including rpool/DATA/vm/alderaan.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/bespin.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/coruscant.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/dagobah.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/dantooine.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/dev.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/monitor.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/office.example.net for recursive snapshot.
Debug: Including rpool/DATA/vm/test.example.net for recursive snapshot.
Debug: Including rpool/ROOT for recursive snapshot.
Debug: Excluding rpool/ROOT/ubuntu-1 because rpool/ROOT includes it recursively.
Doing regular snapshots of rpool rpool/DATA rpool/DATA/vm
Doing recursive snapshots of rpool/DATA/ldap rpool/DATA/postgresql rpool/DATA/vm/alderaan.example.net rpool/DATA/vm/bespin.example.net rpool/DATA/vm/coruscant.example.net rpool/DATA/vm/dagobah.example.net rpool/DATA/vm/dantooine.example.net rpool/DATA/vm/dev.example.net rpool/DATA/vm/monitor.example.net rpool/DATA/vm/office.example.net rpool/DATA/vm/test.example.net rpool/ROOT
Doing a dry run. Not running these commands...
zfs snapshot -o com.sun:auto-snapshot-desc='-'  'rpool@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy  'rpool@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-'  'rpool/DATA@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy  'rpool/DATA@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-'  'rpool/DATA/vm@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy  'rpool/DATA/vm@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/ldap@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/DATA/ldap@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/postgresql@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/DATA/postgresql@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/alderaan.example.net@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/DATA/vm/alderaan.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/bespin.example.net@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/DATA/vm/bespin.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/coruscant.example.net@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/DATA/vm/coruscant.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/dagobah.example.net@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/DATA/vm/dagobah.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/dantooine.example.net@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/DATA/vm/dantooine.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/dev.example.net@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/DATA/vm/dev.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/monitor.example.net@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/DATA/vm/monitor.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/office.example.net@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/DATA/vm/office.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/DATA/vm/test.example.net@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/DATA/vm/test.example.net@zfs-auto-snap_frequent-2013-04-16-2315'
zfs snapshot -o com.sun:auto-snapshot-desc='-' -r 'rpool/ROOT@zfs-auto-snap_frequent-2013-04-16-2348'
zfs destroy -r 'rpool/ROOT@zfs-auto-snap_frequent-2013-04-16-2315'
@zfs-auto-snap_frequent-2013-04-16-2348, 15 created, 15 destroyed, 0 warnings.

real	3m30.995s
user	0m0.152s
sys	0m0.792s
root@tatooine:/root# 

I'm not an awk god, so I tried a bit until I got a working version. There might be better ways. I also don't know if this catches every adge case, but it is a start for improvement. :)
  • Loading branch information
@tisoft
tisoft committed Apr 16, 2013
1 parent cc9f180 commit b6fba51
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/zfs-auto-snapshot.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -339,7 +339,7 @@ ZFS_LIST=$(env LC_ALL=C zfs list -H -t filesystem,volume -s name \
-o name,com.sun:auto-snapshot,com.sun:auto-snapshot:"$opt_label") \
|| { print_log error "zfs list $?: $ZFS_LIST"; exit 136; }

SNAPSHOTS_OLD=$(env LC_ALL=C zfs list -H -t snapshot -S creation -o name) \
SNAPSHOTS_OLD=$(env LC_ALL=C zfs list -H -t snapshot -o name -s name|grep $opt_prefix |awk '{ print substr( $0, length($0) - 14, length($0) ) " " $0}' |sort -r -k1,1 -k2,2|awk '{ print substr( $0, 17, length($0) )}') \
|| { print_log error "zfs list $?: $SNAPSHOTS_OLD"; exit 137; }


Expand Down

0 comments on commit b6fba51

Please sign in to comment.