mergerfs.balance reverse filesystem walk

[qweasdzxc787]

Would be nice to have a command line switch to reverse walk the filesystem rather than the moving the first item found. I have rearranged all my files with rsync and the files are all at the beginning of each disk. If I run mergerfs.balance after I add an empty disk, it will create large empty spaces at the beginning of all the existing disks and creates a speed mismatch between the disks when performing a SnapRAID sync.

Basically, I would like mergerfs.balance to move the newest files first, leaving the oldest files in place and keeping all free space at the end of the disks so they are balanced performance wise.

[qweasdzxc787]

Accessing the sub-second file modification timestamps would be the best way to find the newest files, similar to what SnapRAID does.

[qweasdzxc787]

How about this:

os.stat_float_times ( True )

dated_files = [(os.path.getmtime(fn), os.path.basename(FN) for fn in os.listdir(path)]
dated_files.sort()
dated_files.reverse()
newest = dated_files[0][1]

[trapexit]

I can look at doing that but it's somewhat niche. Under normal usage patterns newer doesn't mean l end of disk.

Does it really make that much of a difference? I'd think fragmentation in general would be a bigger issue.

[qweasdzxc787]

For example, an 8TB Seagate Archive drive has a max read of almost 200 MB/s at the beginning of the disk and a minimum read speed around 80 MB/s at the end of the disk. If one drive is reading near the end while the others are reading from the beginning, that bottlenecks all the drives.

I've been thinking further about this, it's probably most efficient to build up the filelists as a snapshot at the beginning of the process, and moving array entries to the corresponding disk they have been moved to (in a stack fashion, to maintain the newness ordering).

It's niche, yes, but if you've been filling up all your disks evenly (mfs policy) and add a new blank drive to the array, then it makes sense from a SnapRAID performance perspective to make all the disks physically utilized the same. In a WORM hard drive usage pattern, you can generally expect newer files to be towards the end of the disk.

[trapexit]

I've several archive drives and I can't say I've noticed that behavior but I've not paid that much attention.

That'll obviously increase the startup cost quite a bit but it's not difficult. I'll take a stab at it.

[chapmanjacobd]

Initially, I wanted to make a PR for the mergerfs.balance script, but I already have a tool to scan files and save them into a SQLITE db, and I publish that package to pypi so it was a lot easier to just roll with that.

But there are a few major differences between the tool I wrote and mergerfs.balance:

1. mergerfs.balance works for balancing disks but it does not focus on distributing specific folders evenly between disks.

2. If the disks are not fairly well balanced to begin with then mergerfs.balance starts to act like mergerfs.consolidate. For example, my drives were almost full then I added a new empty disk to my mergerfs array.

3. My tool, library scatter, requires a full disk scan before working while the mergerfs.balance tool starts much faster and with fewer dependencies.

Here is how you can create a fs db:

pip install xklb
library fsadd --filesystem fs.db /mnt/d1/* &
library fsadd --filesystem fs.db /mnt/d2/* &
...
library fsadd --filesystem fs.db /mnt/d7/* &
wait

I have five million files and this took about half an hour...

After everything is done you can run library scatter to plan file movement. Depending on the size of data this might take two seconds to two minutes. No files will be moved without you manually copy-pasting the rsync commands. You can freely play around with this command and try different options before running rsync.

In your case @qweasdzxc787 you might do something like this, which would check the balance between the most recently modified 10,000 files:

library scatter -m /mnt/d1:/mnt/d2:/mnt/d3:/mnt/d4/:/mnt/d5:/mnt/d6:/mnt/d7 --sort 'time_modified desc' --group size --limit 10000 fs.db / 
Current path distribution:
╒═════════╤══════════════╤══════════════╤═══════════════╤════════════════╤═════════════════╤════════════════╕
│ mount   │   file_count │ total_size   │ median_size   │ time_created   │ time_modified   │ time_scanned   │
╞═════════╪══════════════╪══════════════╪═══════════════╪════════════════╪═════════════════╪════════════════╡
│ /mnt/d1 │         2806 │ 17.0 GB      │ 1.3 MB        │ Jan 27         │ Jan 27          │ Jan 31         │
├─────────┼──────────────┼──────────────┼───────────────┼────────────────┼─────────────────┼────────────────┤
│ /mnt/d2 │         2836 │ 8.6 GB       │ 1.3 MB        │ Jan 27         │ Jan 27          │ Jan 31         │
├─────────┼──────────────┼──────────────┼───────────────┼────────────────┼─────────────────┼────────────────┤
│ /mnt/d3 │         1049 │ 2.9 GB       │ 287.7 kB      │ Jan 29         │ Jan 27          │ Jan 31         │
├─────────┼──────────────┼──────────────┼───────────────┼────────────────┼─────────────────┼────────────────┤
│ /mnt/d4 │         2604 │ 20.1 GB      │ 403.2 kB      │ Jan 31         │ Jan 26          │ Jan 31         │
├─────────┼──────────────┼──────────────┼───────────────┼────────────────┼─────────────────┼────────────────┤
│ /mnt/d5 │          705 │ 112.4 GB     │ 59.3 MB       │ yesterday      │ Jan 25          │ today          │
╘═════════╧══════════════╧══════════════╧═══════════════╧════════════════╧═════════════════╧════════════════╛

Simulated path distribution:
494 files should be moved
9506 files should not be moved
╒═════════╤══════════════╤══════════════╤═══════════════╤════════════════╤═════════════════╤════════════════╕
│ mount   │   file_count │ total_size   │ median_size   │ time_created   │ time_modified   │ time_scanned   │
╞═════════╪══════════════╪══════════════╪═══════════════╪════════════════╪═════════════════╪════════════════╡
│ /mnt/d1 │         2887 │ 30.8 GB      │ 1.4 MB        │ Jan 27         │ Jan 27          │ Jan 31         │
├─────────┼──────────────┼──────────────┼───────────────┼────────────────┼─────────────────┼────────────────┤
│ /mnt/d2 │         2919 │ 21.3 GB      │ 1.3 MB        │ Jan 27         │ Jan 27          │ Jan 31         │
├─────────┼──────────────┼──────────────┼───────────────┼────────────────┼─────────────────┼────────────────┤
│ /mnt/d3 │         1148 │ 25.5 GB      │ 318.6 kB      │ Jan 29         │ Jan 27          │ Jan 31         │
├─────────┼──────────────┼──────────────┼───────────────┼────────────────┼─────────────────┼────────────────┤
│ /mnt/d4 │         2670 │ 32.4 GB      │ 411.7 kB      │ Jan 31         │ Jan 26          │ Jan 31         │
├─────────┼──────────────┼──────────────┼───────────────┼────────────────┼─────────────────┼────────────────┤
│ /mnt/d5 │          211 │ 24.5 GB      │ 525.2 kB      │ today          │ yesterday       │ today          │
├─────────┼──────────────┼──────────────┼───────────────┼────────────────┼─────────────────┼────────────────┤
│ /mnt/d6 │           69 │ 11.3 GB      │ 111.7 MB      │ yesterday      │ Jan 25          │ today          │
├─────────┼──────────────┼──────────────┼───────────────┼────────────────┼─────────────────┼────────────────┤
│ /mnt/d7 │           96 │ 15.2 GB      │ 107.9 MB      │ yesterday      │ Jan 25          │ today          │
╘═════════╧══════════════╧══════════════╧═══════════════╧════════════════╧═════════════════╧════════════════╛

######### Commands to run #########
### Move 96 files to /mnt/d7: ###
rsync -aE --xattrs --info=progress2 --no-inc-recursive --remove-source-files --files-from=/tmp/tmpmm8g95qw / /mnt/d7
### Move 69 files to /mnt/d6: ###
rsync -aE --xattrs --info=progress2 --no-inc-recursive --remove-source-files --files-from=/tmp/tmp16m8kj28 / /mnt/d6
### Move 99 files to /mnt/d3: ###
rsync -aE --xattrs --info=progress2 --no-inc-recursive --remove-source-files --files-from=/tmp/tmpsc49r7oo / /mnt/d3
### Move 81 files to /mnt/d1: ###
rsync -aE --xattrs --info=progress2 --no-inc-recursive --remove-source-files --files-from=/tmp/tmp96iks8ot / /mnt/d1
### Move 83 files to /mnt/d2: ###
rsync -aE --xattrs --info=progress2 --no-inc-recursive --remove-source-files --files-from=/tmp/tmpmdppltyw / /mnt/d2
### Move 66 files to /mnt/d4: ###
rsync -aE --xattrs --info=progress2 --no-inc-recursive --remove-source-files --files-from=/tmp/tmpdujeaue9 / /mnt/d4

Afterward running the rsync commands you will need to update the fs.db before you run scatter again:

library fsupdate fs.db

https://github.com/chapmanjacobd/library/blob/baea099bb56f9f15d41acc727c0b82f8c5b6c2fc/scripts/scatter.py