Add instructions for using ntfsclone #3292
Conversation
docs/faq.rst
Outdated
| -------------------------------------------------- | ||
|
|
||
| Full disk images are as large as the full disk when uncompressed and don't get much | ||
| smaller post-deduplication after heavy use. This is because virtually all file systems |
There was a problem hiding this comment.
guess one can't tell the dedup reduction that general (the user might have tons of duplicate data in existing files).
but it gets clearer what you mean in the following text, so maybe fix phrasing above like "and might not get...".
docs/faq.rst
Outdated
|
|
||
| ntfsclone can only operate on filesystems with the journal cleared (i.e. turned-off | ||
| machines) which somewhat limits its utility in the case of VM snapshots. However, | ||
| when it can be used, its special image format is even more efficient than just zeroing |
There was a problem hiding this comment.
is it still more efficient over time, like when you daily use the VM and backup it with borg?
There was a problem hiding this comment.
I would need to write a test script (edit some files in NTFS partitons & create archives with zeroing vs. ntfsclone format, rinse & repeat, compare final sizes) but the ntfsclone man page recommends always using the special format if you can control both backup and restore (as we do here). ntfsclone can clone to sparse files, so maybe once #14 is implemented that could simplify these instructions.
There was a problem hiding this comment.
guess the stuff we need to compare for efficiency are these:
- zeroing free space + running borg on the block device with --read-special (reading all the zeros will take a while, but they will dedup / compress to almost nothing. also, everything else stays in place and unchanged sectors will likely dedup ok with future block device images)
- using ntfsclone in special format (maybe dedups badly with future dumps due to special format?)
docs/faq.rst
Outdated
| borg create repo::hostname-partinfo-mbr-gap sfdisk.txt mbr_gap | ||
| sfdisk -l -o Device,Type /dev/sdX | grep NTFS | cut -d' ' -f1 | while read x; do | ||
| ntfsclone -s -o - $x | borg create repo::hostname-part$(echo $x | grep -Eo "[0-9]+$") - | ||
| done |
There was a problem hiding this comment.
hmm, this reads a bit strange as this seems to be about a physical disk / machine and last thing you mentioned above was a VM.
maybe add a "check this!" in the comment of the dd, it depends on layout. there is no problem if we save a bit more than needed, but there might be a problem if we do not save the whole gap (if it is larger).
you practically tested the script?
There was a problem hiding this comment.
I tested it on a MBR-formatted loopback device with a few NTFS partitions, I am testing it with a full windows installation right now.
On second thought I think saving the MBR with sfdisk is kind of redundant, when copying all space leading up to the first partition should do virtually the same thing. I can parse that out of the sfdisk output (it seems to always be 2048 bytes on MBR disks). Maybe instead I should use dd to copy up to the first partition and then use partprobe to notify the kernel of the new partition layout instead of using sfdisk at all.
Should these instructions back up every partition, NTFS or not, using ntfsclone for ntfs ones? At that point this is becoming more full bash script territory instead of just a snippet.
There was a problem hiding this comment.
I just tested it on a Windows VM, which obviously used the MBR format. I added several partitions and checked restore, all the files were in the same places on the secondary NTFS & fat32 partitions and Windows booted (obviously I didn't meticulously check every windows system file but if it booted I'm assuming they stayed in place).
docs/faq.rst
Outdated
|
|
||
| borg extract repo::hostname-partinfo-mbr-gap | ||
| sfdisk /dev/sdX < sfdisk.txt | ||
| dd if=mbr_gap of=/dev/sdX bs=1M |
There was a problem hiding this comment.
the dd is writing over the mbr, which was just edited by sfdisk.
also maybe rather use count=2048, like above.
docs/faq.rst
Outdated
| dd if=mbr_gap of=/dev/sdX bs=1M | ||
| sfdisk -l -o Device,Type /dev/sdX | grep NTFS | cut -d' ' -f1 | while read x; do | ||
| borg extract --stdout repo::hostname-part$(echo $x | grep -Eo "[0-9]+$") | ntfsclone -r -O $x - | ||
| ntfsclone -s -o - $x | borg create repo::hostname-part - |
There was a problem hiding this comment.
It shouldn't be there, I copy-pasted the wrong line from my test script :P
milkey-mouse
force-pushed
the
document-ntfsclone
branch
2 times, most recently
from
59fb29b to
fbc5be4
Compare
|
shall I collapse the ntfsclone fixup? |
|
yes. maybe we should even move the whole stuff into docs/misc/disk-imaging.txt or so? gets rather long for a faq entry and includes quite some scripting... |
milkey-mouse
force-pushed
the
document-ntfsclone
branch
from
eebdee0 to
b0d6803
Compare
|
I agree, especially if we add a section with benchmarks for ntfsclone special/ntfsclone zeroed/zerofree/raw --read-special, we will need a separate docs page. |
|
When you say to move everything into a |
|
yes. alternatively, it could be an own chapter about disk imaging. not sure what's better, a docs section is maybe easier discoverable, so it would not need a link from faq. |
|
The new page could go under the Deployment section, then we can keep its formatting as well |
milkey-mouse
force-pushed
the
document-ntfsclone
branch
from
976572c to
b07a712
Compare
milkey-mouse
force-pushed
the
document-ntfsclone
branch
from
b07a712 to
c0edc60
Compare
|
If existing content is being moved to a new place, we don't want links to 1.1 docs to break. Should this still be backported? |
|
if someone wants permanent links, a link to a specific version can be used (like 1.1.2). |
Fixes #81 and fixes #3259.