[Research] Boot option from SSD

[openoms]

All applications with their own users could be built on the disk to spare SDcard write cycles.
In case of USB3 connection and SSD there is a significant gain on the speed as well.

For example with ThunderHub I've tested:

# create thunderhub user
sudo adduser --disabled-password --gecos "" thunderhub
# move the home directory to the disk
sudo usermod -m -d /mnt/hdd/app-storage/thunderhub thunderhub
# symlink to default folder
sudo ln -s /mnt/hdd/app-storage/thunderhub /home/thunderhub

but resulted in npm permission errors on installation.

Same when instead of the symlink I have swapped all /home/thunderhub paths to /mnt/hdd/app-storage/thunderhub

Will try with other services.

The ones with long build times like ElectRS and RTL would benefit greatly being built on the disk.

[openoms]

Running completely off the SSD is now possible as @hermanlim described in #1205 (comment)
Link to how to: https://www.tomshardware.com/how-to/boot-raspberry-pi-4-usb

Seems that it would need a dedicated (first) partition on the SSD so would only be realistic for new setups.

In that case the SDcard could be kept solely for firmware+OS updates and maybe for infrequent security backups for the SCB and other valuable data.
The SDcard contents could be cloned to the boot partition of the SSD during a first boot, update or recovery -> more work on the blitz.datadrive.sh

@rootzoll what do you think?

[rootzoll]

@openoms yes had the same realization. So the sd card will become as a way to insert new updates, but the OS is running from the SSD. This way we can even use cheaper sd card again, which would also lower the price :)

We can do this together with introducting a new default SSD partition layout similar as we do for experimental BTRFS ... so we would have:

• a os-partition (around 32GBs)
• a data-data partition
• a data-storage partition
• a temp-partition

[hermanlim]

USB thumb drive which is being used for the experimental BTRFS setup could also be partitioned to allow it to serve as an update vector. This would remove completely the need for the SD card once boot from USB is enabled by default on RPi4...

Even if not using BTRFS, I believe more people find it easier to access something that can write onto USB thumbdrive media as opposed to SD card since pretty much all computers come with USB drives but not all computers come with SD slot.

[rootzoll]

Running an additional thumb drive (for example as raid) should still be optional because of the extra power that is needed on the USB ... in the past power drop outs were one of the biggest troubles we had people getting corrupted databases. Even if the sd card reader of the Pi is in the end drawing on the same power its always the same standard component ... if we make an extra usb thumb drive part of our standard setup, people will connect all kind od USB storages as that drive and we risk lots of support requests. So I am just carefull here.

Also people know how to copy files over to a usb drive - but not how to write images to a usb drive. So an extra USB drive would just be dumb data drive with an image file on it ... not a bootable device and even if the USB thum drive would be bootable - so how would be get started with the RaspiBlitz on a fresh setup?

Running a RaspberryPi one basic skill is to learn how to write sd cards with images ... for all kind of projects. So I think even if this is an extra step to learn for people - its an empowering one and we should keep to that basic technique for default.

For the future for people that get delivered a pre-build node and want more convenience on update I could image a update process ... like the OS running on SSD, if an new image is available it will be downloaded thru TOR, then the new image gets written to the sd card and then the system reboots itself, starting that sd card running the update.

The update should not be automatic (user action still being required) and there should be no central person/server making delivery of the update (could be a changeable subscription with the shop/personoftrust delivering the prebuild node or something like that) ... but I think having the sd card still as the default/fallback boot medium of the RaspberryPi in the mix makes sense.

[geco91]

How can we create that one bootable partition on the SSD? To recap, we wanted to target for a future SSD layout:

• a os-partition (around 32GBs)
• a data-data partition
• a data-storage partition
• a temp-partition

All guides to make a RaspberryPi booting from SSD say that you should "clone" the sd card to your SSD or write the raspberryPi OS image to the SSD - which triggers the following question:

The sd card contains multiple partitions already - see:

So do we need to target for a layout like this?

• a boot partition (small)
• a os-partition (around 32GBs)
• a data-data partition
• a data-storage partition
• a temp-partition

That maybe possible on setup up fresh - first clone and then extend partitions. But how will we do updates? Can we clone/copy/replace single partitions? Any input here is welcome.

[openoms]

I think realisticly it would be the same as when switching to a BTRFS filesystem: https://github.com/rootzoll/raspiblitz/blob/v1.6/FAQ.md#how-to-use-btrfs-on-raspiblitz using the MIGRATION method.
Cannot guarantee the integrity of the data when moving partitions and it would also take a long time. The best is to repartition the disk and copy the verified blockchain back over from an external source (over the network as that is the most desktop-OS-independent way).

For testing and advanced users it would be nice to be able to prepare the disk with the new partition layout preloaded with the blockchain data (app-storage partition) and even maybe the app-data to simulate a recovery scenario.

[sr-gi]

I was trying to go with a sd-less setup for my Pi 4 as part of an upgrade I'm currently doing and found some issues that may be worth noting.

After cloning the sd into the SSD and changing the boot options to pick the USB 3.0 ports if the sd was not found I noticed the boot process was really slow and inconsistent (ranging from ~40 secs to several minutes). I tried to plug the sd again to make sure nothing was wrong and the pi booted instantly.

After a while I thought it may be due to power issues with the SSD, so I tried to connect it to a powered USB hub. I only had a 2.0 version available so I didn't have much hope.

I was quite surprised after seeing the issue was fixed, and the system booted consistently in around 30 secs. May try with a USB 3.0 hub later this week, since using the 2.0 is not a viable option.

The setup I'm using (in case it may be relevant):

Raspberry 4B, 4GB
Oficial power adapter
SanDisk Plus 2TB 2.5"
Orico HD case USB 3.0 (https://a.aliexpress.com/_B1CoZQ)

Update:

For the sake of research I copied the sd into a USB 3.0 flash drive and tried booting from there. The pi booted without issues consistently in ~25 secs.

[openoms]

@sr-gi that USB adapter might not work with the USB3 port of the RPi4 due to the common UASP implementation issue.
I would recommend using either a tested adapter (from our shopping list) or this X825 board: https://geekworm.com/products/raspberry-pi-4-model-b-x825-2-5-inch-sata-hdd-ssd-expansion-board which can be powered separately by a 5V barrel connector so power issues are excluded.

[nyxnor]

Cannot guarantee the integrity of the data when moving partitions and it would also take a long time. The best is to repartition the disk and copy the verified blockchain back over from an external source (over the network as that is the most desktop-OS-independent way).

For when the blockchain is already downloadded: But what if there is no need to move partition with data, just reducing from the main partition of the hdd 256MB and allocating on boot partition for the hdd, same 256MB. The boot fileS are like 200MB, boot partition 256MB. It could be unmounted from the hdd some space and then created a boot partition in the hdd. As it is not envolving transfering, but allocating.

To be done in Linux:
https://www.golinuxcloud.com/extend-resize-primary-partition-non-lvm-linux/
https://itsfoss.com/partition-managers-linux/
https://gparted.org/display-doc.php?name=help-manual#gparted-unmount-partition
As for GParted, there is a disclaimer about losing data, as it has to be a caution sign.

To be done on windows, not formatting the ext4, but it can move exy4, resize, allocate the ext4
https://www.diskpart.com/windows-10/ext4-partition-windows.html

microSD card copier on the terminal
https://howtoraspberrypi.com/create-sd-card-raspberry-pi-command-line-linux/

As the drive contains 450-460 GB, I didn´t found a video of someone trying this.

Process:
Boots normaly with microSD, resize (reduce) one of the hdd partition, allocate the unmounted partition. microSD card copier from the CLI. Choose to boot from USB as its demonstrated on a lot of tutorials, shutdown, unplug the card and hope it works. Is it viable this way?

[openoms]

@Ben10Brasileiro I was under the impression that the boot partition would need to be the first on the disk therefore moving all data would be necessary, but could test if booting works from the end of the disk.

[nyxnor]

@openoms Understood. Didnt think this way, maybe its necessary to be in the beggining of the drive. Will search.

[openoms]

Note that booting from NVMe now should be possible: http://www.jeffgeerling.com/blog/2021/raspberry-pi-can-boot-nvme-ssds-now although I'd prefer robust disk redundancy over the SSD boot: #2104

[openoms]

@Ben10Brasileiro actually only the bootlader (/boot) needs to be on the SDcard then the /root can be configured to be on the SSD.
This way would still boot from the SDcard, but run the OS from the faster and more reliable disk (?which could even be a self healing ZFS RAID?).
The OS then could be copied over when booting from a new SDcard which then would set the next boot to happen from the SSD again.
(This is just a thought experiment for now and might be incorrect, would need to try to see it working.)

[nyxnor]

@openoms I like this idea. Challenge accepted.
Overall, I think the microSD could just have a /boot and a backup partition for lnd sbc and other services files if needed.
If this is accomplished, I think we would see a big improvement, relying less on the microSD random I/O speeds. But of course need to be tested. Can try this till 1.7 release, Im going to do some research to not rely on you, but if you got any news, bring it on.

[rootzoll]

I think this is still an important goal to reach .. also might be worse a look how Umbrel is now doing it:
"The new update runs only barebones OS on the microSD card, and shifts all the apps, containers & I/O intensive processes from the microSD card to the SSD"
https://twitter.com/getumbrel/status/1385593994860654594?s=20

With prio on WebUI for v1.8 I am pushing this research to 1.8.1 and if sd card changes are needed an implementation is to target for v1.9 then.

Also check the discussion in this older issue about the same topic: #430

[david-bakin]

BTW, a minor reason for running off of USB (thumb drive or the big SSD): If you're using an external case for your Pi - e.g., CryptoCloak - the darn SD card is buried within the case where it's not easy to reach (you have to disassemble the case to get to it).

[xanoni]

I thought I had added a comment here, but maybe it was a different issue ... I run my node without SD card, it's all on a single SSD. I initially started with a SD card, but then migrated all contents to a LUKS encrypted SSD (only the ~300 MB boot partition is unencrypted).

Didn't have any issues, the Raspbian Kernel supports booting from USB storage fine. Exactly the same as with SD card.

[xanoni]

By the way, the lnd team makes the following argument against using SD cards:

The following (non-exhaustive) list of things can lead to data corruption:
[...]
lnd's main data directory being written on an SD card or USB thumb drive
(SD cards and USB thumb drives must be considered unsafe for critical files
that are written to very often, as the channel DB is).

[openoms]

@xanoni indeed, that is why the LND directory is on the SSD together with all other unique data.

The SDcard is only used as and easily swappable OS drive. Nothing is lost when changed and provides a reliable recovery option.

[xanoni]

that is why the LND directory is on the SSD together with all other unique data.

Yea, I should know this but forgot in that moment. But still, it's not unthinkable that sensitive data ends up in /root, or that corruption of non-sensitive data causes a failure that corrupts sensitive data.

Anyway, all I'm saying is that SSDs make sense, and that my SSD setup was just as easy as with an SD card. (The only extra complication was the encryption.)