Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DietPi-RAMdisk | Replace with vmtouch to use native fs caching #3402

Merged
merged 133 commits into from
Mar 7, 2020
Merged

Conversation

MichaIng
Copy link
Owner

@MichaIng MichaIng commented Feb 28, 2020

Status: Review

ToDo:

  • Create runtime stat/flag files in /run/dietpi

Commit list/description:

  • DietPi-RAMdisk | Remove script due to replacement with vmtouch

+ DietPi-RAMdisk | Remove script due to replacement with vmtouch
@MichaIng MichaIng added this to the v6.29 milestone Feb 28, 2020
+ DietPi-RAMdisk | Remove service as it will be replaced with vmtouch-based fs caching
+ DietPi-PreBoot | Remove DietPi-RAMdisk requisite, execute script from disk location
+ DietPi-Boot | Execute script from disk location
+ DietPi-RAMlog | Remove DietPi mount usage
+ DietPi-WiFi-Monitor | Remove usage of DietPi mount and dependency on DietPi-RAMdisk
+ DietPi-PostBoot | Remove usage of DietPi mount
+ DietPi-Login | Remove DietPi mount usage
+ DietPi-Cron.daily | Remove DietPi mount point usage
@Joulinar
Copy link
Collaborator

@MichaIng
let me know of you need testing

+ DietPi-Cron.hourly | Remove DietPi mount point usage
Network | Disable power management for network devices only, assure via SUBSYSTEM. Run it on actual added WiFi adapter instead of fixed wlan0 via "%k", which matches KERNEL name.
Network | Syntax
+ DietPi-PREP | Remove /DietPi mount point usage and DietPi-RAMdisk implementation
+ RPi | Remove DietPi-RAMdisk note
+ Odroid C2 | Remove DietPi-RAMdisk note from boot.ini
+ Odroid XU4 | Remove DietPi-RAMdisk note from boot.ini
+ dietpi.txt | Remove DietPi-RAMdisk note
+ DietPi-Fan_control | Remove /DietPi mount point usage
+ DietPi-Change_hostname | Remove /DietPi mount point usage
+ DietPi-Create_MySQL_DB | Remove /DietPi mount point usage
+ DietPi-Banner | Remove /DietPi mount point usage
+ DietPi-Benchmark | Remove /DietPi mount point usage
+ DietPi-Globals | Remove /DietPi mount usage
+ DietPi-Logclear | Remove /DietPi mount usage
MichaIng added 15 commits March 6, 2020 00:41
+ DietPi-Set_hardware | Read network info from new runtime dir
+ DietPi-Cloudshell | Read network info from new runtime dir
+ DietPi-Survey | Read network info from new runtime dir
+ DietPi-Benchmark | Read network info from new runtime dir
+ DietPi-Patch | Apply .network location change
+ DietPi-Boot | Make .network file world-writeable initially, required for DietPi-Login to update it as non-root user
+ DietPi-Bugreport | Include new runtime dir into upload archive
+ DietPi-Bugreport | Fix boot/kernel config file names
+ DietPi-Bugreport | Add APT preferences files and simplify *.d/ drop-in config inclusions
+ DietPi-WiFiDB | Use new runtime dir to read network info
+ DietPi-NordVPN | Wait for network-online.target, which is now available on all supported distro versions
+ DietPi-Globals | Read network info from new runtime dir
+ DietPi-Globals | Check if G_PROGRAM_NAME has a non-empty string assigned, instead of checking if it is declared only, before using it in titles etc.
+ DietPi-Globals | Move G_FP_LOG_APT to /tmp tmpfs, since it is removed by G_ERROR_HANDLER in every case anyway, after in case printing error log to console. More persistent log can be as well found in /etc/apt/ and /etc/dpkg.log.
+ DietPi-Software | HAProxy: Update sysvinit service to comply with current DietPi and Debian versions and our current code standards. ToDo: There is a systemd unit included with the package, hence is our sysvinit service even in use?
+ DietPi-Software | Read network info from new runtime dir
+ DietPi-Patch | Update .network info file locations in existing software services
+ DietPi-Patch | Remove obsolete /var/lib/dietpi/.ntpd_override
+ DietPi-Update | Syntax
@MichaIng MichaIng requested review from Fourdee and Joulinar March 6, 2020 13:06
@MichaIng
Copy link
Owner Author

MichaIng commented Mar 6, 2020

Okay, everything ready from my side:

  • DietPi-RAMdisk is history
  • Migration during update has been tested with several older images, should be failsafe
  • Migration for Jessie systems is blocked, has been tested with one older and one newer Jessie image as well
  • vmtouch is installed instead, locking DietPi scripts and dietpi.txt in RAM to preserve execution and parsing performance of RAMdisk. However it can be freely configured via /etc/default/vmtouch to lock anything or nothing or do another of its features instead. As well it is totally optional and can be disabled (systemctl disable vmtouch) or purged (apt purge vmtouch) without affecting DietPi functionality, only our scripts might be a bid slower when executed rarely and the first time during boot.

Do not write runtime files to disk, I moved those to a new runtime tmpfs dir /run/dietpi which do not need to be boot-persistent. This complies with Debian defaults, FHS etc as well:

  • Runtime files, required during service or system runtime: /run
  • Temporary files, e.g. per-script files, expected to be removed after usage: /tmp

+ DietPi-PREP | Install vmtouch for new images
+ DietPi-PREP | Download DietPi archive as tarball
@Joulinar
Copy link
Collaborator

Joulinar commented Mar 6, 2020

@MichaIng
Did a test on a Stretch as well as Buster VM. Both updates to v6.29 finished as follow:

[  OK  ] DietPi-Update | Incremental patching to v6.29.0 completed
[ SUB2 ] DietPi-Update > Completed
[ INFO ] DietPi-Update | Current version : v6.29.0
[ INFO ] DietPi-Update | Latest version  : v6.29.0
[  OK  ] DietPi-Survey | Sending survey data
[ SUB3 ] DietPi-Services > restart
[  OK  ] DietPi-Services | restart : airsonic
[  OK  ] DietPi-Services | restart : cron
Terminated
root@DietPiVM3:~# 

All fine, except the Terminated at the end 😄

Not sure if this is something but I found this on journalctl on both systems.

Mar 06 22:24:16 DietPiVM3 systemd[1]: tmp.mount: Directory /tmp to mount over is not empty, mounting anyway.
Mar 06 22:24:16 DietPiVM3 systemd[1]: Mounting /tmp...
Mar 06 22:24:16 DietPiVM3 systemd[1]: Mounted /tmp.

@MichaIng
Copy link
Owner Author

MichaIng commented Mar 7, 2020

@Joulinar
Many thanks for testing. Yes the termination is expected. Due to the changes it is required to restart DietPi-Update (new code) from within the patch_file. When the child instance has finished, the parent instance must be terminated to not finish another time with old code. However I'll add some message that this is expected.

Done: 9895c3d + edafa69

tmp.mount: Directory /tmp to mount over is not empty

We clear the /tmp dir when creating new images, however it can happen by times that something is written to the non-tmpfs /tmp dir when it mounts late at boot or unexpectedly unmounts or such. To solve:

mkdir /mnt/root
mount /dev/sda1 /mnt/root
rm -Rfv /mnt/root/tmp/{,.??,.[^.]}*
umount /mnt/root
rmdir /mnt/root

+ DietPi-Patch | Add an info before terminating obsolete DietPi-Update parent instance, else the "Terminated" print might users feel rattled: #3402 (comment)
MichaIng added a commit that referenced this pull request Mar 7, 2020
+ DietPi-Pre-patch |  Add an info before terminating obsolete DietPi-Update parent instance, else the "Terminated" print might users feel rattled: #3402 (comment)
@Joulinar
Copy link
Collaborator

Joulinar commented Mar 7, 2020

@MichaIng
looks good now

[  OK  ] DietPi-Update | Incremental patching to v6.29.0 completed
[ SUB2 ] DietPi-Update > Completed
[ INFO ] DietPi-Update | Current version : v6.29.0
[ INFO ] DietPi-Update | Latest version  : v6.29.0
[  OK  ] DietPi-Survey | Sending survey data
[ SUB3 ] DietPi-Services > restart
[  OK  ] DietPi-Services | restart : redis-server
[  OK  ] DietPi-Services | restart : mariadb
[  OK  ] DietPi-Services | restart : php7.3-fpm
[  OK  ] DietPi-Services | restart : lighttpd
[  OK  ] DietPi-Services | restart : logitechmediaserver
[  OK  ] DietPi-Services | restart : cron
[  OK  ] DietPi-Services | ncc maintenance:mode --off
[  OK  ] DietPi-Patch | Everything done! Terminating the obsolete DietPi-Update parent instance...
Terminated
root@DietPiVM1:~#

Question: should there be an automated reboot initiated?

@MichaIng
Copy link
Owner Author

MichaIng commented Mar 7, 2020

@Joulinar

Question: should there be an automated reboot initiated?

Nope, only on first boot, if this is the firstrun update and the image is pre-v6.20.

@Joulinar
Copy link
Collaborator

Joulinar commented Mar 7, 2020

Roger that

@MichaIng MichaIng merged commit 727a5a7 into dev Mar 7, 2020
@MichaIng MichaIng deleted the ramdisk branch March 7, 2020 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DietPi-RAMdisk | Replace with vmtouch to use native fs caching
2 participants