Weird corrupption of dietpi commands #3268

samolego · 2019-12-17T16:51:32Z

Creating a bug report/issue

I've encountered some strange behaviour; I've updated my system to debian buster and ran dietpi script (#1285) for conversion. Everything went smoothly, but when I rebooted / shutted down and turned on the system, I got the following error;
-bash: /DietPi/dietpi/func/dietpi-globals: No such file or directory [FAILED] DietPi-Login | Failed to load DietPi-Globals. Skipping DietPi login scripts...

This was shown on every SSH instance I opened up, instead of "normal" dietpi info. I thought I've messed sth. up, so I reinstalled the system. Re-ran the dietpi conversion script. Shutted down the system and powered it back on. Same issue. I thouht I messed sth. again and did it the third time. This time I didn't shutdown the system for 3 days, everything was running smoothly. Today I decided to reboot it. Same problem.
Further checking shows that /DietPi folder is empty, leading to none of the commands working.

Required Information

DietPi version | cannot run any dietpi cmd, but I'm sure it was latest, since I installed it freshly
Distro version | 10.2
Kernel version | Linux minecraft-server 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux
SBC device | Atomic Pi
Power supply used | 5V 3A
SDcard used | Has internal memory

Additional Information (if applicable)

Was the software title installed freshly or updated/migrated?
At first updated, then installed freshly twice.
Can this issue be replicated on a fresh installation of DietPi?
I've done fresh installation of debian buster twice and reproduced it both times.

Steps to reproduce

Install fresh debian buster, x86_64
Run script to convert to dietpi (PREP_SYSTEM_FOR_DIETPI.sh | Automate #1285)
Reboot / shutdown system - mine happened after rebooting for 2nd / 3rd time

Expected behaviour

Dietpi commands still existing

Actual behaviour

-bash: /DietPi/dietpi/func/dietpi-globals: No such file or directory [FAILED] DietPi-Login | Failed to load DietPi-Globals. Skipping DietPi login scripts...

The text was updated successfully, but these errors were encountered:

MichaIng · 2019-12-17T17:29:12Z

@samolego
Many thanks for your report.

I guess the RAMdisk failed to load. Could you paste:

journalctl -u DietPi.mount
journalctl -u dietpi-ramdisk
cat /var/tmp/dietpi/logs/dietpi-ramdisk.log

samolego · 2019-12-17T19:52:40Z

Here's the output;
journalctl -u DietPi.mount
-- Logs begin at Tue 2019-12-17 16:21:04 GMT, end at Tue 2019-12-17 19:00:57 GMT. -- Dec 17 16:21:04 minecraft-server systemd[1]: Mounting /DietPi... Dec 17 16:21:04 minecraft-server systemd[1]: Mounted /DietPi.
journalctl -u dietpi-ramdisk
-- Logs begin at Tue 2019-12-17 16:21:04 GMT, end at Tue 2019-12-17 19:00:57 GMT. -- Dec 17 16:21:05 minecraft-server systemd[1]: Starting DietPi-RAMdisk... Dec 17 16:21:05 minecraft-server dash[365]: /bin/dash: 1: cannot create /var/tmp/dietpi/logs/dietpi-ramdisk.log: Read-only file system Dec 17 16:21:05 minecraft-server systemd[1]: dietpi-ramdisk.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Dec 17 16:21:05 minecraft-server systemd[1]: dietpi-ramdisk.service: Failed with result 'exit-code'. Dec 17 16:21:05 minecraft-server systemd[1]: Failed to start DietPi-RAMdisk.
cat /var/tmp/dietpi/logs/dietpi-ramdisk.log
Sat Dec 14 09:56:45 UTC 2019 | DietPi-RAMdisk: Copying DietPi to RAM... Sat Dec 14 09:56:46 UTC 2019 | DietPi-RAMdisk: Copied DietPi to RAM. Sat Dec 14 10:09:34 GMT 2019 | DietPi-RAMdisk: Storing DietPi to disk... Sat Dec 14 10:09:34 GMT 2019 | DietPi-RAMdisk: Stored DietPi to disk. Sat Dec 14 10:09:35 GMT 2019 | DietPi-RAMdisk: Copying DietPi to RAM... Sat Dec 14 10:09:35 GMT 2019 | DietPi-RAMdisk: Copied DietPi to RAM. Sat Dec 14 10:11:49 GMT 2019 | DietPi-RAMdisk: Storing DietPi to disk... Sat Dec 14 10:11:49 GMT 2019 | DietPi-RAMdisk: Stored DietPi to disk. Sat Dec 14 10:11:50 GMT 2019 | DietPi-RAMdisk: Copying DietPi to RAM... Sat Dec 14 10:11:50 GMT 2019 | DietPi-RAMdisk: Copied DietPi to RAM. Tue Dec 17 16:21:04 GMT 2019 | DietPi-RAMdisk: Storing DietPi to disk... Tue Dec 17 16:21:04 GMT 2019 | DietPi-RAMdisk: Stored DietPi to disk.

MichaIng · 2019-12-17T22:03:08Z

@samolego

cannot create /var/tmp/dietpi/logs/dietpi-ramdisk.log: Read-only file system

Strange, is /var or /var/tmp a dedicated mount on your system?

df
cat /etc/fstab

samolego · 2019-12-18T06:04:03Z

df

Filesystem     1K-blocks    Used Available Use% Mounted on
udev              959876       0    959876   0% /dev
tmpfs             195316    3156    192160   2% /run
/dev/mmcblk0p2  12470936 3638992   8178736  31% /
tmpfs             976580       0    976580   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs             976580       0    976580   0% /sys/fs/cgroup
tmpfs            1047552       0   1047552   0% /tmp
tmpfs              10240       0     10240   0% /DietPi
tmpfs              51200       8     51192   1% /var/log
cat /etc/fstab
# Please use "dietpi-drive_manager" to setup mounts
#----------------------------------------------------------------
# NETWORK
#----------------------------------------------------------------


#----------------------------------------------------------------
# TMPFS
#----------------------------------------------------------------
tmpfs /tmp tmpfs defaults,size=1023M,noatime,nodev,nosuid,mode=1777 0 0
tmpfs /DietPi tmpfs size=10m,noatime,lazytime,nodev,nosuid,mode=1755
tmpfs /var/log tmpfs defaults,size=50m,noatime,nodev,nosuid,mode=1777 0 0

#----------------------------------------------------------------
# MISC: ecryptfs, vboxsf (VirtualBox shared folder), bind mounts
#----------------------------------------------------------------


#----------------------------------------------------------------
# SWAPFILE
#----------------------------------------------------------------
/var/swap none swap sw 0 0

#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------

MichaIng · 2019-12-18T11:58:18Z

@samolego
Hmm, you run DietPi on an SDcard on x86_64? However it seems to be not inside the fstab, which is a bid strange. Also since it is partition two, did you setup the originating Debian with two partitions, e.g. first partition for kernel+bootloader?

Please try to start the dietpi-ramdisk.service now and check back if it succeeded:

systemctl start dietpi-ramdisk
ls -l /DietPi

samolego · 2019-12-18T12:43:53Z

No, I don't run DietPi on an external SD, it's installed on the built-in memory (Atomic Pi has 16 GB built-in). I sadly cannot remember how I set up the partitioning, but I'm pretty sure I left it to default configuration during installation process.

The starting of dietpi ramdisk hasn't succeeded;
systemctl start dietpi-ramdisk
Job for dietpi-ramdisk.service failed because the control process exited with error code. See "systemctl status dietpi-ramdisk.service" and "journalctl -xe" for details.

ls -l /DietPi
total 0

And systemctl status dietpi ramdisk.service shows:

   Loaded: loaded (/etc/systemd/system/dietpi-ramdisk.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-12-18 11:45:16 GMT;
7min ago
  Process: 913 ExecStartPre=/bin/mkdir -p /var/tmp/dietpi/logs (code=exited, status=0/SUCCESS)
  Process: 914 ExecStart=/bin/dash -c /boot/dietpi/func/dietpi-ramdisk 0 2>&1 >> /var/tmp/dietpi/logs/dietpi-ramdisk.log (code=exited, status=2)
 Main PID: 914 (code=exited, status=2)

Dec 18 11:45:16 minecraft-server systemd[1]: Starting DietPi-RAMdisk...
Dec 18 11:45:16 minecraft-server dash[914]: /bin/dash: 1: cannot create /var/tmp/dietpi/logs/dietpi-ramdisk.log: Read-only file system
Dec 18 11:45:16 minecraft-server systemd[1]: dietpi-ramdisk.
service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 18 11:45:16 minecraft-server systemd[1]: dietpi-ramdisk.
service: Failed with result 'exit-code'.
Dec 18 11:45:16 minecraft-server systemd[1]: Failed to start
 DietPi-RAMdisk.

MichaIng · 2019-12-18T14:50:01Z

@samolego
Okay, so the root file system is indeed not R/W mounted. So you currently cannot edit /etc/fstab, right?

Not sure why the required fstab entry is missing, however lets try to fix this manually first:

mount -o rw,remount /dev/mmcblk0p2 /

If this works, then edit /etc/fstab, e.g. nano /etc/fstab and add the following last line:

/dev/mmcblk0p2 / ext4 noatime,lazytime 0 1

(I assume its ext4 file system?)
I guess /dev/mmcblk0p1 contains the boot partition? Check if /boot directory is empty: ls -Al /boot
If so, do: mount /dev/mmcblk0p1 /boot
Check now that /boot partition is not empty anymore and contains e.g. dietpi.txt and some other files. If so, add the related fstab entry as additional line:

/dev/mmcblk0p1 /boot auto noatime,lazytime 0 2

Now you should be able to start dietpi-ramdisk.service:

systemctl restart dietpi-ramdisk

This should lead to /DietPi directory being not empty anymore: ls -Al /DietPi

samolego · 2019-12-18T15:44:41Z

The service started without errors, however, when I rebooted the device, same thing happened as before.

/etc/fstab is indeed read only after booting up. The /boot was not empty, however, /DietPi was. So I did all the steps again (removed the lines in /etc/fstab, rebooted and re-added them). So rn I can start ramdisk service again. Any more clues?

Thanks for all the help.

MichaIng · 2019-12-18T15:50:19Z

@samolego
Wait, you added the lines to /etc/fstab and saved the file, which succeeded, but then after reboot the root file system was still mounted as read-only?
Is the root file system really ext4? journalctl -u -.mount should give a hint. To have fstab entry compatible with all file systems:

/dev/mmcblk0p2 / auto noatime,lazytime 0 1

File system can be checked via:

df -T

samolego · 2019-12-18T17:26:54Z

but then after reboot the root file system was still mounted as read-only?
Exactly.

If I do the steps from your previous comment and reopen SSH, system functions normally. But reboot mounts root fs as RO ...

df -T proves that root fs is ext4

udev           devtmpfs    959876       0    959876   0% /dev
tmpfs          tmpfs       195316    3156    192160   2% /run
/dev/mmcblk0p2 ext4      12470936 3638992   8178736  31% /
tmpfs          tmpfs       976580       0    976580   0% /dev/shm
tmpfs          tmpfs         5120       0      5120   0% /run/lock
tmpfs          tmpfs       976580       0    976580   0% /sys/fs/cgroup
tmpfs          tmpfs      1047552       0   1047552   0% /tmp
tmpfs          tmpfs        10240    1360      8880  14% /DietPi
tmpfs          tmpfs        51200       8     51192   1% /var/log

Oh, and
journalctl -u -.mount shows "no entries"

-- Logs begin at Wed 2019-12-18 16:29:49 GMT, end at Wed 2019-12-18 16:35:52 GMT. ---- No entries --

MichaIng · 2019-12-18T17:38:10Z

@samolego
After doing the changes, could you please paste again, to assure correct fstab entry:

cat /etc/fstab

samolego · 2019-12-18T17:39:37Z

Here it is

# Please use "dietpi-drive_manager" to setup mounts
#----------------------------------------------------------------
# NETWORK
#----------------------------------------------------------------


#----------------------------------------------------------------
# TMPFS
#----------------------------------------------------------------
tmpfs /tmp tmpfs defaults,size=1023M,noatime,nodev,nosuid,mode=1777 0 0
tmpfs /DietPi tmpfs size=10m,noatime,lazytime,nodev,nosuid,mode=1755
tmpfs /var/log tmpfs defaults,size=50m,noatime,nodev,nosuid,mode=1777 0 0

#----------------------------------------------------------------
# MISC: ecryptfs, vboxsf (VirtualBox shared folder), bind mounts
#----------------------------------------------------------------


#----------------------------------------------------------------
# SWAPFILE
#----------------------------------------------------------------
/var/swap none swap sw 0 0

#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------
#/dev/mmcblk0p2 / ext4 noatime,lazytime 0 1
/dev/mmcblk0p1 /boot auto noatime,lazytime 0 2
/dev/mmcblk0p2 / auto noatime,lazytime 0 1

MichaIng · 2019-12-18T17:45:22Z

@samolego
Can you run and paste:

systemctl daemon-reload
cat /run/systemd/generator/-.mount
systemctl status systemd-remount-fs

samolego · 2019-12-18T17:48:22Z

cat /run/systemd/generator/-.mount

# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
Where=/
What=/dev/mmcblk0p2
Options=noatime,lazytime

systemctl status systemd-remount-fs

● systemd-remount-fs.service - Remount Root and Kernel File Systems
   Loaded: loaded (/lib/systemd/system/systemd-remount-fs.service; static; vendor preset: enabled)
   Active: active (exited) since Wed 2019-12-18 16:29:49 GMT; 26min ago
     Docs: man:systemd-remount-fs.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
 Main PID: 252 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 2155)
   Memory: 0B
   CGroup: /system.slice/systemd-remount-fs.service

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

MichaIng · 2019-12-18T18:03:48Z

@samolego

16:29:49

Does this match the boot time or better suits to when you remounted the drive manually?
Also you can check journalctl and check for errors around the initial mount of this drive. systemd-remount-fs is the service that should remount the root file system with R/W access, at least when the fstab entry exists.

R/W should be default, but who knows, to assure it is set, you could adjust the fstab entries: rw,noatime,lazytime

samolego · 2019-12-18T18:24:25Z

Yeah, the time matches boot time, since I have wrong timezone set up rn :-P.

EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)

First error seems to be with swap:

Dec 18 16:29:49 minecraft-server swapon[263]: swapon: /var/swap: swapon failed: Read-only file system

There seem to be no systemd errors ... will try to reboot with added rw option to fstab
EDIT: problem isn't gone with rw added

MichaIng · 2019-12-18T19:01:50Z

@samolego
Jep, swap files of course cannot be created on R/O file systems. Okay so for some strange reason, rw seems to be not default in your case. This means, to assure that, we should add this as default to our fresh fstab.

Okay, DietPi boot scripts should then succeeds, including initial firstrun update+install prompts, right?
When done, could you do:

cp -a /etc/fstab /etc/fstab_bak
dietpi-drive_manager 4
cat /etc/fstab

This is what we call on DietPi-PREP to create a fresh fstab based on current attached drives. It "should" contain root and boot mount, however good to test since it was not the case when you ran the script.

If those are missing, revert:

mv /etc/fstab_bak /etc/fstab

samolego · 2019-12-18T21:19:18Z

So, if I understood correctly - should I reinstall the dietpi with PREP script and after installation procces is finished run those cmds?
Will do it tomorrowm since it's a bit late rn ;-)

MichaIng · 2019-12-19T00:18:23Z

@samolego
Nope no need to reinstall, if thinks work now. The commands are to test the fstab reset command only, to see if it now does what it should on your system. If not I'd do a closer look into the code.

samolego · 2019-12-19T09:48:45Z

I'm not sure if I presented it to you correctly - if I reboot the system, dietpi cmds die again, so not sure what you meant by DietPi boot scripts should then succeed

dietpi-drive_manager 4

[ INFO ] DietPi-Drive_Manager | Detecting drives, please wait...
/DietPi/dietpi/dietpi-drive_manager: line 207: blkid: command not found
/DietPi/dietpi/dietpi-drive_manager: line 208: blkid: command not found
/DietPi/dietpi/dietpi-drive_manager: line 213: blkid: command not found
[ INFO ] DietPi-Drive_Manager |  - Detected mounted drive: /dev/mmcblk0p2 > /
/DietPi/dietpi/dietpi-drive_manager: line 304: blkid: command not found
[ INFO ] DietPi-Drive_Manager | autofs4 module not available in kernel, x-systemd.automount has been disabled

cat /etc/fstab

# Please use "dietpi-drive_manager" to setup mounts
#----------------------------------------------------------------
# NETWORK
#----------------------------------------------------------------


#----------------------------------------------------------------
# TMPFS
#----------------------------------------------------------------
tmpfs /tmp tmpfs noatime,lazytime,nodev,nosuid,mode=1777
tmpfs /DietPi tmpfs size=10m,noatime,lazytime,nodev,nosuid,mode=1755
tmpfs /var/log tmpfs size=50m,noatime,lazytime,nodev,nosuid,mode=1777

#----------------------------------------------------------------
# MISC: ecryptfs, vboxsf (VirtualBox shared folder), bind mounts
#----------------------------------------------------------------


#----------------------------------------------------------------
# SWAPFILE
#----------------------------------------------------------------


#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------

MichaIng · 2019-12-19T13:37:30Z

@samolego

blkid: command not found

There we have the issue, however this is possible. blkid is part of the essential util-linux, which is definitely installed, as otherwise the whole system would be unable to load. Please try to recover this binary: apt install --reinstall util-linux

I'm not sure if I presented it to you correctly - if I reboot the system, dietpi cmds die again, so not sure what you meant by DietPi boot scripts should then succeed

You mean when you add the /etc/fstab entries, after reboot rootfs is still mounted R/O?
After running dietpi-drive_manager 4 of course, since due to missing blkid it fails to add root and boot mounts to fresh fstab. Hence the backup of fstab to restore the entries: mv /etc/fstab_bak /etc/fstab
This was just for testing and now the reason is clear 😉.

samolego · 2019-12-19T14:28:05Z

You mean when you add the /etc/fstab entries, after reboot rootfs is still mounted R/O?

No, but I had to restart
systemctl restart dietpi-ramdisk

Anyways, seems to be working now. I will try to reboot a few times :-). Thanks so much!

samolego · 2019-12-19T14:31:50Z

Rebooted a few times successfully! I cannot believe it was a package failure. Thanks for all the help & quick responses. Keep being awesome :-P.

+ DietPi-Drive_Manager | Add rw flag to mount options. This should be default but seems to be not in rare cases: #3268

MichaIng · 2019-12-20T12:20:01Z

R/W flag is now added explicitly to fstab: c7b82ca
Changelog: c8243de

MichaIng added the Investigating 🤔 label Dec 17, 2019

samolego closed this as completed Dec 19, 2019

MichaIng removed the Investigating 🤔 label Dec 20, 2019

MichaIng added a commit that referenced this issue Dec 20, 2019

v6.27

c7b82ca

+ DietPi-Drive_Manager | Add rw flag to mount options. This should be default but seems to be not in rare cases: #3268

MichaIng mentioned this issue Dec 20, 2019

Beta v6.27.1 #3270

Merged

MichaIng mentioned this issue Dec 30, 2019

Release v6.27 #3290

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Weird corrupption of dietpi commands #3268

Weird corrupption of dietpi commands #3268

samolego commented Dec 17, 2019 •

edited

Loading

MichaIng commented Dec 17, 2019

samolego commented Dec 17, 2019

MichaIng commented Dec 17, 2019

samolego commented Dec 18, 2019 •

edited by MichaIng

Loading

MichaIng commented Dec 18, 2019

samolego commented Dec 18, 2019 •

edited

Loading

MichaIng commented Dec 18, 2019 •

edited

Loading

samolego commented Dec 18, 2019

MichaIng commented Dec 18, 2019 •

edited

Loading

samolego commented Dec 18, 2019 •

edited

Loading

MichaIng commented Dec 18, 2019

samolego commented Dec 18, 2019

MichaIng commented Dec 18, 2019

samolego commented Dec 18, 2019

MichaIng commented Dec 18, 2019

samolego commented Dec 18, 2019 •

edited

Loading

MichaIng commented Dec 18, 2019

samolego commented Dec 18, 2019

MichaIng commented Dec 19, 2019 •

edited

Loading

samolego commented Dec 19, 2019 •

edited

Loading

MichaIng commented Dec 19, 2019

samolego commented Dec 19, 2019

samolego commented Dec 19, 2019

MichaIng commented Dec 20, 2019

Weird corrupption of dietpi commands #3268

Weird corrupption of dietpi commands #3268

Comments

samolego commented Dec 17, 2019 • edited Loading

Creating a bug report/issue

Required Information

Additional Information (if applicable)

Steps to reproduce

Expected behaviour

Actual behaviour

MichaIng commented Dec 17, 2019

samolego commented Dec 17, 2019

MichaIng commented Dec 17, 2019

samolego commented Dec 18, 2019 • edited by MichaIng Loading

MichaIng commented Dec 18, 2019

samolego commented Dec 18, 2019 • edited Loading

MichaIng commented Dec 18, 2019 • edited Loading

samolego commented Dec 18, 2019

MichaIng commented Dec 18, 2019 • edited Loading

samolego commented Dec 18, 2019 • edited Loading

MichaIng commented Dec 18, 2019

samolego commented Dec 18, 2019

MichaIng commented Dec 18, 2019

samolego commented Dec 18, 2019

MichaIng commented Dec 18, 2019

samolego commented Dec 18, 2019 • edited Loading

MichaIng commented Dec 18, 2019

samolego commented Dec 18, 2019

MichaIng commented Dec 19, 2019 • edited Loading

samolego commented Dec 19, 2019 • edited Loading

MichaIng commented Dec 19, 2019

samolego commented Dec 19, 2019

samolego commented Dec 19, 2019

MichaIng commented Dec 20, 2019

samolego commented Dec 17, 2019 •

edited

Loading

samolego commented Dec 18, 2019 •

edited by MichaIng

Loading

samolego commented Dec 18, 2019 •

edited

Loading

MichaIng commented Dec 18, 2019 •

edited

Loading

MichaIng commented Dec 18, 2019 •

edited

Loading

samolego commented Dec 18, 2019 •

edited

Loading

samolego commented Dec 18, 2019 •

edited

Loading

MichaIng commented Dec 19, 2019 •

edited

Loading

samolego commented Dec 19, 2019 •

edited

Loading