DietPi-Globals | Address long APT phases without output

[Arghh]

Is your feature request related to a problem? Please describe:

The Step "apt upgrade, please wait" while running "dietpi-update" can take a long time. 20 + Minutes because of lot of kernel upgrades. I dont know if the upgrade is stuck, because there is no output. Without any output on that step it is very hard to say if the update is stuck or working.

Describe the solution you'd like:

Would it be possible to show the apt upgrade console output while updating DietPi? If apt upgrade already shows output then I think my update is stuck.

Additional context

DietPi-Update
─────────────────────────────────────────────────────
Phase: Upgrading APT packages

[ INFO ] DietPi-Update | APT update, please wait...
Hit:1 http://raspbian.raspberrypi.org/raspbian buster InRelease
Hit:2 https://archive.raspberrypi.org/debian buster InRelease
Reading package lists...
[  OK  ] DietPi-Update | APT update
[ INFO ] DietPi-Update | APT upgrade, please wait...

This last line output is what i mean. Does this step show console update or not?

Thanks you very much!

[ravenclaw900]

Hello,
Note that while that step doesn't show the

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done

output, if there are any packages that need to be upgraded, it should show that happening.

[ INFO ] APT upgrade, please wait...
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 22671 files and directories currently installed.)
Preparing to unpack .../wget_1.20.1-1.1_arm64.deb ...
Unpacking wget (1.20.1-1.1) over (1.18-5+deb9u3) ...
Setting up wget (1.20.1-1.1) ...
[  OK  ] APT upgrade

[Arghh]

Indeed. Dietpi doesn't show the download info everything else gets shown. MY connection to apt server is very slow while it downloaded the kernel it seemed like the update was stuck. when I run "apt upgrade" it shows also the packages been downloaded. Can we get that info added to the "dietpi-update" progress?
That would be my feature request.

[Joulinar]

well it's not dedicated to dietpi-update. basically the apt downloading information is hide at dietpi-software as well

If I'm not mistaken it's the functionality from dietpi-globals > G_AGUP & G_AGUG. As you see, package download is not shwon

root@DietPi3:~# G_AGUP
[ INFO ] APT update, please wait...
Hit:1 http://raspbian.raspberrypi.org/raspbian buster InRelease
Hit:2 https://archive.raspberrypi.org/debian buster InRelease
Reading package lists...
[  OK  ] APT update
root@DietPi3:~# G_AGUG
[ INFO ] APT upgrade, please wait...
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 22160 files and directories currently installed.)
Preparing to unpack .../raspberrypi-kernel_1.20210303-1_armhf.deb ...
Adding 'diversion of /boot/kernel.img to /usr/share/rpikernelhack/kernel.img

[MichaIng]

One reason is that the "processing" output looks very bad in logs. If it recognises the STDOUT is no terminal, it switches to a percentage output 10%, 20%, 30% each on one line.

As said in the other issue, these steps are also slow because of disable APT cache and list compression. When we reduce the G_AGUP calls in general, we can in turn change the default on cache and compression and speed the processing up, which then makes output obsolete.

I'll run a bunch of tests to fine a good balance.

[MichaIng]

Another related request: #4344

Quoting myself:

I think we need to revert the default APT list compression that we enabled a few versions ago. While this reduces disk writes notably, it leads to those long parsing times. I'll first add another change to reduce G_AGUP calls, storing info/timestamp of the last call, so that subsequent calls are in case skipped, then add a toggle for list compression to dietpi-config, changing the default to plain text lists. Or I'll leave lists compressed, but re-enable the binary cache file. Needs some testing which step has best performance per disk writes ratio 🙂.

Related request to add APT list compression toggle: #3926

[Joulinar]

As you closed the other one, I will post on this 😃

---

on my RPi4 64bit test system, compression is already disabled. however it still take ages today. I guess it's because download speed of apt server is quite slow atm. As the download is not shown, it could lead to the impression of a hanging installation. 🙄

Get:1 https://deb.debian.org/debian buster/main arm64 libstdc++-8-dev arm64 8.3.0-6 [1502 kB]
Get:2 https://deb.debian.org/debian buster/main arm64 g++-8 arm64 8.3.0-6 [8176 kB]
61% [2 g++-8 138 kB/8176 kB 2%]                                                                 16.3 kB/s 7min 57s   

woop woop 16.3 kB/s 🤣

[MichaIng]

Okay, that is another issue then, out of our control (regarding speed). The first output of apt-get upgrade however is indeed about the database parsing, after which the downloads are done:

[ INFO ] APT upgrade, please wait...
(Reading database ... 26179 files and directories currently installed.)
Preparing to unpack .../libperl5.32_5.32.1-4_amd64.deb ...
Unpacking libperl5.32:amd64 (5.32.1-4) over (5.32.1-3) ...
Preparing to unpack .../perl_5.32.1-4_amd64.deb ...
Unpacking perl (5.32.1-4) over (5.32.1-3) ...
Preparing to unpack .../perl-base_5.32.1-4_amd64.deb ...
Unpacking perl-base (5.32.1-4) over (5.32.1-3) ...
Setting up perl-base (5.32.1-4) ...
(Reading database ... 26179 files and directories currently installed.)
Preparing to unpack .../perl-modules-5.32_5.32.1-4_all.deb ...
Unpacking perl-modules-5.32 (5.32.1-4) over (5.32.1-3) ...
Setting up perl-modules-5.32 (5.32.1-4) ...
Setting up libperl5.32:amd64 (5.32.1-4) ...
Setting up perl (5.32.1-4) ...
Processing triggers for libc-bin (2.31-11) ...
[  OK  ] APT upgrade

EDIT: Ah nope, you're right, downloads are done before the "(Reading database ..." line, however after parsing package lists:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  perl-base
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1628 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://deb.debian.org/debian bullseye/main amd64 perl-base amd64 5.32.1-4 [1628 kB]
Fetched 1628 kB in 1s (2817 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 21731 files and directories currently installed.)
Preparing to unpack .../perl-base_5.32.1-4_amd64.deb ...
Unpacking perl-base (5.32.1-4) over (5.32.1-3) ...
Setting up perl-base (5.32.1-4) ...

G_DIETPI-NOTIFY uses some method to check whether its printing to an interactive terminal or not, to in case remove colour codes and other escape sequences. Theoretically we can use the same method to let G_AG* output being more verbose.
... brings the issue that currently it is piped into tee to log for the error handler, which will look very messy then. Another idea is to replace the tee pipe with script <command>, which allows to log the output but wrapping the command into a capable terminal which handles escape sequences (control and colour codes). I wanted to do that anyway so that G_EXEC_OUTPUT=1 G_EXEC <command> shows colours, e.g. nice for various builds where they are lost currently with the pipe.

Okay I think we have a few starting points:

• Reduce APT calls in general, avoid multiple G_AGUP shortly after another, potentially skip even installs/upgrades or already installed packages? Both requires an override force option, in case the sources list entries or pinning has been changed.
• Add APT list compression toggle.
• Re-evaluate best defaults regarding APT list compression and APT cache. Since source package cache is disabled regardless of regular package cache, writes are already halfed since some versions ago.
• Wrap G_EXEC_OUTPUT=1 G_EXEC error handled command calls into a logging wrapper terminal, like script, so that colours and control sequences are used, but only final console output is logged to file (no raw cryptic escape sequences). Use G_DIETPI-NOTIFY method to detect interactive terminal (or make it an own function) to add -qq or -y (or -q -y) apt-get flags.

[Joulinar]

Ah nope, you're right, downloads are done before the "(Reading database ..." line, however after parsing package lists

yes exactly. Atm this all is hidden and not visible for the end user. he just see [ INFO ] APT upgrade, please wait..., not knowing is it the parsing of the package lists or the download or does it died? Especially if you have bigger downloads, like kernel, combined with a slow apt package server ...

[MichaIng]

PR up to allow disabling APT list compression via dietpi-config: #5447

Also with last version already we increased the verbosity of APT calls in DietPi scripts, so that more details, e.g.:

# apt-get -qq upgrade | tee
# apt-get upgrade | tee
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Since out G_AG* commands are piped through tee, there is no percentage processing output, but without the -qq double quite flag, more steps are shown.