META | DietPi-Benchmark optional uploads

[Fourdee]

Similar to dietpi-survey. Have an option to run:

• RootFS transfer rate
• CPU performance
• RAM performance

Then option to upload scores to http://dietpi.com/benchmark
Simple database to compare your scores against HW models etc.

@MichaIng
Thoughts?

[Fourdee]

REF: https://github.com/Fourdee/DietPi/issues/1253#issuecomment-346881878

[MichaIng]

@Fourdee
Nice idea. We can combine this with dietpi-survey, e.g. dietpi-benchmark allows user to choose upload, then creates some flag (which could be simply the data within /tmp/dietpi-benchmark/upload.txt) and starts dietpi-survey. dietpi-survey then checks for the flag and in case adds benchmark results die survey upload.

Hmm, but needs different handling, as relation between device data and results is needed 🤔.

[Fourdee]

@MichaIng

Excellent 👍

Once https://github.com/Fourdee/DietPi/issues/1768 is completed, i'll make a start on this.

[LexiconCode]

Consider implementing this into the stats for each device under the download section of dietpi.com
It's very hard to get a picture of real-world performance of a SBC before purchase not to mention accessory hardware performance in relation to a particular SBC.

[Fourdee]

@LexiconCode

Great idea, love it 👍 We'll definitely consider doing this.

My PHP experience is somewhat limited, so we may have to simply link the URL in the downloads section.

[Fourdee]

@MichaIng
Benchmark suite code done, just need to impliment how we are to upload this to dietpi.com. Use of survey and send, if the generated results file exists?

Currently, the file exported is below:

cat << _EOF_ > /var/lib/dietpi/dietpi-benchmark/upload_results
$CPUBENCH_TIME
$rootfs_write
$rootfs_read
$ram_write
$ram_read
_EOF_

root@DietPi:~# cat /var/lib/dietpi/dietpi-benchmark/upload_results
9.27
37.1
198
542
387

[LexiconCode]

Is it possible for rootfs benchmark to determine what type of hardware like MMC, hard drive, SD card?

In addition which interface it's using USB 2. 0, 3. 0, sata...

[Fourdee]

@LexiconCode

Is it possible for rootfs benchmark to determine what type of hardware like MMC, hard drive, SD card?

Unsure, however, good suggestion, i'll take a look 👍

[MichaIng]

@Fourdee
Ah sorry, was a bid busy since last weekend, missed that one.

First I agree with @LexiconCode that the values have more meaning, if connected with related hardware info.

• On SBCs CPU and RAM are fixed, thus here $G_HW_MODEL[_DESCRIPTION] would be sufficient.
• But on native PC + VM we should find out which hardware is in use, e.g. grep -m1 '^model name' <<< "$(</proc/cpuinfo)" | sed 's/model name[[:blank:]]*: //'?
• But couldn't find a way to get memory bandwidth or name info, besides using additional tools: lshw or dmidecode
• For disk speed and all machines are at least some basic drive and connection info helpful to sort/interpret the values, as mentioned. mmc can be identified via root mount /dev/* name, but no idea how to get all these infos without additional tools.
  • cat /sys/block/mmcblk0/device/name does not always exist
  • cat /sys/block/sda/device/model sometimes empty, also does not always exist
  • Couldn't find USB version or something so far.

About file scheme:

• If @userdeveloper98 finds the time to add it to the new survey page, theoretically it doesn't matter too much.
• But just to stay flexible it is never wrong to add it in a shell readable way again. But compared to the java database, we need to think how to present the data. E.g.:
  • Calculate average values for each device, also to support our performance rating. Then the values would need to be added e.g. like this:
    aBENCH_CPU[$G_HW_MODEL_DESCRIPTION]=$(( ${aBENCH_CPU[$G_HW_MODEL_DESCRIPTION]:=0} + $CPU_BENCH_RESULT )) && (( ${aDEVICE_COUNT[$G_HW_MODEL_DESCRIPTION]}++))
    After looping through all upload files:
    for i in ${#aDEVICE_COUNT[@]}; do ${aBENCH_CPU_AVG[$i]}=$(( ${aBENCH_CPU[$i]} / ${aDEVICE_COUNT[$i]} )); done
  • Show a result spread over all devices:
    ((aBENCH_CPU[$CPU_BENCH_RESULT]++))
    The result would need to be rounded first, grouped into reasonable score intervals, so we can present it in some sort of histogram.
• Or we do both 😄.
• But sourcing these files then does not contain any additional hardware info. For this I see no other chance then creating $UNIQUE_ID indexed arrays and adding the above mentioned infos as well to allow differentiation.

[Fourdee]

@MichaIng

We could probably do this using a 2D array?

((aBENCH_CPU_TIME[$G_HW_MODEL_DESCRIPTION,$(sed -n 1p $fp_dietpi_benchmark)]++))

That way, results are linked to the $G_HW_MODEL_DESCRIPTION

# -------------------------
# DietPi-Benchmark
# -------------------------
((aBENCH_CPU_TIME[Odroid C2 (aarch64),17.15]++))
((aBENCH_ROOTFS_WRITE[Odroid C2 (aarch64),26.6]++))
((aBENCH_ROOTFS_READ[Odroid C2 (aarch64),110]++))
((aBENCH_RAM_WRITE[Odroid C2 (aarch64),473]++))
((aBENCH_RAM_READ[Odroid C2 (aarch64),480]++))

---

I'am still unsure how you scrape the above into the survey file, but, i'd probably do it this way. Using UUID as the array index

aBENCH_DEVICE[$UUID]=$G_HW_MODEL_DESCRIPTION
aBENCH_CPU_TIME[$UUID]=$(sed -n 1p $fp_dietpi_benchmark)

---

Or even:

for (( i=0;i<total_file_count;i++))
do
. UUID $i
done

from

# -------------------------
# DietPi-Benchmark
# -------------------------
index=$1
aBENCH_DEVICE[$index]=$G_HW_MODEL_DESCRIPTION
aBENCH_CPU_TIME[$index]=$(sed -n 1p $fp_dietpi_benchmark)
aBENCH_ROOTFS_WRITE[$index]=$(sed -n 2p $fp_dietpi_benchmark)
aBENCH_ROOTFS_READ[$index]=$(sed -n 3p $fp_dietpi_benchmark)
aBENCH_RAM_WRITE[$index]=$(sed -n 4p $fp_dietpi_benchmark)
aBENCH_RAM_READ[$index]=$(sed -n 5p $fp_dietpi_benchmark)

Then we can work with the array's above.

root@DietPi:~# cat bench
# -------------------------
# DietPi-Benchmark
# -------------------------
index=$1
aBENCH_DEVICE[$index]='Odroid C2 (aarch64)'
aBENCH_CPU_TIME[$index]='17.15'
aBENCH_ROOTFS_WRITE[$index]='26.6'
aBENCH_ROOTFS_READ[$index]='110'
aBENCH_RAM_WRITE[$index]='473'
aBENCH_RAM_READ[$index]='480'
root@DietPi:~# . bench 0

[Fourdee]

Ok made a start on this code for this, basic code working well.

We end up with arrays of:

BENCH_RESULT_CPU_MIN[$HW_MODEL]
BENCH_RESULT_CPU_MAX[$HW_MODEL]
BENCH_RESULT_CPU_AVG[$HW_MODEL]

Which we can use to print some nice HTML stuff afterwards (which I have no clue where to start lol)

root@DietPi:/var/www/downloads/testing# ./dietpi-survey_report
#system 1
17.09
72.1
110
468
489

#system 2 (copy of system 1 upload)
17.09
72.1
110
468
489

BENCH_RESULT_CPU_AVG for HW_MODEL 12 = 17.0

Temps not printing once added to array, but the original var is?

[Fourdee]

@MichaIng

Whoop whoop 🥂

[Fourdee]

@MichaIng

However this could be presented much nicer on the new java survey page with nice charts.
Maybe @userdeveloper98 finds some time to implement this by times? 😜

Yep agree, we'll need to wait and see if @userdeveloper98 is up for it :)

[Fourdee]

Getting alot of this during update:
Was occurring before any of my changes.

TOSTART_OPTION[]: bad array subscript
/tmp/dietpi-survey_report/11f218f5-2ba7-4835-af54-b3e1da5f2e91.txt: line 17: aSOFTWARE[${aSOFTWARE_NAME6_10[]}]++: bad substitution

---

line 19: aSOFTWARE[]: bad array subscript

#!/bin/bash
# -------------------------
((aSURVEY_VERSION[6]++))
# -------------------------
((aSURVEY_SENTCOUNT[5]++))
((aDIETPI_VERSION[6.9]++))
((aDEVICE_NAME[RPi 3 Model B (armv7l)]++))
((aCPU_ARCH[armv7l]++))
((aCPU_COUNT[4]++))
((aDISTRO_VERSION[stretch]++))
((aAUTOSTART_OPTION[${aAUTOSTART_NAME[5]}]++))
((aAUTO_SETUP_AUTOMATED[0]++))
((aNETWORK_INTERFACE[eth0]++))
# -------------------------
# DietPi-Software Installed
# -------------------------
((aSOFTWARE[${aSOFTWARE_NAME6_9[5]}]++))
((aSOFTWARE[${aSOFTWARE_NAME6_9[6]}]++))
((aSOFTWARE[${aSOFTWARE_NAME6_9[31]}]++))
((aSOFTWARE[${aSOFTWARE_NAME6_9[44]}]++))
((aSOFTWARE[${aSOFTWARE_NAME6_9[96]}]++))
((aSOFTWARE[${aSOFTWARE_NAME6_9[103]}]++))
((aSOFTWARE[${aSOFTWARE_NAME6_9[104]}]++))
((aSOFTWARE[${aSOFTWARE_NAME6_9[152]}]++))

---

line 17: aSOFTWARE[${aSOFTWARE_NAME6_10[]}]++: bad substitution

#!/bin/bash
# -------------------------
((aSURVEY_VERSION[6]++))
# -------------------------
((aSURVEY_SENTCOUNT[1]++))
((aDIETPI_VERSION[6.10]++))
((aDEVICE_NAME[NanoPi NEO 2 (aarch64)]++))
((aCPU_ARCH[aarch64]++))
((aCPU_COUNT[4]++))
((aDISTRO_VERSION[stretch]++))
((aAUTOSTART_OPTION[${aAUTOSTART_NAME[0]}]++))
((aAUTO_SETUP_AUTOMATED[0]++))
((aNETWORK_INTERFACE[eth0]++))
# -------------------------
# DietPi-Software Installed
# -------------------------
((aSOFTWARE[${aSOFTWARE_NAME6_10[]}]++))

lol?

Ok we need to check min/max.

((aAUTOSTART_OPTION[${aAUTOSTART_NAME[28]}]++))

[MichaIng]

@Fourdee
Jep known issues.

On earlier versions for it was possible that survey added an empty string as installed software ID. I guess it was due to first run survey, if not .installed file was created yet, or at least nothing yet marked as installed.

The other issue is due to DietPi subversion integer being wrong sometimes in updater. So DietPi is actually v6.10+, thus survey uploads new survey file scheme, but version string added is wrong, so no special aSOFTWARE_NAME array exist for this.

I anyway though that we could clean the survey files a bid, removing those that were not changed for time X. But not urgent, since the errors above do not break anything, just a few software installs are dropped.

[Fourdee]

6103e95d-3a09-4729-b7dd-6bd9f3cb2e7c.txt: line 17: aSOFTWARE[]: bad array subscript
((aSOFTWARE[${aSOFTWARE_NAME_12[5]}]++))

[Fourdee]

@MichaIng

since the errors above do not break anything

Adding checks for bad files and skipping them 👍

root@DietPi:/var/www/downloads/testing# /etc/cron.minutely/dietpi-survey_report
Bad file contains array with no index ([]), skipping: /tmp/dietpi-survey_report/00fa6266-cc32-410c-9017-df2ce5003b85.txt

since the errors above do not break anything

Might possibly be the cause why only 1 benchmark result is showing, i've done 4, unsure yet:
https://dietpi.com/survey/#benchmark

Appears, the upload completes, but file does not exist on server.

[  OK  ] DietPi-Survey | Connection test: ssh.dietpi.com
#!/bin/bash
# -------------------------
((aSURVEY_VERSION[6]++))
# -------------------------
((aSURVEY_SENTCOUNT[23]++))
((aDIETPI_VERSION[6.17]++))
((aDEVICE_NAME[RPi 3 Model B+ (armv7l)]++))
((aCPU_ARCH[armv7l]++))
((aCPU_COUNT[4]++))
((aDISTRO_VERSION[stretch]++))
((aAUTOSTART_OPTION[${aAUTOSTART_NAME[0]}]++))
((aAUTO_SETUP_AUTOMATED[0]++))
((aNETWORK_INTERFACE[eth0]++))
# -------------------------
# DietPi-Software Installed
# -------------------------
((aSOFTWARE[${aSOFTWARE_NAME6_17[0]}]++))
((aSOFTWARE[${aSOFTWARE_NAME6_17[103]}]++))
((aSOFTWARE[${aSOFTWARE_NAME6_17[105]}]++))
# -------------------------
# DietPi-Benchmark
# -------------------------
BENCH_HW_MODEL=3
BENCH_CPU=21.30
BENCH_ROOTFS_WRITE=7.2
BENCH_ROOTFS_READ=20.0
BENCH_RAM_WRITE=506
BENCH_RAM_READ=159
BENCH_CPU_TEMP_START=44
BENCH_CPU_TEMP_END=59

[ .... ] DietPi-Survey | (1/5) Testing connection to ssh.dietpi.com, please wait[  OK  ] DietPi-Survey | Connection test: ssh.dietpi.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   830    0     0  100   830      0    812  0:00:01  0:00:01 --:--:--   812
100   830    0     0  100   830      0    812  0:00:01  0:00:01 --:--:--   812
[  OK  ] DietPi-Survey | Successfully sent survey data

root@DietPi:/var/www/downloads/testing# sync
root@DietPi:/var/www/downloads/testing# nano /home/dietpi-survey/survey/dce3cbc9-d3c5-4506-a654-c475184786bd.txt
#no file

Aha
http://www.manpagez.com/man/5/sshd_config/
MaxSessions

Idiot moment, resolved below 🈯️

[Fourdee]

Completed, all issues resolved 🈯️

You can run a quick benchmark and upload with:

/DietPi/dietpi/func/dietpi-benchmark 2

View results:
https://dietpi.com/survey#benchmark

[MichaIng]

@Fourdee
What do you think about creating results file in a shell source-able way, so it can be directly cat into the dietpi-survey upload file and sourced by other scripts as well. Generally I like to do this for most of our data/settings/results files, as sourcing should always be much faster then using sed or grep to catch content. Also the same values are then forced to have the same variable names across scripts, to have increased consistency.
https://github.com/Fourdee/DietPi/blob/dev/dietpi/func/dietpi-benchmark#L253-L261

[Fourdee]

@MichaIng

Yep, makes sense, i'll make the change.

[Fourdee]

@MichaIng

Done, cleaned up and now sourced vars.

Tested fine.

[MichaIng]

@Fourdee
Currently upload is done automatically, so no "optional uploads". Can we have a switch to have results saved locally or upload them? Or two separate selections for bench only and upload only?

I am thinking as well, if it is okay to bench single devices as well, e.g. CPU only, but then we need to handle empty entries in report.

And currently on VM, 1° CPU temperature is shown. Would be good to not only show N/A on report page, but as well in dietpi-benchmark menu header/results window. At best the temp test is simple skipped then in case of empty values, show N/A.

[Fourdee]

@MichaIng

I am thinking as well, if it is okay to bench single devices as well, e.g. CPU only, but then we need to handle empty entries in report.

The whole purpose of this was to reduce available options, and, support benchmark uploads via 1 menu item.

I removed the option to manually run CPU/rootfs/RAM as its contained within the suite option, which, we want users to run as share/view their results.

Currently upload is done automatically, so no "optional uploads"

Yep, intended, its part of the survey. However, we should change the "Ok" to run bench, have it a "Yes/No":

And currently on VM, 1° CPU temperature is shown.

Yep, i'll see if I can clean it up.

[Fourdee]

🈯️

[Fourdee]

Completed.

[MichaIng]

@Fourdee

The whole purpose of this was to reduce available options, and, support benchmark uploads via 1 menu item.

Hmm, I still think we should allow to run benchmarks without upload. Even that our survey is anonym and safe, I don't like it from end user perspective to be forced to transfer any data. Nothing against reducing options for simplicity, but there should be always a choice, if it's about sharing your data.
E.g. I purged free version of 3dmark instantly from my Windows, after recognizing that bench results are automatically uploaded 😉.

[Fourdee]

@MichaIng

Thought the same eariler.

2 tests:

Ok so:

• Opted in = automatic upload of scores
• Opted out = Prompt to opt in, if user still opted out, no upload, however, scores are now shown after test.

I'll make the change.

[Fourdee]

Completed.