Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The DietPi installation does not follow the *nix Filesystem Hierarchy Standard #1297

Closed
TerryE opened this issue Dec 12, 2017 · 9 comments
Closed

Comments

@TerryE
Copy link

TerryE commented Dec 12, 2017

This is a general configuration / architectural issue that started out as a comment in #1294, but this has a wider scope so it really merits its own issue.

Expected behaviour:

Linux adopts a convention Filesystem Hierarchy Standard (FHS) for what is type of files are located in /etc, /usr/local/lib,/var/lib. DietPi should follow FHS, as this convention have been developed for sound reasons and following it makes for better system configuration and facilitates experienced sysadmins using the package. For example /etc is normally used solely for piecewise static configuration data and is by default o:r accessible, with all files in the /etc tree owned by and only writeable by root. Where data (such as identification data) is not publicly readable but access is required by service account, then this is mode 640 and the files are service account's group so the application can read it (see for example /etc/at.deny).

Packages such as etckeeper can be used to to manage the system configuraton, if the installation follows these conventions.

Actual behaviour:

DietPi stores non-configuration data in /etc such as /etc/dietpi/logs which breaks the conventions and makes it impractical to use etckeeper. These log snapshots are really dietpi application data and therefore would be better located in /var/lib/dietpi/logs, and so on.

Likewise /usr/local/libs should be reserved for shared binary libraries used by /usr/local/bin programs.

The share hierarchies of /usr are a better fit except that all files in these hierarchies are again root-owned so this can't be written to by an unprivileged service account. The node_modules hierarchy doesn't fit this read-only pattern so doesn't belong here. Where applications need applications directories that they can write to, then these are kept in a /var/lib subdirectory owned by the service account, e.g. /var/lib/mysql which is normally mode 700 or 755 with a 700 subdirectory for private data. So here we should follow the mysql pattern were services run in a service account and in the case of node red, for example, use /var/lib/npm and /var/lib/node-red.

Before exploring / expanding this issue further I am interested in @Fourdee Dan's comments.

@TerryE TerryE changed the title The DietPi installation does not follow the \*nix Filesystem Hierarchy Standard The DietPi installation does not follow the *nix Filesystem Hierarchy Standard Dec 12, 2017
@k-plan
Copy link
Contributor

k-plan commented Dec 12, 2017

Before exploring / expanding this issue further I am interested in @Fourdee Dan's comments.

Until @Fourdee give any comment:

Lock conversation on this issue.

Locking the conversation means:

  • Other users can’t add new comments to this issue.
  • You and other collaborators with access to this repository can still leave comments that others can see.
  • You can always unlock this issue again in the future.

Repository owner locked and limited conversation to collaborators Dec 12, 2017
@Fourdee Fourdee self-assigned this Dec 13, 2017
@Fourdee
Copy link
Collaborator

Fourdee commented Dec 13, 2017

@TerryE

Many thanks for the info 👍

I'll browse through FHS and make a decision on if we are adopt it. I'll post here once I have a answer :)

@Fourdee Fourdee added this to the v160 milestone Dec 13, 2017
@Fourdee
Copy link
Collaborator

Fourdee commented Dec 16, 2017

Notes:

  • /etc/dietpi/logs, due to /var/log default as RAMlog for performance, and, SD write reduction. We use this for storing logs which would avoid the 1hour automatic clear (to free up memory).
  • /etc I agree with "/etc to static configuration files and may not contain binaries". Solution would be to install everything to /mnt/dietpi_userdata as standard, or binaries only in /usr/bin

I think the simple solution here (in regards to DietPi logs and non apt/dpkg installations eg: amiberry) is to install/configure everything in /mnt/dietpi_userdata. This way, we can customize the installation to our needs and optimizations, without breaking FHS. It would also benefit uses with /mnt/dietpi_userdata setup for USB drives with higher IO than rootFS.
We use /mnt/dietpi_userdata for some items (eg: mysql/nextcloud/owncloud data), but not all, i'll need to review them.

@TerryE
Yep, i agree 👍
Leave it with me, i'll go through the DietPi specific items 1st, then we can review any outstanding queries.

Repository owner unlocked this conversation Dec 17, 2017
@Fourdee
Copy link
Collaborator

Fourdee commented Dec 17, 2017

@Fourdee
Copy link
Collaborator

Fourdee commented Dec 17, 2017

Moved from https://github.com/Fourdee/DietPi/issues/1309#issue-282622893


MichaIng wrote:

@Fourdee
Sorry for opening up new unlocked issue 😅, just wanted to add the /opt folder into discussion for DietPi files. At least I learned about this to be used for non repo package software (compared to /etc) and according to the debian.org article it is also meant to include binaries: https://wiki.debian.org/FilesystemHierarchyStandard

If consider /usr/bin, maybe consider /usr/local/(bin), which looks most intended for stuff like that and is clean on default installations. It also already works great for ownCloud/Nextcloud occ/ncc commands 😛. Then we don't need our binaries as alias anymore and they will also work inside scripts. But no benefit from ramdisk.

But my main idea was just to copy our optimized software configs out of ramdisk, as they really don't need to be there (so real performance benefit) but consume RAM, that we have to consider while writing configs into their own files. Okay it's just about a couple of MB still, but my further ideas are to clean up dietpi-software more by moving configurations into their own files 😉.

@Fourdee
Copy link
Collaborator

Fourdee commented Dec 17, 2017

Ok going with:

  • /var/tmp/dietpi/logs for DietPi logs outside of /var/log, that we dont want clearing every 1 hour with dietpi-ramlog
  • /var/lib/dietpi/dietpi-ramlog_store for storing RAMlog /var/log files during reboots

Commit done, i'll tweak if needed.

@TerryE @MichaIng Any objections, suggestions?

Then remains checking everything else we use /etc/dietpi and discussing.

@TerryE
Copy link
Author

TerryE commented Dec 17, 2017

Yup make them owned by dietpi:dietpi and mode 660, then any account in the dietpi group can manage the files.

@Fourdee
Copy link
Collaborator

Fourdee commented Dec 27, 2017

Ok,completed for v160 /etc/dietpi is no more!:

  • /var/tmp/dietpi/logs for DietPi logs outside of /var/log, that we dont want clearing every 1 hour with dietpi-ramlog
  • Replaced /etc/dietpi throughout DietPi scripts with /var/lib/dietpi
  • dietpi-software installations of non-systemd services which used /etc/dietpi. Have been moved to /var/lib/dietpi/dietpi-software/services
  • dietpi-software installations of custom scripts (eg: Pi-SPC) which used /etc/dietpi. Have been moved to /var/lib/dietpi/dietpi-software/installed

Will leave open, to allow me to run some final testing.

@Fourdee
Copy link
Collaborator

Fourdee commented Dec 30, 2017

Completed.

I'll do final in-depth testing on v160 prior to release.

@Fourdee Fourdee closed this as completed Dec 30, 2017
@Fourdee Fourdee mentioned this issue Jan 28, 2018
Fourdee referenced this issue Jan 28, 2018
**v6.0** (previously v160)
(28/01/18)

**Important Information:**
**All DietPi images have been re-created. Existing installations (v159 or lower), can no longer be updated, or supported. To continue support, users must install the latest v6.0 image.**
 - https://github.com/Fourdee/DietPi/issues/1385
 - All images are now Debian Stretch (excluding Odroid's)
 - ARMbian based images are now mainline kernel 4.13+.
 - Native PC (EFI): is now an ISO, with clonezilla bundled. Simplifies installation via Rufus write: https://github.com/Fourdee/DietPi/issues/1171#issuecomment-336522021
 - If you are happy with your existing installation of v159 (or lower), you are not required to install the v6.0 image, however, we cannot continue to provide support for v159 (or lower) installations.

**Minor notes:**
The XMAS tree has now been taken down, stored away on github history for next year. Hope you all had a good one :D

**Changes / Improvements / Optimizations:**

General | DietPi RPi kernel, now reverted to stock RPi kernel: https://github.com/Fourdee/DietPi/issues/1378

General | We have completed much needed backbone work for DietPi, which will allow for improved expansion in source code. This includes the use of dietpi-globals.

DietPi-Globals | New script which optimizes most used DietPi commands and vars, throughout our scripts. Also exported to bash session, please type 'G_' then press 'TAB' to see a full list of options: https://github.com/Fourdee/DietPi/issues/1311

General | FHS compliance completed. /etc/dietpi has moved to /var/lib/dietpi. RAMlog store has moved to /var/tmp/dietpi: https://github.com/Fourdee/DietPi/issues/1297#issuecomment-352241193

General | We have refreshed our terminal messages look & feel, oriented on RPi boot messages, and with process animation: https://github.com/Fourdee/DietPi/pull/1377

General | wget: Now set to prefer IPv4 by default (generally faster, can be changed by 'CONFIG_PREFER_IPVERSION' in dietpi.txt): https://github.com/Fourdee/DietPi/issues/1285#issuecomment-353230187

General | APT: Now set to force IPv4 by default (generally faster, can be changed by 'CONFIG_PREFER_IPVERSION' in dietpi.txt): https://github.com/Fourdee/DietPi/issues/1285#issuecomment-353230187

General | SparkySBC: CPU gov default changed to Performance, reports of increased stability.

General | Swapfile generation is now completed during 1st run of dietpi-software (previously boot stage): https://github.com/Fourdee/DietPi/issues/1270#issue-278797206

General | DietPi-Funtime: Removed from DietPi. Although it looked pretty, it did absolutely nothing (except slow down a program)

DietPi-Automation | All dietpi.txt entries have been renamed and cleaned up.

DietPi-Automation | dietpi.txt: CONFIG_NTP_MODE will now be applied during 1st run of device: https://github.com/Fourdee/DietPi/issues/1379

DietPi-Boot | Improved the method of initial FS_partition and FS_expansion during 1st run, via systemD services. 'fs_force_resize=' in dietpi.txt is no longer supported: https://github.com/Fourdee/DietPi/issues/1285#issuecomment-352159930

DietPi-Banner | IP: Will now also list the active network adapter used (eg: eth0/wlan0)

DietPi-Config | Dion Audio LOCO V1/V2: Soundcards added for RPi.

DietPi-Config | Locale: en_GB.UTF-8 is now automatically installed, alongside user selected choice. Required for DietPi scripts to function.

DietPi-Drive_Manager | Added support for exFAT, many thanks @MichaIng : https://github.com/Fourdee/DietPi/pull/1312

DietPi-Globals | Global variables and functions are now exported during login. Please see the sourcecode for more information: https://github.com/Fourdee/DietPi/issues/1311

DietPi-Set_Hardware | Sparky SBC: enable aotg.aotg1_speed compatibility setting for USB 1.1, when USB-DAC configured: https://github.com/Fourdee/DietPi/issues/1301

DietPi-Set_Software | "pool" directive is now used for NTPD: https://github.com/Fourdee/DietPi/pull/1404

DietPi-Software | NAA Daemon: Updated to latest (3.5.2-36). Existing installs will be patched automatically: https://github.com/Fourdee/DietPi/issues/1305

DietPi-Software | PHP-FPM: Increased from "$CPU_CORES_TOTAL" to "pm.max_children = $(( $CPU_CORES_TOTAL * 3 ))". This should avoid failed forking of PHP-FPM processes/requests : https://github.com/Fourdee/DietPi/issues/1298

DietPi-Software | ownCloud/Nextcloud: Added option to choose data directory via dietpi.txt pre installation: https://github.com/Fourdee/DietPi/issues/1314#issuecomment-352782055

DietPi-Software | ownCloud/Nextcloud: Switch to pretty URLs (without "index.php") on Apache

DietPi-Software | ownCloud/Nextcloud: Automated backup restoring on install and creation und uninstall to ownCloud/Nextcloud data directory

DietPi-Software | ownCloud: Switch to non-package/archive installation. This allows usage of preferred web based updater.

DietPi-Software | Nextcloud: Resolved OPcache admin panel warnings now also on Lighttpd

DietPi-Software | UrBackup: Installation updated to latest version 2.1.20. For new installations only: https://github.com/Fourdee/DietPi/issues/1335

DietPi-Software | NodeRed: Corrected user which nodered runs under, now runs as its own user, created during install: https://github.com/Fourdee/DietPi/issues/1294#issuecomment-354314318

DietPi-Software | SqueezeBox/LMS (Stretch): Installation resolved: https://github.com/Fourdee/DietPi/issues/1124

DietPi-Software | MySQL: Completely remove MySQL from DietPi in favour of MariaDB: https://github.com/Fourdee/DietPi/issues/1397

DietPi-Software | Ampache: MySQL DB and configs have been updated (adds correct userdata folder for music by default): https://github.com/Fourdee/DietPi/issues/1420

run_ntpd | Added support for systemd-timesyncd completion/detection: https://github.com/Fourdee/DietPi/issues/1379

**Bug Fixes:**

General | Fixed two systemd error messages during shutdown and boot: https://github.com/Fourdee/DietPi/issues/1330

DietPi-Automation | Resolved an issue where AUTO_SETUP_TIMEZONE was not being applied correctly, thanks @k-plan: https://github.com/Fourdee/DietPi/issues/1285#issuecomment-356310496

DietPi-Automation | dietpi.txt: CONFIG_NTP_MIRROR will now be applied to systemd-timesyncd configuration: https://github.com/Fourdee/DietPi/issues/1379

DietPi-Config | Resolved an issue with WiFi Country code, failing to set on some devices: https://github.com/Fourdee/DietPi/issues/838

DietPi-Config | Resolved an issue where disabling IPv6 didn't have an effect on AMD64 devices: https://github.com/Fourdee/DietPi/issues/1343#issuecomment-359652751

DietPi-Services | dietpi-wifi-monitor: Is no longer controlled, to prevent WiFi drop during software installs/updates etc: https://github.com/Fourdee/DietPi/issues/1288#issuecomment-350653480

DietPi-Software | General: MySQL using software titles now have their own database user, instead of accessing as "root": https://github.com/Fourdee/DietPi/issues/1397#issuecomment-359655198

DietPi-Software | qBittorrent: Resolved an issue with inability to log into web interface: https://github.com/Fourdee/DietPi/issues/1366

DietPi-Software | Resolved an issue where our custom LD_LIBRARY_PATH would cause APT failures. LD_LIBRARY_PATH has now been reverted, apologies if this effected your system: https://github.com/Fourdee/DietPi/issues/1329

DietPi-Software | Resolved an issue where APT installations would fail if services were masked. All known 

DietPi software services, will be enabled/unmasked, before installation: https://github.com/Fourdee/DietPi/issues/1320

DietPi-Software | WiFi Hotspot (Stretch): Resolved an issue where hostapd would fail to run due to missing libssl1.0.0 lib, not available in repos: https://github.com/Fourdee/DietPi/issues/1299

DietPi-Software | Shairport-sync (Stretch): Resolved an issue where this would fail to install, due to pre-req URLS becomming invalid: https://github.com/Fourdee/DietPi/issues/1303

DietPi-Software | Plex Media Server: Resolved uninstall to include /var/lib/plexmediaserver in removal (which is not completed via apt purge).

DietPi-Software | MariaDB: Resolved an issue where MariaDB would fail to uninstall correctly: https://github.com/Fourdee/DietPi/pull/1280

DietPi-Software | Aira2 (Stretch): Resolved installation, now used APT installation: https://github.com/Fourdee/DietPi/issues/1310

DietPi-Software | Mosquitto: Resolved various issues with failed install, due to Mosq repo not being maintained (deb's missing from repo header list, requires non-stretch available packages). deb's are now hosted on dietpi.com: https://github.com/Fourdee/DietPi/issues/1306

DietPi-Software | ownCloud/Nextcloud: Fixed an installation issue on Jessie with MariaDB: https://github.com/Fourdee/DietPi/pull/1319

DietPi-Software | Google AIY: Updated install to gitbranch=voicekit. Many thanks to @mpember for the heads up: https://github.com/Fourdee/DietPi/issues/1065#issuecomment-354304388

DietPi-Software | OpenJDK: Replaces OracleJDK: https://github.com/Fourdee/DietPi/issues/1401

DietPi-Update | dietpi.txt is now checked for missing entries, and, will now be patched during the update: https://github.com/Fourdee/DietPi/issues/1292#issuecomment-350818969

Sparky SBC | Kernel updated, which resolves issues with HQPlayer playback: https://www.computeraudiophile.com/forums/topic/32132-allo-sparky-usbridge/?do=findComment&comment=753100

**Allo Web Interface v5:**

Sparky SBC: Matrix Audio X-SPDIF 2, native DSD is now added to kernel, many thanks @sudeep: sparkysbc/Linux#3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants