[enh] New config-panel mechanism

[zamentur]

The problem

VPN Client need to upload some files in config panel.
YunoHost/issues#1589
The config panel current system is to complicated (too much redundant code to write).

Solution

CLI / API design

This CLI / API has been rewrited in order to find a way to get a future uniform CLI/API for Global, Domain, App, User configuration/settings.

The old design of yunohost app setting APP KEY [-v VALUE] [-d] is not ok cause set actions are define with a GET HTPP verb...

Currently, it's like yunohost settings aka global settings but on steroid.

yunohost app config get APP [KEY] [--mode {classic|full|export}]

Get a description of available forms with current values

$ yunohost app config get vpnclient

Get a specific value

$ yunohost app config get vpnclient manual.vpn.server_port

Export values in a file as json

$ yunohost app config get vpnclient manual -m export --output-as json > /root/verysimple.cube

Why define --mode instead of a --fields option ?

I think it's boring for user to specify each fields wanted. I think we could offers some kind of pre defined views called mode here.

yunohost app config set APP [KEY] [-v VALUE] [-a QUERY_STRING] [-f ARGS_FILE]

Change settings inside a specific form

In interactive way

$ yunohost app config set vpnclient manual

Note: Current values are prefilled, so you can just type enter to keep like it was.

Prefilled with some answers

$ yunohost app config set vpnclient manual --args "server_port=443&server_proto=tcp"

Or

$ yunohost app config set vpnclient manual --args-file /root/verysimple.cube

Change a specific setting

$ yunohost app config set vpnclient manual.vpn.server_port -v 443

Config panel format

You can see an example of toml here: https://github.com/YunoHost/test_apps/blob/master/config_app_ynh/config_panel.toml

Default config script

With simple use case, packagers doesn't need anymore to write a config script, the toml is enough. If the config script doesn't exist, it will be added with this default code:

#!/bin/bash
source /usr/share/yunohost/helpers
ynh_abort_if_errors
final_path=$(ynh_app_setting_get $app final_path)
ynh_app_config_run $1

Option short key have to be unique in all the config_panel.toml

So you can't have

[manual.vpn.server_ip]
[advanced.dns.server_ip]

Indeed the real variable name is server_ip and here you have a conflict.
This, simplifies some part of the code, and avoid to have to use long name and do ./_ conversion.
In future, it could be decide to manipulate with FORM_ID like "manual_server_ip" or "advanced_server_ip"

No retrocompatibility

All previous config panel should be rewritten into 1.0 version.
The main breaking change is environment variables are shorten and lower case.
However, write a config panel in this new version is easier than 0.1 format.

New question types

email, url, date, time, color, range

This types are quite trivial, it refers to HTML5 dedicated input. Some pre-validation are defined for them.

text

This type allows to have a multiline input.
In cli mode, it runs your default editor to allow you to edit the multiline text.

tags

You could now select tags like here https://bootstrap-vue.org/docs/components/form-tags
You can defined an allowed list of choices with choices argument
The pattern attribute applies on all element

file

An input file question. It allows only single file (no multifile or folder).

The file mechanism inside moulinette is not used, cause these file questions are defined in toml not in action map :/
Instead we use a crazy base64 method. So this file input is not designed to support big file upload.

markdown and alert

Markdown and alert component allow to display some static or dynamic info to user.

Read and write values

bind argument mechanism

This new argument allows to define where read and write the value bind to the question.

Default behaviour

If you have not defined a specific getter/setter (see bellow), and without source argument it will read and save the value in app settings yaml file.

Read / write in a var of a configuration file

If you want to read and save the value into a variable (called like the option name) of a file (json, yaml, ini, php, py ...) you can do:

bind = ":__FINALPATH__/conf.json"

If you want to read and save the value into an other variable (email in the example) of a file (json, yaml, ini, php, py ...) you can do:

bind = "email:__FINALPATH__/conf.json"

Note: This mechanism is language agnostic, however it's monoline: you can't save multiline text or file in a variable with this method. If you need to save multiline content in a configuration variable, you should do it via a specific getter/setter.

If you want to read and save the value into an other variable (email in the example) after a regex in a file (json, yaml, ini, php, py ...) you can do:

bind = "^\[user\]>email:__FINALPATH__/conf.json"

Read / write an entire file

If you have a question of type fileor text you could want to save the content into a specific path on the system.

bind = "__FINALPATH__/logo.png"

Specific getter / setter

Simple getter

Here is an example of custom getter you can put in a config script

get__current_date() {
echo "$(date)"
}

Getter which rewrite an option

A more advanced example, where we rewrite the option argument
scripts/config

get__status() {
    if [ -f "/sys/class/net/tun0/operstate" ] && [ "$(cat /sys/class/net/tun0/operstate)" == "up" ]
    then
    cat << EOF
style: success
ask:
  en: Your VPN is running :)
EOF
    else
    cat << EOF
style: danger
ask:
  en: Your VPN is down
EOF
    fi
}

config_panel.toml

        [main.cube.status]
        ask = "Custom getter alert"
        type = "alert"
        style = "info"
        bind = "null" # no behaviour on

Setter

set__admin_password() {
    if [ -z "${admin_password}" ]
    then
        python -c "import crypt; print(crypt.crypt(\"${admin_password}\", \"\$6\$saltsalt\$\"))" > $final_path/password
        ynh_print_info "The hash of your password has been registered in $final_path/password"
    fi
}

I suggest to display info for user about change that has been made to help them to understand a bit what's going.

Validation and internal values

Validation can be made with regex through pattern argument

        pattern.regexp = '^.+@.+$'
        pattern.error = 'An email is required for this field'

You can also restrict several types with a choices list.

        choices.option1 = "Plop1"
        choices.option2 = "Plop2"
        choices.option3 = "Plop3"

Some other type specific argument exist like

type	validation arguments
number, range	min, max, step
file	accept
tags	limit
boolean	yes no

Finally, if you need specific or multi variable validation, you can use custom validators function:

validate__admin_password() {
    if [[ "${#admin_password}" -lt 8 ]]; then echo 'Too short password'; fi
}

Note: this validators is a bad example, cause password type questions are already checked...

visible

A visible property allow to display section or question based on a condition

[manual.auth.password]
visible = "login && ! private_key"

Note: visible are not managed in CLI

redact confidential info

Password are redact automatically in operation log, but you can add redact args for a specific option if needed.

Other actions than read, validate and save

Restart a service at the end

You can use the services key to specify which service need to be reloaded or restarted

services = [ 'nginx', '__APP__' ]

This argument could be on panel, section, or question.

Overwrite config panel mechanism

All main configuration helpers are overwritable, example:

ynh_app_config_apply() {
    
    # Stop vpn client
    touch /tmp/.ynh-vpnclient-stopped
    systemctl stop ynh-vpnclient

    _ynh_app_config_apply

    # Start vpn client
    systemctl start ynh-vpnclient
    rm -f /tmp/.ynh-vpnclient-stopped

}

New config helpers

ynh_get_var

ynh_set_var

ynh_app_config_get

ynh_app_config_show

ynh_app_config_validate

ynh_app_config_apply

ynh_app_config_run

Philosophy

I think we should discuss in next meeting how to avoid bad use of all of these things to avoid to lose users with too big/complicated configuration panels

PR Status

• Write english locale
• Test app install
• Boolean : what if we want to save 'yes' into the config file ?
• Is config_panel the right terms ? may be helpers should be called like ynh_app_forms_get ?
• panels in the code could be called forms instead...
• Unit test
• Don't generate each time the file checksum for upgrade
• list all reserved keywords and trigger an error if an option use one of this keyword
• Documentation for packagers

How to test

You need :

• [enh] Add prefill and multiline in prompt moulinette#290
• New config panel with file and other nice formitem yunohost-admin#366
• https://github.com/YunoHost/test_apps/tree/master/config_app_ynh

Validation

• Principle agreement 0/2 :
• Quick review 0/1 :
• Simple test 0/1 :
• Deep review 0/1 :

[Psycojoker]

We uses bottle as the web backend in the moulinette, it might be worth it looking at the builtin features for file upload like http://bottlepy.org/docs/dev/api.html?highlight=fileupload#bottle.FileUpload and see if we can integrate that directly into yunohost code '-' (and that's secured reviewed code for bottle ... normally)

[zamentur]

We uses bottle as the web backend in the moulinette, it might be worth it looking at the builtin features for file upload like http://bottlepy.org/docs/dev/api.html?highlight=fileupload#bottle.FileUpload and see if we can integrate that directly into yunohost code '-' (and that's secured reviewed code for bottle ... normally)

I already use Bottle for file argument in moulinette (see the PR about importing user with a CSV file). However, here it's a bit more difficult cause file questions are contained in configpanel.toml not in actionmap, that's why i use this base64 approach.