Improve optimization for "diacritics-modifiers"

[Glitchy-Tozier]

At the moment, keys like `, ^, ˇ, ', ¸, ~, etc. are only treated as their individual characters.
This drastically reduces this optimizer's potential to be used in other languages.

Example: Spanish uses

• ´ + a
• ´ + e
• ´ + i
• ´ + o
• ´ + u

These Acute Accents are not optimized for.
There are similar, but worse problems in languages such as French, Portugese, Swedish, Polish, and so on.

My suggestion: Properly optimize for all possible combinations that key might produce.
Example with the ^-key:

• ^ + space = ^ (Caret)
• ^ + ^ = ̂ (Circumflex)
• ^ + allowed letters, such as e = ê
• ^ + disallowed lettrs, such as r = nothing

[dariogoetz]

I think this is somewhat related to combos #44 as the diacretics are also used not simultaneously but before the letter characters.

As I noted in that issue, I will currently not be able to tackle such a more complex topic.

[Glitchy-Tozier]

I'm aware that you're short on time (as am I), but I figured I'd catalog this issue. :)

1. When implementing this feature, it might be a good idea to look for smart solutions that leverage the fact that unicode-characters such as ("◌̂") exist. Currently, we're using the wrong character ("^").
2. The closest thing I found to what would be great was the deunicode-crate, but it only removes diacritics, without also returning the diacritical mark itself.

[dariogoetz]

I believe that diacretics can be realized using "one-shot" layers. I added a standard_diacritics.yml as an example. The modifiers can now be specified both via MatrixPosition and a symbol making it possible to also use higher-layer symbols for one-shot layers (useful e.g. for á on a standard qwerty that requires a shift press).

[Glitchy-Tozier]

Interesting, I'll take a look sometime the next weeks!
I saw you did quite a bit of work, in many aspects of the optimizer! What's the logic behind those modifiers? How do you detect whether letters require those modifiers?

[dariogoetz]

The modifier type can be specified in the keyboard.yml config when specifying modifier locations. If a layer is activated using OneShot modifiers, it means that the keys specified in the config have to be typed (pressed and released) one after the other to activate the layer (in particular not "held").
This is just the regular configuration for the layers. So to have diacritics, you simply add one or several additional diacritics-layers on top of the existing ones (for Neo2-based layouts this would be a seventh and an eighth layer for forward and backward diacritics, see the example config; if you also want "hat" diacritics, you need another ninth one).

It is now also possible to specify modifiers both using MatrixPositions, so [col, row], and char. This enables you to let the modifier position move with the underlying layout in an optimization, if that is required (for the diacritics, for instance, the forward tick diacritics shall always be generated with the "`" key regardless of its position.

[Glitchy-Tozier]

Seems pretty neat, looking at the config!
Three questions:

1. What was the thought behind adding an additional setting for letting the modifier move? Isn't this covered by the fixed_keys-option (or something with a similar name)?
2. Does this new system have any important drawbacks, such as a loss of speed?
   • If yes, what are the drawbacks?
   • If no, why not incorporate diacritics into the default keyboard-configs?
3. How would you define behavior that mixes multiple layers, like:
   1. Press ^
   2. Press SHIFT+o
   3. ...to type Ô?

[dariogoetz]

Seems pretty neat, looking at the config! Three questions:

1. What was the thought behind adding an additional setting for letting the modifier move? Isn't this covered by the `fixed_keys`-option (or something with a similar name)?

The fixed_keys-option specifies, which symbols are not permuted when specifying a layout string (and thus also during the optimization). If the modifier is specified by a symbol on a fixed key, it will not move.

Previously, the modifier was specified by providing a MatrixPosition and therefore be independent of the layout. This, for instance, allows to keep the modifiers on the home row (home-row-mods) while still optimizing which letters shall be located on the home row.

The reason why it is now possible to also specify a symbol rather than a MatrixPosition is to allow diacritics to be generated by a higher-layer symbol (e.g. one of the french accents on a German QWERTZ-layout requires a shift-key). Also, now it is possible to optimize the location of the modifier.

2. Does this new system have any important drawbacks, such as a loss of speed?
   
   * If yes, what are the drawbacks?
   * If no, why not incorporate diacritics into the default keyboard-configs?

As soon as there is at least one one_shot layer configured, there is a ~30% performance penalty involved.

3. How would you define behavior that mixes multiple layers, like:
   
   1. Press `^`
   2. Press SHIFT+`o`
   3. ...to type `Ô`?

Mixing one_shot and hold layers is not possible. So real-world uppercase diacritics is not an option. Possible would be "one after the other"

1. Press ^
2. Press SHIFT
3. Press o
4. ...to type Ô

[Glitchy-Tozier]

The reason why it is now possible to also specify a symbol rather than a MatrixPosition is to allow diacritics to be generated by a higher-layer symbol (e.g. one of the french accents on a German QWERTZ-layout requires a shift-key). Also, now it is possible to optimize the location of the modifier.

Neat!

Mixing one_shot and hold layers is not possible. So real-world uppercase diacritics is not an option. Possible would be "one after the other"

1. Press ^
2. Press SHIFT
3. Press o
4. ...to type Ô

I still don't quite get that part. where would you put the Ô in this list?

base_layout:
  keys:
    # Row 0 (number row)
    ...
    # Row 2 (home row)
    - - ["⇩"]
      - ["u", "U", "\\", "⇱", "", "⊂"]
      - ["i", "I", "/", "⇠", "ι", "∫", "ì", "í"]
      - ["a", "A", "{",  "⇣", "α", "∀", "à", "á"]
      - ["e", "E", "}", "⇢", "ε", "∃", "è", "é"]
      - ["o", "O", "*", "⇲", "ο", "∈", "ò", "ó"]
      - ["s", "S", "?", "¿", "σ", "Σ"]
      - ["n", "N", "(", "4", "ν", "ℕ"]
      - ["r", "R", ")", "5", "ρ", "ℝ"]
      - ["t", "T", "-", "6", "τ", "∂"]
      - ["d", "D", ":", ",", "δ", "Δ"]
      - ["⇘"]
      - ["´", "~", "/", "˝", "", "˘"]
      - ["\n"]
    ...

[dariogoetz]

You would put it behind the - ["o", "O", "*", "⇲", "ο", "∈", "ò", "ó"]

base_layout:
  keys:
    # Row 0 (number row)
    ...
    # Row 2 (home row)
    - - ["⇩"]
      - ["u", "U", "\\", "⇱", "", "⊂"]
      - ["i", "I", "/", "⇠", "ι", "∫", "ì", "í"]
      - ["a", "A", "{",  "⇣", "α", "∀", "à", "á"]
      - ["e", "E", "}", "⇢", "ε", "∃", "è", "é"]
      - ["o", "O", "*", "⇲", "ο", "∈", "ò", "ó", "Ô"]
      - ["s", "S", "?", "¿", "σ", "Σ"]
      - ["n", "N", "(", "4", "ν", "ℕ"]
      - ["r", "R", ")", "5", "ρ", "ℝ"]
      - ["t", "T", "-", "6", "τ", "∂"]
      - ["d", "D", ":", ",", "δ", "Δ"]
      - ["⇘"]
      - ["´", "~", "/", "˝", "", "˘"]
      - ["\n"]
    ...

and then add another layer in modifiers

    - Left:
        type: one_shot
        value: ["^", "⇧"]
      Right:
        type: one_shot
        value: ["^", "⇗"]

This would activate the new layer containing upper case ^ diacritics by hitting ^, then Shift, and then the character (o)

For a full set of diacritics, you would need to add

1. a layer for the lowercase backward diacritics, e.g. "è" (already in the config)
2. a layer for the lowercase forward diacritics, e.g. "é" (already in the config)
3. a layer for the lowercase "hat" diacritics, e.g. "ê"
4. a layer for the uppercase backward diacritics, e.g. "È"
5. a layer for the uppercase forward diacritics, e.g. "É"
6. a layer for the uppercase "hat" diacritics, e.g. "Ê"

[Glitchy-Tozier]

Got it! Eventually, I might help flesh out standard_diacritics.yml with all the diacritics NEO-keyboards can type.
Should we close this issue?

[dariogoetz]

That would be awesome. Thanks.

If you have the time, maybe you can double check the involved performance hit.