app: config: Add support for appending to the config string #768

carlescufi · 2024-11-14T10:50:13Z

In some cases, and specifically in the manifest.group-filter and manifest.project-filter options, it is sometimes useful to be able to append to a value instead of replacing it completely.

For example, assuming one wants to add to an existing group filter, without this patch the user needs to do:

(assuming the group filter is currently +unstable,-optional, and the
user wants to add +extras).
$ west config manifest.group-filter
$ west config manifest.group-filter +unstable,-optional,+extras

With this patch instead:

$ west config -a manifest.group-filter ,+extras

src/west/app/config.py

pdgendt · 2024-11-14T11:04:28Z

I do have a concern about the appending if the value doesn't exist yet.

Something like:

$ west config -a manifest.group-filter ,+extras

Could result in an error because of the leading comma.

carlescufi · 2024-11-14T11:07:57Z

I do have a concern about the appending if the value doesn't exist yet.

I also thought about that, but did not want to overcomplicate the code. But yes, let me do that as well.

src/west/app/config.py

pdgendt

LGTM, can you add tests?

carlescufi · 2024-11-14T14:31:53Z

LGTM, can you add tests?

yep, on it!

marc-hb

LGTM, can you add tests?

yep, on it!

I think these won't involve shlex.split() because Zephyr will not be involved but please use some whitespace anyway in test data for more "stress".

src/west/app/config.py

carlescufi · 2024-11-15T18:10:34Z

tests/test_config.py

@@ -229,6 +229,9 @@ def test_local_creation_with_topdir():
    assert 'pytest' not in cfg(f=GLOBAL)
    assert cfg(f=LOCAL, topdir=str(topdir))['pytest']['key'] == 'val'

+def test_append_basic():


tests are WIP!

src/west/app/config.py

Simplify the error handling of the different action flags (-l, -d and -D) by using a parser mutually exclusive group. Signed-off-by: Carles Cufi <[email protected]>

In some cases, and specifically in the manifest.group-filter and manifest.project-filter options, it is sometimes useful to be able to append to a value instead of replacing it completely. For example, assuming one wants to add to an existing group filter, without this patch the user needs to do: (assuming the group filter is currently +unstable,-optional, and the user wants to add +extras). > west config manifest.group-filter > west config manifest.group-filter +unstable,-optional,+extras With this patch instead: > west config -a manifest.group-filter ,+extras Signed-off-by: Carles Cufi <[email protected]>

src/west/app/config.py

pdgendt

TEST APPROVAL, DO NOT MERGE

It was only a test, PR can be merged with formatting failures, as expected.

marc-hb

@carlescufi please try to follow the formatting suggestions from ruff for NEW or CHANGED code but please avoid git churn in unchanged code, thanks!

src/west/app/config.py

carlescufi · 2024-11-29T08:59:37Z

@carlescufi please try to follow the formatting suggestions from ruff for NEW or CHANGED code but please avoid git churn in unchanged code, thanks!

Honestly I have to say I disagree with this approach, and I already told @pdgendt. I think we either reformat the codebase to match ruff or we don't add the (IMHO misleading) check that is there today in CI.

kartben · 2024-11-29T14:51:03Z

@carlescufi please try to follow the formatting suggestions from ruff for NEW or CHANGED code but please avoid git churn in unchanged code, thanks!

Honestly I have to say I disagree with this approach, and I already told @pdgendt. I think we either reformat the codebase to match ruff or we don't add the (IMHO misleading) check that is there today in CI.

@pdgendt my guess is you just forgot to add a ruff-excludes.toml file, did you not? The way it's done today it's preventing folks to contribute until they fix the entire file they're touching, which sounds broken (and unintentional). FWIW this bit me here #713 too.

pdgendt · 2024-11-29T14:54:06Z

@pdgendt my guess is you just forgot to add a ruff-excludes.toml file, did you not? The way it's done today it's preventing folks to contribute until they fix the entire file they're touching, which sounds broken (and unintentional). FWIW this bit me here #713 too.

Not really, this was the outcome after some offline discussions with @marc-hb, which were also reasonable, but not the same as the approach that was taken for Zephyr itself.

kartben · 2024-11-29T14:58:39Z

@pdgendt my guess is you just forgot to add a ruff-excludes.toml file, did you not? The way it's done today it's preventing folks to contribute until they fix the entire file they're touching, which sounds broken (and unintentional). FWIW this bit me here #713 too.

Not really, this was the outcome after some offline discussions with @marc-hb, which were also reasonable, but not the same as the approach that was taken for Zephyr itself.

So you mean you will be just ignoring the CI failures for the time being if they are for lines that were not changed by the contributor? Sorry if I am missing something obvious, but I don't understand why it was done differently than for Zephyr itself (plus, of course, I have no idea about the content and outcome of the offline discussions you're referring to ;))

pdgendt · 2024-11-29T15:03:26Z

So you mean you will be just ignoring the CI failures for the time being if they are for lines that were not changed by the contributor? Sorry if I am missing something obvious, but I don't understand why it was done differently than for Zephyr itself (plus, of course, I have no idea about the content and outcome of the offline discussions you're referring to ;))

Yeah, I understand the confusion, a GH job can only be green or red, nothing in between. The current idea was; contributors can be asked to update the format, but if only a few lines are touched, it can get merged without it.

There aren't as many contributors to west as there are for Zephyr luckily...

pdgendt · 2024-11-29T15:05:28Z

A large part of the discussion is online: #766

marc-hb · 2024-11-30T18:59:43Z

I think we (1) either reformat the codebase to match ruff or we don't add the (IMHO misleading) check that is there today in CI.

(1) means the git history is practically destroyed. Especially problematic considering the main author left.
(2) means no formatting progress.

So you mean you will be just ignoring the CI failures for the time being if they are for lines that were not changed by the contributor?

Yes, formatting issues are obviously not blocking. As mentioned by Pieter, Github has unfortunately no yellow color. BTW this is not just a problem with formatting; every warning in every linter has the same CI problem.

Sorry if I am missing something obvious,

Agree it could/should be more obvious. Pieter could you maybe rename "Format check" to "Format _suggestion"? Basically: try to express "yellow" in words.

but I don't understand why it was done differently than for Zephyr itself (plus, of course, I have no idea about the content and outcome of the offline discussions you're referring to ;))

We're even then because I don't know how it was done for Zephyr itself :-)

In #766 I gave a real-world example where this approach worked well. It was not formatting but pylint and shellcheck; from a CI perspective the process is the same.

my guess is you just forgot to add a ruff-excludes.toml file

An exclude file is interesting, is that how Zephyr solves this problem in general? Not just with formatting but with every linter? Unlike (1) and (2), this allows some form of incremental progress but there are still smaller "big bangs" where the entire git history gets broken on a file-by-file basis. west has unfortunately a small number of giant files.

An exclude file does not solve the problem of changes in the code or the configuration of the formatting tool itself.

marc-hb · 2024-11-30T19:22:21Z

To be clear: I do not share Pieter's (and others') "enthusiasm" for formatting. Unlike C/C++, Python already enforces indentation = a little bit of formatting and in practice that seems enough to me, I mean I rarely ever scratch my head trying to understand badly formatted Python code. So, ruff only looks like a nice-to-have but nothing more.

On the other hand, I use git blame/rebase/cherry-pick/merge all the time and degrading those commands affects productivity greatly. Even worse when there's a reformatting every year because of some ruff upgrade or configuration change.

pdgendt · 2024-12-02T08:33:17Z

To be clear: I do not share Pieter's (and others') "enthusiasm" for formatting. Unlike C/C++, Python already enforces indentation = a little bit of formatting and in practice that seems enough to me, I mean I rarely ever scratch my head trying to understand badly formatted Python code. So, ruff only looks like a nice-to-have but nothing more.

I would say the opposite, I "hate" formatting, as in, I don't want to spend time on it. Working on a project where code is in a certain style, I need to "learn" how formatting is done, and carefully curate whatever modifications or additions I do.
I want to focus on the content rather than the looks, and in the best case scenario, tools to do formatting for me.

For python projects I have worked with black in the past, and the ruff formatter is like black, but faster. They have a commitment towards compatibility too, so I have some trust in that the formatting doesn't change drastically.

On the other hand, I use git blame/rebase/cherry-pick/merge all the time and degrading those commands affects productivity greatly. Even worse when there's a reformatting every year because of some ruff upgrade or configuration change.

And I agree, that's why I'm not a fan of blindly applying it to all files, but gradually. Either by exclusion or non-forcing checks. Both have downsides, but I'm happy to adjust.

Currently tooling in and around Zephyr is the wild-wild-west. Maintainers get to call the shots, and for good reason, but this does have the downside of different approaches in each. I introduced ruff in west and Zephyr hoping to get a more uniform tooling codebase. And formatting in an uncompromising way.

marc-hb · 2024-12-02T15:56:07Z

I would say the opposite, I "hate" formatting, as in, I don't want to spend time on it.
[...]
Currently tooling in and around Zephyr is the wild-wild-west. Maintainers get to call the shots, and for good reason, but this does have the downside of different approaches in each.

I don't know about Zephyr more broadly, but in west there had never been a formatter and there was no "time spent" on formatting either. There could have been some suggestions to try this or that formatting tool to help with "extreme" cases (e.g. very long lines) but if there were, it was so rare that I don't remember any such event. So I don't think formatting has ever caused anyone to "spend any time" in west. I don't remember that for Python in Zephyr either.

As I said, C/C++ offers a lot more editing "creativity" than Python so maybe formatting has been a problem there but for west specifically I'm afraid ruff is a solution in search of a problem.

What would be perfectly non-intrusive is some sort of ruff_format_patch tool that touches only new or changed code like checkpatch.pl does. BTW that's what allows checkpatch not to be worried about being "--subjective", or having contradictions or inconsistencies across checkpatch versions: because it looks only at changed lines. This makes a massive... difference! (pun intended)
From https://www.kernel.org/doc/html/latest/dev-tools/checkpatch.html

Checkpatch is not always right. Your judgement takes precedence over checkpatch messages. If your code looks better with the violations, then its probably best left alone.

pdgendt · 2024-12-02T16:12:08Z

I don't know about Zephyr more broadly, but in west there had never been a formatter and there was no "time spent" on formatting either. There could have been some suggestions to try this or that formatting tool to help with "extreme" cases (e.g. very long lines) but if there were, it was so rare that I don't remember any such event. So I don't think formatting has ever caused anyone to "spend any time" in west. I don't remember that for Python in Zephyr either.

By "time spent", I meant what I do while developing, not commenting on contributions. I have developed in many different languages, mostly out of curiosity and learning new things. And languages like go, rust, zig, ... (yes all the new kids) come with formatting guidelines and tooling out of the box.
There is no nit-picking, or preference of appearance as the language has set clear defaults.

Python has the PEP8 guideline which allows much more than what black or ruff currently output, but we currently don't comply either.

As I said, C/C++ offers a lot more editing "creativity" than Python so maybe formatting has been a problem there but for west specifically I'm afraid ruff is a solution in search of a problem.

That was my earlier point on trying have uniform code across projects under the Zephyr umbrella, and without tooling to enforce it, it's always a never-ending discussion.

What would be perfectly non-intrusive is some sort of ruff_format_patch tool that touches only new or changed code like checkpatch.pl does. BTW that's what allows checkpatch not to be worried about being "subjective", or having contradictions or inconsistencies across checkpatch versions: because it looks only at changed lines. It makes a massive... difference! (pun intended)

kernel.org/doc/html/latest/dev-tools/checkpatch.html

Checkpatch is not always right. Your judgement takes precedence over checkpatch messages. If your code looks better with the violations, then its probably best left alone.

You can turn off formatting just as easy for code blocks, but this shouldn't be the default IMO.

marc-hb · 2024-12-02T17:25:46Z

And languages like go, rust, zig, ... (yes all the new kids) come with formatting guidelines and tooling out of the box.

I like that too - but as you know that's not the reality for Python. Except for one very important thing: indentation.

and without tooling to enforce it, it's always a never-ending discussion.

I agree it becomes a never-ending discussion without tooling BUT only if maintainers decide to engage in that discussion in the first place. That has never been the case with west until now. I've been around for a few years and I didn't see any formatting discussion with Python in Zephyr either. Indentation likely helped.

marc-hb · 2024-12-02T21:03:31Z

... engage in that discussion in the first place. That has never been the case with west until now. [...] Indentation likely helped.

I just remembered that west had some codestyle checks in CI since forever. That likely helped with readability and reduced formatting needs even more. Formatting is just a small, less important part of code style and readability.

carlescufi requested a review from pdgendt November 14, 2024 10:50

carlescufi force-pushed the config-append branch from 9301226 to 717a09c Compare November 14, 2024 10:53