-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Configuration generates invalid yaml for multiline and object content #179
Comments
I was able to add to the ocm project a test case which I believe reproduced the problem. You can see what I added in this PR open-component-model/ocm#734 |
Think this may be a goccy/go-yaml bug. Either that or ocm is using replace incorrectly. |
adding @Skarlso |
Yeah, we are just using whateverm OCM does :) |
Uwe found the issue in the corresponding YAML library. goccy/go-yaml#447 |
@mandelsoft created goccy/go-yaml#447, but since @Skarlso found that the same issue has been reported already two years ago without any fix, most-likely we need to find a solution on our own. @mandelsoft mentioned that he may know an older lib which might act as basis for a fix. |
@dee0 confirmed that with using a certain format as input, the lib produces the wanted output. We'll close this issue now and open a new one, checking for a lib that can handle things better. |
Hey @morri-son I just tried a new ocm-controller build with @Skarlso's 'don't marshal primitives' change and I think the ocm-controller is still unusable for me. This because object data is not stored correctly by the configuration. Object data is still yielding invalid yaml. e.g. I get
but it needs to be
I have not tried the 'formatting the text the right way' trick for the multiline string data yet. And I don't know how I can reasonably do that. In the 'last mile config' we won't know how much indenting is required and in spiff++ I am not seeing a good way to determine and then apply the required amount of indenting. What I maybe able to do is to use spiff++'s 'asjson' function to double quote the multiline string. I'll try to give that a shot this weekend. |
Uhh. Probably our best option is to switch to yq lib. But it's going to be disruptive. |
Hey @Skarlso Actually it seems like it is pretty easy. At least if the change is limited to subst.go. While I have not pushed it up to github yet, I made the change in subst.go last night. While some of the tests in subst_test.go failed it looked like it was just due to formatting differences in the output. I plan on taking a closer look at the failures today. Hopefully I'll be able to resolve them and I'll push my change up to my fork. Oh, and I think as part of this I want to make sure the tests are covering
Regarding these tesst additions my thinking is
|
I didn't say it's hard. It's more like tedious. Also a completely breaking change. I did implement it using yq. I was able to throw out most of the code. The only missing feature was the struct based thing. The braking change is that the path must start with '.' And values need to be quoted. Which might break some other things and will definitely break existing tests and demos and component versions. |
adding @hilmarf and @fabianburth to follow the discussion. We can plan to switch to another lib, but as Gergely mentioned it will be both, requiring changes on implementation as well after that breaking change, on the consumer side... |
I even included a simple example on how to use yqlib on the yq repository. mikefarah/yq#2021 |
Here https://github.com/open-component-model/ocm/actions/runs/8873092383/job/24358384683?pr=734 The path doesn't need to start with a '.'. Consider, with goccy the path needed to start with '$.' and the substitution code took care of this under the covers. So in similar fashion the code I pushed is adding the required '.' under the covers. So that isn't a breaking change. Unless consumers are doing something silly like depending on the precise formatting of the output YAML or JSON code they shouldn't care about this change. And on that note, I switched the tests to use MatchYAML and MatchJSON instead of Equals on strings. Testing for string equality
Regarding the switch to CandidateNode in the SubstitutionTarget implementation. This is necessary to efficiently handle multiple updates to the same target. While I have pushed the changes up and all the existing tests are passing, things I would still like to do before considering this ready for handover to anyone are
|
Yes, you can do that in the background, but I would rather not do any "fixing of the yaml path" which is invisible to the user. That just opens up a bunch of problems when trying to debug failures. Any hidden modification of the yaml path is a problem. I would rather be up-front about what is required to make things work. Also regarding your implementation, that's almost the same as I wrote, with the expectation that I didn't bother with all the tree parsing. The default all parser can handle all of that and can be re-run as in the example I linked in. |
ok folks, then we keep this issue open until we really figured out which way we plan to go in the future, using doccy or anything else, e.g. yg. Once the discussion has finalised, we'll spin up a new issue with the exact steps to be done. |
yah, I think this issue is fine. Dan has some great ideas and we can spit-ball in this issue. :) |
Thanks :) As for making debugging easier,
Then a can run the yq cli myself to validate whatever substitutions are taking place Fwiw, this is pretty much what I was doing at the start of this except I was using go playground because I didn't goccy doesn't appear to have a cli. Actually, elaborating a bit more on what I was doing
So I didn't need to maintain my own ingress and extraction scripts and if there log messages that told me what I need to extract and what the expression used was I think that would go a long way to making it so that problems were easier to diagnose. |
This morning I fixed the lint error however I see there is a test failure. That test doesn't fail locally. This evening I added all the tests I said I wanted to add. And I confirmed that ocm-controller build with the changes in my ocm PR works. That is, the service I was trying to onboard to ocm+ocm-controller+flux deployment actually was deployed. :) I hope to have time tomorrow evening to look into that failing test + do a sanity check of the changes in the PR. Oh, and at least when I am running the tests within vs code yqlib is very chatty. I'll check if that is the case during normal execution and if so look into reducing its verbosity. |
There is a logger that it uses that you can set to debug mode at the begin to display it. |
// GetLogger returns the yq logger instance.
func GetLogger() *logging.Logger {
return log
} This is the thing that you need to call and set it to debug. |
At least within the context of this ticket, I think I have made all the changes I would like to see in OCM. This evening I
So from my perspective what is left is
|
## Description Fixes open-component-model/ocm-project#179 ## What type of PR is this? (check all applicable) - [ ] 🍕 Feature - [x ] 🐛 Bug Fix - [ ] 📝 Documentation Update - [ ] 🎨 Style - [ ] 🧑💻 Code Refactor - [ ] 🔥 Performance Improvements - [x ] ✅ Test - [ ] 🤖 Build - [ ] 🔁 CI - [ ] 📦 Chore (Release) - [ ] ⏩ Revert ## Related Tickets & Documents <!-- Please use this format link issue numbers: Fixes #123 https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> - Related Issue # 179 - Closes # (179) - Fixes # (179) > Remove if not applicable ## Screenshots <!-- Visual changes require screenshots --> ## Added tests? - [x ] 👍 yes - [ ] 🙅 no, because they aren't needed - [ ] 🙋 no, because I need help - [ ] Separate ticket for tests # (issue/pr) Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration ## Added to documentation? - [ ] 📜 README.md - [ x] 🙅 no documentation needed ## Checklist: - [x ] My code follows the style guidelines of this project - [ x] I have performed a self-review of my code - [ x] I have commented my code, particularly in hard-to-understand areas - [ x] I have made corresponding changes to the documentation - [ x] My changes generate no new warnings - [ x] I have added tests that prove my fix is effective or that my feature works - [ x] New and existing unit tests pass locally with my changes - [ x] Any dependent changes have been merged and published in downstream modules --------- Co-authored-by: Gergely Brautigam <[email protected]>
What happened:
My release failed with
ocm-system dmi-broker 10h False Could not load chart: cannot load values.yaml: error converting YAML to JSON: yaml: line 23: could not find expected ':'
Each of my dictionary and multiline string values were improperly indented. In the screenshot below you can see on the right hand side my file after Configuration was applied. Note that certs.rs.crt is improperly indented. On the left is a copy that I fixed 'by hand' and which is parsable.
The values.yaml in the dir resource in my component version contains
My config.yaml in my component version, that is my config for the ocm controller, contains
defaults
rules
Running the command
kubectl get -n broker configmap dmi-broker -o=yaml | yq '.data.config' | yq -C | less -iR
to look at the content of my configmap that is used in my cluster I see the following
Image below shows the 'pipeline' I have defined for the ocm-controller
I believe this is an ocm-controller bug because
What you expected to happen:
I expected the release to work of course :)
How to reproduce it (as minimally and precisely as possible):
I believe you just need to have a rule that is setting a multiline string value or an object value.
Anything else we need to know:
Not that I can think of
Environment:
My ocm-controller was built from this hash of this fork
https://github.com/dee0sap/ocm-controller/tree/228d0be45540590ea51b712d3fb21d2c2082ef72
because I need my quick fix for #68
The text was updated successfully, but these errors were encountered: