Preserve formatting with in place writing

[tamalsaha]

Describe the bug
yq w -1 command removes blank lines and changes spacing before comments.

Input Yaml
Concise yaml document(s) (as simple as possible to show the bug)
data1.yml:

# Log level for operator
logLevel: 3

# Annotations applied to operator deployment
annotations: {}

# Annotations passed to operator pod(s).
podAnnotations: {}

# Node labels for pod assignment
nodeSelector:  # +doc-gen:break
  beta.kubernetes.io/os: linux
  beta.kubernetes.io/arch: amd64

Command
The command you ran:

yq w -i test.yaml logLevel 4

Actual behavior

# Log level for operator
logLevel: 4
# Annotations applied to operator deployment
annotations: {}
# Annotations passed to operator pod(s).
podAnnotations: {}
# Node labels for pod assignment
nodeSelector: # +doc-gen:break
  beta.kubernetes.io/os: linux
  beta.kubernetes.io/arch: amd64

Expected behavior

# Log level for operator
logLevel: 4

# Annotations applied to operator deployment
annotations: {}

# Annotations passed to operator pod(s).
podAnnotations: {}

# Node labels for pod assignment
nodeSelector:  # +doc-gen:break
  beta.kubernetes.io/os: linux
  beta.kubernetes.io/arch: amd64

Additional context
Add any other context about the problem here.

$ yq --version
yq version 3.3.2

[mikefarah]

Yep - this is due to the underlying go-yaml parser, I see someone has already raised an issue there:

go-yaml/yaml#627

[tamalsaha]

Thanks for the confirmation. Something else is also broken between v3.3.0 (working) and v3.3.2. I am running the following command:

yq r api/crds/installer.voyagermesh.com_voyageroperators.v1.yaml spec.versions[0].schema.openAPIV3Schema.properties.spec > /tmp/voyager-values.openapiv3_schema.yaml

You can see the files here: https://gist.github.com/tamalsaha/eeee8d9d2db6623bf1c77717ba3b2c2d

In v3.3.0, the formatting of strings remain unchanged (newlines are preserved). In v3.3.2, the newlines are gone.

[mikefarah]

Ah yeah I updated the go-yaml library in 3.3.1 which had a number of changes and fixes. That is an open issue against the lib (go-yaml/yaml#627)

[tamalsaha]

Looks like go-yaml decided to not fix the issue, rather just document it. 😞

[aakrem]

Would be helpful to preserve the original formatting!

[twbecker]

It would also be nice to be able to preserve document markers (---). Seems like a great tool but unfortunately not being able to preserve virtually all the formatting is a dealbreaker for us.

[markzporter]

Just wanted to second this, love the tool but also love our blank lines 😁

[Justinzobel]

It also changes our yml files from 4 spaces indentation to 2 :(

[ryenus]

@mikefarah, FYI, just saw yaml/libyaml#269 (comment), notably there's https://github.com/pantoniou/libfyaml

This is not on the TODO list for libyaml. None of us has time for this. I recommend to try out https://github.com/pantoniou/libfyaml which has support for this.

[escummy]

After exploring methods to update YAML files while preserving blank lines and comments, the following approach proved to be effective using yq in conjunction with sed, we can achieve the desired updates without altering the structure:

This method has been tested to work effectively with both yq version 4.x.x and version 3.x.x

(Initial) --> data1.yml

# Log level for operator
logLevel: 3

# Annotations applied to operator deployment
annotations: {}

# Annotations passed to operator pod(s).
podAnnotations: {}

# Node labels for pod assignment
nodeSelector:  # +doc-gen:break
  beta.kubernetes.io/os: linux
  beta.kubernetes.io/arch: amd64

• First, this command replaces all blank lines in the YAML file with a placeholder comment to prevent their removal.

# Replace blank lines with a temporary marker
sed -i '/^$/s// #BLANK_LINE/' /data1.yml

• Next, it updates the logLevel field using yq (Use the proper command depending your yq --version)

# Perform the update using 'yq' version 3.x.x
yq w -i /data1.yml logLevel 4

# Perform the update using 'yq' version 4.x.x
yq eval '.logLevel = 4' -i /data1.yml

• Finally, it restores the original blank lines by removing the placeholder comments.

# Restore blank lines
sed -i "s/ *#BLANK_LINE//g" /data1.yml

Then, the expected behavior is achieved by creating the same YAML file as described above, preserving the initial blank lines and ensuring that formatting or comments are not inadvertently disrupted, thus maintaining its clarity and structure.

(New) --> data1.yml

# Log level for operator
logLevel: 4

# Annotations applied to operator deployment
annotations: {}

# Annotations passed to operator pod(s).
podAnnotations: {}

# Node labels for pod assignment
nodeSelector:  # +doc-gen:break
  beta.kubernetes.io/os: linux
  beta.kubernetes.io/arch: amd64

(Tested)
By following these steps, you can ensure that modifications to your YAML file.

I hope this helps!