ratchet update exit with errors #80

ruzickap · 2024-04-18T03:29:04Z

New issue?

I have reviewed the Known Issues and confirm this is not a known issue.

TL;DR

I'm getting error panic: runtime error: index out of range [40] with length 40 when running ratchet update mega-linter.yml

Expected behavior

It should update the GitHub Action file or give me the "understandable" error.

Observed behavior

No response

Version

0.9.1

On what operating system are you using ratchet?

Mac OS

CI/CD workflow file

---
name: mega-linter

on:
  workflow_dispatch:
  push:
    branches-ignore:
      - main

permissions: read-all

jobs:
  mega-linter:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - name: Checkout Code
        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

      - name: Restore lychee cache
        uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2
        with:
          path: .lycheecache
          key: cache-lychee-${{ github.sha }}
          restore-keys: cache-lychee-

      - name: Extract commands from markdown files
        run: |
          set -euxo pipefail
          echo '#!/usr/bin/env bash' > README.sh
          find . -name '*.md' -print0 | while IFS= read -r -d '' FILE; do
            # Extract:  ... 
            sed -n "/^\`\`\`\(bash\|shell\)$/,/^\`\`\`$/p" "${FILE}" | sed '/^```*/d' >> README.sh
            # Extract:   ```bash ... ```
            sed -n "/^  \`\`\`\(bash\|shell\)$/,/^  \`\`\`$/p" "${FILE}" | sed '/^  ```*/d; s/^  //' >> README.sh
          done
          ls -la README.sh
          chmod a+x README.sh

      - name: 💡 MegaLinter
        uses: oxsecurity/megalinter@a7a0163b6c8ff7474a283d99a706e27483ddd80f # v7.10.0
        env:
          GITHUB_COMMENT_REPORTER: false
          # Disabled due to error: [GitHub Status Reporter] Error posting Status for REPOSITORY with ...: 403
          GITHUB_STATUS_REPORTER: false
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Relevant log output

❯ ratchet update mega-linter.yml
panic: runtime error: index out of range [40] with length 40

goroutine 1 [running]:
github.com/sethvargo/ratchet/command.fixIndentation(0x1400011e0f0)
	github.com/sethvargo/ratchet/command/command.go:180 +0x404
github.com/sethvargo/ratchet/command.loadYAMLFiles({0x104df0358, 0x14000200080}, {0x14000200050, 0x1, 0x3?}, 0x1)
	github.com/sethvargo/ratchet/command/command.go:147 +0x1d4
github.com/sethvargo/ratchet/command.(*UpdateCommand).Run(0x14000198060, {0x104df30b0, 0x1400003b180}, {0x140000122c0, 0x1, 0x1})
	github.com/sethvargo/ratchet/command/update.go:72 +0x144
github.com/sethvargo/ratchet/command.Run({0x104df30b0, 0x1400003b180}, {0x140000122b0?, 0x2?, 0x2?})
	github.com/sethvargo/ratchet/command/command.go:58 +0x278
main.realMain(...)
	github.com/sethvargo/ratchet/main.go:26
main.main()
	github.com/sethvargo/ratchet/main.go:18 +0xc0

Additional information

Steps to reproduce the error:

$ cd /tmp/
$ wget https://raw.githubusercontent.com/ruzickap/my-git-projects/main/gh-repo-defaults/my-defaults/.github/workflows/mega-linter.yml
$ ratchet update mega-linter.yml
panic: runtime error: index out of range [40] with length 40

goroutine 1 [running]:
github.com/sethvargo/ratchet/command.fixIndentation(0x140001981e0)
	github.com/sethvargo/ratchet/command/command.go:180 +0x404
github.com/sethvargo/ratchet/command.loadYAMLFiles({0x104970358, 0x14000110080}, {0x14000110050, 0x1, 0x3?}, 0x1)
	github.com/sethvargo/ratchet/command/command.go:147 +0x1d4
github.com/sethvargo/ratchet/command.(*UpdateCommand).Run(0x14000198060, {0x1049730b0, 0x1400003b180}, {0x140000122c0, 0x1, 0x1})
	github.com/sethvargo/ratchet/command/update.go:72 +0x144
github.com/sethvargo/ratchet/command.Run({0x1049730b0, 0x1400003b180}, {0x140000122b0?, 0x2?, 0x2?})
	github.com/sethvargo/ratchet/command/command.go:58 +0x278
main.realMain(...)
	github.com/sethvargo/ratchet/main.go:26
main.main()
	github.com/sethvargo/ratchet/main.go:18 +0xc0

The text was updated successfully, but these errors were encountered:

sethvargo · 2024-04-18T14:55:47Z

I know why this is happening, but I don't know WHY this is happening. The document separator (---) is messing up the line parsing. @ruzickap you can remove the document separator and everything will be happy. I need to think about how to fix this edge case though

ruzickap · 2024-04-18T17:03:12Z

Thank you, but starting the yaml files with --- is something considered as "good practice" and various yaml linters reports it (https://yamllint.readthedocs.io/en/latest/rules.html#module-yamllint.rules.document_start).

I will fait for the final fix...

Thank you

sethvargo · 2024-04-18T20:21:50Z

Yes, I know. You can remove it, run ratchet, and manually re-add it.

Instead of depending on a diff to restore newlines, this completely changes the algorithm to compare the input contents with the raw render YAML contents (before any mutations) to identify changes. We compute the places where newlines need to be injected, run Ratchet, and then insert those newlines back after modification. There are also now many more tests and a test harness that is more sustainable for adding tests and edge cases. This also introduces an intentionally-undocumented environment variable that dumps debugging information about whitespace computations when set. This will help with issue debugging. This results in a slight behavior change: YAML contents _will_ have consistently-formatted indentation now, but all newline whitespace changes are preserved, including block scalars. Fixes GH-80

sethvargo · 2024-04-19T22:24:59Z

#81

stdedos · 2024-04-25T12:39:56Z

I am having a number of unrelated errors with most of these files - even after removing ---:
https://github.com/pylint-dev/pylint-pytest/tree/master/.github

sethvargo · 2024-04-25T17:12:48Z

@stdedos I just cloned and ran ratchet against that repo at HEAD using the code in #81 and it ran without errors:

➜  pylint-pytest master ❃ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   .github/workflows/release.yml

no changes added to commit (use "git add" and/or "git commit -a")
➜  pylint-pytest master ❃ git diff
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 10b8cd1..4003613 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -21,10 +21,10 @@ jobs:
       url: https://pypi.org/project/pylint-pytest/
     steps:
       - name: Check out code from Github
-        uses: actions/checkout@v4
+        uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # ratchet:actions/checkout@v4
       - name: Set up Python ${{ env.DEFAULT_PYTHON }}
         id: python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # ratchet:actions/setup-python@v5
         with:
           python-version: ${{ env.DEFAULT_PYTHON }}
           check-latest: true

stdedos · 2024-04-25T18:55:08Z

Did you, by any chance, happen to "just pick the one that works"? 😅

$ find .github/ -name '*.yml' -or -name '*.yaml' | xargs -I{} ./ratchet pin {}
panic: runtime error: index out of range [95] with length 95

goroutine 1 [running]:
github.com/sethvargo/ratchet/command.fixIndentation(0xc0001121b0)
	github.com/sethvargo/ratchet/command/command.go:180 +0x4ed
github.com/sethvargo/ratchet/command.loadYAMLFiles({0x82c500, 0xc00010c070}, {0xc00010c040, 0x1, 0x416113?}, 0x1)
	github.com/sethvargo/ratchet/command/command.go:147 +0x255
github.com/sethvargo/ratchet/command.(*PinCommand).Run(0xc000015da0, {0x82f248, 0xc0000ab140}, {0xc000014140, 0x1, 0x1})
	github.com/sethvargo/ratchet/command/pin.go:83 +0x1cb
github.com/sethvargo/ratchet/command.Run({0x82f248, 0xc0000ab140}, {0xc000014130?, 0x2?, 0x2?})
	github.com/sethvargo/ratchet/command/command.go:58 +0x212
main.realMain(...)
	github.com/sethvargo/ratchet/main.go:26
main.main()
	github.com/sethvargo/ratchet/main.go:18 +0xba
panic: runtime error: index out of range [9] with length 9

goroutine 1 [running]:
github.com/sethvargo/ratchet/command.fixIndentation(0xc00028c0c0)
	github.com/sethvargo/ratchet/command/command.go:180 +0x4ed
github.com/sethvargo/ratchet/command.loadYAMLFiles({0x82c500, 0xc00011c070}, {0xc00011c040, 0x1, 0x416113?}, 0x1)
	github.com/sethvargo/ratchet/command/command.go:147 +0x255
github.com/sethvargo/ratchet/command.(*PinCommand).Run(0xc000191d70, {0x82f248, 0xc0001a3140}, {0xc000190110, 0x1, 0x1})
	github.com/sethvargo/ratchet/command/pin.go:83 +0x1cb
github.com/sethvargo/ratchet/command.Run({0x82f248, 0xc0001a3140}, {0xc000190100?, 0x2?, 0x2?})
	github.com/sethvargo/ratchet/command/command.go:58 +0x212
main.realMain(...)
	github.com/sethvargo/ratchet/main.go:26
main.main()
	github.com/sethvargo/ratchet/main.go:18 +0xba
panic: runtime error: index out of range [65] with length 65

goroutine 1 [running]:
github.com/sethvargo/ratchet/command.fixIndentation(0xc0000141e0)
	github.com/sethvargo/ratchet/command/command.go:180 +0x4ed
github.com/sethvargo/ratchet/command.loadYAMLFiles({0x82c500, 0xc0000320b0}, {0xc000032080, 0x1, 0x100000000000000?}, 0x1)
	github.com/sethvargo/ratchet/command/command.go:147 +0x255
github.com/sethvargo/ratchet/command.(*PinCommand).Run(0xc00011bd70, {0x82f248, 0xc00012b140}, {0xc00011a110, 0x1, 0x1})
	github.com/sethvargo/ratchet/command/pin.go:83 +0x1cb
github.com/sethvargo/ratchet/command.Run({0x82f248, 0xc00012b140}, {0xc00011a100?, 0x2?, 0x2?})
	github.com/sethvargo/ratchet/command/command.go:58 +0x212
main.realMain(...)
	github.com/sethvargo/ratchet/main.go:26
main.main()
	github.com/sethvargo/ratchet/main.go:18 +0xba

sethvargo · 2024-04-25T19:11:59Z

I'm using ratchet compiled from #81:

➜  Development  git clone https://github.com/pylint-dev/pylint-pytest
Cloning into 'pylint-pytest'...
remote: Enumerating objects: 1039, done.
remote: Counting objects: 100% (393/393), done.
remote: Compressing objects: 100% (230/230), done.
remote: Total 1039 (delta 223), reused 251 (delta 163), pack-reused 646
Receiving objects: 100% (1039/1039), 229.28 KiB | 14.33 MiB/s, done.
Resolving deltas: 100% (596/596), done.
➜  Development  cd pylint-pytest
➜  pylint-pytest master ✔ ratchet -v
ratchet (devel) (d905c9cd9c2f8741d02765f56b4d31681599e0fd, darwin/arm64)
➜  pylint-pytest master ✔ ratchet pin .github/workflows/*.{yml,yaml}
➜  pylint-pytest master ❃ git diff
diff --git a/.github/workflows/checks.yaml b/.github/workflows/checks.yaml
index 2ee5938..a66daa0 100644
--- a/.github/workflows/checks.yaml
+++ b/.github/workflows/checks.yaml
@@ -28,10 +28,10 @@ jobs:
       pre-commit-key: ${{ steps.generate-pre-commit-key.outputs.key }}
     steps:
       - name: Check out code from GitHub
-        uses: actions/checkout@v4
+        uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # ratchet:actions/checkout@v4
       - name: Set up Python ${{ env.DEFAULT_PYTHON }}
         id: python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # ratchet:actions/setup-python@v5
         with:
           python-version: ${{ env.DEFAULT_PYTHON }}
           check-latest: true
@@ -42,7 +42,7 @@ jobs:
             hashFiles('pyproject.toml', 'requirements/**.txt') }}" >> $GITHUB_OUTPUT
       - name: Restore Python virtual environment
         id: cache-venv
-        uses: actions/[email protected]
+        uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # ratchet:actions/[email protected]
         with:
           path: .venv
           key: >-
@@ -55,7 +55,7 @@ jobs:
             hashFiles('.pre-commit-config.yaml') }}" >> $GITHUB_OUTPUT
       - name: Restore pre-commit environment
         id: cache-precommit
-        uses: actions/[email protected]
+        uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # ratchet:actions/[email protected]
         with:
           path: ${{ env.PRE_COMMIT_CACHE }}
           key: >-
@@ -71,23 +71,23 @@ jobs:
     needs: prepare-base
     steps:
       - name: Check out code from GitHub
-        uses: actions/checkout@v4
+        uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # ratchet:actions/checkout@v4
       - name: Set up Python ${{ env.DEFAULT_PYTHON }}
         id: python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # ratchet:actions/setup-python@v5
         with:
           python-version: ${{ env.DEFAULT_PYTHON }}

# ...

Also note, you don't need xargs - see the command I ran above:

ratchet pin .github/workflows/*.{yml,yaml}

I'm pretty confident that merging #81 and releasing will resolve your issue.

Instead of depending on a diff to restore newlines, this completely changes the algorithm to compare the input contents with the raw render YAML contents (before any mutations) to identify changes. We compute the places where newlines need to be injected, run Ratchet, and then insert those newlines back after modification. There are also now many more tests and a test harness that is more sustainable for adding tests and edge cases. This also introduces an intentionally-undocumented environment variable that dumps debugging information about whitespace computations when set. This will help with issue debugging. This results in a slight behavior change: YAML contents _will_ have consistently-formatted indentation now, but all newline whitespace changes are preserved, including block scalars. Fixes GH-80

stdedos · 2024-04-25T19:49:15Z

On a tanget (and since you don't have Discussions), what is the logic of # ratchet:actions/[email protected] vs # ratchet:actions/setup-python@v5 supposed to be?

I think GHA parses "partial tags" (@v5) to support "auto-updating minors etc". However, any sha1 you may write can possibly only be a specific version.

ruzickap added the bug Something isn't working label Apr 18, 2024

sethvargo mentioned this issue Apr 19, 2024

Refactor formatting adjustments #81

Merged

sethvargo closed this as completed in #81 Apr 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ratchet update exit with errors #80

ratchet update exit with errors #80

ruzickap commented Apr 18, 2024

sethvargo commented Apr 18, 2024

ruzickap commented Apr 18, 2024

sethvargo commented Apr 18, 2024

sethvargo commented Apr 19, 2024

stdedos commented Apr 25, 2024

sethvargo commented Apr 25, 2024

stdedos commented Apr 25, 2024 •

edited

Loading

sethvargo commented Apr 25, 2024

stdedos commented Apr 25, 2024

ratchet update exit with errors #80

ratchet update exit with errors #80

Comments

ruzickap commented Apr 18, 2024

New issue?

TL;DR

Expected behavior

Observed behavior

Version

On what operating system are you using ratchet?

CI/CD workflow file

Relevant log output

Additional information

sethvargo commented Apr 18, 2024

ruzickap commented Apr 18, 2024

sethvargo commented Apr 18, 2024

sethvargo commented Apr 19, 2024

stdedos commented Apr 25, 2024

sethvargo commented Apr 25, 2024

stdedos commented Apr 25, 2024 • edited Loading

sethvargo commented Apr 25, 2024

stdedos commented Apr 25, 2024

stdedos commented Apr 25, 2024 •

edited

Loading