Failing to filtering out symlinks (using the clean-ignore example)

[PMPP]

Hi @newren ! Great work, I came across your tool, just after ignoring the git's suggestion to use, and after spending over 10hours waiting for filter-branch to finish what I wanted it to do, here I am!

What I am trying to achieve:

• Big monorepo
• Extract series of folders and root files into a new repo, keep its history

From:

/folder-1/folder-a
/folder-1/folder-c
/folder-2/folder-aa
/folder-2/folder-cc
file-a

To:

/folder-1/folder-a
/folder-2/folder-aa
file-a

I know the current repo used to have a folder-b and folder-bb inside the folders that I will keep, but I don't want to bring them along in the history.

with filter-branch I just git removed everything I didn't need then git ls-files > keep-these-ones , and tried to remove anything that was not in that list. For my case this gives me enough history, to make sense of past choices.

The example for clean-ignore roughly fits my case, or I wanted to believe it, so I could move on with my day.
.gitignore all the folders I don't want, add a couple of old folders that I know they used to exist, and we removed them and just run clean-ignore.

curl 'https://raw.githubusercontent.com/newren/git-filter-repo/main/git-filter-repo' > git-filter-repo
cp git-filter-repo git_filter_repo.py
curl 'https://raw.githubusercontent.com/newren/git-filter-repo/refs/heads/main/contrib/filter-repo-demos/clean-ignore' > clean-ignore
python3 clean-ignore --force

The issue I am facing, "seems" to happen due to symlinks that exist in the repo, where perhaps clean-ignore attempts to remove the symlink, but the target of the symlink is something that is not being .gitignore, resulting in:

Parsed 1655 commitsfatal: pathspec 'frontend/some-app/app/Resources/translations/messages.ms.po' is beyond a symbolic link
Traceback (most recent call last):
  File "/home/pmpp/please-god-make-it-work/my-repo/clean-ignore", line 81, in <module>
    main()
  File "/home/pmpp/please-god-make-it-work/my-repo/clean-ignore", line 77, in main
    filter.run()
  File "/home/pmpp/please-god-make-it-work/my-repo/git_filter_repo.py", line 4892, in run
    self._parser.run(self._input, self._output)
  File "/home/pmpp/please-god-make-it-work/my-repo/git_filter_repo.py", line 1527, in run
    self._parse_commit()
  File "/home/pmpp/please-god-make-it-work/my-repo/git_filter_repo.py", line 1378, in _parse_commit
    self._commit_callback(commit, aux_info)
  File "/home/pmpp/please-god-make-it-work/my-repo/git_filter_repo.py", line 4108, in _tweak_commit
    self._commit_callback(commit, self.callback_metadata(aux_info))
  File "/home/pmpp/please-god-make-it-work/my-repo/clean-ignore", line 68, in skip_ignores
    bad = self.get_ignored(filenames)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pmpp/please-god-make-it-work/my-repo/clean-ignore", line 52, in get_ignored
    (source, linenum, pattern, pathname) = result.split(b"\0")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 4, got 1)
fatal: stream ends early
fast-import: dumping crash report to .git/fast_import_crash_84569

And the dump cat .git/fast_import_crash_84569:

fast-import crash report:
    fast-import process: 84569
    parent process     : 84561
    at 2025-01-29 03:55:56 +0000

fatal: stream ends early

Most Recent Commands Before Crash
---------------------------------

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000
*

Active Branch LRU
-----------------
    active_branches = 1 cur, 5 max

  pos  clock name
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   1)      0 refs/tags/v1.1.4

Inactive Branches
-----------------
refs/tags/v1.1.4:
  status      : active dirty
  tip commit  : 0000000000000000000000000000000000000000
  old tree    : 0000000000000000000000000000000000000000
  cur tree    : 0000000000000000000000000000000000000000
  commit clock: 0
  last pack   :


Marks
-----
:1 1a8f11f2d9485a41810124eb52b2f6d730667b0a

-------------------
END OF CRASH REPORT

I didn't manage to find any issues related to symlinks, nor reference in the docs to address them, so here to inform this happens (perhaps its expected and I don't know better) and ... always open to suggestions if there is a better way to achieve my goal :)

Thank you for giving such a great tool to the community!