Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scan GitHub and GitLab refs that aren't pulled by default #1588

Open
6 tasks
rgmz opened this issue Aug 1, 2023 · 2 comments · May be fixed by #1918
Open
6 tasks

Scan GitHub and GitLab refs that aren't pulled by default #1588

rgmz opened this issue Aug 1, 2023 · 2 comments · May be fixed by #1918

Comments

@rgmz
Copy link
Contributor

rgmz commented Aug 1, 2023

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

This article featured on HackerNews reminded me that both GitHub and GitLab have refs for (merge|pull) requests that — to my knowledge — aren't pulled by default. This is especially juicy for TruffleHog because PRs may contain commits or alternate history with valid secrets that technically no longer exist in the repository.

GitHub

$  git clone https://github.com/trufflesecurity/trufflehog.git
$ cd trufflehog
$ git ls-remote
From https://github.com/trufflesecurity/trufflehog.git
69021f59c57b6a49bafcc8d1827e355405330014        HEAD
aa3ba817175e1bcdf852f11ffa327c1d134ecb26        refs/heads/0x1/add-on-prem-verification-flag
ade5d91d5ca94e996377b1909a802dd9dbc51b6b        refs/heads/1560-error-during-git-scan-using-pre-commit
...
c3644ccaf1bb0799ef919a5897b7737ebff428db        refs/pull/1/head
180b9c288bfc2b0f0543f91dfd74740f4f1d410e        refs/pull/100/head
c36defae7818b738389815dd32a64db398e2d833        refs/pull/100/merge
...
3048c6429d80094b42fd44919e3d6768536178ce        refs/tags/2.0.97
6d2dc5e965f9d5c23acea0c0c6377b06a8ae6c7c        refs/tags/v0.1.0-alpha
fda044631b344997a4556f52aadbd7c8275d0802        refs/tags/v3.0.0

GitLab

$ git clone [email protected]:pdftk-java/pdftk.git
$ cd pdftk
$ git ls-remote
From [email protected]:pdftk-java/pdftk.git
be326bcca502dcdcbfd6dc63f31976ea5aa7ffc9        HEAD
0d30918c940cae11e48aeb9c86857bb1455fd52a        refs/heads/ci
6c6fccca904b9234fa20e5c372e4573c0442156d        refs/heads/ci_native
...
53680180fef65bc6cd9fedc90b345ef354ab8c2c        refs/merge-requests/1/head
a86eca79300f9b0db0d2af2b6cb411168e8df16a        refs/merge-requests/12/head
696c8d99b25552123344bda752d3df6b8929adab        refs/merge-requests/12/merge
...
e80603a9130240019417eec30d7edad6cecba325        refs/tags/v3.0.0
4e20310b472b52cca12ad94252f90cff6cfebad6        refs/tags/v3.0.1

Others

  • Bitbucket
  • Gitea
  • Gogs
  • Sourcehut
  • Forgejo
  • ?

Problem to be Addressed

Find valid secrets that may be hidden in old PR histories.

Description of the Preferred Solution

Off the top of my head, I do not have any ideas on how to tackle this. It seems that you need to manually query for (git ls-remote) and checkout each ref in a detached state. The history will not show up via git log otherwise.

Scanning refs could result in the same commits being scanned twice. I am not sure if this would produce duplicate results, or if we already deduplicate results.

Additional Context

N/A

References

N/A

@rgmz
Copy link
Contributor Author

rgmz commented Sep 5, 2023

This seems trivial to implement:

# assuming the origin is GitHub, which can be checked with `git ls-remote`
git fetch origin "+refs/pull/*:refs/heads/pull/*"

@0x736E
Copy link

0x736E commented Sep 15, 2023

can this behaviour be confirmed to be present in TruffleHog?

@rgmz rgmz linked a pull request Oct 18, 2023 that will close this issue
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging a pull request may close this issue.

2 participants