Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Log skipped files on debug level #3383

Merged
merged 2 commits into from
Oct 8, 2024
Merged

Log skipped files on debug level #3383

merged 2 commits into from
Oct 8, 2024

Conversation

rgmz
Copy link
Contributor

@rgmz rgmz commented Oct 7, 2024 •
edited
Loading

Description:

This was inspired by my comment here. Trace is extremely noisy1, such information should be logged on the debug level IMO.

$ ./trufflehog github ... --debug
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2024-10-07T19:50:42-04:00	info-2	trufflehog	starting scanner workers	{"count": 4}
2024-10-07T19:50:42-04:00	info-2	trufflehog	starting detector workers	{"count": 32}
2024-10-07T19:50:42-04:00	info-2	trufflehog	starting verificationOverlap workers	{"count": 4}
2024-10-07T19:50:42-04:00	info-2	trufflehog	starting notifier workers	{"count": 4}
2024-10-07T19:50:42-04:00	info-0	trufflehog	running source	{"source_manager_worker_id": "skxXF", "with_units": true}
2024-10-07T19:50:42-04:00	info-2	trufflehog	enumerating source	{"source_manager_worker_id": "skxXF"}
2024-10-07T19:50:42-04:00	info-2	trufflehog	Caching repository info	{"source_manager_worker_id": "skxXF"}
2024-10-07T19:50:42-04:00	info-1	trufflehog	Enumerating with token	{"source_manager_worker_id": "skxXF"}
2024-10-07T19:50:42-04:00	info-2	trufflehog	attempting to clone repo	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git"}
2024-10-07T19:50:42-04:00	info-0	trufflehog	Completed enumeration	{"source_manager_worker_id": "skxXF", "num_repos": 1, "num_orgs": 0, "num_members": 0}
2024-10-07T19:50:46-04:00	info-1	trufflehog	successfully cloned repo	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "subcommand": "git clone", "repo": "https://github.com/cohere-ai/tokenizer.git", "path": "/tmp/trufflehog-41733-1261391252", "args": []}
2024-10-07T19:50:46-04:00	info-0	trufflehog	scanning repo	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git"}
2024-10-07T19:50:46-04:00	info-2	trufflehog	skipping file: extension is ignored	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "mime": "application/x-sharedlib", "timeout": 60, "ext": ".so"}
2024-10-07T19:50:46-04:00	info-2	trufflehog	skipping file: extension is ignored	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "mime": "application/x-sharedlib", "timeout": 60, "ext": ".so"}
2024-10-07T19:50:46-04:00	info-2	trufflehog	skipping file: extension is ignored	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "mime": "application/x-sharedlib", "timeout": 60, "ext": ".so"}
2024-10-07T19:50:46-04:00	info-2	trufflehog	skipping file: extension is ignored	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "mime": "application/x-sharedlib", "timeout": 60, "ext": ".so"}
2024-10-07T19:50:46-04:00	info-2	trufflehog	skipping file: extension is ignored	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "mime": "application/x-sharedlib", "timeout": 60, "ext": ".so"}
2024-10-07T19:50:46-04:00	info-2	trufflehog	skipping file: extension is ignored	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "mime": "application/x-sharedlib", "timeout": 60, "ext": ".so"}
2024-10-07T19:50:46-04:00	info-2	trufflehog	skipping file: extension is ignored	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "mime": "application/x-sharedlib", "timeout": 60, "ext": ".so"}
2024-10-07T19:50:47-04:00	info-2	trufflehog	finished parsing git log.	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "total_log_size": 1593783}
2024-10-07T19:50:47-04:00	info-1	trufflehog	scanning staged changes	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "path": "/tmp/trufflehog-41733-1261391252"}
2024-10-07T19:50:47-04:00	info-2	trufflehog	finished parsing git log.	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "total_log_size": 0}
2024-10-07T19:50:47-04:00	info-1	trufflehog	scanning git repo complete	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "path": "/tmp/trufflehog-41733-1261391252", "time_seconds": 1, "commits_scanned": 93}
2024-10-07T19:50:47-04:00	info-2	trufflehog	finished scanning repo	{"source_manager_worker_id": "skxXF", "unit": "https://github.com/cohere-ai/tokenizer.git", "unit_kind": "repo", "repo": "https://github.com/cohere-ai/tokenizer.git", "duration_seconds": 0.892665195}
2024-10-07T19:50:47-04:00	info-0	trufflehog	finished scanning	{"chunks": 2381, "bytes": 11359546, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "4.981928411s", "trufflehog_version": "dev"}

Checklist:

  • Tests passing (make test-community)?
  • Lint passing (make lint this requires golangci-lint)?

Footnotes

  1. Why is debug 2 and trace 5? What is the meaningful difference between 3, 4, and 5? https://github.com/trufflesecurity/trufflehog/blob/dcf8363eaa78aab4242849351c7a53f93662420e/main.go#L267-L271 ↩

@rgmz rgmz requested a review from a team as a code owner October 7, 2024 23:46
@rgmz rgmz force-pushed the chore/log-levels branch from 19119fd to fa3d63f Compare October 7, 2024 23:47
@rgmz rgmz requested a review from a team as a code owner October 7, 2024 23:47
@rgmz
chore(github): remove duplicate 'scanning repo' message
a904023 
The exact same thing is logged in git.ScanCommits.
ahrav
ahrav approved these changes Oct 8, 2024
View reviewed changes
Copy link
Collaborator

@ahrav ahrav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for getting this updated.

@ahrav ahrav merged commit 23afcd7 into trufflesecurity:main Oct 8, 2024
13 checks passed
@rgmz rgmz deleted the chore/log-levels branch October 8, 2024 18:21
abmussani added a commit to abmussani/trufflehog that referenced this pull request Oct 9, 2024
@abmussani
Merge branch 'main' into fix-privatelink-snowflake-detector
1276dde 
* main: (79 commits)
  Log skipped files on debug level (trufflesecurity#3383)
  build: update retracted bluemonday ver (trufflesecurity#3369)
  Fix git binary handling and add a smoke test (trufflesecurity#3379)
  fix(deps): update module google.golang.org/protobuf to v1.35.1 (trufflesecurity#3382)
  Added Cisco Meraki API Key detector (trufflesecurity#3367)
  improved the agora detector (trufflesecurity#3360)
  fix(deps): update module github.com/xanzy/go-gitlab to v0.110.0 (trufflesecurity#3376)
  fix(deps): update golang.org/x/exp digest to 225e2ab (trufflesecurity#3371)
  fix(deps): update module golang.org/x/net to v0.30.0 (trufflesecurity#3373)
  fix(deps): update module golang.org/x/crypto to v0.28.0 (trufflesecurity#3372)
  chore(deps): update sigstore/cosign-installer action to v3.7.0 (trufflesecurity#3368)
  fix(deps): update module cloud.google.com/go/storage to v1.44.0 (trufflesecurity#3366)
  fix(deps): update module github.com/schollz/progressbar/v3 to v3.16.1 (trufflesecurity#3365)
  [refactor] - Decouple Metrics From Cache Implementation (trufflesecurity#3355)
  fix(deps): update module github.com/snowflakedb/gosnowflake to v1.11.2 (trufflesecurity#3363)
  Updated Cosign Install URL (trufflesecurity#3364)
  fix(deps): update module github.com/jedib0t/go-pretty/v6 to v6.6.0 (trufflesecurity#3361)
  Added Pattern test cases for detectors (trufflesecurity#3354)
  remove size check (trufflesecurity#3351)
  fix(deps): update module go.mongodb.org/mongo-driver to v1.17.1 (trufflesecurity#3357)
  ...

# Conflicts:
#	go.sum
@blsaccess blsaccess mentioned this pull request Oct 11, 2024
Merged
abmussani added a commit to abmussani/trufflehog that referenced this pull request Oct 14, 2024
@abmussani
Merge branch 'main' into box-detector
9c66a31 
* main: (127 commits)
  Update SaladCloud description (trufflesecurity#3399)
  fix tests (trufflesecurity#3400)
  [chore] Update custom detector default description (trufflesecurity#3398)
  add description to salad (trufflesecurity#3397)
  Add detector for SaladCloud API Keys (trufflesecurity#3273)
  fix(deps): update module github.com/xanzy/go-gitlab to v0.111.0 (trufflesecurity#3393)
  Add SliceContainsString common util (trufflesecurity#3395)
  fix: pr template link to golangci-lint (trufflesecurity#3392)
  fix(deps): update golang.org/x/exp digest to f66d83c (trufflesecurity#3389)
  Separate detector tests into unit/integration (trufflesecurity#3274)
  Manually upgrade github dep (trufflesecurity#3387)
  Updated Fastly Personal Token Detector (trufflesecurity#3386)
  fix(deps): update module google.golang.org/api to v0.200.0 (trufflesecurity#3391)
  [Fix] Snowflake privatelink Support (trufflesecurity#3286)
  Enhanced the easyinsight detector (trufflesecurity#3384)
  Log skipped files on debug level (trufflesecurity#3383)
  build: update retracted bluemonday ver (trufflesecurity#3369)
  Fix git binary handling and add a smoke test (trufflesecurity#3379)
  fix(deps): update module google.golang.org/protobuf to v1.35.1 (trufflesecurity#3382)
  Added Cisco Meraki API Key detector (trufflesecurity#3367)
  ...

# Conflicts:
#	pkg/engine/defaults.go
#	pkg/pb/detectorspb/detectors.pb.go
#	proto/detectors.proto
@ahrav ahrav mentioned this pull request Nov 12, 2024
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ahrav ahrav ahrav approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rgmz @ahrav