Update Azure application credentials detector #2985

rgmz · 2024-06-18T14:42:37Z

Description:

This closes #1979, closes #2394, closes #3039, and closes #3362,

Depends on #2976.

Checklist:

Tests passing (make test-community)?
Lint passing (make lint this requires golangci-lint)?

CLAassistant · 2024-06-18T14:42:43Z

All committers have signed the CLA.

rgmz

Flagging outstanding stuff.

pkg/detectors/azure_entra/common.go

pkg/detectors/azure_entra/serviceprincipal/sp.go

pkg/detectors/azure_entra/serviceprincipal/v1/spv1.go

pkg/detectors/azure_entra/serviceprincipal/v2/spv2.go

pkg/detectors/uuids.txt

proto/detectors.proto

ankushgoel27 · 2024-10-30T07:44:29Z

I am seeing this error

{"level":"error","ts":"2024-10-30T06:42:32Z","logger":"trufflehog","msg":"goroutine 315 [running]:\nruntime/debug.Stack()\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/debug/stack.go:26 +0x5e\ngithub.co
m/trufflesecurity/trufflehog/v3/pkg/common.Recover({0x52cffe0, 0xc001c1fc50})\n\t/root/trufflehog/pkg/common/recover.go:17 +0x5b\npanic({0x416f7c0?, 0x783b830?})\n\t/root/go/pkg/mod/golang.org/[email protected]/src/ru
ntime/panic.go:785 +0x132\ngithub.com/trufflesecurity/trufflehog/v3/pkg/detectors/azure_entra/serviceprincipal/v2.isValidTenant.func1()\n\t/root/trufflehog/pkg/detectors/azure_entra/serviceprincipal/v2/spv2.go:253 +0x1b\npanic({0x416f7c0
?, 0x783b830?})\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/panic.go:785 +0x132\ngithub.com/trufflesecurity/trufflehog/v3/pkg/detectors/azure_entra/serviceprincipal/v2.isValidTenant({0x7689f31a6050, 0
xc025dddd40}, 0xc0013df710, {0xc026331384, 0x22})\n\t/root/trufflehog/pkg/detectors/azure_entra/serviceprincipal/v2/spv2.go:257 +0x112\ngithub.com/trufflesecurity/trufflehog/v3/pkg/detectors/azure_entra/serviceprincipal/v2.ProcessData({0
x7689f31a6050, 0xc025dddd40}, 0xc025dddd70, 0xc026792060, 0xc026792450, 0x1, 0xc0013df710)\n\t/root/trufflehog/pkg/detectors/azure_entra/serviceprincipal/v2/spv2.go:99 +0x573\ngithub.com/trufflesecurity/trufflehog/v3/pkg/detectors/azure_
entra/serviceprincipal/v1.Scanner.FromData({0xc0252efa70?, {}}, {0x7689f31a6050, 0xc025dddd40}, 0x1, {0xc010ab7500?, 0xc02464eee8?, 0x0?})\n\t/root/trufflehog/pkg/detectors/azure_entra/serviceprincipal/v1/spv1.go:64 +0x10a\ngithub.com/tr
ufflesecurity/trufflehog/v3/pkg/engine.(*Engine).detectChunk(0xc001402340, {0x52cffe0, 0xc001c1fc50}, {0xc01a8de000, {{0xc010ab7500, 0x3400, 0x3400}, {0x496dc24, 0x13}, 0x1, ...}, ...})\n\t/root/trufflehog/pkg/engine/engine.go:1055 +0x33
d\ngithub.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).detectorWorker(0xc001402340, {0x52cffe0, 0xc001c1fc50})\n\t/root/trufflehog/pkg/engine/engine.go:1024 +0x150\ngithub.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).
startDetectorWorkers.func1()\n\t/root/trufflehog/pkg/engine/engine.go:670 +0xdc\ncreated by github.com/trufflesecurity/trufflehog/v3/pkg/engine.(*Engine).startDetectorWorkers in goroutine 1\n\t/root/trufflehog/pkg/engine/engine.go:666 +0
x10f\n","detector_worker_id":"eC7Xi","recover":"runtime error: invalid memory address or nil pointer dereference","error":"panic"}

rgmz · 2024-10-30T11:03:47Z

0x132\ngithub.com/trufflesecurity/trufflehog/v3/pkg/detectors/azure_entra/serviceprincipal/v2.isValidTenant.func1()\n\t/root/trufflehog/pkg/detectors/azure_entra/serviceprincipal/v2/spv2.go:253

This was removed. Make sure you pull in the latest changes.

ahrav

A handful of non-blocking comments, but otherwise LGTM! Thanks for adding these.

pkg/detectors/azure_entra/common.go

pkg/detectors/azure_entra/serviceprincipal/sp.go

pkg/detectors/azure_entra/serviceprincipal/v1/spv1.go

pkg/detectors/azure_entra/serviceprincipal/v2/spv2.go

feat(azure): finish pending items for review

rgmz · 2024-11-20T20:55:11Z

A handful of non-blocking comments, but otherwise LGTM! Thanks for adding these.

Thanks @ahrav. I've made the requested changes.

rgmz · 2024-11-20T21:04:43Z

Test failure seems to be unrelated?

--- FAIL: TestAPKHandler (7.35s)
    --- FAIL: TestAPKHandler/apk_with_3_leaked_keys (7.35s)
        apk_test.go:63: 
            	Error Trace:	/home/runner/work/trufflehog/trufflehog/pkg/handlers/apk_test.go:63
            	Error:      	Not equal: 
            	            	expected: 942
            	            	actual  : 947
            	Test:       	TestAPKHandler/apk_with_3_leaked_keys
2024-11-20T21:02:48Z	error	context	error writing to data channel	{"mime": "text/plain; charset=utf-8", "timeout": 60, "error": "context canceled"}
2024-11-20T21:02:50Z	error	context	non-critical error processing chunk	{"timeout": 5, "mime": "application/zip", "timeout": 60, "error": "error extracting archive with format: .zip: determining stream size: fast-forwarding to end: simulated error during newFileReader"}
2024-11-20T21:02:50Z	error	context	non-critical error processing chunk	{"error": "error processing file"}
2024-11-20T21:02:50Z	error	context	non-critical error processing chunk	{"error": "EOF"}
FAIL
FAIL	github.com/trufflesecurity/trufflehog/v3/pkg/handlers	10.883s

ahrav · 2024-11-20T21:07:13Z

Test failure seems to be unrelated?

--- FAIL: TestAPKHandler (7.35s)
    --- FAIL: TestAPKHandler/apk_with_3_leaked_keys (7.35s)
        apk_test.go:63: 
            	Error Trace:	/home/runner/work/trufflehog/trufflehog/pkg/handlers/apk_test.go:63
            	Error:      	Not equal: 
            	            	expected: 942
            	            	actual  : 947
            	Test:       	TestAPKHandler/apk_with_3_leaked_keys
2024-11-20T21:02:48Z	error	context	error writing to data channel	{"mime": "text/plain; charset=utf-8", "timeout": 60, "error": "context canceled"}
2024-11-20T21:02:50Z	error	context	non-critical error processing chunk	{"timeout": 5, "mime": "application/zip", "timeout": 60, "error": "error extracting archive with format: .zip: determining stream size: fast-forwarding to end: simulated error during newFileReader"}
2024-11-20T21:02:50Z	error	context	non-critical error processing chunk	{"error": "error processing file"}
2024-11-20T21:02:50Z	error	context	non-critical error processing chunk	{"error": "EOF"}
FAIL
FAIL	github.com/trufflesecurity/trufflehog/v3/pkg/handlers	10.883s

Yea, doesn't look related.

rgmz · 2024-11-20T23:39:12Z

Yea, doesn't look related.

@ahrav It doesn't fail if I remove this entry from defaults.go.

		&ayrshare.Scanner{},
-		&azure_serviceprincipal_v1.Scanner{},
		&azure_serviceprincipal_v2.Scanner{},

I think it's because @joeleonjr pulls in DefaultDetectors() for APK scanning (#3517). I don't quite know why that's causing the test failure.

ahrav · 2024-11-20T23:51:28Z

Yea, doesn't look related.

@ahrav It doesn't fail if I remove this entry from defaults.go.
		&ayrshare.Scanner{},
-		&azure_serviceprincipal_v1.Scanner{},
		&azure_serviceprincipal_v2.Scanner{},
I think it's because @joeleonjr pulls in DefaultDetectors() for APK scanning (#3517). I don't quite know why that's causing the test failure.

Oh, I see what's happening. My guess is that the APK handler builds the ahoCorasick data structure using all the keywords from the detectors. Adding a new detector could potentially disrupt this test.

This change likely causes the archive handler to identify an additional piece of data—or, in this case, an entire file—due to the keywords included in the detector you mentioned.

I'll create a PR to fix the test. Thanks for helping me track this down.

ahrav · 2024-11-20T23:58:20Z

#3641

rgmz force-pushed the feat/azure-ad2 branch 18 times, most recently from edf16dd to 3acae1a Compare June 21, 2024 02:56

rgmz force-pushed the feat/azure-ad2 branch 2 times, most recently from 4a02d1a to 4a99e74 Compare July 1, 2024 18:39

rgmz mentioned this pull request Jul 2, 2024

Update Azure secretPatFmt #3039

Closed

2 tasks

rgmz mentioned this pull request Sep 7, 2024

Azure Detector Regex fix #3271

Closed

2 tasks

rgmz mentioned this pull request Oct 3, 2024

[Fix] Update Azure Client, Tenant Ids and Client Secret Detection Regexs. #3362

Closed

2 tasks

rgmz force-pushed the feat/azure-ad2 branch from 4a99e74 to 20b3f37 Compare October 9, 2024 16:30

rgmz commented Oct 9, 2024

View reviewed changes

rgmz marked this pull request as ready for review October 9, 2024 18:38

rgmz requested review from a team as code owners October 9, 2024 18:38

rgmz force-pushed the feat/azure-ad2 branch from d69f4c2 to c331a07 Compare October 9, 2024 21:03

rgmz force-pushed the feat/azure-ad2 branch 2 times, most recently from f603ceb to a2e26ac Compare November 3, 2024 15:48

rgmz requested a review from a team as a code owner November 3, 2024 15:48

rgmz force-pushed the feat/azure-ad2 branch 4 times, most recently from 438e11d to 9ae4378 Compare November 11, 2024 19:24

rgmz force-pushed the feat/azure-ad2 branch from 9ae4378 to 0c120db Compare November 20, 2024 02:38

rgmz requested a review from a team as a code owner November 20, 2024 02:38

rgmz force-pushed the feat/azure-ad2 branch 3 times, most recently from bcbb73f to 5244159 Compare November 20, 2024 03:19

rgmz mentioned this pull request Nov 20, 2024

Create detector for Azure refresh tokens #2978

Open

3 tasks

rgmz force-pushed the feat/azure-ad2 branch from 5244159 to d9f59e6 Compare November 20, 2024 17:05

ahrav approved these changes Nov 20, 2024

View reviewed changes

rgmz force-pushed the feat/azure-ad2 branch 2 times, most recently from 2c0b903 to 437e6d3 Compare November 20, 2024 20:49

feat(detectors): create azure_entra base package

9631cd7

feat(azure): finish pending items for review

rgmz force-pushed the feat/azure-ad2 branch from 437e6d3 to 9631cd7 Compare November 20, 2024 20:54

ahrav merged commit 4e243a0 into trufflesecurity:main Nov 20, 2024
12 of 13 checks passed

rgmz deleted the feat/azure-ad2 branch November 20, 2024 21:07

blsaccess mentioned this pull request Nov 21, 2024

Update trufflehog to 3.84.0 blacklanternsecurity/bbot#1995

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Azure application credentials detector #2985

Update Azure application credentials detector #2985

rgmz commented Jun 18, 2024 •

edited

Loading

CLAassistant commented Jun 18, 2024 •

edited

Loading

rgmz left a comment

ankushgoel27 commented Oct 30, 2024 •

edited

Loading

rgmz commented Oct 30, 2024

ahrav left a comment

rgmz commented Nov 20, 2024

rgmz commented Nov 20, 2024

ahrav commented Nov 20, 2024

rgmz commented Nov 20, 2024 •

edited

Loading

ahrav commented Nov 20, 2024

ahrav commented Nov 20, 2024

Update Azure application credentials detector #2985

Update Azure application credentials detector #2985

Conversation

rgmz commented Jun 18, 2024 • edited Loading

Description:

Checklist:

CLAassistant commented Jun 18, 2024 • edited Loading

rgmz left a comment

Choose a reason for hiding this comment

ankushgoel27 commented Oct 30, 2024 • edited Loading

rgmz commented Oct 30, 2024

ahrav left a comment

Choose a reason for hiding this comment

rgmz commented Nov 20, 2024

rgmz commented Nov 20, 2024

ahrav commented Nov 20, 2024

rgmz commented Nov 20, 2024 • edited Loading

ahrav commented Nov 20, 2024

ahrav commented Nov 20, 2024

rgmz commented Jun 18, 2024 •

edited

Loading

CLAassistant commented Jun 18, 2024 •

edited

Loading

ankushgoel27 commented Oct 30, 2024 •

edited

Loading

rgmz commented Nov 20, 2024 •

edited

Loading