Remove invalid user EIDs and UIDs from bid request #3891

Pubmatic-Supriya-Patil · 2024-08-28T06:08:01Z

This PR solves issue #3859

Pubmatic-Supriya-Patil · 2024-09-04T03:56:48Z

Hi @bsardo @bretg
Please review this PR and let me know if any changes are required.

SyntaxNode · 2024-11-01T03:26:26Z

errortypes/code.go

@@ -35,6 +35,8 @@ const (
 	InvalidBidResponseDSAWarningCode
 	SecCookieDeprecationLenWarningCode
 	SecBrowsingTopicsWarningCode
+	InvalidUserEIDsWarningCode
+	InvalidUserUIDsWarningCode


Do these new warning codes match with PBS-Java? We try to keep in sync.

Can you please help me to get those error codes. Do we have any document or PR to check new warning codes match with PBS-Java.

As per prebid-server-java/pull/3465, I understand that the same 999 warning code is being used when removing eids arrays. In the Prebid Server Go repository, 999 corresponds to UnknownErrorCode in the error codes reference. Are we intending to use UnknownErrorCode here as well?

In this PR, I introduced InvalidUserEIDsWarningCode (10013) and InvalidUserUIDsWarningCode (10014). If we are okay with a generic error code, we could use UnknownErrorCode (999). However, please confirm whether we want separate error codes for InvalidUserEIDs and InvalidUserUIDs, or if using 999 would suffice.

Kindly provide your input on this.

SyntaxNode · 2024-11-01T03:27:01Z

endpoints/openrtb2/sample-requests/invalid-whole/user-ext-eids-uids-id-empty.json

@@ -1,42 +0,0 @@
-{


Instead of removing these tests cases, please modify them to prove the new behavior.

SyntaxNode · 2024-11-01T03:28:31Z

endpoints/openrtb2/auction.go

@@ -1296,8 +1296,14 @@ func (deps *endpointDeps) validateUser(req *openrtb_ext.RequestWrapper, aliases
 	// Check Universal User ID
 	eids := userExt.GetEid()
 	if eids != nil {
-		eidsValue := *eids
-		for eidIndex, eid := range eidsValue {
+


Nitpick; Please remove the leading empty line.

SyntaxNode · 2024-11-01T03:30:17Z

endpoints/openrtb2/auction.go

+
+		if len(eidErrors) > 0 {
+			errL = append(errL, eidErrors...)
+		}


Likely fine to keep this logic here for now. We want to separate validation from fixing in the future. We can refactor this new logic along with the rest later. Thoughts @bsardo?

SyntaxNode · 2024-11-01T03:32:18Z

endpoints/openrtb2/auction.go

+			validEIDs = append(validEIDs, eid)
+		} else {
+			errorsList = append(errorsList, &errortypes.Warning{
+				Message:     fmt.Sprintf("Removed EID with empty UIDs (source: %s)", eid.Source),


Please keep with the established error message format. Perhaps something like:

request.user.ext.eids[0] (source: %s) contains only empty uids and is removed from the request

SyntaxNode · 2024-11-01T03:37:56Z

endpoints/openrtb2/auction_test.go

+		expectedErrorMessages []string
+	}{
+		{
+			name: "Valid EID with non-empty UID",


Go test names cannot contain spaces so they are replaced with a dash character. This makes it hard to locate the failing test based on its name. Recommend using simpler test names like:

one-eid-one-uid-valid

one-eid-one-uid-empty

many-mixed

one-eid-many-uid-empty

The parent test name is within scope so it's easy to locate in source. Please apply this guidance to other tests added in this PR.

SyntaxNode · 2024-11-01T03:41:27Z

endpoints/openrtb2/auction_test.go

+		t.Run(tc.name, func(t *testing.T) {
+			validEIDs, errorsList := validateEIDs(tc.input)
+
+			if len(validEIDs) != len(tc.expected) {


Please use testify assert statements where possible. This can be written as follows which will compare the contents instead of just the array lengths, yielding more concrete tests.

assert.ElementsMatch(e, tc.expected, validEIDs)

SyntaxNode · 2024-11-01T03:42:36Z

endpoints/openrtb2/auction_test.go

+			expectedValidUIDs: nil,
+			expectedErrors:    nil,
+		},
+	}


Please add a test case for a nil input.

bsardo · 2024-11-04T16:40:16Z

Hi @Pubmatic-Supriya-Patil, can you please merge with master (no rebase) and resolve conflicts? Thanks!

SyntaxNode · 2024-11-05T16:30:23Z

Thank you for contributing this feature Please address the comments and we will re-review.

…ver into Fix_3859

Pubmatic-Supriya-Patil · 2024-12-02T09:22:23Z

Hi @Pubmatic-Supriya-Patil, can you please merge with master (no rebase) and resolve conflicts? Thanks!

HI @bsardo I have updated branch and addressed review comments.

endpoints/openrtb2/auction.go

VeronikaSolovei9 · 2024-12-10T04:32:25Z

endpoints/openrtb2/auction.go

+			for uidIndex, uid := range eid.UIDs {
+				if uid.ID == "" {
+					return append(errL, fmt.Errorf("request.user.eids[%d].uids[%d] missing required field: \"id\"", eidIndex, uidIndex))
+				}


At this time we will not have empty eid.ID, correct? there is a check for this already in line 1334.

Also do we need to set "clean" EIDs back to req.User.EIDs?
Something like req.User.EIDs = validEids after the for loop?

Please correct me if I'm wrong. We don't want to fail the entire request if there is at least one invalid EID, correct? Or do we want to drop invalid ones and continue the execution?
If we want to continue the execution - please add this fix and json tests where we validate "clean" user.EIDs and error messages.

Can we remove if len(eid.UIDs) == 0 { validation also from validateUser function. This is also not possible because validateEids wil return error for emtpy uids. Please provide your suggestions @VeronikaSolovei9

No no, I didn't ask to delete if len(eid.UIDs) == 0 { - it is needed here. I only asked about if we want to add valid aids to the request. And you fixed this by adding this line: req.User.EIDs = validEids

Okay. Thanks. Please review PR

…ver into Fix_3859

ShriprasadM · 2025-01-13T15:24:31Z

@Pubmatic-Supriya-Patil : Please respond to this comments.

bsardo · 2025-01-15T18:37:03Z

@Pubmatic-Supriya-Patil can you please merge with master? That should resolve the failing validate-merge check.

…ver into Fix_3859

VeronikaSolovei9 · 2025-01-17T08:59:35Z

endpoints/openrtb2/auction.go

+		for eidIndex, eid := range validEids {
+			if eid.Source == "" {
+				return append(errL, fmt.Errorf("request.user.eids[%d] missing required field: \"source\"", eidIndex))
+			}

-		for uidIndex, uid := range eid.UIDs {
-			if uid.ID == "" {
-				return append(errL, fmt.Errorf("request.user.eids[%d].uids[%d] missing required field: \"id\"", eidIndex, uidIndex))
+			if len(eid.UIDs) == 0 {
+				return append(errL, fmt.Errorf("request.user.eids[%d].uids must contain at least one element or be undefined", eidIndex))


This code fails the request if eids look like this:

"eids": [ { "source": "uidapi.com", "uids": [ { "id": "" }, { "id": "id111111" } ] }, { "source": "liveramp.com", "uids": [ { "id": "id22222" } ] }, { "source": "", <<< this is the problem "uids": [ { "id": "123" }, { "id": "1234" } ] } ]

All eids are added to the valid list, because they have at least one valid id. The last eid doesn't have a source and instead of removing this eid from the list of valid aids - we return fatal error and fail the request with unexpected message:

Invalid request: request.user.eids[0].uids[0] contains empty ids and is removed from the request Invalid request: request.user.eids[2] missing required field: "source"

Possible fix: move these 2 checks to the validateEIDs function after validateUIDs.
Add message to the error list and don't add this aid to the list of valid eids.

As per my understanding, the logic for removing invalid IDs is specific to eids. We have retained the existing logic for checking the source. However, it is unclear whether we need to check for an empty source and remove that entry while allowing other valid eids. @VeronikaSolovei9, could you please confirm this scenario and clarify the expected behavior in this case?

I will ask team about the expected behavior.

My take is that EID.source is required. IMO, makes sense to remove empty sources.

The team discussed offline. We're in agreement that if EID.source is invalid we should simply remove the corresponding EID without failing the request. We should also add a warning message.

Thanks for the update. I’ll incorporate the required changes.

Remove invalid user EIDs and UIDs from bid request

0f955e9

Pubmatic-Supriya-Patil mentioned this pull request Aug 28, 2024

Drop the User EID object if the all the ID values are empty #3859

Open

bsardo self-assigned this Oct 21, 2024

SyntaxNode reviewed Nov 1, 2024

View reviewed changes

Merge branch 'master' of github.com:Pubmatic-Supriya-Patil/prebid-ser…

9352604

…ver into Fix_3859

bsardo assigned VeronikaSolovei9 Dec 4, 2024

VeronikaSolovei9 reviewed Dec 10, 2024

View reviewed changes

endpoints/openrtb2/auction.go Outdated Show resolved Hide resolved

VeronikaSolovei9 reviewed Dec 10, 2024

View reviewed changes

Merge branch 'master' of github.com:Pubmatic-Supriya-Patil/prebid-ser…

541cbed

…ver into Fix_3859

Removed redundant check for uid.id validation

22e6b21

Merge branch 'master' of github.com:Pubmatic-Supriya-Patil/prebid-ser…

db31c94

…ver into Fix_3859

Pubmatic-Supriya-Patil requested review from VeronikaSolovei9 and SyntaxNode January 16, 2025 04:27

VeronikaSolovei9 reviewed Jan 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove invalid user EIDs and UIDs from bid request #3891

Remove invalid user EIDs and UIDs from bid request #3891

Pubmatic-Supriya-Patil commented Aug 28, 2024

Pubmatic-Supriya-Patil commented Sep 4, 2024

SyntaxNode Nov 1, 2024

Pubmatic-Supriya-Patil Dec 2, 2024

Pubmatic-Supriya-Patil Dec 2, 2024

SyntaxNode Nov 1, 2024

SyntaxNode Nov 1, 2024

SyntaxNode Nov 1, 2024

SyntaxNode Nov 1, 2024

SyntaxNode Nov 1, 2024

SyntaxNode Nov 1, 2024

SyntaxNode Nov 1, 2024

bsardo commented Nov 4, 2024

SyntaxNode commented Nov 5, 2024

Pubmatic-Supriya-Patil commented Dec 2, 2024

VeronikaSolovei9 Dec 10, 2024 •

edited

Loading

VeronikaSolovei9 Dec 10, 2024 •

edited

Loading

Pubmatic-Supriya-Patil Jan 14, 2025

VeronikaSolovei9 Jan 17, 2025

Pubmatic-Supriya-Patil Jan 17, 2025

ShriprasadM commented Jan 13, 2025

bsardo commented Jan 15, 2025

VeronikaSolovei9 Jan 17, 2025

Pubmatic-Supriya-Patil Jan 17, 2025

VeronikaSolovei9 Jan 20, 2025

bretg Jan 21, 2025

bsardo Jan 21, 2025

Pubmatic-Supriya-Patil Jan 22, 2025

Remove invalid user EIDs and UIDs from bid request #3891

Are you sure you want to change the base?

Remove invalid user EIDs and UIDs from bid request #3891

Conversation

Pubmatic-Supriya-Patil commented Aug 28, 2024

Pubmatic-Supriya-Patil commented Sep 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsardo commented Nov 4, 2024

SyntaxNode commented Nov 5, 2024

Pubmatic-Supriya-Patil commented Dec 2, 2024

VeronikaSolovei9 Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

VeronikaSolovei9 Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShriprasadM commented Jan 13, 2025

bsardo commented Jan 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VeronikaSolovei9 Dec 10, 2024 •

edited

Loading

VeronikaSolovei9 Dec 10, 2024 •

edited

Loading