Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle proportions for RA and uplift metrics #1090

Merged
merged 5 commits into from
Mar 23, 2023
Merged

Conversation

lukesonnet
Copy link
Collaborator

@lukesonnet lukesonnet commented Mar 20, 2023

Features and Changes

  • Refines regression adjustment approach for proportions. Required two changes: handling compute_theta for proportion statistics and using the unadjusted mean for computing the variance of the relative change (uplift). Note this second change affects sample mean tests as well, and should make it more conservative.
  • Also handles case where variance is missing in one of the periods and reverts to non-RA (but doesn't alert the front-end to this fact).

Dependencies

n/a

Testing

Screenshots

n/a

@github-actions
Copy link

Your preview environment pr-1090-bttf has been deployed.

Preview environment endpoints are available at:

(stat_b.mean / stat_a.mean) - 1 if stat_a.mean > 0 else 0
(stat_b.mean - stat_a.mean) / stat_a.unadjusted_mean
if stat_a.unadjusted_mean > 0
else 0
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only change that is on an actively used path, and unadjusted_mean should always == mean for non-RA metrics.

@lukesonnet lukesonnet merged commit a018475 into main Mar 23, 2023
@lukesonnet lukesonnet deleted the lsonnet/ra_prop branch March 23, 2023 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants