Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make sure the app doesn't break with the new delivery_email field #3196

Closed
timabbott opened this issue Dec 7, 2018 · 7 comments
Closed

Make sure the app doesn't break with the new delivery_email field #3196

timabbott opened this issue Dec 7, 2018 · 7 comments
Labels

Comments

@timabbott
Copy link
Member

As part of adding (hacky) support for not sharing users' real email addresses with everyone in the organization, we've added a new field, delivery_email (which in most configurations just equals the email, but ). The basic model is that the email field will continue to be a human-readable identifier for a user, and for most client purposes, they can just ignore the delivery_email field and things will transparently work; users generally will only have access to their own delivery_email.

I expect that this hacky approach (where the email field may not be the user's actual email address) is going to be how things work for a long time, because the email field name is used in so many places that migrating away from and then renaming it likely won't happen for a year or more from when we start doing so.

It's likely that the mobile apps don't need any work; for the webapp frontend, we needed to do just these two commits: zulip/zulip@67f0b1b and zulip/zulip@bc74aba (for the "my account" settings page). (Well, we'll possibly need to add a few more changes for the administration UI, but the mobile app doesn't have one).

But we should do some testing of the mobile app for this to make sure something reasonable happens in this configuration (which is available in the development environment now thanks to the commit above); I suppose if there's a place where we display "your email address", that should be changed to use .delivery_email. No changes should be needed for the auth/login stage; for that, everything uses the value of delivery_email under the old name email.

(We'd like to turn this feature on for chat.zulip.org within the next few weeks assuming no blockers from e.g. mobile)

@gnprice
Copy link
Member

gnprice commented Dec 21, 2018

One other aspect of this worth making explicit: the transition from the existing configuration to the new one. When an organization makes this transition, the email of every user will change. Because we (mis)use the user's email as an identifier, this would be pretty confusing for the app.

Our planned solution is to have the server expire all event queues for the organization (for mobile clients... maybe all non-webapp clients? Tim told me last week he believes the webapp will have no trouble with the transition.) That should cause the app to reload everything, so there's no confusion.

This is an important area to test, though -- there's a lot of kinds of data involved, and there may be cases where we try to still use pre-transition data combined with post-transition data, which will make things confused.

@gnprice
Copy link
Member

gnprice commented Jul 5, 2019

Update: This setting is available for org admins on zulipchat.com to choose, though marked "beta":
https://zulipchat.com/help/restrict-visibility-of-email-addresses
Not sure of when it was deployed, or its status in server releases.

We just got a very helpful, detailed report from a puzzled org admin of a set of symptoms that turned out to be triggered by this setting. They found that with the setting enabled (i.e. with email addresses visible only to admins), all users in the organization saw the following symptoms on mobile (lettering mine, to clarify discussion below):

(a) Private messages disappearing after sending, I think this is noted in issue #3117
(b) Incoming messages will create notifications but not appear in app until the app is closed and reopened
(c) Unread message counts don't clear until the app is closed
(d) Sent or received messages in any private message feed create unread counts in all pm feeds
(e) User image does not appear as icon for far right tab or in associated screen

On the last point (e), the avatar specifically is blank.

One further symptom found in discussion:
(f) on the PM conversations screen, each conversation shows the user's own name, in addition to whoever else is in it.

I believe symptoms (e) and (f) are probably because we try to use the user's email address from something like auth.email to look up their avatar, or to filter them out of the list of users in a PM conversation, respectively.

Not sure about (d). That sounds like it might be the same bug, compounded with some additional bug in a reducer.

The first three symptoms (a), (b), (c) all sound like we're failing to get events from the server. Not sure why this setting would lead to that failure mode, but apparently it does.

@timabbott
Copy link
Member Author

This has been available for beta-testing by a few orgs (I think because this one asked for it) for a period of a couple weeks. It is not in any server release. We should definitely consider making this work properly in the mobile app a priority issue, because we have been getting regular requests for this feature and will definitely want to include it in the next release.

IIRC there is an event sent when delivery_email changes when an organization has this setting turned on, so if the issue is just in handling that, we should be able to have the app do a full refresh in that case if that addresses the issue; it should be a quite rare change.

@gnprice
Copy link
Member

gnprice commented Jul 8, 2019

That's helpful context, thanks.

IIRC there is an event sent when delivery_email changes when an organization has this setting turned on, so if the issue is just in handling that, we should be able to have the app do a full refresh in that case if that addresses the issue; it should be a quite rare change.

Cool. We'll want to do that, but unfortunately this will be more complex to implement than just the transition -- major steps include

  • Really going through and finding all the places that we identify the current user by their email. These will be tricky to fully track down.
  • Fixing those. This probably means building more data structures to use IDs where we've been using emails.

@gnprice gnprice added P0 critical Highest priority and removed P1 high-priority labels Jul 8, 2019
gnprice added a commit to gnprice/zulip-mobile that referenced this issue Jul 9, 2019
This probably doesn't totally work as is -- there are probably some
places we want the email before we have the /register response, as
well as places we want the delivery_email.  But I think it's the core
of the fix to zulip#3196.
gnprice added a commit to gnprice/zulip-mobile that referenced this issue Jul 12, 2019
This probably doesn't totally work as is -- there are probably some
places we want the email before we have the /register response, as
well as places we want the delivery_email.  But I think it's the core
of the fix to zulip#3196.
gnprice added a commit to gnprice/zulip-mobile that referenced this issue Jul 13, 2019
This probably doesn't totally work as is -- there are probably some
places we want the email before we have the /register response, as
well as places we want the delivery_email.  But I think it's the core
of the fix to zulip#3196.
gnprice added a commit to gnprice/zulip-mobile that referenced this issue Jul 13, 2019
This probably doesn't totally work as is -- there are probably some
places we want the email before we have the /register response, as
well as places we want the delivery_email.  But I think it's the core
of the fix to zulip#3196.
gnprice added a commit that referenced this issue Jul 16, 2019
This brings this UI in line with the corresponding UI in the webapp,
which is the card that pops up on clicking on a user's name or avatar.

The user's email address can be helpful information, but their name is
usually more helpful.  Also, under an org setting that's currently in
beta after being a frequent request for a long time, users' real email
addresses won't even be available to normal users -- so the `email`
property will be an opaque address made up by the server.  (See #3196.)
So we certainly don't want to show that here.

While we're here, also swap the order of the name and the status
indicator, again following the webapp.
gnprice added a commit that referenced this issue Jul 16, 2019
Part of #3196 -- this is an example of code that will break when
hidden-emails are enabled, because the (real) email used for auth
will be different from the (server-generated, fake) email used as
an identifier for the user.

Found by grepping for `auth.email`.  There are still a few more
matches for that pattern, which with one exception (in src/api/)
are also broken; those we'll fix in the next few commits.
gnprice added a commit that referenced this issue Jul 16, 2019
Part of #3196 -- this is another bit of code that will break when
the hidden-emails setting is enabled.
gnprice added a commit that referenced this issue Jul 16, 2019
Part of #3196 -- this is another bit where we were using the user's
(real) auth email where their (possibly different and fake)
identifier-email is what we need.
@gnprice
Copy link
Member

gnprice commented Jul 16, 2019

Just pushed (as bd6c155..2407f3f) a series of fixes to various small places where we were using the current user's real email (the one used in authentication) where what we needed was their identifier-email. Each of these was a spot that would break when the hidden-emails setting is enabled. Now they all get the identifier-email through a common codepath, getOwnEmail.

Still in a draft branch is the change to make getOwnEmail actually consult the email from /register, rather than return the real email from authentication. Until that change, the above fixes are almost all no-ops. As of Friday, that change provoked some bugs at the moment of switching accounts, or logging into an account; still working through those. Hopefully the changes I just pushed will clear out some of the underbrush for debugging those other issues.

I also observed today a puzzling issue where, shortly after enabling hidden-emails on an org, I believe the server was still returning unobfuscated real emails in the display_recipient properties of a /messages response. Details in chat. In the discussion there, @timabbott suggested it might be a server-side caching issue; more debugging required.

gnprice added a commit to gnprice/zulip-mobile that referenced this issue Jul 16, 2019
This probably doesn't totally work as is -- there are probably some
places we want the email before we have the /register response, as
well as places we want the delivery_email.  But I think it's the core
of the fix to zulip#3196.
gnprice added a commit that referenced this issue Jul 17, 2019
With the "hidden emails" feature (aka `email_address_visibility`)
that's now in beta on the server, the user's "email" as used as an
identifier for them in the Zulip API can be an opaque one made up by
the server -- different from `auth.email` aka `delivery_email`.  So we
need to switch our concept of "own email" to use the identifier-email;
this is #3196.

The server tells us this user's identifier-email in the `/register`
response, so one necessary step is to remember it.  Do that.

In principle we should then update this value on EVENT_USER_UPDATE.
But (a) the same is true of several other values here, and more
importantly (b) we already don't handle EVENT_USER_UPDATE at all,
either in our `users` state-subtree or in the various denormalized
spots where we store other users' emails, avatars, etc.  So punt.
Changing emails is pretty uncommon; and in the particular case where
it's happening for all an org's users en masse because hidden-emails
was turned on or off, the only near-term-feasible solution will
probably be to have the server just invalidate clients' event queues.
@gnprice
Copy link
Member

gnprice commented Jul 17, 2019

Just pushed 139b7d1..c5b9370, and I believe this is now fixed! There may be yet-unknown bugs where we still use auth.email aka the delivery_email where we want the identifer-email, or vice versa; we'll file and fix those separately as we see them.

Behavior is still not good at transition time, in the period shortly after turning the setting on (or off) for an org. There are two pieces to this:

  • The server-side caching issue mentioned above. I did some further debugging yesterday, described in the linked chat thread, and I think we have a diagnosis and a couple of ways to fix it.

    This is a bug in the data the server is returning -- a hole in the intended privacy properties of this setting, though fortunately a temporary one for I think up to 24 hours after the setting is enabled -- but its effect is especially visible on mobile because we use emails to identify users in lots of places, where the webapp has I think gotten farther in migrating away from that.

  • Updating all the client-side data that identifies users by email address. This is actually a lot like (our current diagnosis of) that server-side caching issue, in that a root cause is that user emails (and avatar_urls, etc.) are denormalized into a variety of spots scattered across our data structures, so that it's not trivial to track them all down and update them.

    The solution we've talked about before, which I think is reasonable and is the only near-term-feasible one, is to have the server simply expire all event queues for the org when this setting is changed, so that we reload all data from the server.

    (That still leaves a bug where we don't properly handle users changing their email address -- either the self-user, or any other user. This is analogous to Doesn’t update my own avatar #3397 and is part of the umbrella issue Ensure all server events and their operations are handled #3408; the updates come in realm_user events with opcode update. But that's an uncommon occurrence, it isn't easy to fix given the denormalization, and so I don't think it's a priority compared to our other open issues.)

@timabbott
Copy link
Member Author

@gnprice just a note -- the plan I'd imagined was for the client to destroy its event queue when receiving an event of the email_address_policy changing; the webapp I believe should be able to (possibly with small changes to address bugs) handle such transitions without a reload because of how it already handles email address changes correctly.

(We have a handful of events that on the webapp trigger a reload for similar reasons)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants