Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Projects to relay #1460

Merged
merged 81 commits into from
Dec 12, 2017
Merged

Projects to relay #1460

merged 81 commits into from
Dec 12, 2017

Conversation

mattkrick
Copy link
Member

@mattkrick mattkrick commented Nov 1, 2017

it's time to bite the bullet. this will fix up faulty cache invalidations, speed up dashboard loading by a bunch, and enable us to point to a specific card on the dashboard, which is necessary when a notification says "you've got a new card, click to see it".

this refactor is really painful because Relay makes you use a "Global ID" for entities, which is base64(type, id). That means if a function can get called from something populated by both cashay & relay, then we have to handle both. It's also a challenge because foreign keys need to be converted to Global IDs, too. For example, if a team has an activeFacilitator then team.activeFacilitator must get matched with a TeamMember.id. A project has a teamMemberId, too. So, now that it comes from Relay we have to make sure the IDs are correct there, too.
... All of that to say, despite being 200 changed files, I'm really surprised it isn't bigger.

tests:

  • UI should be exactly the same, except load no less than 400% faster (also implements optimistic UI on just about everything)
  • meeting facilitator shouldn't lose their facilitator role if they lose connectivity for a second (we now check every 5 seconds, instead of immediately)
  • fix Notification bell count only updates on refresh #1536 where notification subscriptions don't work right

things out of scope, but still need fixing (for another, smaller, PR):

  • a teamMemberRemoved subscription. without it, if a teamMember is removed, lists won't update in real time.
  • colocate notification data needs to speicific components. without it, it'll get a little unruly
  • implement dataLoader throughout. this will reduce the number of lookups for all hits to graphql.

@mattkrick mattkrick requested a review from dan-f December 8, 2017 06:00
@mattkrick mattkrick changed the title Epic 8 - Projects to relay [WIP] Epic 8 - Projects to relay Dec 8, 2017
@mattkrick mattkrick changed the title Epic 8 - Projects to relay Projects to relay Dec 8, 2017
@dan-f
Copy link
Contributor

dan-f commented Dec 8, 2017

Upon completion of a meeting with just myself in which I added a project during the updates phase and also reordered a few projects during the updates phase.

screen shot 2017-12-08 at 12 27 00 pm

Reloading the page results in the same error.

@dan-f
Copy link
Contributor

dan-f commented Dec 8, 2017

Not seeing my name during the agenda phase of the meeting

screen shot 2017-12-08 at 12 35 47 pm

@dan-f
Copy link
Contributor

dan-f commented Dec 8, 2017

I wrapped up the meeting as the facilitator, but my other attendee ends up stuck in the meeting still.

screen shot 2017-12-08 at 12 37 05 pm

@mattkrick
Copy link
Member Author

mattkrick commented Dec 8, 2017

NICE!

  • fixed the dates in the first one (relay doesn't like the Date type, so everything needs to support ISO8601 strings)
  • added picture & preferredName to the MeetingAgendaItems fragment
  • that 3rd one is gonna be trickier, still working on repro

@mattkrick
Copy link
Member Author

mattkrick commented Dec 8, 2017

MOAR BUGZ:

  • optimistic greeting is a string of draftJS json
  • callingMember is sometimes undefined when on the dev server

@mattkrick
Copy link
Member Author

okie dokie, all ready!

@mattkrick
Copy link
Member Author

@jordanh @ackernaut if you guys have the time, i'd love to get some more fingers on this! it's currently up on the dev server. especially try to break the meeting.

Copy link
Contributor

@dan-f dan-f left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Matt!

You've definitely kept yourself busy here! I'm still doing my review, but figured I'd share some of my comments so far. Some caveats:

  • Due to the enormous scope of this work, I haven't been able to go into too much detail, though there are some lines of code that sort of arbitrarily caught my eye to poke at ¯_(ツ)_/¯
  • Though I'm mostly done looking at the backend changes, I have some reservations that I'm going to try to put to words in a bit around the concept of the shared data loader
  • I'm wondering about when you decide to put data resolution logic in a GraphQL type definition vs in the query modules.

That's it for now! More on data loaders and the relay / cache layer later today.

};


export default class RethinkDataLoader {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding of DataLoader based on the README was that the API client is intended to have one unique loader per request, per data object. So callng this thing "RethinkDataLoader" is a bit misleading. It's more of a grab-bag of data loaders. I'd be tempted to simply export the various loaders separately.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, the readme isn't great about realworld examples, however there is one example here: https://github.com/facebook/dataloader#caching-per-request

this is the standard practice in other open source projects, although we all suffer with what to call it! There's a dataloader, then a dataloader bag, then a shared data loader is a bag of dataloader bags.

there's no value in exporting the separately because you don't know what you'll need at the time of creation, unless in misunderstood?

if (filterFn(value)) {
if (value.operationId) {
if (!dataLoader) console.log('NO DL', channelName);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if !dataLoader, we still hit L18.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah good call, i'll just remove this & the else clause, they were just to help in dev.

@@ -38,7 +38,7 @@ describe('endMeeting', () => {
teamMember: r.table('TeamMember').getAll(teamId, {index: 'teamId'}).orderBy('preferredName')
}, dynamicSerializer);
expect(db).toMatchSnapshot();
expect(mockFn).toBeCalledWith(teamId);
expect(dataLoader.isShared()).toEqual(true);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels a bit like testing the implementation. As long as we get the data we asked for, we're good, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right you are!
the point of sharing the dataloader is to make sure we don't hit the database n+1 times (n subscribers + 1 mutation). we'll still get the right result if we don't share, but it'll be much more expensive & virtually impossible to pinpoint the source. If you want i can remove


// RESOLUTION
const now = new Date();
const agendaItem = await r.table('AgendaItem').insert({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just giving a heads-up: I've been asynchronously working on adding data constructors and functions for operating on well-defined data so that we can reduce the usage of ad-hoc object literals and ID generation: https://github.com/ParabolInc/action/tree/data-constructors-and-accessors/src/universal/data

})
},
...agenda,
...meeting,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these definitely unused?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep! everything with a spread operator is the old format. as i switch stuff to relay, I remove the legacy stuff. The relay convention is to store this stuff on the "viewer" (read User). so it'd be like:

viewer {
  team(teamId: "123") {
    agendaItems {
      id
      content
    }
  }
}

it's actually pretty great, relations are a lot easier to see in code & since everything is stored under the viewer you never need to check for the userId authorization because that's baked in.

type: GraphQLISO8601Type,
description: 'the datetime cursor'
}
// private: {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code comment reminder

@@ -103,6 +107,12 @@ const User = new GraphQLObjectType({
type: GraphQLString,
description: 'The application-specific name, defaults to nickname'
},
// presence: {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code comment reminder

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed in a subsequent PR (don't wanna cause too many conflicts)

@@ -111,6 +121,31 @@ const User = new GraphQLObjectType({
return (userId === source.id) ? source.tms : undefined;
}
},
teams: {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point, I'm confused by when we decide to handle field resolution in server/graphql/queries modules vs. in server/graphql/types modules.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Type vs. Field. let's chat it up if it's still a little foggy, once you get it down it'll make GraphQL seem a lot cooler!

@@ -37,6 +37,10 @@ const reactivateTeamMembersAndMakeNotifications = async (invitees, inviter, team
reactivatedUsers.forEach(({id: userId, tms}) => {
getPubSub().publish(`${NEW_AUTH_TOKEN}.${userId}`, {newAuthToken: tmsSignToken({sub: userId}, tms)});
});
reactivatedTeamMembers.forEach((teamMember) => {
const teamMemberAdded = {teamMember};
getPubSub().publish(`${TEAM_MEMBER_ADDED}.${teamId}`, {teamMemberAdded, ...subOptions});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are inactive team members actually removed from teams?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we set an isNotRemoved: false flag on it. That way, archived projects still have an owner even when the person is long gone.

@@ -21,16 +22,16 @@ const isTmsValid = (tmsFromDB = [], tmsFromToken = []) => {
return true;
};

export default function scConnectionHandler(exchange) {
export default function scConnectionHandler(exchange, sharedDataLoader) {
return async function connectionHandler(socket) {
// socket.on('message', message => {
// if (message === '#2') return;
// console.log('SOCKET SAYS:', message);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yeah, good for debuggin, but this'll be gone soon.

Copy link
Contributor

@dan-f dan-f left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, I think I've sorted out my remaining thoughts on the backend changes. It's getting late here, so I'll finish up with the front-end/relay part of the review tomorrow morning.

The dataloader concept was new to me, so I spent a bit reading up on their docs. Seems like a really reasonable concept, and a simple caching layer. Thanks for adding it!

I have some suggestions on changing how we integrate dataloader into our architecture. In general terms, I think that the current approach in this PR creates our dataloader instances at an event in the application lifecycle which is a bit too early and could result in a shared memory bugs for us down the road.

From the dataloader readme:

DataLoader caching does not replace Redis, Memcache, or any other shared application-level cache. DataLoader is first and foremost a data loading mechanism, and its cache only serves the purpose of not repeatedly loading the same data in the context of a single request to your Application.

Avoid multiple requests from different users using the DataLoader instance, which could result in cached data incorrectly appearing in each request. Typically, DataLoader instances are created when a Request begins, and are not used once the Request ends.

So the authors recommend to manage dataloader instances at the request lifecycle level, meaning each request should get its own instance of dataloader.

However, based on my understanding of SocketCluster, we're currently creating one SharedDataLoader per worker at the moment the worker is born (worker.js L41). That piece of memory is then read and written to by every HTTP and websocket connection which is handed off to that worker.

I suggest we do the following:

  • For HTTP requests:
    • Create our set of dataloaders at the moment that the request is born. I'm no expert in express, so perhaps this means adding some sort of middleware such that we can pass the dataloaders through the middleware stack, or simply creating the dataloaders in the per-request httpGraphQLHandler.
  • For WebSocket connections:
    • My understanding was that the requests to subscribe/unsubscribe don't do much data fetching on their own, so let's not worry about caching data in these cases yet.
    • For the procedure which handles GraphQL requests over WebSockets (scGraphQLHandler) we're essentially imposing a request/response model on top of websockets. So at the moment the "request" (e.g. "graphql" message) comes in, we should create our dataloaders, and those loaders should then be garbage-collected when that procedure returns.
  • I took a look at the 'shared-dataloader' library, and I can't quite tell what the main sell is. Since the recommended usage of 'dataloder' is to not state across request lifetime boundaries, I'd say we should probably just use plain ol' dataloader.

editors: {
type: new GraphQLList(ProjectEditorDetails),
description: 'a list of users currently editing the project (fed by a subscription, so queries return null)',
resolve: ({editors = []}) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, in the future, we could keep this data in an in-memory cache such as redis

@@ -38,7 +44,11 @@ const Project = new GraphQLObjectType({
type: GraphQLFloat,
description: 'the shared sort order for projects on the team dash & user dash'
},
status: {type: new GraphQLNonNull(ProjectStatusEnum), description: 'The status of the project'},
// TODO make this nonnull again
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's keeping it from being non-null?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a bug in the relay compiler that makes fragments order dependent :-(.
The more i work in GraphQL, the more i dislike nonnulls, so i may just keep it out.

@mattkrick
Copy link
Member Author

mattkrick commented Dec 12, 2017

I'm writing a medium article on what shared-dataloader does, so i'll be brief:
In short,

Hope that clears some stuff up! Let's chat about dataloaders & fields vs. types tomorrow

@dan-f
Copy link
Contributor

dan-f commented Dec 12, 2017

no, the shared dataloader does NOT create dataloaders at the worker level. it just creates the bag.

Duly noted! I think the singular naming was confusing me a bit. So we have a SharedDataLoader which has many DataLoaders, and it just so happens that our RethinkDataLoader itself is a bag of many data loaders. Perhaps we can iterate a bit on the naming? RethinkDataLoaders with an s?

I'm still not convinced that we should be implementing a global database cache in the application itself. Let's talk more about this today! Ping me when you wanna chat, and until then I'll be continuing with the review.

Copy link
Contributor

@dan-f dan-f left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright! I think I've done what I can with the front-end review. This wraps up my feedback! Here goes:

Patterns:

  • Can we define our naming conventions for fragment containers, query roots, redux containers, etc.? They're all sort of "container" components, but I don't know if I've seen a consistent pattern yet.
  • With the relay mutations, there's a pattern of using the optimisticUpdater rather than the more declarative optimisticResponse. Is there something keeping us from using optimisticResponse?

Documentation:

  • Being newer to this codebase, it would be really useful for me if our modules, functions, classes, components, etc. could have some sort of minimal documentation. I realize that code changes over time and docs get stale, and I'm not asking for JSDoc level documentation. But I think something along the lines of, e.g., // this container component renders a component during the error, loading, and ready states of it's data lifecycle would be really useful for speeding up these reviews and also for just remembering when to use which components.
    • While the source code tells me how we've implemented something, a good name and a quick comment tells me why we're implementing it, what it does, and when I should use it.
    • This would be especially useful for cases in which we're overriding a framework module.
  • Let's avoid using PropTypes.any. If we know that we're expecting a field with some set of values, then we know its type. If the type of a field is some ad-hoc class instance, we can always use PropTypes.instanceof. At the very least a comment accompanying a PropTypes.any declaration would be much appreciated.

{suggestions && suggestions.map((suggestion, idx) => {
return (
// eslint-disable-next-line
<div key={idx} onMouseDown={dontTellDraft} onClick={handleSelect(idx)}>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than adding click handlers to divs, let's use the PlainButton component.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the plain button still has some styles that i don't want since here, since the child is non deterministic. I should make an PlainListItem component!

NotificationsClearedSubscription
];

const DashboardWrapper = ({atmosphere, dispatch, history, location}) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this Component is more of a container component why don't we call this something like DashboardContainer and move it to the containers directory?

Copy link
Member Author

@mattkrick mattkrick Dec 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, & it's changed a lot since this PR! Captured in #1559 so i don't bork the future changes

@@ -61,4 +65,13 @@ const styleThunk = (custom, {isEditing}) => ({
}
});

export default withStyles(styleThunk)(EditingStatus);
export default createFragmentContainer(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Time to move to containers?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So hear me out, how I think of this is there is an EditingStatus component that is wrapped in a container, and that container is nothing more than the thing the HOC provides. This is the relay-approved way of doing things because the fragment name must match the component name that it's tied to, so if we tied it to EditingStatusContainer, we'd be breaking that convention.

import entitizeText from 'universal/utils/draftjs/entitizeText';


class EditorInputWrapper extends Component {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure this can be written as a functional component or extend PureComponent.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, regarding naming, if this is the component we should be using when we want to edit DraftJS content, let's just call it something like Editor, since the views it's wrapping are opaque to the caller.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Editor is the name of the component that Draft-js uses.
ProjectEditor is what we use when we wanna use draft-js inside a Project
This is an editor that I envision we'll use whenever we wanna swap out an input box. So I should probably call in InputEditor? but then we have an Editable that is an input. I'll rename & add comments in #1559

@@ -11,7 +11,7 @@ const PlainButton = ({styles, ...props}) => (
);

PlainButton.propTypes = {
children: PropTypes.arrayOf(PropTypes.element),
children: PropTypes.any,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, looks like this should be PropTypes.node!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!

@@ -24,7 +24,7 @@ const InvoiceRoot = ({atmosphere, match: {params: {invoiceId}}}) => {
variables={{invoiceId}}
render={({error, props: queryProps}) => {
return (
<TransitionGroup appear style={{overflow: 'hidden'}}>
<TransitionGroup appear component={null}>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

component={null}?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the things i do for backwards compatibility... reactjs/react-transition-group#242

team
} = props;
class MeetingAgendaItems extends Component {
state = {};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

state = { agendaProjects: [] }

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

match: PropTypes.object.isRequired
};

export default connect()(withRouter(withAtmosphere(MeetingRoot)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than calling react-redux's connect on a parent which itself doesn't need dispatch, can we let the children that need access to dispatch connect themselves?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's a good practice! In this case, MeetingRoot does need dispatch to feed to the subParams because a certain subscription needs it (NotificationsAdded). May be a code smell to do it like this, but I don't know how else to couple the subscription to the query (and unfortunately they need coupling since we need to flush the query if the sub dies).

const myUserId = myTeamMemberId && myTeamMemberId.split('::')[0];
const isMyMeetingSection = myTeamMemberId === currentTeamMember.id;
class MeetingUpdates extends Component {
state = {};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be declaring the shape of state, even if it's initially empty.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

if (isTempId(projectId)) return undefined;
const {viewerId} = environment;
const updater = (store) => {
removeFromProjectConnections(store, viewerId, projectId, teamId);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than imperatively working with the store, can we not just use the NODE_DELETE config for this CRUD operation?

Copy link
Member Author

@mattkrick mattkrick Dec 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh such a good question.
so those configs are a carry over from relay classic-- back then, they didn't have the imperative API or the proxy stores. you did everything with these configs. and. people. hated. it.

  1. let's say Matt joins the team. do you append him or prepend him to the team members array? Neither, he is in the freakin middle. where's that option?
  2. let's say you use a GraphQLList instead of a Connection. You have to use the imperative approach, anyways.
  3. let's say you're doing something aside from the most basic add/delete, like maybe you need some arguments in your pathToConnection, but it's not working. How do you trace your steps?
  4. IIRC RANGE_DELETE removes the node from the array, but doesn't GC the node, yet. NODE_DELETE deletes the node, and so now your array has an undefined in it, which propagates to your view layer & now you're wondering why a card is undefined... it's no fun.

FWIW, the todo-modern example uses the imperative approach while the classic todo uses the config approach, so i take that as a "yeah, we like this way better, but there's no reason to have to rewrite your old code if it's working". Personally, I don't really care that much, but I'd prefer to write everything the same way instead of starting out using a config, finding it breaks or the business logic changes, then using the imperative approach anyways.

@mattkrick mattkrick merged commit 88b6715 into epic8.5 Dec 12, 2017
@mattkrick mattkrick deleted the projects-to-relay branch January 19, 2018 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants