Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ui/cluster-ui: fix polling in fingerprints pages #85772

Merged
merged 1 commit into from
Aug 16, 2022

Conversation

xinhaoz
Copy link
Member

@xinhaoz xinhaoz commented Aug 8, 2022

Fixes: #85236

Previously, SQL statement and transaction stats would be refreshed
every 5 minutes via sagas in CC and the cached api reducer in
db-console. This method relied on a refresh data call that resided
in shouldComponentUpdate, which was ignored by the respective
polling managers when the time interval was not complete. This
pattern was hacky as (unguarded calls in shouldComponentUpdate
are typically avoided in React. Polling in these pages were removed
with the introduciton of persisted stats, however we would like to
reintroduce polling when the selected time interval is `Latest xx..'
(i.e. not a custom interval). The removal of this polling introduced
a bug in the CC fingerprint pages, as the saga effects for when the
data was received would invalidate the data after the polling interval.
Now that the data was never refreshed, the page would get stuck on
the 'loading data' page.

This commit reintroduces polling via a setTimeout in the page
components, rather than through cached data reducer and sagasn for CC.
Data in the fingerprints overview pages is now refreshed every
5 minutes for non-custom time ranges. The data invalidation in
CC is also cleaned up such that a call to invalidate data only
happens right before a request to fetch data (to signify new data
is being loaded).

Release note (bug fix): the statements and transaction fingerprint
will no longer get stuck on the loading page in CC after 5 minutes
idling on the page

Release note (ui change): the statements and transaction fingerprint
now refresh data every 5 minutes for non-custom time ranges

Release justification: bug fix

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@xinhaoz xinhaoz changed the title cluster-ui: prevent polling sql stats in CC after 5m ui/cluster-ui: prevent polling sql stats every 5m Aug 8, 2022
@xinhaoz xinhaoz force-pushed the cc-stmts-refresh branch 3 times, most recently from 87b4467 to e4aa72e Compare August 11, 2022 20:03
@xinhaoz xinhaoz marked this pull request as ready for review August 11, 2022 20:12
Copy link
Contributor

@maryliag maryliag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @maryliag and @xinhaoz)


-- commits line 11 at r1:
nit: request
when the time selected is "last hour", "last month" and so on, we still want to keep refreshing, we just don't refresh if is a custom value
we need


pkg/ui/workspaces/cluster-ui/src/store/sqlStats/sqlStats.sagas.ts line 80 at r1 (raw file):

  try {
    yield call(resetSQLStats);
    yield put(sqlStatsActions.invalidated());

why is this no longer required?

@xinhaoz xinhaoz force-pushed the cc-stmts-refresh branch 3 times, most recently from 1c36edc to 1d85377 Compare August 15, 2022 19:21
@xinhaoz xinhaoz changed the title ui/cluster-ui: prevent polling sql stats every 5m ui/cluster-ui: stop unwanted fetch unbounded date range in fingerprints pages Aug 15, 2022
Copy link
Member Author

@xinhaoz xinhaoz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @maryliag and @xinhaoz)


-- commits line 11 at r1:

Previously, maryliag (Marylia Gutierrez) wrote…

nit: request
when the time selected is "last hour", "last month" and so on, we still want to keep refreshing, we just don't refresh if is a custom value
we need

Changed my approach to handle this!


pkg/ui/workspaces/cluster-ui/src/store/sqlStats/sqlStats.sagas.ts line 65 at r2 (raw file):

export function* resetSQLStatsSaga(action: PayloadAction<StatementsRequest>) {
  try {
    yield call(resetSQLStats);

At a point in the past, a call toinvalidated would refresh the stats (and may have been a requirement in order for the request to go through). This is no longer necessary, and as of this PR, invalidated is called directly in the request action to signify that new data is being fetched.

@xinhaoz xinhaoz requested a review from a team August 15, 2022 19:49
Copy link
Contributor

@maryliag maryliag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 5 files at r1, 9 of 11 files at r2, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @maryliag and @xinhaoz)


pkg/ui/workspaces/cluster-ui/src/store/sqlStats/sqlStats.sagas.ts line 35 at r2 (raw file):

  action: PayloadAction<StatementsRequest>,
): any {
  yield put(sqlStatsActions.invalidated());

is you invalidate before each request, doesn't this means that if you have a scale as "last 1h", when you're about to refresh, the info on the page will disappear, since is no longer valid? having the same problem of blank page being refreshed?

@xinhaoz xinhaoz changed the title ui/cluster-ui: stop unwanted fetch unbounded date range in fingerprints pages ui/cluster-ui: fix polling in fingerprints pages Aug 15, 2022
@xinhaoz
Copy link
Member Author

xinhaoz commented Aug 15, 2022

pkg/ui/workspaces/cluster-ui/src/store/sqlStats/sqlStats.sagas.ts line 35 at r2 (raw file):

Previously, maryliag (Marylia Gutierrez) wrote…

is you invalidate before each request, doesn't this means that if you have a scale as "last 1h", when you're about to refresh, the info on the page will disappear, since is no longer valid? having the same problem of blank page being refreshed?

Yes the invalidation is necessary to show the loading spinner. I took another look and my reasoning for the bug was wrong, so I've updated the PR. I initially thought throttleWithReset was responsible for the bug and was making the api calls. I then remembered we previously had the refresh in shouldComponentUpdate, and so throttleWithReset was just for preventing those calls from making a request before the delay was up. The real issue was the invalidation in the receivedSqlStats saga, which fired after the cache invalidation period. This was likely to trigger the loading screen, or maybe leftover behaviour from a time when we only fetched data if it was invalid (this is what cachedDataReducer does). So since we were invalidating the data without ever refreshing it before, the pages would get stuck on loading. I think one improvement here (but out of scope for this PR) would be to introduce a new inFlight param to the state to better separate data validity and when a request is in flight (this also exists in the db-console side already).

Now that the invalidation is moved right before we make the request, we shouldn't have problems with that anymore. Also removing throttleWIthReset still makes sense since we use the timeout in the pages directly to manage the polling interval now.

Copy link
Contributor

@maryliag maryliag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a comment to change the fixes to partially addresses, otherwise :lgtm:

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @maryliag and @xinhaoz)


pkg/ui/workspaces/cluster-ui/src/store/sqlStats/sqlStats.sagas.ts line 35 at r2 (raw file):

Previously, xinhaoz (Xin Hao Zhang) wrote…

Yes the invalidation is necessary to show the loading spinner. I took another look and my reasoning for the bug was wrong, so I've updated the PR. I initially thought throttleWithReset was responsible for the bug and was making the api calls. I then remembered we previously had the refresh in shouldComponentUpdate, and so throttleWithReset was just for preventing those calls from making a request before the delay was up. The real issue was the invalidation in the receivedSqlStats saga, which fired after the cache invalidation. This was likely to trigger the loading screen, or maybe leftover behaviour from a time when we only fetched data if it was invalid (this is what cachedDataReducer does). So since we were invalidating the data without ever refreshing it before, the pages would get stuck on loading. I think one improvement here (but out of scope for this PR) would be to introduce a new inFlight param to the state to better separate data validity and when a request is in flight (this also exists in the db-console side already).

Now that the invalidation is moved right before we make the request, we shouldn't have problems with that anymore. Also removing throttleWIthReset still makes sense since we use the timeout in the pages directly to manage the polling interval now.

I assume the scope of the PR was to fix the problem of "showing a loading instead of the actual data on the page". Seems like this PR fixes the part about the data actually being refreshed, and we need another PR to continue showing the data even during a refresh (which is specially important on the cases we have a lot of data and it takes minutes to load)
In that case, change the description to partially addresses the issue, and once you have the second PR, then it can be considered fixed

@xinhaoz
Copy link
Member Author

xinhaoz commented Aug 15, 2022

pkg/ui/workspaces/cluster-ui/src/store/sqlStats/sqlStats.sagas.ts line 35 at r2 (raw file):

Previously, maryliag (Marylia Gutierrez) wrote…

I assume the scope of the PR was to fix the problem of "showing a loading instead of the actual data on the page". Seems like this PR fixes the part about the data actually being refreshed, and we need another PR to continue showing the data even during a refresh (which is specially important on the cases we have a lot of data and it takes minutes to load)
In that case, change the description to partially addresses the issue, and once you have the second PR, then it can be considered fixed

Oh we can introduce that behaviour quite easily. I always thought we wanted to show the loading spinner when a new request was pending though -- which specific cases do we not want to show that? Is it just when the source of the request is from polling rather than a change in time range?

Copy link
Contributor

@maryliag maryliag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @maryliag and @xinhaoz)


pkg/ui/workspaces/cluster-ui/src/store/sqlStats/sqlStats.sagas.ts line 35 at r2 (raw file):

Previously, xinhaoz (Xin Hao Zhang) wrote…

Oh we can introduce that behaviour quite easily. I always thought we wanted to show the loading spinner when a new request was pending though -- which specific cases do we not want to show that? Is it just when the source of the request is from polling rather than a change in time range?

That's correct. Basically, if there was some user interaction (change date, refresh page), we can show the loading. If it's just us updating the data, we don't want to distract the user, so that case shouldn't display any loading

@xinhaoz xinhaoz force-pushed the cc-stmts-refresh branch 4 times, most recently from 300fe1e to 99fc418 Compare August 16, 2022 15:24
@xinhaoz xinhaoz requested review from a team and maryliag August 16, 2022 15:24
@xinhaoz
Copy link
Member Author

xinhaoz commented Aug 16, 2022

Updated to match behaviour above, so requesting a re-review!

Copy link
Contributor

@maryliag maryliag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thank you for updating it! :lgtm:

Reviewed 1 of 11 files at r2, 9 of 9 files at r4, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @maryliag)

Fixes: cockroachdb#85236

Previously, SQL statement and transaction stats would be refreshed
every 5 minutes via sagas in CC and the cached api reducer in
db-console. This method relied on a refresh data call that resided
in `shouldComponentUpdate`, which was ignored by the respective
polling managers when the time interval was not complete. This
pattern was hacky as (unguarded calls in `shouldComponentUpdate`
are typically avoided in React. Polling in these pages were removed
with the introduciton of persisted stats, however we would like to
reintroduce polling when the selected time interval is `Latest xx..'
(i.e. not a custom interval). The removal of this polling introduced
a bug in the CC fingerprint pages, as the saga effects for when the
data was received would invalidate the data after the polling interval.
Now that the data was never refreshed, the page would get stuck on
the 'loading data' page.

This commit reintroduces polling via a `setTimeout` in the page
components, rather than through cached data reducer and sagasn for CC.
Data in the fingerprints overview pages is  now refreshed every
5 minutes for non-custom time ranges. The data invalidation in
CC is also cleaned up such that we don't invalidate data after a
delay in the receive effect. When data is refreshed via polling, we
do not show the loading spinner.

Release note (bug fix): the statements and transaction fingerprint
will no longer get stuck on the loading page in CC after 5 minutes
idling on the page

Release note (ui change): the statements and transaction fingerprint
now refresh data every 5 minutes for non-custom time ranges

Release justification: bug fix
@xinhaoz
Copy link
Member Author

xinhaoz commented Aug 16, 2022

CI failure doesn't look related so I'm merging, TFTR!
bors r+

@craig craig bot merged commit 9856a01 into cockroachdb:master Aug 16, 2022
@craig
Copy link
Contributor

craig bot commented Aug 16, 2022

Build succeeded:

@maryliag
Copy link
Contributor

maryliag commented Sep 8, 2022

@xinhaoz can you backport this to 22.1 ?

@xinhaoz xinhaoz deleted the cc-stmts-refresh branch September 21, 2022 21:14
ericharmeling added a commit to ericharmeling/cockroach that referenced this pull request Jan 9, 2023
This commit moves polling to the statement and transaction
details page components from the cached data reducer and sagas,
using the same approach as cockroachdb#85772.

Fixes cockroachdb#91297.

Release note (ui change): The statement fingerprint details page in
the Console no longer infinitely loads after 5 minutes.
craig bot pushed a commit that referenced this pull request Jan 10, 2023
92596: ui: fix polling for statement and transaction details pages r=ericharmeling a=ericharmeling

This commit moves polling to the statement and transaction details page components from the cached data reducer and sagas, using the same approach as #85772.

Fixes #91297.

Loom (DB Console and CC Console): https://www.loom.com/share/d3002ad54463448aaa3b8c9d74e31d10

Release note (ui change): The statement fingerprint details page in the Console no longer infinitely loads after 5 minutes.

Co-authored-by: Eric Harmeling <[email protected]>
ericharmeling added a commit to ericharmeling/cockroach that referenced this pull request Jan 13, 2023
This commit moves polling to the statement and transaction
details page components from the cached data reducer and sagas,
using the same approach as cockroachdb#85772.

Fixes cockroachdb#91297.

Release note (ui change): The statement fingerprint details page in
the Console no longer infinitely loads after 5 minutes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

fix store refresh on cache invalidation period for CC console pages
3 participants