feat: cancel db query on stop #15403

koszti · 2021-06-27T00:38:43Z

SUMMARY

This is addressing #15402

SQL Lab has Stop button but it only updates the state of superset without actually stopping the database query. Leaving the queries running in the background affect database performance. This PR is to stop running queries in the backend databases when:

The Stop button pressed in SQL Lab
Navigated to a new site, refreshed or closed the SQLLab browser window. This is optional and configurable per database at Database -> Performance -> Cancel query on window unload event.

It currently supports mysql, postgres and snowflake databases but new database support can be added easily. The PR is supporting sync and async query execution modes.

This is a refactored and extended version of the unmerged and inactive #7611

TESTING INSTRUCTIONS

Run a long running query in one of the supported drivers. For example MySQL: SELECT sleep(60)
Press the Stop button, close the tab or navigate to another Superset page
Running query should get killed automatically in the backend database. For example MySQL SHOW PROCESSLIST should not show the query from point 1)

ADDITIONAL INFORMATION

Has associated issue: [sqllab] Stop button should kill running query in the database #15402
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

rumbin · 2021-06-28T05:46:38Z

Slightly off-topic:
How does the Stop button in Explore handle this situation?

koszti · 2021-06-28T08:50:55Z

Slightly off-topic:
How does the Stop button in Explore handle this situation?

It behaves the same and Stop button in Explore doesn't stop the query in the database. I'm planning to implement the same functionality for Explore as well by reusing some components of this PR once it get merged. This PR is focusing only on SQLLab.

codecov · 2021-06-28T12:08:51Z

Codecov Report

Merging #15403 (517c792) into master (9ed8ce5) will decrease coverage by 0.23%.
The diff coverage is 71.26%.

@@            Coverage Diff             @@
##           master   #15403      +/-   ##
==========================================
- Coverage   76.95%   76.71%   -0.24%     
==========================================
  Files         976      976              
  Lines       51326    51410      +84     
  Branches     6912     6918       +6     
==========================================
- Hits        39498    39441      -57     
- Misses      11607    11748     +141     
  Partials      221      221

Flag	Coverage Δ
hive	`?`
javascript	`71.38% <30.76%> (-0.02%)`	⬇️
mysql	`81.55% <77.02%> (-0.02%)`	⬇️
postgres	`81.57% <77.02%> (-0.02%)`	⬇️
presto	`?`
python	`81.66% <78.37%> (-0.46%)`	⬇️
sqlite	`81.18% <78.37%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset-frontend/src/SqlLab/reducers/sqlLab.js	`34.81% <0.00%> (-0.15%)`	⬇️
...set-frontend/src/views/CRUD/data/database/types.ts	`100.00% <ø> (ø)`
superset/databases/api.py	`92.95% <ø> (ø)`
superset/views/core.py	`75.01% <0.00%> (-0.11%)`	⬇️
...erset-frontend/src/SqlLab/components/SqlEditor.jsx	`53.74% <22.22%> (-1.31%)`	⬇️
superset/sql_lab.py	`79.92% <46.15%> (-3.56%)`	⬇️
.../CRUD/data/database/DatabaseModal/ExtraOptions.tsx	`93.18% <100.00%> (+0.32%)`	⬆️
superset/db_engine_specs/base.py	`88.25% <100.00%> (-0.28%)`	⬇️
superset/db_engine_specs/mysql.py	`97.61% <100.00%> (+0.43%)`	⬆️
superset/db_engine_specs/postgres.py	`97.36% <100.00%> (+0.33%)`	⬆️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9ed8ce5...517c792. Read the comment docs.

superset/db_engine_specs/base.py

suddjian

This code looks good to me, thank you for the awesome contribution!

Would you mind adding a few tests for the new functionality?

…ery-on-stop

koszti · 2021-07-03T00:24:23Z

This code looks good to me, thank you for the awesome contribution!

Would you mind adding a few tests for the new functionality?

@suddjian I added a few tests by 6d20937 and it's passing locally. Can you please approve the CI workflow to run it in GHA?

koszti · 2021-07-05T11:13:08Z

@suddjian any chance to get this merged?

betodealmeida · 2021-07-06T16:55:28Z

Huh, I'm pretty sure the stop button used to work with Hive/Presto, killing the query. I wonder if we had a regression at some point.

yousoph · 2021-07-06T21:39:01Z

I'm not sure if navigating to another page should cancel the query - I can see wanting to navigate away from a long running query and coming back to it.

koszti · 2021-07-07T00:08:22Z

Huh, I'm pretty sure the stop button used to work with Hive/Presto, killing the query.

@betodealmeida I think you're right. Hive and presto drivers are running queries in async mode with custom cursor handlers. They periodically selects the query status from the superset db in every second and if the query status is STOPPED then it cancels the query by calling the cursor.cancel() function. Hive here, Presto here. This method unfortunately is not necessarily working for other db engines. Mysql, postgres and snowflake for example are running queries in sync mode where polling is not available so we can't check periodically the query status from the superset database. With this PR when pressing the stop button an extra message will be sent to the corresponding db_engine_spec to cancel the query by calling DB specific kill commands.

The behaviour of Db engines that doesn't implement the cancel_query method remains the same, so no changes in hive and presto queries. They're still running in async mode, polling query statuses and terminating queries by cursor.cancel() whenever the stop button pressed.

I'm not sure if navigating to another page should cancel the query - I can see wanting to navigate away from a long running query and coming back to it.

@yousoph you can still keep running the long queries by not closing the browser tab. This is a very similar behaviour when running queries in a desktop SQL client. If we close a desktop SQL client then the running queries are terminated. I believe that SQL Lab should behave the same and running queries need to be terminated when navigating to another window or when closing the browser. This is a default behaviour in Looker (and maybe in Tableau?) as well.

A bit more context about this PR:

We're about to give SQL Lab access to 2000 registered users and expecting about 100 concurrent users in every hour. Manual queries accidentally will be be self joins, cross joins, ones with missing WHERE conditions or simple very long running queries. People will press the stop button or close the browser windows in the hope their queries got killed to let other people to run queries. In concurrency sensitive databases (like snowflake or redshift), stuck queries blocking people to run new queries and features like this PR is crucial.

Please let me know your thoughts. 🙇‍♂️

eschutho · 2021-07-07T00:14:13Z

superset-frontend/src/SqlLab/components/SqlEditor.jsx

@@ -222,6 +224,7 @@ class SqlEditor extends React.PureComponent {

  componentWillUnmount() {
    window.removeEventListener('resize', this.handleWindowResize);
+    window.removeEventListener('beforeunload', this.onBeforeUnload.bind(this));


don't think you need the bind here, since this has already been bound when the method was declared.

thanks, fixed by efb86e5

Oh, true. And also I think it actually won't work with .bind since that returns a new function.

eschutho · 2021-07-07T00:14:33Z

superset-frontend/src/SqlLab/components/SqlEditor.jsx

@@ -212,6 +213,7 @@ class SqlEditor extends React.PureComponent {
    this.setState({ height: this.getSqlEditorHeight() });

    window.addEventListener('resize', this.handleWindowResize);
+    window.addEventListener('beforeunload', this.onBeforeUnload.bind(this));


same thing.. don't think you need the bind.

thanks fixed by efb86e5

can you please approve the CI workflow so github actions will run again? 🙇‍♂️

eschutho · 2021-07-07T00:37:03Z

@yousoph you can still keep running the long queries by not closing the browser tab. This is a very similar behaviour when running queries in a desktop SQL client. If we close a desktop SQL client then the running queries are terminated. I believe that SQL Lab should behave the same and running queries need to be terminated when navigating to another window or when closing the browser. This is a default behaviour in Looker (and maybe in Tableau?) as well.

I agree on ending the query when closing the browser, but I believe this will also end the query when you go to a different tab outside of sql lab as well.
Tested on Firefox by running a query and then clicking on the "Chart" tab:

koszti · 2021-07-07T00:46:14Z

I believe this will also end the query when you go to a different tab outside of sql lab as well.

That's true. Clicking on a non sqllab link (like Charts or Dashboards) is terminating the running query but opening a new tab in the browser should keep it running. What do you think is that acceptable and can it be the default behaviour? Or maybe we should make it configurable?

betodealmeida · 2021-07-07T02:05:05Z

I believe this will also end the query when you go to a different tab outside of sql lab as well.

That's true. Clicking on a non sqllab link (like Charts or Dashboards) is terminating the running query but opening a new tab in the browser should keep it running. What do you think is that acceptable and can it be the default behaviour? Or maybe we should make it configurable?

I think configurable is a good approach, since it changes the current behavior. And just to confirm, when you leave the page it only cancels sync queries, not async?

eschutho · 2021-07-07T19:11:09Z

I believe this will also end the query when you go to a different tab outside of sql lab as well.

That's true. Clicking on a non sqllab link (like Charts or Dashboards) is terminating the running query but opening a new tab in the browser should keep it running. What do you think is that acceptable and can it be the default behaviour? Or maybe we should make it configurable?

I think configurable is a good approach, since it changes the current behavior. And just to confirm, when you leave the page it only cancels sync queries, not async?

I only tested on sync queries so far, yes.

eschutho · 2021-07-07T19:13:18Z

Also testing with Postgres, I'm seeing an error on a long-running query. Small nit, maybe @yousoph can comment, but on shorter queries, there's a bit of a race condition going on. I can hit "stop", see the confirmation that it was stopped but then a few seconds later, still see the results. LMK if you want a video.
But here's the error on the wb_health_population example data set, which is larger:

…ery-on-stop

koszti · 2021-07-08T11:18:40Z

The popup appears only in Firefox. It's a little bit annoying indeed but looking into the beforeunload spec this is what might happen in Chrome as well soon.

👍 I'll make killing queries on window close events configurable in the DB config window and setting it to False as default. I hope that's fine and will update this PRs soon.

eschutho · 2021-07-08T18:05:20Z

@koszti if you want to look at the error above, I may have some simple workaround suggestions for the alert. Since FF is going to alert anyway with no real meaningful message, we could go ahead and put a confirm message for all browsers to ask the person if they want us to close their db connection when they leave. That also somewhat takes care of the configuration issue because people will have a choice. @betodealmeida @yousoph any thoughts on that?
For the race condition on the client side, I think a good solution would be to check the query status again before displaying the results, but that may be something that we can do in a followup PR.

yousoph · 2021-07-08T21:47:21Z

Since FF is going to alert anyway with no real meaningful message, we could go ahead and put a confirm message for all browsers to ask the person if they want us to close their db connection when they leave. That also somewhat takes care of the configuration issue because people will have a choice

I like this idea!

etr2460

Same question as Beto, but this looks pretty good otherwise. cc @john-bodley for python review, but i have no concerns besides the comments here

etr2460 · 2021-07-08T21:56:57Z

superset/db_engine_specs/base.py

@@ -1304,6 +1304,28 @@ def get_column_spec(
            )
        return None

+    @classmethod
+    def get_cancel_query_payload(cls, cursor: Any, query: Query) -> Any:


Can we improve the types here (e.g. better than Any)?

the cursor is db driver specific, defined in the driver and it's hard to set anything better than Any that remains compatible with every driver. The cursor is currently using Any in the entire codebase. But please correct me if I'm wrong or if you have better idea that doesn't require major refactoring of the codebase.

Looks like now the return type here is typed, so this lgtm

etr2460 · 2021-07-08T21:57:03Z

superset/db_engine_specs/base.py

+        return None
+
+    @classmethod
+    def cancel_query(cls, cursor: Any, query: Query, payload: Any) -> None:


same type question here

etr2460 · 2021-07-08T21:57:33Z

superset/db_engine_specs/mysql.py

+    def get_cancel_query_payload(cls, cursor: Any, query: Query) -> Any:
+        cursor.execute("SELECT CONNECTION_ID()")
+        row = cursor.fetchone()
+        return row[0]
+
+    @classmethod
+    def cancel_query(cls, cursor: Any, query: Query, payload: Any) -> None:


same typing questions here

etr2460 · 2021-07-08T21:59:30Z

superset/db_engine_specs/postgres.py

+        cursor.execute(
+            "SELECT pg_terminate_backend(pid) "
+            "FROM pg_stat_activity "
+            "WHERE pid <> pg_backend_pid() and usename='%s'" % payload


yeah, same question here. ideally we can only cancel this query

etr2460 · 2021-07-08T21:59:53Z

superset/sql_lab.py

@@ -438,12 +449,18 @@ def execute_sql_statements(  # pylint: disable=too-many-arguments, too-many-loca
    with closing(engine.raw_connection()) as conn:
        # closing the connection closes the cursor as well
        cursor = conn.cursor()
+        cancel_query_payload = db_engine_spec.get_cancel_query_payload(cursor, query)
+        logger.info(cancel_query_payload)


maybe remove the logger?

removed by ec39477

etr2460 · 2021-07-08T22:02:28Z

superset/sql_lab.py

+
+
+def cancel_query(query: Query, user_name: Optional[str] = None) -> None:
+    """Cancal a running query."""


spelling nit: cancel

also please update the docblock to match all the others with arguments

improved by 4706b43

john-bodley · 2021-07-08T22:36:55Z

superset/db_engine_specs/base.py

+    @classmethod
+    def get_cancel_query_payload(cls, cursor: Any, query: Query) -> Any:
+        """
+        Returns None if query can not be cancelled.


Could you provide a more descriptive description of what this function does, especially for the non-base case?

addressed by ef8436b

john-bodley · 2021-07-08T22:37:37Z

superset/db_engine_specs/base.py

+        :param cursor: Cursor instance in which the query will be executed
+        :param query: Query instance
+        :return: Type of the payload can vary depends on databases
+        but must be jsonable. None if query can't be cancelled.


None if query can't be cancelled

could possibly be implied by better type hints.

addressed by ef8436b

john-bodley · 2021-07-08T22:45:20Z

superset/db_engine_specs/snowflake.py

+
+    @classmethod
+    def cancel_query(cls, cursor: Any, query: Query, payload: Any) -> None:
+        cursor.execute("SELECT SYSTEM$CANCEL_ALL_QUERIES(%s)" % payload)


Could we use a Python 3 format string instead?

addressed by bfc8d60

koszti · 2021-07-09T01:23:05Z

Since FF is going to alert anyway with no real meaningful message, we could go ahead and put a confirm message for all browsers to ask the person if they want us to close their db connection when they leave. That also somewhat takes care of the configuration issue because people will have a choice

I like this idea!

I liked it too, it's elegant but turns out there are some problems.

We can't check what the user answered in the page leave confirmation popup hence we can't stop the query conditionally. Checking the user's answer requires to introduce the unload event but in the unload event the react props are no longer available so we can't call the this.stopQuery() function. :(

Using standard window.confirm would be a workaround but unfortunately we can't use window.confirm in beforeunload events. Using custom confirmation messages in the page leave popup is not allowed either.

I made the following changes in 54da5e4:

It's checking the the query status and it triggers the mechanism only if the query is still running. This is to eliminate some race conditions.

The only solution I can see to make the auto-kill optional is to introduce the "Cancel db query on window close" configurable option in the databases table but that'd require some changes in the core /api/v1/database/ endpoint. I can add a new property into the extra JSON field to avoid db migration. But please let me know if you can see any simpler solution, before I start doing it. Changes in the core database API endpoint response could affect many other things.

PedroMartinSteenstrup · 2021-07-09T11:56:27Z

I'm not sure if navigating to another page should cancel the query - I can see wanting to navigate away from a long running query and coming back to it.

At the moment, editing a dashboard / chart and navigating away will prompt Changes that you made may not be saved. and not store a state that can be restored by going back to the same page.
I'm feeling it is not intuitive why the SQL Lab should behave differently, especially in a browser environment where opening a new tab is just a click away, and where BI end-users are cognizant of the fact that the processes are active while the browser tab is too. I personally did not even notice this was possible until you mentioned it.
Within the SQL Lab, it is possible to move between different query tabs without killing them, which is intuitive.

betodealmeida

This looks great! I have a small question, and may main concern is that I think we should never cancel a running query if the DB is configured to be async.

betodealmeida · 2021-07-09T14:22:44Z

superset-frontend/src/SqlLab/components/SqlEditor.jsx

+  onBeforeUnload(event) {
+    if (this.props.latestQuery?.state === 'running') {
+      event.preventDefault();
+      this.stopQuery();


One more question about this: can we make sure that we stop only queries when the DB is in sync mode? if the DB is in async mode the expectation I don't think we should ever cancel the query by navigating away from the browser.

Can we make it at least configurable? I’m happy to go for the extra mile if needed and add a new extra param to the db config if no objections. We really need to cancel all db queries both in sync and async modes.

This is because we’d like to go with async mode to utilise the celery workers but we still expect a lot of accidental cross joins, missing WHERE clauses and unwanted long running queries. In this case people will just close the browser and that’s making other people impossible running new queries. DBs like snowflake start queuing queries very quickly and unwanted long running queries easily could block using the entire db for long hours. Or DBAs have to kill unwanted queries manually.

Furthermore if it works only in sync mode then the stop button is basically doing nothing in async mode with non hive/presto databases.

Let me know your thoughts 🙇

Sure, if there's a legit use case let's make it configurable. I like the idea of having a per-DB option, this way for Hive we could have it off (since queries can take hours), and for Postgres (say) we could have it on.

@yousoph, any thoughts here?

Furthermore if it works only in sync mode then the stop button is basically doing nothing in async mode with non hive/presto databases.

Oh, sorry if I wasn't clear, I meant stopping a query by navigating away. If we press the stop button we should still cancel async queries, regardless of the configuration flag.

This is now configurable at Database -> Performance -> Cancel query on window unload event and defaults to False to keep the current behaviour. Commit: e5223ec

superset/sql_lab.py

betodealmeida · 2021-07-09T14:26:26Z

superset/db_engine_specs/base.py

+        return None
+
+    @classmethod
+    def cancel_query(cls, cursor: Any, query: Query, cancel_query_id: str) -> None:


Is it possible to return a boolean indicating if the query was canceled?

addressed by 517c792

Co-authored-by: Beto Dealmeida <[email protected]>

…ery-on-stop

koszti · 2021-07-11T02:21:06Z

Added a new option to Database -> Performance to optionally cancel running queries when closing browser window or navigating to another site:

Commit: e5223ec

betodealmeida

This is awesome! <3

john-bodley

Thanks for adding this feature. I wasn't aware that the STOP button in SQL Lab didn't actually stop the query. From a UX perspective, given that not all engines are supported, it seems the frontend should only show the STOP button for eligible databases.

superset/db_engine_specs/base.py

john-bodley · 2021-07-21T03:40:42Z

superset/db_engine_specs/mysql.py

+        """
+        try:
+            cursor.execute(f"KILL CONNECTION {cancel_query_id}")
+        except Exception:  # pylint: disable=broad-except


Catching broad exceptions (per the Pylint message) is undesirable. This should most likely be sqlalchemy.exc.DBAPIError.

john-bodley · 2021-07-21T04:22:49Z

superset/db_engine_specs/snowflake.py

+        :param query: Query instance
+        :return: Snowflake Session ID
+        """
+        cursor.execute("SELECT CURRENT_SESSION()")


@koszti does this work? The reason I ask is per the documentation, CURRENT_SESSION

Returns a unique system identifier for the Snowflake session corresponding to the present connection

yet the connection associated with said cursor doesn't not the same connection which instantiated the query (in all likelihood for async queries that connection resides on a Celery host). Thus I would suspect there are actually no queries running within the current session.

The same is true for the other engines.

The query that you're referring to is running when the query initiated and not when the stop button pressed.
It's using the same cursor that is created in the first place in execute_sql_statements here. The returned session id then saved into the query table in the superset backend database.

When the stop button pressed, we get the saved session id from the query table and kill the query by another SQL. In this way queries can be killed from every celery worker.

It's tested on Postgres, MySQL and Snowflake engines with sync and async query mode.

Apologies. I misread the code logic. I originally thought the new cursor which is used to stop the query was being used to obtain the session identifier.

john-bodley · 2021-07-21T16:45:28Z

@koszti at Airbnb we predominantly use Presto (where only the system connector is able to kill queries) and though the new behavior is more reflective of what's actually happening, i.e., the query isn't stopped it does result in a subtle UX "regression". Specifically users have to wait for the query to complete before attempting to run another query, whereas previously hitting STOP would free up the UI (even though the query was actually never stopped).

* feat: cancel db query on stop * fix pylint * Add unit tests * Do not bind multiple times * Stop only running queries * Postgres to cancel only the required query * Remove extra log * Add docstring * Better types, docstring and naming * Use python3 format strings * Update superset/sql_lab.py Co-authored-by: Beto Dealmeida <[email protected]> * Add cancel_query_on_windows_unload option to database * Return cancel_query as bool Co-authored-by: Beto Dealmeida <[email protected]>

feat: cancel db query on stop

c6ec8e8

pull-request-size bot added the size/L label Jun 27, 2021

koszti changed the title ~~feat: cancel db query on stop~~ [WIP] feat: cancel db query on stop Jun 27, 2021

koszti changed the title ~~[WIP] feat: cancel db query on stop~~ feat: cancel db query on stop Jun 27, 2021

koszti and others added 2 commits June 28, 2021 14:25

fix pylint

822e2f3

Merge branch 'master' into cancel-db-query-on-stop

1f4df58

suddjian reviewed Jul 2, 2021

View reviewed changes

superset/db_engine_specs/base.py Show resolved Hide resolved

suddjian reviewed Jul 2, 2021

View reviewed changes

koszti added 2 commits July 3, 2021 01:10

Add unit tests

6d20937

Merge branch 'master' of github.com:koszti/superset into cancel-db-qu…

cda7c0e

…ery-on-stop

eschutho requested a review from betodealmeida July 6, 2021 02:04

eschutho reviewed Jul 7, 2021

View reviewed changes

Do not bind multiple times

efb86e5

Merge branch 'master' of github.com:koszti/superset into cancel-db-qu…

0f91637

…ery-on-stop

etr2460 reviewed Jul 8, 2021

View reviewed changes

john-bodley reviewed Jul 8, 2021

View reviewed changes

Stop only running queries

54da5e4

koszti added 5 commits July 9, 2021 03:04

Postgres to cancel only the required query

0f49471

Remove extra log

ec39477

Add docstring

4706b43

Better types, docstring and naming

ef8436b

Use python3 format strings

bfc8d60

betodealmeida requested changes Jul 9, 2021

View reviewed changes

koszti and others added 4 commits July 9, 2021 16:47

Update superset/sql_lab.py

e1fc4b3

Co-authored-by: Beto Dealmeida <[email protected]>

Add cancel_query_on_windows_unload option to database

e5223ec

Merge branch 'master' of github.com:koszti/superset into cancel-db-qu…

78bb78d

…ery-on-stop

Return cancel_query as bool

517c792

betodealmeida approved these changes Jul 13, 2021

View reviewed changes

srinify merged commit 02032ee into apache:master Jul 13, 2021

john-bodley reviewed Jul 21, 2021

View reviewed changes

john-bodley mentioned this pull request Jul 23, 2021

fix(15403): Re-enable canceling query for Hive and Presto #15878

Merged

8 tasks

saurabh3091 mentioned this pull request Feb 17, 2022

Stop button for queries doesn't work in SQL Lab. #18787

Closed

3 tasks

hzxa21 mentioned this pull request Dec 13, 2022

bug: unspported function: "pg_backend_pid" when using SQL Lab risingwavelabs/sqlalchemy-risingwave#19

Open

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.3.0 labels Mar 12, 2024



		def cancel_query(query: Query, user_name: Optional[str] = None) -> None:
		"""Cancal a running query."""

feat: cancel db query on stop #15403

feat: cancel db query on stop #15403

Conversation

koszti commented Jun 27, 2021 • edited Loading

SUMMARY

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

rumbin commented Jun 28, 2021

koszti commented Jun 28, 2021 • edited Loading

codecov bot commented Jun 28, 2021 • edited Loading

Codecov Report

suddjian left a comment

Choose a reason for hiding this comment

koszti commented Jul 3, 2021 • edited Loading

koszti commented Jul 5, 2021 • edited Loading

betodealmeida commented Jul 6, 2021

yousoph commented Jul 6, 2021

koszti commented Jul 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eschutho commented Jul 7, 2021 • edited Loading

koszti commented Jul 7, 2021

betodealmeida commented Jul 7, 2021

eschutho commented Jul 7, 2021

eschutho commented Jul 7, 2021

koszti commented Jul 8, 2021 • edited Loading

eschutho commented Jul 8, 2021

yousoph commented Jul 8, 2021

etr2460 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koszti Jul 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etr2460 Jul 8, 2021 • edited by john-bodley Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koszti commented Jul 9, 2021

PedroMartinSteenstrup commented Jul 9, 2021

betodealmeida left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koszti Jul 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koszti commented Jul 11, 2021 • edited Loading

betodealmeida left a comment

Choose a reason for hiding this comment

john-bodley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

john-bodley Jul 21, 2021 • edited Loading

Choose a reason for hiding this comment

koszti Jul 21, 2021 • edited Loading

Choose a reason for hiding this comment

john-bodley Jul 21, 2021 • edited Loading

Choose a reason for hiding this comment

john-bodley commented Jul 21, 2021

koszti commented Jun 27, 2021 •

edited

Loading

koszti commented Jun 28, 2021 •

edited

Loading

codecov bot commented Jun 28, 2021 •

edited

Loading

koszti commented Jul 3, 2021 •

edited

Loading

koszti commented Jul 5, 2021 •

edited

Loading

koszti commented Jul 7, 2021 •

edited

Loading

eschutho commented Jul 7, 2021 •

edited

Loading

koszti commented Jul 8, 2021 •

edited

Loading

koszti Jul 9, 2021 •

edited

Loading

etr2460 Jul 8, 2021 •

edited by john-bodley

Loading

koszti Jul 9, 2021 •

edited

Loading

koszti commented Jul 11, 2021 •

edited

Loading

john-bodley Jul 21, 2021 •

edited

Loading

koszti Jul 21, 2021 •

edited

Loading

john-bodley Jul 21, 2021 •

edited

Loading