Adding progress bar display func

[meztez]

For #199.

[meztez]

Not printing, looking for further insight. @krlmlr? Any ideas.

[krlmlr]

Thanks for working on it! I think this should be an R option with a callback that is called when the option is set, see duckdb.materialize_callback for an example.

[krlmlr]

Or, perhaps even a slot in the duckdb_connection class?

[meztez]

Well, I'm almost done with the callback. So I'll test that first.

[meztez]

Oh, DUCKDB_DISABLE_PRINT flag probably does not help
https://github.com/duckdb/duckdb-r/blob/a664862817a0ee9a741357d54e973ca2a2cf0e4e/src/Makevars.in#L18C46-L18C66

[meztez]

All right, it works, now ironing out the bugs.

[meztez]

library(duckdb)
library(cli)

progress <- function(x) {
  if (cli::cli_progress_num() == 0) {
    cli::cli_progress_bar("Duckdb SQL", total = 100, .envir = .GlobalEnv)
  }
  cli::cli_progress_update(set = x, .envir = .GlobalEnv)
  if (x > 100) {
    cli::cli_progress_done(.envir = .GlobalEnv)
  }
}
options("duckdb.progress_display" = progress)
conn <- duckdb::dbConnect(duckdb::duckdb())
duckdb::dbSendQuery(conn, "SET progress_bar_time = 0;")
q <- "CREATE OR REPLACE TABLE BOB AS (
      SELECT * FROM 'ldbc-sf300-comments-creationDate.parquet')"
duckdb::dbSendQuery(conn, q)

[meztez]

I'm done on this one. Let me know if this works for you.

[e-kotov]

Testing with {spanishoddata}:

library(spanishoddata)
library(duckdb)
library(tidyverse)


x_dates <- c("2022-01-01", "2022-01-02", "2022-01-03", "2022-01-04")
x <- spod_get(type = "od", zones = "distr", dates = x_dates)

dbGetQuery(x$src$con, "SELECT current_setting('enable_progress_bar');")
dbSendQuery(x$src$con, "SET enable_progress_bar = true;")
dbGetQuery(x$src$con, "SELECT current_setting('enable_progress_bar');")


progress <- function(x) {
  if (cli::cli_progress_num() == 0) {
    cli::cli_progress_bar("Duckdb SQL", total = 100, .envir = .GlobalEnv)
  }
  cli::cli_progress_update(set = x, .envir = .GlobalEnv)
  if (x > 100) {
    cli::cli_progress_done(.envir = .GlobalEnv)
  }
}

options("duckdb.progress_display" = progress)
duckdb::dbSendQuery(x$src$con, "SET progress_bar_time = 0;")

xx <- x |> group_by(id_origin, date, activity_origin) |> summarise(mean_trips = mean(n_trips)) |> collect()

And it works!

@meztez do we have to manually define the progress function though...? what is the final idea of this PR? I would expect that progress bar just 'magically' appears as soon as we do:

dbGetQuery(x$src$con, "SELECT current_setting('enable_progress_bar');")

p.s. in my case x$src$con is because spod_get returns tbl_duckdb_connection, so you have to reach out to the connection itself.

[meztez]

It could provide a dummy default. It's just a function(x) called with progress percentage from within duckdb-r.

I'm not the package maintainer and I just needed it for a deliverable, so whatever works is fine by me.

[krlmlr]

Thanks for the PR!

Looking at the implementation, I think the callback function should be a slot in the connection object. There could be basic reporting (opt-out, in interactive mode only) in the duckdb R package, and more sophisticated progress in duckplyr.

[e-kotov]

I'm not the package maintainer and I just needed it for a deliverable, so whatever works is fine by me.

@meztez totally makes sense. Thanks for the work in the internals to make this possible!

Looking forward for this to be merged!

[HenrikBengtsson]

In the above examples, (x > 100) indicates that the processing is complete. Shouldn't that be (x >= 100)? I think it's more common to consider 100% to indicate "done" than "still processing".

[meztez]

In the above examples, (x > 100) indicates that the processing is complete. Shouldn't that be (x >= 100)? I think it's more common to consider 100% to indicate "done" than "still processing".

progress <- function(x) {
  if (x < 100 && cli::cli_progress_num() == 0) {
    cli::cli_progress_bar("Duckdb SQL", total = 100, .envir = .GlobalEnv, )
  }
  cli::cli_progress_update(set = x, .envir = .GlobalEnv)
}

options("duckdb.progress_display" = progress)