GCP resumable upload error #883

stuvet · 2022-07-21T20:51:42Z

Prework

Read and agree to the code of conduct and contributing guidelines.
Confirm that your issue is most likely a genuine bug in stantargets and not a known limitation, usage error, or bug in another package that stantargets depends on.
If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
- Runnable: post enough R code and data so any onlooker can create the error on their own computer.
- Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
- Readable: format your code according to the tidyverse style guide.

Description

For CmdstanMCMC objects > 5Mb &repository='gcs' the sampling step within tar_stan_mcmc runs fine, but reliably throws:

Error: _store_ Must supply either retry_object or all of upload_url, file and type from googleCloudStorageR::gcs_retry_upload.

This is resolved with either repository='local' or repository='aws', either of which complete without error.

Also note the deprecation warning when using cmdstanr 0.5.2.1 (see bottom of reprex)

Expected result

Target successfully completes upload to gcp.

Reprex

library(targets)
tar_script({
    stanfile <- "model.stan"

    tar_option_set(
      resources = tar_resources(
        gcp = tar_resources_gcp(bucket = "my-gcs-bucket"), 
        aws = tar_resources_aws(bucket = "my-aws-bucket")
      ), 
      cue = tar_cue(mode = "always"), 
      format = "qs", 
      repository = "gcp"
    )

    cmdstanr::cmdstan_version()
    cmdstanr::check_cmdstan_toolchain()
    cmdstanr::write_stan_file("
       data {          
         int<lower=0> N_rep; // Used to increase output file size to trigger resumable uploads for the CmdstanMCMC object
         int<lower=1> N;
         vector[N] x;          
         vector[N] y;        
       }
       parameters {          
         real alpha;         
         real beta;          
         real<lower=0> sigma;        
       }       
       model {          
         alpha ~ std_normal();          
         beta ~ std_normal();          
         sigma ~ std_normal();          
         y ~ normal(alpha + x * beta , sigma);       
         }       
       generated quantities {          
         array[N_rep] real x_rand = uniform_rng(rep_vector(0.0, N_rep), rep_vector(100.0, N_rep));         
         array[N_rep] real y_pred = normal_rng(alpha + to_vector(x_rand) * beta, sigma);        
       } ", dir = getwd(), basename = stanfile)

    list(
       stantargets::tar_stan_mcmc(
         fit, 
         stanfile, 
         data = list(N_rep = 100, N = 100, x = 1:100, y = 2 * (1:100) + rnorm(100, 0, 1))
       )
    )
})
tar_make()
#> The C++ toolchain required for CmdStan is setup properly!
#> • start target fit_file_model
#> • built target fit_file_model
#> • start target fit_data
#> • built target fit_data
#> • start target fit_mcmc_model
#> Running MCMC with 4 sequential chains...
#> 
#> Chain 1 Iteration:    1 / 2000 [  0%]  (Warmup) 
#> Chain 1 Iteration:  100 / 2000 [  5%]  (Warmup) 
#> Chain 1 Iteration:  200 / 2000 [ 10%]  (Warmup) 
#> Chain 1 Iteration:  300 / 2000 [ 15%]  (Warmup) 
#> Chain 1 Iteration:  400 / 2000 [ 20%]  (Warmup) 
#> Chain 1 Iteration:  500 / 2000 [ 25%]  (Warmup) 
#> Chain 1 Iteration:  600 / 2000 [ 30%]  (Warmup) 
#> Chain 1 Iteration:  700 / 2000 [ 35%]  (Warmup) 
#> Chain 1 Iteration:  800 / 2000 [ 40%]  (Warmup) 
#> Chain 1 Iteration:  900 / 2000 [ 45%]  (Warmup) 
#> Chain 1 Iteration: 1000 / 2000 [ 50%]  (Warmup) 
#> Chain 1 Iteration: 1001 / 2000 [ 50%]  (Sampling) 
#> Chain 1 Iteration: 1100 / 2000 [ 55%]  (Sampling) 
#> Chain 1 Iteration: 1200 / 2000 [ 60%]  (Sampling) 
#> Chain 1 Iteration: 1300 / 2000 [ 65%]  (Sampling) 
#> Chain 1 Iteration: 1400 / 2000 [ 70%]  (Sampling) 
#> Chain 1 Iteration: 1500 / 2000 [ 75%]  (Sampling) 
#> Chain 1 Iteration: 1600 / 2000 [ 80%]  (Sampling) 
#> Chain 1 Iteration: 1700 / 2000 [ 85%]  (Sampling) 
#> Chain 1 Iteration: 1800 / 2000 [ 90%]  (Sampling) 
#> Chain 1 Iteration: 1900 / 2000 [ 95%]  (Sampling) 
#> Chain 1 Iteration: 2000 / 2000 [100%]  (Sampling) 
#> Chain 1 finished in 0.3 seconds.
#> Chain 2 Iteration:    1 / 2000 [  0%]  (Warmup) 
#> Chain 2 Iteration:  100 / 2000 [  5%]  (Warmup) 
#> Chain 2 Iteration:  200 / 2000 [ 10%]  (Warmup) 
#> Chain 2 Iteration:  300 / 2000 [ 15%]  (Warmup) 
#> Chain 2 Iteration:  400 / 2000 [ 20%]  (Warmup) 
#> Chain 2 Iteration:  500 / 2000 [ 25%]  (Warmup) 
#> Chain 2 Iteration:  600 / 2000 [ 30%]  (Warmup) 
#> Chain 2 Iteration:  700 / 2000 [ 35%]  (Warmup) 
#> Chain 2 Iteration:  800 / 2000 [ 40%]  (Warmup) 
#> Chain 2 Iteration:  900 / 2000 [ 45%]  (Warmup) 
#> Chain 2 Iteration: 1000 / 2000 [ 50%]  (Warmup) 
#> Chain 2 Iteration: 1001 / 2000 [ 50%]  (Sampling) 
#> Chain 2 Iteration: 1100 / 2000 [ 55%]  (Sampling) 
#> Chain 2 Iteration: 1200 / 2000 [ 60%]  (Sampling) 
#> Chain 2 Iteration: 1300 / 2000 [ 65%]  (Sampling) 
#> Chain 2 Iteration: 1400 / 2000 [ 70%]  (Sampling) 
#> Chain 2 Iteration: 1500 / 2000 [ 75%]  (Sampling) 
#> Chain 2 Iteration: 1600 / 2000 [ 80%]  (Sampling) 
#> Chain 2 Iteration: 1700 / 2000 [ 85%]  (Sampling) 
#> Chain 2 Iteration: 1800 / 2000 [ 90%]  (Sampling) 
#> Chain 2 Iteration: 1900 / 2000 [ 95%]  (Sampling) 
#> Chain 2 Iteration: 2000 / 2000 [100%]  (Sampling) 
#> Chain 2 finished in 0.3 seconds.
#> Chain 3 Iteration:    1 / 2000 [  0%]  (Warmup) 
#> Chain 3 Iteration:  100 / 2000 [  5%]  (Warmup) 
#> Chain 3 Iteration:  200 / 2000 [ 10%]  (Warmup) 
#> Chain 3 Iteration:  300 / 2000 [ 15%]  (Warmup) 
#> Chain 3 Iteration:  400 / 2000 [ 20%]  (Warmup) 
#> Chain 3 Iteration:  500 / 2000 [ 25%]  (Warmup) 
#> Chain 3 Iteration:  600 / 2000 [ 30%]  (Warmup) 
#> Chain 3 Iteration:  700 / 2000 [ 35%]  (Warmup) 
#> Chain 3 Iteration:  800 / 2000 [ 40%]  (Warmup) 
#> Chain 3 Iteration:  900 / 2000 [ 45%]  (Warmup) 
#> Chain 3 Iteration: 1000 / 2000 [ 50%]  (Warmup) 
#> Chain 3 Iteration: 1001 / 2000 [ 50%]  (Sampling) 
#> Chain 3 Iteration: 1100 / 2000 [ 55%]  (Sampling) 
#> Chain 3 Iteration: 1200 / 2000 [ 60%]  (Sampling) 
#> Chain 3 Iteration: 1300 / 2000 [ 65%]  (Sampling) 
#> Chain 3 Iteration: 1400 / 2000 [ 70%]  (Sampling) 
#> Chain 3 Iteration: 1500 / 2000 [ 75%]  (Sampling) 
#> Chain 3 Iteration: 1600 / 2000 [ 80%]  (Sampling) 
#> Chain 3 Iteration: 1700 / 2000 [ 85%]  (Sampling) 
#> Chain 3 Iteration: 1800 / 2000 [ 90%]  (Sampling) 
#> Chain 3 Iteration: 1900 / 2000 [ 95%]  (Sampling) 
#> Chain 3 Iteration: 2000 / 2000 [100%]  (Sampling) 
#> Chain 3 finished in 0.3 seconds.
#> Chain 4 Iteration:    1 / 2000 [  0%]  (Warmup) 
#> Chain 4 Iteration:  100 / 2000 [  5%]  (Warmup) 
#> Chain 4 Iteration:  200 / 2000 [ 10%]  (Warmup) 
#> Chain 4 Iteration:  300 / 2000 [ 15%]  (Warmup) 
#> Chain 4 Iteration:  400 / 2000 [ 20%]  (Warmup) 
#> Chain 4 Iteration:  500 / 2000 [ 25%]  (Warmup) 
#> Chain 4 Iteration:  600 / 2000 [ 30%]  (Warmup) 
#> Chain 4 Iteration:  700 / 2000 [ 35%]  (Warmup) 
#> Chain 4 Iteration:  800 / 2000 [ 40%]  (Warmup) 
#> Chain 4 Iteration:  900 / 2000 [ 45%]  (Warmup) 
#> Chain 4 Iteration: 1000 / 2000 [ 50%]  (Warmup) 
#> Chain 4 Iteration: 1001 / 2000 [ 50%]  (Sampling) 
#> Chain 4 Iteration: 1100 / 2000 [ 55%]  (Sampling) 
#> Chain 4 Iteration: 1200 / 2000 [ 60%]  (Sampling) 
#> Chain 4 Iteration: 1300 / 2000 [ 65%]  (Sampling) 
#> Chain 4 Iteration: 1400 / 2000 [ 70%]  (Sampling) 
#> Chain 4 Iteration: 1500 / 2000 [ 75%]  (Sampling) 
#> Chain 4 Iteration: 1600 / 2000 [ 80%]  (Sampling) 
#> Chain 4 Iteration: 1700 / 2000 [ 85%]  (Sampling) 
#> Chain 4 Iteration: 1800 / 2000 [ 90%]  (Sampling) 
#> Chain 4 Iteration: 1900 / 2000 [ 95%]  (Sampling) 
#> Chain 4 Iteration: 2000 / 2000 [100%]  (Sampling) 
#> Chain 4 finished in 0.3 seconds.
#> 
#> All 4 chains finished successfully.
#> Mean chain execution time: 0.3 seconds.
#> Total execution time: 1.5 seconds.
#> 
#> ✖ error target fit_mcmc_model
#> • end pipeline: 24.583 seconds
#> Error : _store_ Must supply either retry_object or all of upload_url, file and type
#> In addition: Warning messages:
#> 1: 'validate_csv' is deprecated. Please use 'diagnostics' instead. 
#> 2: 1 targets produced warnings. Run tar_meta(fields = warnings, complete_only = TRUE) for the messages.
#> ✖ Problem with the pipeline.
#> Error:
#> ! problem with the pipeline.

^{Created on 2022-07-21 by the reprex package (v2.0.1)}

Setup

Centos 7 in GCP
R: 4.2.0
Targets: f37af16
Stantargets: 4ee5367
Cmdstan: 2.30.0
CmdstanR: 0.5.2.1
googleCloudStorageR: 0.7.0
Repository: gcp (working as expected for all other targets)

Session

R version 4.2.0 (2022-04-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS:   /apps/R/4.2.0/lib64/R/lib/libRblas.so
LAPACK: /apps/R/4.2.0/lib64/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] googleCloudStorageR_0.7.0 fstcore_0.9.12            dplyr_1.0.9               targets_0.12.1.9000      

loaded via a namespace (and not attached):
  [1] matrixStats_0.62.0     fs_1.5.2               bigrquery_1.4.0        bit64_4.0.5            httr_1.4.3            
  [6] rstan_2.26.13          googleAuthR_2.0.0      R.cache_0.15.0         tensorA_0.36.2         tools_4.2.0           
 [11] backports_1.4.1        utf8_1.2.2             R6_2.5.1               DBI_1.1.3              colorspace_2.0-3      
 [16] withr_2.5.0            gridExtra_2.3          prettyunits_1.1.1      tidyselect_1.1.2       processx_3.7.0        
 [21] bit_4.0.4              curl_4.3.2             compiler_4.2.0         cli_3.3.0              paws_0.1.12           
 [26] stringfish_0.15.7      posterior_1.2.2        scales_1.2.0           checkmate_2.1.0        readr_2.1.2           
 [31] callr_3.7.1            askpass_1.1            StanHeaders_2.26.13    digest_0.6.29          rmarkdown_2.14        
 [36] R.utils_2.12.0         pkgconfig_2.0.3        htmltools_0.5.2        parallelly_1.32.0      fst_0.9.8             
 [41] styler_1.7.0           dbplyr_2.2.1           fastmap_1.1.0          highr_0.9              rlang_1.0.4           
 [46] rstudioapi_0.13        tarchetypes_0.6.0.9000 farver_2.1.1           generics_0.1.3         RApiSerialize_0.1.0   
 [51] jsonlite_1.8.0         zip_2.2.0              R.oo_1.25.0            distributional_0.3.0   inline_0.3.19         
 [56] magrittr_2.0.3         loo_2.5.1              Rcpp_1.0.9             munsell_0.5.0          fansi_1.0.3           
 [61] abind_1.4-5            lifecycle_1.0.1        R.methodsS3_1.8.2      yaml_2.3.5             pkgbuild_1.3.1        
 [66] stantargets_0.0.4      grid_4.2.0             parallel_4.2.0         listenv_0.8.0          crayon_1.5.1          
 [71] hms_1.1.1              knitr_1.39             ps_1.7.1               pillar_1.7.0           igraph_1.3.2          
 [76] base64url_1.4          stats4_4.2.0           codetools_0.2-18       reprex_2.0.1           glue_1.6.2            
 [81] evaluate_0.15          V8_4.2.0               data.table_1.14.2      RcppParallel_5.1.5     vctrs_0.4.1           
 [86] tzdb_0.3.0             gtable_0.3.0           openssl_2.0.2          purrr_0.3.4            qs_0.25.3             
 [91] future_1.26.1          assertthat_0.2.1       cachem_1.0.6           ggplot2_3.3.6          xfun_0.31             
 [96] gargle_1.2.0           tibble_3.1.7           memoise_2.0.1          globals_0.15.1         ellipsis_0.3.2        
[101] cmdstanr_0.5.2.1

The text was updated successfully, but these errors were encountered:

stuvet added the type: bug label Jul 21, 2022

stuvet assigned wlandau Jul 21, 2022

wlandau transferred this issue from ropensci/stantargets Jul 22, 2022

wlandau removed the type: bug label Jul 22, 2022

ropensci locked and limited conversation to collaborators Jul 22, 2022

wlandau converted this issue into discussion #884 Jul 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

This issue was moved to a discussion.

GCP resumable upload error #883

GCP resumable upload error #883

stuvet commented Jul 21, 2022 •

edited

Loading

This issue was moved to a discussion.

This issue was moved to a discussion.

GCP resumable upload error #883

GCP resumable upload error #883

Comments

stuvet commented Jul 21, 2022 • edited Loading

Prework

Description

Expected result

Reprex

Setup

Session

This issue was moved to a discussion.

stuvet commented Jul 21, 2022 •

edited

Loading