[OPTIMIZATION] Profiling of influx_query()

[ckatsulis]

Dominik,

First, a thanks for you effort and attention to this package. Hopefully you will be getting more and more help in the upcoming months as influxdb continues to gain ground.

Observation

One interesting observation with the package is the time it takes to pull down large data sets relative to the network activity on the querying host where the R call is made.

I have noticed large lags between when the data connection ceasing activity and when the R call eventually returns a result. As I'm on 155Mbits/sec download, I see my network connection peak at around 35-50Mbits for about 10-20 seconds, then nothing. I then believe it takes R several minutes to assemble what has been pulled. Is this consistent with your experience?

Investigation

I tried to get a profile using profvis and profr, however it causes my windows R session to crash. Are you able to try this on your end for windows?

However, I was able to get this to work for my linux machine. I wanted to give you one from about 100k+ lines, but the file size was around 6MB and takes 35 seconds on my little VM. This was the result from a call with around 10k lines:

The problem looks fairly well nested in some character / string handling. In my experience, it might benefit all if those calls could be replaced with optimized Rcpp equivalents. I don't know if you have a need for speed, but a change would drastically elevate R's competitiveness to python or the like in these situations.

profile = dget("profile.txt")
profile.txt

System Info

> sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] profvis_0.3.3     dygraphs_1.1.1.4  basePricer_0.1.0  profr_0.3.1       nnls_1.4          xts_0.10-1        zoo_1.8-0         influxdbr_0.11.18

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.11    lattice_0.20-35 digest_0.6.12   R6_2.2.2        grid_3.4.0      plyr_1.8.4      magrittr_1.5    httr_1.3.0      stringi_1.1.5   curl_2.7        tools_3.4.0     stringr_1.2.0  
[13] htmlwidgets_0.8 purrr_0.2.2.2   compiler_3.4.0  htmltools_0.3.6

[dleutnant]

@ckatsulis Thanks for your profiling!!! Could you please check the performance again with latest dev version 8b0568d

[ckatsulis]

You are the man! How do I install that specific dev version with devtools?

…

On Mon, Aug 21, 2017 at 4:24 AM, Dominik Leutnant ***@***.***> wrote: @ckatsulis <https://github.com/ckatsulis> Thanks for your profiling!!! Could you please check the performance again with latest dev version 8b0568d <8b0568d> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHKwqnfUq0wnAxHWNpCQ6mr8Z2tPG-Vzks5saUy3gaJpZM4O8t0d> .

[dleutnant]

devtools::install_github("dleutnant/influxdbr@dev")

[ckatsulis]

That works exactly as anticipated. Thanks again for the quick turn around! I'm guessing 20x improvement.

You have recently changed your main wiki page, but you used to remind patrons how to install the dev version until you can push these to CRAN. People are definately gonna want these last three improvements!

[dleutnant]

CRAN maintainers are quite fast these days: New release of influxdbr 0.13.0. is already on CRAN! Thanks for your support, @ckatsulis !

[ckatsulis]

Quick moving... Thanks for letting me know.

…

On Tue, Aug 22, 2017 at 12:37 AM, Dominik Leutnant ***@***.*** > wrote: CRAN maintainers are quite fast these days: New release of influxdbr is already on CRAN! Thanks for your support, @ckatsulis <https://github.com/ckatsulis> ! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHKwqnPN3YmEGbZ-O7MfnjGWw8xNBbBNks5samkygaJpZM4O8t0d> .

[dleutnant]

I'm going to close this issue for now as it addressed a specific parsing problem (which has been solved).

[ckatsulis]

Thanks again!

…

On Thu, Aug 31, 2017 at 3:37 AM, Dominik Leutnant ***@***.***> wrote: Closed #28 <#28>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHKwqqNQG38xN5gX8wLVFbyubHnYCRD8ks5sdnDSgaJpZM4O8t0d> .