Throttling not working properly with purrr::map2()

[judith-bourque]

Issue

The rate limit for the API is 45 req / min.

When I run more than 45 queries with purrr::map2(), I get the following error message:

> purrr::map2(rep("132.203.167.188", 50), c("status,message,country,city"), locateip::locate_ip)
Error in `purrr::map2()`:
ℹ In index: 46.
Caused by error in `httr2::req_perform()`:
! HTTP 429 Too Many Requests.
Backtrace:
 1. purrr::map2(...)
 2. purrr:::map2_("list", .x, .y, .f, ..., .progress = .progress)
 6. locateip (local) .f(.x[[i]], .y[[i]], ...)
 7. locateip:::get_location(...)
 8. httr2::req_perform(...)

Troubleshooting

• Check that there's a httr2::req_throttle() with rate limit of 45 req / min
  

  locateip/R/locate.R

  Line 67 in ad2a717

  httr2::req_throttle(45 / 60)

• Check that the performed request still has a rate limit (this test passes)
• Use httr2::throttle_status()
• Read Rate limiting section: for an API with 10 req / min limit, it says to set the throttle to 10/60, which is exactly what's in this code
• Re-run reprex with throttle way below limit (like 30 req / min) AND real sample of 50 IPs

Proposed solutions

• Create a function to batch calls with the JSON endpoint #126
• Check value of X-RI header before sending next request #122
• Add cache to store previous query results
• Use httr2::req_retry() to control when/how to retry #124

[judith-bourque]

httr2::throttle_status() doesn't seem to take into acount loops when using purrr::map2()... or does it? It returns the last request, nothing more.

resp <- purrr::map2(rep("132.203.167.188", 3), c("status,message,country,city"), locateip::locate_ip)
httr2::throttle_status()
#>        realm        last_request
#> 1 ip-api.com 2023-06-14 12:36:00

Created on 2023-06-14 with reprex v2.0.2

[judith-bourque]

Tried putting throttle to 30/60, got this error:

Could it be because I'm making multiple identical calls?

resp <- purrr::map2(rep("132.203.167.188", 50), c("status,message,country,city"), locateip::locate_ip)
#> Error in `purrr::map2()`:
#> ℹ In index: 31.
#> Caused by error in `httr2::req_perform()`:
#> ! HTTP 502 Bad Gateway.
#> Backtrace:
#>      ▆
#>   1. └─purrr::map2(...)
#>   2.   └─purrr:::map2_("list", .x, .y, .f, ..., .progress = .progress)
#>   3.     ├─purrr:::with_indexed_errors(...)
#>   4.     │ └─base::withCallingHandlers(...)
#>   5.     ├─purrr:::call_with_cleanup(...)
#>   6.     └─locateip (local) .f(.x[[i]], .y[[i]], ...)
#>   7.       └─locateip:::get_location(...)
#>   8.         └─httr2::req_perform(...)
#>   9.           └─httr2:::resp_abort(resp, error_body(req, resp), call = error_call)
#>  10.             └─rlang::abort(...)
httr2::throttle_status()
#>        realm        last_request
#> 1 ip-api.com 2023-06-14 12:51:56

[judith-bourque]

I re-ran it with a sample of 50 unique IPs and got a valid response 🥳

Not sure how to create a reprex without sharing real IP adresses, but this gives an idea...

# Data format example
# ip_tibble <- tibble::tibble(ip = c("123.123.123"))

# Install package with rate limit of 40 req / min
remotes::install_github("clessn/locateip", ref = remotes::github_pull("121"))

ip_location <-
  purrr::map2(
    ip_tibble$ip,
    c("status,message,country,regionName,city,lat,lon,query"),
    locateip::locate_ip
  )

[judith-bourque]

Using a for loop with a longer wait time, I was able to do 200+ requests without any 429 errors.

ips <- rep("132.203.167.188", 200)

for (i in seq_along(ips) {
    ip <- ips[[i]]
    resp <- locateip::locate_ip(ip, fields = c("status,message,country,regionName,city,lat,lon,query")
    readr::write_rds(resp, paste0(i, ".rds"))
    Sys.sleep(60)
}