-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot load datasets in BioC 3.19 (HTTP 403 error) #23
Comments
Hm. I don't have any problems, even when forcibly re-downloading, e.g., with: se <- fetchReference("immgen", "2024-02-26", realize.assays = TRUE, overwrite=TRUE) |
Hmm, it's happening for me locally on home network and when run on a WEHI server. scRNAseq::GrunHSCData()
Error: SignatureDoesNotMatch (HTTP 403). The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. |
Just tried at work, doesn't happen to me on either Mac or Linux. Dunno what's going on. You'll have to do some debugging on your end. The relevant function is |
I'll see if I can get someone else to reproduce it. #> traceback()
10: stop(error)
9: handler$fn(request)
8: run(request, retry)
7: retry(request)
6: send_request(request)
5: s$list_objects_v2(config$bucket, Prefix = actual.prefix, ContinuationToken = token)
4: listFiles(project, asset, version, config = config)
3: saveVersion(package, name, version, cache = cache, overwrite = overwrite)
2: fetchReference("immgen", "2024-02-26", realize.assays = TRUE)
1: ImmGenData() at which point: #> config$bucket
[1] "gypsum-prod"
#> actual.prefix
[1] "celldex/immgen/2024-02-26/"
#> token
NULL |
Maybe paws.storage is prioritizing AWS credentials from elsewhere, instead of using that ones supplied by
Run down until |
Not sure I've understood what you're asking for. # Replicating what happens what I run celldex::ImmGenData()
config <- gypsum::publicS3Config(cache = gypsum:::cacheDirectory())
s <- gypsum:::create_s3(config)
actual.prefix <- "celldex/immgen/2024-02-26/"
token <- NULL
listing <- s$list_objects_v2(config$bucket, Prefix = actual.prefix,
ContinuationToken = token)
# https://github.com/LTLA/celldex/issues/23#issuecomment-2089328586
config0 <- gypsum::publicS3Config()
s <- gypsum:::create_s3(config0)
debug(s$list_objects_v2)
listing <- s$list_objects_v2(config$bucket, Prefix = actual.prefix)
debugging in: s$list_objects_v2(config$bucket, Prefix = actual.prefix)
debug: {
op <- new_operation(name = "ListObjectsV2", http_method = "GET",
http_path = "/{Bucket}?list-type=2", paginator = list(input_token = "ContinuationToken",
limit_key = "MaxKeys", output_token = "NextContinuationToken",
result_key = list("Contents", "CommonPrefixes")))
input <- .s3$list_objects_v2_input(Bucket = Bucket, Delimiter = Delimiter,
EncodingType = EncodingType, MaxKeys = MaxKeys, Prefix = Prefix,
ContinuationToken = ContinuationToken, FetchOwner = FetchOwner,
StartAfter = StartAfter, RequestPayer = RequestPayer,
ExpectedBucketOwner = ExpectedBucketOwner, OptionalObjectAttributes = OptionalObjectAttributes)
output <- .s3$list_objects_v2_output()
config <- get_config()
svc <- .s3$service(config)
request <- new_request(svc, op, input, output)
response <- send_request(request)
return(response)
}
Browse[1]>
debug: op <- new_operation(name = "ListObjectsV2", http_method = "GET",
http_path = "/{Bucket}?list-type=2", paginator = list(input_token = "ContinuationToken",
limit_key = "MaxKeys", output_token = "NextContinuationToken",
result_key = list("Contents", "CommonPrefixes")))
Browse[1]>
debug: input <- .s3$list_objects_v2_input(Bucket = Bucket, Delimiter = Delimiter,
EncodingType = EncodingType, MaxKeys = MaxKeys, Prefix = Prefix,
ContinuationToken = ContinuationToken, FetchOwner = FetchOwner,
StartAfter = StartAfter, RequestPayer = RequestPayer, ExpectedBucketOwner = ExpectedBucketOwner,
OptionalObjectAttributes = OptionalObjectAttributes)
Browse[1]>
debug: output <- .s3$list_objects_v2_output()
Browse[1]>
debug: config <- get_config()
Browse[1]>
debug: svc <- .s3$service(config)
Browse[1]> names(config$credentials$creds)
[1] "access_key_id" "secret_access_key" "session_token" "access_token" "expiration" "provider_name"
Browse[1]> names(config0)
[1] "endpoint" "bucket" "key" "secret"
Browse[1]> identical(as.character(config$credentials$creds$access_key_id), config0$key)
[1] TRUE
Browse[1]> identical(as.character(config$credentials$creds$secret_access_key), config0$secret)
[1] TRUE I can't find any other AWS credentials on any of the 2 laptops and 1 server where I'm having problems with this. |
Well, I dunno what's going on with the S3 API. So I just modified gypsum to download files and such via HTTP instead. Try installing the latest |
FWIW my new suspicion is that config <- gypsum::publicS3Config()
s <- paws.storage::s3(
endpoint=config$endpoint,
region="us-east-1", # swapped with "auto"
credentials=list(
creds=list(
access_key_id=config$key,
secret_access_key=config$secret
)
)
)
s$list_objects_v2(config$bucket, Delimiter="/") With and without "auto" in the |
No difference apart from the obvious in specifying the config <- gypsum::publicS3Config()
s1 <- paws.storage::s3(
endpoint=config$endpoint,
region="us-east-1", # swapped with "auto"
credentials=list(
creds=list(
access_key_id=config$key,
secret_access_key=config$secret
)
)
)
s1$list_objects_v2(config$bucket, Delimiter="/")
#> $IsTruncated
#> [1] FALSE
#>
#> $Contents
#> list()
#>
#> $Name
#> [1] "gypsum-prod"
#>
#> $Prefix
#> character(0)
#>
#> $Delimiter
#> [1] "/"
#>
#> $MaxKeys
#> [1] 1000
#>
#> $CommonPrefixes
#> $CommonPrefixes[[1]]
#> $CommonPrefixes[[1]]$Prefix
#> [1] "..logs/"
#>
#>
#> $CommonPrefixes[[2]]
#> $CommonPrefixes[[2]]$Prefix
#> [1] "celldex/"
#>
#>
#> $CommonPrefixes[[3]]
#> $CommonPrefixes[[3]]$Prefix
#> [1] "scRNAseq/"
#>
#>
#> $CommonPrefixes[[4]]
#> $CommonPrefixes[[4]]$Prefix
#> [1] "test-R-perms/"
#>
#>
#> $CommonPrefixes[[5]]
#> $CommonPrefixes[[5]]$Prefix
#> [1] "test-R-quota/"
#>
#>
#> $CommonPrefixes[[6]]
#> $CommonPrefixes[[6]]$Prefix
#> [1] "test-R/"
#>
#>
#>
#> $EncodingType
#> character(0)
#>
#> $KeyCount
#> [1] 6
#>
#> $ContinuationToken
#> character(0)
#>
#> $NextContinuationToken
#> character(0)
#>
#> $StartAfter
#> character(0)
#>
#> $RequestCharged
#> character(0)
s2 <- paws.storage::s3(
endpoint=config$endpoint,
# region="us-east-1", # swapped with "auto"
region="auto", # swapped with "auto"
credentials=list(
creds=list(
access_key_id=config$key,
secret_access_key=config$secret
)
)
)
s2$list_objects_v2(config$bucket, Delimiter="/")
#> $IsTruncated
#> [1] FALSE
#>
#> $Contents
#> list()
#>
#> $Name
#> [1] "gypsum-prod"
#>
#> $Prefix
#> character(0)
#>
#> $Delimiter
#> [1] "/"
#>
#> $MaxKeys
#> [1] 1000
#>
#> $CommonPrefixes
#> $CommonPrefixes[[1]]
#> $CommonPrefixes[[1]]$Prefix
#> [1] "..logs/"
#>
#>
#> $CommonPrefixes[[2]]
#> $CommonPrefixes[[2]]$Prefix
#> [1] "celldex/"
#>
#>
#> $CommonPrefixes[[3]]
#> $CommonPrefixes[[3]]$Prefix
#> [1] "scRNAseq/"
#>
#>
#> $CommonPrefixes[[4]]
#> $CommonPrefixes[[4]]$Prefix
#> [1] "test-R-perms/"
#>
#>
#> $CommonPrefixes[[5]]
#> $CommonPrefixes[[5]]$Prefix
#> [1] "test-R-quota/"
#>
#>
#> $CommonPrefixes[[6]]
#> $CommonPrefixes[[6]]$Prefix
#> [1] "test-R/"
#>
#>
#>
#> $EncodingType
#> character(0)
#>
#> $KeyCount
#> [1] 6
#>
#> $ContinuationToken
#> character(0)
#>
#> $NextContinuationToken
#> character(0)
#>
#> $StartAfter
#> character(0)
#>
#> $RequestCharged
#> character(0)
all.equal(s1, s2)
#> [1] "Component \".internal\": Component \"config\": Component \"region\": 1 string mismatch" Session infosessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.4.0 (2024-04-24)
#> os macOS Sonoma 14.4.1
#> system aarch64, darwin20
#> ui X11
#> language (EN)
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz Australia/Melbourne
#> date 2024-05-08
#> pandoc 3.1.11 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date (UTC) lib source
#> cli 3.6.2 2023-12-11 [1] CRAN (R 4.4.0)
#> crayon 1.5.2 2022-09-29 [1] CRAN (R 4.4.0)
#> curl 5.2.1 2024-03-01 [1] CRAN (R 4.4.0)
#> digest 0.6.35 2024-03-11 [1] CRAN (R 4.4.0)
#> evaluate 0.23 2023-11-01 [1] CRAN (R 4.4.0)
#> fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.4.0)
#> filelock 1.0.3 2023-12-11 [1] CRAN (R 4.4.0)
#> fs 1.6.4 2024-04-25 [1] CRAN (R 4.4.0)
#> glue 1.7.0 2024-01-09 [1] CRAN (R 4.4.0)
#> gypsum 1.0.0 2024-04-30 [1] Bioconductor 3.19 (R 4.4.0)
#> htmltools 0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0)
#> httr 1.4.7 2023-08-15 [1] CRAN (R 4.4.0)
#> httr2 1.0.1 2024-04-01 [1] CRAN (R 4.4.0)
#> jsonlite 1.8.8 2023-12-04 [1] CRAN (R 4.4.0)
#> knitr 1.46 2024-04-06 [1] CRAN (R 4.4.0)
#> lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.4.0)
#> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.4.0)
#> paws.common 0.7.2 2024-04-09 [1] CRAN (R 4.4.0)
#> paws.storage 0.5.0 2024-01-09 [1] CRAN (R 4.4.0)
#> R6 2.5.1 2021-08-19 [1] CRAN (R 4.4.0)
#> rappdirs 0.3.3 2021-01-31 [1] CRAN (R 4.4.0)
#> Rcpp 1.0.12 2024-01-09 [1] CRAN (R 4.4.0)
#> reprex 2.1.0 2024-01-11 [1] CRAN (R 4.4.0)
#> rlang 1.1.3 2024-01-10 [1] CRAN (R 4.4.0)
#> rmarkdown 2.26 2024-03-05 [1] CRAN (R 4.4.0)
#> rstudioapi 0.16.0 2024-03-24 [1] CRAN (R 4.4.0)
#> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.4.0)
#> withr 3.0.0 2024-01-16 [1] CRAN (R 4.4.0)
#> xfun 0.43 2024-03-25 [1] CRAN (R 4.4.0)
#> xml2 1.3.6 2023-12-04 [1] CRAN (R 4.4.0)
#> yaml 2.3.8 2023-12-11 [1] CRAN (R 4.4.0)
#>
#> [1] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library
#>
#> ────────────────────────────────────────────────────────────────────────────── But using current |
Not sure if this is a temporary thing and/or related to the in-progress release of BioC 3.19:
Created on 2024-05-01 with reprex v2.1.0
Session info
The text was updated successfully, but these errors were encountered: