Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix_382 #383

Merged
merged 10 commits into from
Jun 14, 2023
Merged

fix_382 #383

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a879bf7
Update getQuote.R
ethanbsmith Apr 19, 2023
b50e2d4
Update getQuote.R
ethanbsmith Apr 19, 2023
dc8ecd5
Update getQuote.R
ethanbsmith May 6, 2023
76b3728
downshifted to v6 api
ethanbsmith May 7, 2023
fd345e3
added both v6 and v7 code paths
ethanbsmith May 18, 2023
33cbc8c
updated gdpr scenario error handling
ethanbsmith May 18, 2023
bc27737
Update getQuote.R
ethanbsmith May 18, 2023
9f9b7b2
minor cleanup
ethanbsmith May 19, 2023
b3dbad6
Update getQuote.R
ethanbsmith May 19, 2023
554af1b
Update getQuote.R
ethanbsmith May 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 46 additions & 7 deletions R/getQuote.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,46 @@ function(Symbols,src='yahoo',what, ...) {
df[Symbols,]
}

.yahooSession <- function(force.new = FALSE) {
cache.name <- "_yahoo_curl_session_"
ses <- get0(cache.name, .quantmodEnv) #get cached session

if (is.null(ses) || force.new) {
ses <- list()
ses$h <- curl::new_handle()
#yahoo finance doesn't seem to set cookies without these headers
#and the cookies are needed to get the crumb
curl::handle_setheaders(ses$h, accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7")
URL <- "https://finance.yahoo.com/"
r <- curl::curl_fetch_memory(URL, handle = ses$h)
#yahoo redirects to a consent form w/ a single cookie for GDPR:
#detecting the redirect seems very brittle as its sensitive to the trailing "/"
ses$can.crumb <- ((r$status_code == 200) && (URL == r$url) && (NROW(curl::handle_cookies(ses$h)) > 1))
assign(cache.name, ses, .quantmodEnv) #cache session
}

if (ses$can.crumb) {
#get a crumb so that downstream callers dont have to handle invalid sessions.
#this is a network hop, but very lightweight payload
n <- if (unclass(Sys.time()) %% 1L >= 0.5) 1L else 2L
query.srv <- paste0("https://query", n, ".finance.yahoo.com/v1/test/getcrumb")
r <- curl::curl_fetch_memory(query.srv, handle = ses$h)
if ((r$status_code == 200) && (length(r$content) > 0)) {
ses$crumb = rawToChar(r$content)
} else {
if (!force.new) ses <- .yahooSession(TRUE) else stop("Unable to get yahoo crumb")
Copy link
Owner

@joshuaulrich joshuaulrich Jun 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems backward. I expect to call .yahooSession(TRUE) when force.new is TRUE, else throw an error because we couldn't get a crumb.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its really a state machine. the force.new is never intended to be used directly, only in the recursive call. i can rewrite it as a loop to clean up the signature

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling the function recursively is fine. The logic on this line (51) confused me, but seems correct once I walk through it.

Assume force.new is FALSE. Line 45 checks that we have an active session. Line 51 checks if we're able to get a crumb. If we can't get a crumb, then !force.new is TRUE and we call .yahooSession(TRUE). So that seems like it does what it should.

I think some comments would help me remember what this is doing.

# we were unable to get a crumb
if (force.new) {
    # we couldn't get a crumb with a new session
    stop("unable to get yahoo crumb")
} else {
    # we tried to re-use a session but couldn't get a crumb
    # try to get a crumb using a new session
    ses <- .yahooSession(TRUE)
}

}
}

return(ses)
}

`getQuote.yahoo` <-
function(Symbols,what=standardQuote(),...) {
function(Symbols,what=standardQuote(),session=NULL,...) {
importDefaults("getQuote.yahoo")

length.of.symbols <- length(Symbols)
if (is.null(session)) session <- .yahooSession()

if(length.of.symbols > 200) {
# yahoo only works with 200 symbols or less per call
# we will recursively call getQuote.yahoo to handle each block of 200
Expand All @@ -39,7 +74,7 @@ function(Symbols,what=standardQuote(),...) {
for(i in 1:length(all.symbols)) {
Sys.sleep(0.5)
cat(i,", ")
df <- rbind(df, getQuote.yahoo(all.symbols[[i]],what))
df <- rbind(df, getQuote.yahoo(all.symbols[[i]],what,session=session))
}
cat("...done\n")
return(df)
Expand All @@ -59,11 +94,15 @@ function(Symbols,what=standardQuote(),...) {
# exchange, fullExchangeName, market, sourceInterval, exchangeTimezoneName,
# exchangeTimezoneShortName, gmtOffSetMilliseconds, tradeable, symbol
QFc <- paste0(QF,collapse=',')
URL <- paste0("https://query1.finance.yahoo.com/v7/finance/quote?symbols=",
SymbolsString,
"&fields=",QFc)
if (session$can.crumb) {
URL <- paste0("https://query1.finance.yahoo.com/v7/finance/quote?crumb=", session$crumb)
} else {
URL <- "https://query1.finance.yahoo.com/v6/finance/quote?"
}

URL <- paste0(URL, "&symbols=", SymbolsString, "&fields=", QFc)
# The 'response' data.frame has fields in columns and symbols in rows
response <- jsonlite::fromJSON(curl::curl(URL))
response <- jsonlite::fromJSON(curl::curl(URL, handle = session$h))
if (is.null(response$quoteResponse$error)) {
sq <- response$quoteResponse$result
} else {
Expand Down