Invalid Cookie when trying to get a quote

[pazer001]

Bug Report

Describe the bug

I'm getting an error: "Invalid Cookie" when trying to quote.

Minimal Reproduction

// symbol: ATAO
// symbol: AAMATX
// symbol: AAVVF
// Can be any
const symbolResult = await yahooFinance2.quote(symbol);

Environment

$ node -v
v20.0.0

Additional Context

[gadicc]

Hey all, thanks for all the reports! Looking into this now.

[gadicc]

Hey all, working on this... it's not as simple as I'd hoped, unfortunately. Everyone is welcome to see the progress in #634 and get involved if they'd like to or are able.

However, there is an easy alternative in the meantime. So far it seems only quote() is affected. quoteSummary() still works and provides most (but not all) of the same info, so may be able to help you in the meantime.

e.g.

Quote API (raw JSON from Yahoo that I had backed up)

          {
            "language": "en-US",
            "region": "US",
            "quoteType": "EQUITY",
            "quoteSourceName": "Nasdaq Real Time Price",
            "triggerable": true,
            "customPriceAlertConfidence": "HIGH",
            "currency": "USD",
            "postMarketChangePercent": -0.531021,
            "postMarketTime": 1645664396,
            "postMarketPrice": 159.22,
            "postMarketChange": -0.850006,
            "regularMarketChange": -4.25,
            "regularMarketChangePercent": -2.5864167,
            "regularMarketTime": 1645650002,
            "regularMarketPrice": 160.07,
            "regularMarketDayHigh": 166.15,
            "regularMarketDayRange": "159.75 - 166.15",
            "regularMarketDayLow": 159.75,
            "regularMarketVolume": 90009247,
            "regularMarketPreviousClose": 164.32,
            "bid": 0,
            "ask": 0,
            "bidSize": 9,
            "askSize": 10,
            "fullExchangeName": "NasdaqGS",
            "financialCurrency": "USD",
            "regularMarketOpen": 164.98,
            "averageDailyVolume3Month": 101720751,
            "averageDailyVolume10Day": 79056160,
            "fiftyTwoWeekLowChange": 43.86001,
            "fiftyTwoWeekLowChangePercent": 0.37742025,
            "fiftyTwoWeekRange": "116.21 - 182.94",
            "fiftyTwoWeekHighChange": -22.869995,
            "fiftyTwoWeekHighChangePercent": -0.12501363,
            "fiftyTwoWeekLow": 116.21,
            "fiftyTwoWeekHigh": 182.94,
            "dividendDate": 1644451200,
            "earningsTimestamp": 1643301000,
            "earningsTimestampStart": 1651003200,
            "earningsTimestampEnd": 1651521600,
            "trailingAnnualDividendRate": 0.865,
            "trailingPE": 26.611805,
            "trailingAnnualDividendYield": 0.005264119,
            "epsTrailingTwelveMonths": 6.015,
            "epsForward": 6.56,
            "epsCurrentYear": 6.16,
            "priceEpsCurrentYear": 25.985392,
            "sharesOutstanding": 16319399936,
            "bookValue": 4.402,
            "fiftyDayAverage": 172.2574,
            "fiftyDayAverageChange": -12.187393,
            "fiftyDayAverageChangePercent": -0.070751056,
            "twoHundredDayAverage": 151.27095,
            "twoHundredDayAverageChange": 8.799057,
            "twoHundredDayAverageChangePercent": 0.058167525,
            "marketCap": 2612246347776,
            "forwardPE": 24.400915,
            "priceToBook": 36.363018,
            "sourceInterval": 15,
            "exchangeDataDelayedBy": 0,
            "pageViewGrowthWeekly": -0.038373638,
            "averageAnalystRating": "1.8 - Buy",
            "tradeable": false,
            "firstTradeDateMilliseconds": 345479400000,
            "priceHint": 2,
            "marketState": "PREPRE",
            "exchange": "NMS",
            "shortName": "Apple Inc.",
            "longName": "Apple Inc.",
            "messageBoardId": "finmb_24937",
            "exchangeTimezoneName": "America/New_York",
            "exchangeTimezoneShortName": "EST",
            "gmtOffSetMilliseconds": -18000000,
            "market": "us_market",
            "esgPopulated": false,
            "displayName": "Apple",
            "symbol": "AAPL"
          }

quoteSummary("AAPL"):

{
  summaryDetail: {
    maxAge: 1,
    priceHint: 2,
    previousClose: 167.63,
    open: 166.09,
    dayLow: 165.91,
    dayHigh: 167.4,
    regularMarketPreviousClose: 167.63,
    regularMarketOpen: 166.09,
    regularMarketDayLow: 165.91,
    regularMarketDayHigh: 167.4,
    dividendRate: 0.92,
    dividendYield: 0.0056,
    exDividendDate: 2023-02-10T00:00:00.000Z,
    payoutRatio: 0.1545,
    fiveYearAvgDividendYield: 0.94,
    beta: 1.296885,
    trailingPE: 28.42347,
    forwardPE: 25.32273,
    volume: 12178317,
    regularMarketVolume: 12178317,
    averageVolume: 63909388,
    averageVolume10days: 49926240,
    averageDailyVolume10Day: 49926240,
    bid: 0,
    ask: 0,
    bidSize: 1000,
    askSize: 1800,
    marketCap: 2644314161152,
    fiftyTwoWeekLow: 124.17,
    fiftyTwoWeekHigh: 176.15,
    priceToSalesTrailing12Months: 6.823385,
    fiftyDayAverage: 156.0672,
    twoHundredDayAverage: 150.1536,
    trailingAnnualDividendRate: 0.91,
    trailingAnnualDividendYield: 0.0054286225,
    currency: 'USD',
    fromCurrency: null,
    toCurrency: null,
    lastMarket: null,
    coinMarketCapLink: null,
    algorithm: null,
    tradeable: false
  },
  price: {
    maxAge: 1,
    preMarketChangePercent: -0.00954487,
    preMarketChange: -1.60001,
    preMarketTime: 2023-04-20T13:29:59.000Z,
    preMarketPrice: 166.03,
    preMarketSource: 'FREE_REALTIME',
    regularMarketChangePercent: -0.0029827594,
    regularMarketChange: -0.5,
    regularMarketTime: 2023-04-20T14:32:19.000Z,
    priceHint: 2,
    regularMarketPrice: 167.13,
    regularMarketDayHigh: 167.4,
    regularMarketDayLow: 165.91,
    regularMarketVolume: 12178317,
    averageDailyVolume10Day: 49926240,
    averageDailyVolume3Month: 63909388,
    regularMarketPreviousClose: 167.63,
    regularMarketSource: 'FREE_REALTIME',
    regularMarketOpen: 166.09,
    exchange: 'NMS',
    exchangeName: 'NasdaqGS',
    exchangeDataDelayedBy: 0,
    marketState: 'REGULAR',
    quoteType: 'EQUITY',
    symbol: 'AAPL',
    underlyingSymbol: null,
    shortName: 'Apple Inc.',
    longName: 'Apple Inc.',
    currency: 'USD',
    quoteSourceName: 'Nasdaq Real Time Price',
    currencySymbol: '$',
    fromCurrency: null,
    toCurrency: null,
    lastMarket: null,
    marketCap: 2644314161152
  }
}

[gadicc]

May even be worth publishing an interim release that maps quoteSummary to quote for a "quick fix" until we can sort this out properly. Will look into that tomorrow, I think I'm out of time for today :/

[gadicc]

As reported over with our friends at ghostfolio/ghostfolio#1873, things have started working on their own again (i.e. with the existing release and no changes). As I said there, it's unclear what direction Yahoo is going with this so we'll have to watch out for it in the future, but at least a lot of the code to deal with it is already done now if we need it again :pray

[pazer001]

@gadicc so for now we should wait for an overall solution OR you will collect the data from different sources?

[gadicc]

@pazer001 everything is working 🎉 (with no changes on our side... Yahoo turned off whatever they turned on earlier).

But we'll have to keep an eye on it. When I was working on the code earlier, I noticed some inconsistencies in the data required to make it work, even in the browser... so, they might have just turned it off while they work out some bugs on their side, and then could turn it on again in the future. Unfortunately I can't develop / prepare for such an eventuality unless they turn it back on again... so let's just hope it stays off 🤞

[dtslvr]

I have started to notice Error: Invalid cookie again...

[gadicc]

oh dear 😔 I'm away for the weekend and will only be and to look at this next week 😕 but presumably if it's more stable now it will be easier to investigate further. my prior partial work is still available on the PR if anyone wants to take a stab before then.

…

On Thu, 4 May 2023, 20:46 Thomas Kaul, ***@***.***> wrote: I have started to notice Error: Invalid cookie again... — Reply to this email directly, view it on GitHub <#633 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC5IGTRR65IX65TVXQTQULXEPTN3ANCNFSM6AAAAAAXFAHOTU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[GiorgioBaz]

Yea, its a lot more prominent for me again. Would anyone happen to know of a way to send an array of tickers to quote summary in a similar way to how you can do it with quote? I don't necessarily want to use a for loop to go through them all cos it takes ages

[rodrigopeters]

I have notice Exiting with Error: Invalid Cookie only with quote. If I use quoteSummary it works. Unfortunately quoteSummary does not allow multiple symbols

$ yahoo-finance quote MULT3.SA
Exiting with Error: Invalid Cookie

[Nathan-Franck]

I have notice Exiting with Error: Invalid Cookie only with quote. If I use quoteSummary it works. Unfortunately quoteSummary does not allow multiple symbols

$ yahoo-finance quote MULT3.SA
Exiting with Error: Invalid Cookie

This worked for me! I'm just looking for price data, just spam the api with a message per-ticker and this works great!

[andi-bar157]

can cofirm, error is appearing again out of nothing!

[rodrigopeters]

still same problem:

$ yahoo-finance quote AAPL
Exiting with Error: Invalid Cookie

[GiorgioBaz]

Anyone else getting a invalid JSON response when spanning quotesummary with each ticker?

node_modules\node-fetch\lib\index.js:273

[0] return Body.Promise.reject(new FetchError('invalid json response body at ${_this2.url} reason: ${err.message}', 'invalid-json'));

[0] FetchError: invalid json response body at https://query2.finance.yahoo.com/v10/finance/quoteSummary/AAPL?formatted=false&modules=price reason: Unexpected token T in JSON at position 0`

[dtslvr]

quoteSummary() is working for me, but search() is also affected.

[austinyearlykim]

1. yahoo-finance2 @ v2.3.10
2. "Error: Invalid Cookie" when using .quote()
3. .quoteSummary() is working as an alternative.
4. @gadicc it's really cool how involved you are maintaining this. enjoy your weekend brother

[JosHonig]

Hi guys, many thanks from me too. I really apriciate the work.

Did you see this thread?
https://stackoverflow.com/questions/76059562/yahoo-finance-api-get-quotes-returns-invalid-cookie

I don't know if its a setting or hardcoded. But for me the switch to quoteSummary() is fine for the moment

[gadicc]

Hi all, thanks for all the updates and kind words.

Continuing on from my work from last time, this is now working again with the latest commit to cookies branch which includes the WIP PR #634.

All that's left:

• Use only when needed (just quote; anyone seen anything else? search seems to work fine without it)
• Expire crumb when cookie expires
• Adapt test code to remove crumb from URL to not break cache (and prob same with cookies)
• Tests for all the cookie / crumb stuff

I'm actually away on an overseas trip at the moment, but still going to try to get something out today, as my time for the rest of the week is more limited.

[gadicc]

Hi all, out of time for today, but I think we're in the good place. Updated the above message, only outstanding thing are unit tests for the new code. Of course I tested by hand that it does work :)

I usually don't like to publish new releases without tests on the new code, but I would like to get a working release out. In any event, if people can test the code from the cookies branch, that would be amazing, and with a bit of feedback I'll publish a new release even if I don't get a chance to do the tests.

Of note: only quote() uses the new cookie & crumb code. search() didn't require it in my tests. So please if anyone can give this a spin and report back on your findings, that would be great :)

[gadicc]

Hi again all :)

I know it's easier to test against published packages rather than checking out a development branch... and also wanted to have the published release working on as much as possible. So, all the above is now published in v2.4.0`.

Your feedback will be helpful to confirm that everything is working in a variety of cases. I haven't deleted the http cache of all prior tests as it's more than I have time to look into now, but basic testing in a variety of operations seems to all be working well. So let's see how we go!

[gadicc]

🎉 This issue has been resolved in version 2.4.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[gadicc]

Unfortunately Yahoo responses differ depending on origination IP. So the above fix only works in certain territories and further work will still be needed in EU.

[andi-bar157]

yep!

[Nest] 172 - 05/08/2023, 4:10:55 PM ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172 - 05/08/2023, 4:10:55 PM WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()
[Nest] 172 - 05/08/2023, 4:10:55 PM ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172 - 05/08/2023, 4:10:55 PM WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()
[Nest] 172 - 05/08/2023, 4:10:56 PM ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172 - 05/08/2023, 4:10:56 PM WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()
[Nest] 172 - 05/08/2023, 4:10:57 PM ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172 - 05/08/2023, 4:10:57 PM WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()
[Nest] 172 - 05/08/2023, 4:10:57 PM ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172 - 05/08/2023, 4:10:57 PM WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()
[Nest] 172 - 05/08/2023, 4:10:57 PM ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172 - 05/08/2023, 4:10:57 PM WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()

[gadicc]

Tracking in #637.

Feel like I'm one step away... just got to get all the cookie domains correct.
In second last step, all correct cookies are sent, but on the quote request, some are missing, and we get Invalid cookie.
Ironically for whatever reason, in the test specs, it all works. And maybe there's a clue there.

Unfortunately I'm out of time. I'm away overseas and not sure when I'll get a chance to work on this again, hence this update... in case anyone else wants to take a look in the meantime.

[gadicc]

🎉 This issue has been resolved in version 2.4.1 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[gadicc]

Working! Definitely lots of room for improvement but at least things are working again. All my tests were via a UK VPN. YMMV. If things aren't working for you, please report what country you're in 🙏

[puthre]

Just a heads up, using the previous Yahoo API version works just fine:
curl https://query1.finance.yahoo.com/v6/finance/quote?symbols=AAPL vs

curl https://query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL

[gadicc]

Hey, thanks, @puthre, I did see some others had gone this route.

I suppose this should have been the first thing to check, since our validation is quite comprehensive, and if it still passed, that would have been an easy fix. But also concerned that they might retire the older API so I wanted to just get a proper fix done and dusted, especially if they might start requiring the same process for APIs beyond just quote. Let's see what happens :)

If anyone else feels like implementing a quoteV6() module and see how the results compare and if all validation tests pass, that would be amazing actually, and I'd happily accept a PR for that. The big advantage is that the cookie and crumb stuff - especially from within the EU - involves a lot of extra requests the first time round (and currently, we don't have the ability to save that data for future requests if the node process is restarted).

[sourdzl]

would like to express our gratitude @gadicc and team for the quick resolution here!

[mrWh1te]

Thank you!

[Luminilion]

@gadicc You're doing an amazing work on this! Congrats!

[JosHonig]

I got a lot of Invalid Cookie messages the last days. I use QuoteSummary

[dtslvr]

I got a lot of Invalid Cookie messages the last days. I use QuoteSummary

I can confirm the error for quoteSummary() (ghostfolio/ghostfolio#2148).

[gadicc]

🎉 This issue has been resolved in version 2.4.2 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[gadicc]

Thanks @Luminilion, for the kind words, and @JosHonig and @dtslvr for the updated reports.

So, we only do all the extra cookie and crumb logic when we know we need it, as it requires something like 6 extra requests (in EU at least) before we can start doing the regular queries (once off, per process load; hope to add DB persistance for cookies when I get a chance soon, useful for e.g. serverless).

Up until now, this has been only the quote API, but looks like Yahoo have just rolled this out to quoteSummary too. Fortunately, the fix is trivial (we simply add { query: { needsCrumb: true } } to quoteSummary.ts's call to modeExec().

So, please continue to report if you notice this on any other APIs... and of course, PRs are welcome as usual... in this case it's a one line change :)

[dtslvr]

That was quick 🚀 Thank you very much for the new release @gadicc!

[nocodehummel]

Invalid cookie error also occurred on endpoints;

• calendarEvents,
• balanceSheetHistory,
• incomeStatementHistory,
• cashflowStatementHistory.

Fixed with upgrade from 2.4.1 to 2.4.3
Nice job!!