Yahoo - crumb missing about financials-extension

[mbacchi]

Same:

[szweep]

Looking at a yahoo-XXX.html file in the .financials-extension directory I see that the crumb is 13 characters long. I'm not sure if this varies or is a new format, but changing the regular expression that matches the crumb from 11 characters to 13 solved the issue for me:

diff --git a/src/financials_yahoo.py b/src/financials_yahoo.py
index 4159dec..1533751 100644
--- a/src/financials_yahoo.py
+++ b/src/financials_yahoo.py
@@ -184,7 +184,7 @@ class Yahoo(BaseClient):
                 return 'Yahoo.getRealtime({}, {}) - handleCookiesAndConsent'.format(ticker, datacode)
 
             try:
-                r = r'\bcrumb=([^"]{11})"'
+                r = r'\bcrumb=([^"]{13})"'
                 pattern = re.compile(r)
                 match = pattern.search(text)
                 if match:

[cmallwitz]

Do you have an example of a crumb with length 13? Maybe there are some "url encoded" characters in there... Thanks for the feedback! 27 Aug 2024 17:10:42 Steve Z ***@***.***>:

…

Looking at a yahoo-XXX.html file in the .financials-extension directory I see that the crumb is 13 characters long. I'm not sure if this varies or is a new format, but changing the regular expression that matches the crumb from 11 characters to 13 solved the issue for me: ` diff --git a/src/financials_yahoo.py b/src/financials_yahoo.py index 4159dec..1533751 100644 --- a/src/financials_yahoo.py +++ b/src/financials_yahoo.py @@ -184,7 +184,7 @@ class Yahoo(BaseClient): return 'Yahoo.getRealtime({}, {}) - handleCookiesAndConsent'.format(ticker, datacode) * try: * ** r = r'\bcrumb=([^"]{11})"' * ** r = r'\bcrumb=([^"]{13})"' pattern = re.compile(r) match = pattern.search(text) if match: * ` — Reply to this email directly, view it on GitHub[#109 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AA226TZODG7ZI73VSGXC5KLZTSQIBAVCNFSM6AAAAABJQOP2NCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJSHE4DGNZZGY]. You are receiving this because you are subscribed to this thread. [Tracking image][https://github.com/notifications/beacon/AA226T4FKDWS4SKE576OWMLZTSQIBA5CNFSM6AAAAABJQOP2NCWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUJ3VKPI.gif]

[szweep]

This is what I see:

data-url="https://query1.finance.yahoo.com/v7/finance/quote?fields=fiftyTwoWeekHigh%2CfiftyTwoWeekLow%2CfromCurrency%2CfromExchange%2CheadSymbolAsString%2ClogoUrl%2ClongName%2CmarketCap%2CmessageBoardId%2CoptionsType%2CregularMarketTime%2CregularMarketChange%2CregularMarketChangePercent%2CregularMarketOpen%2CregularMarketPrice%2CregularMarketSource%2CregularMarketVolume%2CpostMarketTime%2CpostMarketPrice%2CpostMarketChange%2CpostMarketChangePercent%2CpreMarketTime%2CpreMarketPrice%2CpreMarketChange%2CpreMarketChangePercent%2CshortName%2CtoCurrency%2CtoExchange%2CunderlyingExchangeSymbol%2CunderlyingSymbol&formatted=true&imgHeights=50&imgLabels=logoUrl&imgWidths=50&symbols=IBM&lang=en-US&region=US&crumb=yrtV8%2FbGiDQ"

So the "%2F" looks like a urlencoded character, but I'm unsure whether this is decoded before the pattern is matched. All I know is that I was seeing the "no crumb" error before the change and it worked afterwards. :)

[szweep]

OK, with further testing I see examples with no urlencoded characters and a length of 11, so my "fix" is no longer working

[szweep]

Something like this allows the matched string to be longer but ensures the crumb is 11 characters after it is decoded:

diff --git a/src/financials_yahoo.py b/src/financials_yahoo.py
index 1533751..502f5f2 100644
--- a/src/financials_yahoo.py
+++ b/src/financials_yahoo.py
@@ -184,11 +184,13 @@ class Yahoo(BaseClient):
                 return 'Yahoo.getRealtime({}, {}) - handleCookiesAndConsent'.format(ticker, datacode)
 
             try:
-                r = r'\bcrumb=([^"]{11})"'
+                r = r'\bcrumb=([^"]{11,17})"'
                 pattern = re.compile(r)
                 match = pattern.search(text)
                 if match:
-                    self.crumb = urllib.parse.unquote(match.group(1))
+                    crumb = urllib.parse.unquote(match.group(1))
+                    if len(crumb) == 11:
+                        self.crumb = crumb
             except BaseException as e:
                 logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode)
                 del self.realtime[ticker]

[cmallwitz]

Let me see if I can apply that to the repo... Thanks! 27 Aug 2024 18:22:46 Steve Z ***@***.***>:

…

Something like this allows the matched string to be longer but ensures the crumb is 11 characters after it is decoded: *diff --git a/src/financials_yahoo.py b/src/financials_yahoo.py index 1533751..502f5f2 100644 --- a/src/financials_yahoo.py +++ b/src/financials_yahoo.py @@ -184,11 +184,13 @@ class Yahoo(BaseClient): return 'Yahoo.getRealtime({}, {}) - handleCookiesAndConsent'.format(ticker, datacode) try: - r = r'\bcrumb=([^"]{11})"' + r = r'\bcrumb=([^"]{11,17})"' pattern = re.compile(r) match = pattern.search(text) if match: - self.crumb = urllib.parse.unquote(match.group(1)) + crumb = urllib.parse.unquote(match.group(1)) + if len(crumb) == 11: + self.crumb = crumb except BaseException as e: logger.exception("BaseException ticker=%s datacode=%s", ticker, datacode) del self.realtime[ticker] * — Reply to this email directly, view it on GitHub[#109 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AA226T4WPMIKREY5X7K575LZTSYWJAVCNFSM6AAAAABJQOP2NCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJTGEZDOMBRG4]. You are receiving this because you commented. [Tracking image][https://github.com/notifications/beacon/AA226T437NSNZEX3G7NM67LZTSYWJA5CNFSM6AAAAABJQOP2NCWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUJ36CGS.gif]

[cmallwitz]

After some feedback from @szweep I have made some more changes to crumb handling. I'm pretty confident this is is improving things, so if anybody wants to have a go try the new release 3.6.2