Yahoo changed its URL and now requires crumb parameter about yahoofinanceapi

[karlwancl]

@mayerwin , thanks for your notification! It may take some time to study and implement the changes, it will be committed once i have completed.

[mayerwin]

Thanks for the fast commit, it works for a first request (e.g. await Yahoo.GetHistoricalAsync(ticker, data.MinDt.ToDateTime(), maxDt.ToDateTime(), YahooFinanceApi.Period.Daily)), however the second (for a different ticker) fails with throwing an exception: "Value cannot be null.\r\nParameter name: Crumb is empty!"

at YahooFinanceApi.Yahoo.<GetResponseStreamAsync>d__13.MoveNext()

[karlwancl]

Oops, it seems yahoo returns empty crumb if there's already a session with the client rather than creating a new session.

Maybe a cache of the cookie/ HTTP client is needed for the case, but it may create problem when using multiple thread. Anyways, thanks for the test. I need more time to dig deep to the problem.

[karlwancl]

Hi @mayerwin , I have added cache for the web client to prevent empty crumb when doing second or later requests, locking mechanism is also added to ensure thread-safety.

Please help to test if the committed code is working for 2nd times onwards and in multithread environment, thanks for your help!

[mayerwin]

Hi, thanks, can you update the package on nuget?

[karlwancl]

Updated

[mayerwin]

ok, so I am not getting 502 errors anymore, however I tried multi-threading and got some exceptions:

System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.ValueCollection.Enumerator.MoveNext()
   at Flurl.Http.FlurlClient.WriteRequestCookies(HttpRequestMessage request) in C:\projects\flurl\src\Flurl.Http.Shared\FlurlClient.cs:line 220
   at Flurl.Http.FlurlClient.<SendAsync>d__28.MoveNext() in C:\projects\flurl\src\Flurl.Http.Shared\FlurlClient.cs:line 199
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Flurl.Http.HttpResponseMessageExtensions.<ReceiveStream>d__4.MoveNext() in C:\projects\flurl\src\Flurl.Http.Shared\HttpResponseMessageExtensions.cs:line 77
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at YahooFinanceApi.Yahoo.<GetResponseStreamAsync>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at YahooFinanceApi.Yahoo.<GetHistoricalAsync>d__14.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()

Here is the test code used:

        Enumerable.Range(0, 100)
                              .AsParallel()
                              .WithDegreeOfParallelism(16)
                              .ForAll(i => {
                                  try {
                                      Yahoo.GetHistoricalAsync(ticker, minDtm, maxDtm, YahooFinanceApi.Period.Daily).Wait();
                                  }
                                  catch (Exception ex) {
                                      Logger.Trace($"Failed: " + ex);
                                  }
                                  Logger.Trace($"Done: " + i);
                              });

[karlwancl]

Thanks for the test! I need some time to figure out the multithread issue. For the time being, please use the api in single-thread environment, thanks!

[karlwancl]

It seems the cached client changes some of its internal state when calling the web url, lock is used for each web call for temporary fix, may improve the implementation when time is available.

[mayerwin]

I am getting the following error since yesterday: "Request to https://finance.yahoo.com/d/quotes.csv?s=MCD&f=ohgl1v failed with status code 301 (Redirect)."

It is redirected to: 301: Permanent redirect to http://download.finance.yahoo.com/d/quotes.csv?s=MCD&f=ohgl1v (and works in the browser)

Can you update the domain (the fix seems simple)?

[karlwancl]

The redirect exception only happens when the project is targeting the old .NET framework, I'm not sure if it is a problem from the package Flurl or httpclient. Anyways, the fix is commited & nuget package is updated, please update your version accordingly, thanks

[mayerwin]

Done, it works again, thanks.