New search chain that doesnt use serpapi about langchain

[ilyazub]

too few free trials, too expensive

Thanks for your work, @hwchase17! Let me know if you need SerpApi search credits for research and development.

[hwchase17]

thanks for offering @ilyazub ! very generous. big fan of serpapi so i'm personally fine paying atm (not doing a ton of research)

closing this issue as we have both a google search and bing alternative

[hwchase17]

https://stackoverflow.com/questions/37083058/programmatically-searching-google-in-python-using-custom-search

[conceptofmind]

@hwchase17 From my initial testing setting up a custom google search API through GCP produces nearly identical scraped results when compared with serpapi. For example, here is a comparison of the scraped text produced: https://www.diffchecker.com/L4oCHMUA

A big benefit is the Custom Search API allows for 10000 requests per day compared to the only 100 a month offered by serpapi.

I will work on comparing the Custom Bing Search API offered by Microsoft with serpapi (Bing) next.

I think the browserless scraping with requests is also a limitation as the service used in the WebGPT example allows for only 1000 instances a month. Will have to do more research into this as well. I think Selenium Webdriver may be a decent alternative.

Thank you for all of your work!