langchain-ai / web-explorer

Web Explorer

This is a lightweight app using the Web Research Retriever.

Setup

You only need to supply a few things.

In settings() function, supply:

• Search: Select the search tool you want to use (e.g., GoogleSearchAPIWrapper).
• Vectorstore: Select the vectorstore and embeddings you want to use (e.g., Chroma, OpenAIEmbeddings).
• Select the LLM you want to use (e.g., ChatOpenAI).

To use st.secrets set enviorment variables in .streamlit/secrets.toml file.

Or, simply add environemnt variables and remove st.secrets:

import os
os.environ["GOOGLE_API_KEY"] = "YOUR_API_KEY"
os.environ["GOOGLE_CSE_ID"] = "YOUR_CSE_ID" 
os.environ["OPENAI_API_BASE"] = "https://api.openai.com/v1"
os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"

For GOOGLE_API_KEY , you could get it from this link.

For GOOGLE_CSE_ID , you could get it from this link

Run

streamlit run web_explorer.py

Example output:

web-explorer's Issues

Streamlit app down

Streamlit app seems to be down - https://web-explorer.streamlit.app/

Extremely slow

I got it working, but it takes ages for it to query results from Google API. Is it necessary to use it? I had to create a custom search engine and create an API key for it, but can I just use the OpenAI API key and not use Google CSE?

AssertionError

Hello All,

I have ran the app successfully but I got this AssertionError Message, Can you help us on this please?

Error Message;
WARNING:langchain.callbacks.manager:Error in PrintRetrievalHandler.on_retriever_start callback: PrintRetrievalHandler.on_retriever_start() takes 2 positional arguments but 3 were given INFO:langchain.retrievers.web_research:Generating questions for Google Search ... INFO:langchain.retrievers.web_research:Questions for Google Search (raw): {'question': 'Central Park', 'text': LineList(lines=['1. What are the top attractions to visit in Central Park?\n', '2. How big is Central Park and what are its boundaries?\n'])} INFO:langchain.retrievers.web_research:Questions for Google Search: ['1. What are the top attractions to visit in Central Park?\n', '2. How big is Central Park and what are its boundaries?\n'] INFO:langchain.retrievers.web_research:Searching for relevat urls ... INFO:langchain.retrievers.web_research:Searching for relevat urls ... INFO:langchain.retrievers.web_research:Search results: [{'title': "Visiting New York's Central Park: 14 Top Attractions | PlanetWare", 'link': 'https://www.planetware.com/new-york-city/new-york-central-park-us-ny-central.htm', 'snippet': 'Apr 14, 2023 ... 1. Belvedere Castle · 2. Central Park Zoo · 3. Strawberry Fields · 4. Bethesda Fountain and Terrace · 5. Conservatory Garden · 6. Loeb Boathouse on\xa0...'}, {'title': 'Top 10 Attractions In Central Park - Central Park', 'link': 'https://centralpark.org/top-10-places-in-central-park/', 'snippet': '10. Conservatory Water · 9. Wollman Rink · 8. Imagine Mosaic – Strawberry Fields · 7. Conservatory Garden · 6. Central Park Reservoir · 5. Bow Bridge · 4. Bethesda\xa0...'}, {'title': 'What are the best places to see and visit in Central Park?', 'link': 'https://www.centralparktours.net/blog/what-are-the-best-places-in-central-park/', 'snippet': 'Apr 17, 2020 ... Top spots in Central Park that are worthy visiting · The Great Lawn · Conservatory Garden · Belvedere Castle · Carousel · Alice in Wonderland.'}] INFO:langchain.retrievers.web_research:Searching for relevat urls ... INFO:langchain.retrievers.web_research:Search results: [{'title': 'Central Park - Wikipedia', 'link': 'https://en.wikipedia.org/wiki/Central_Park', 'snippet': "Its size and cultural position make it a model for the world's urban parks. Its influence earned Central Park the designations of National Historic Landmark in\xa0..."}, {'title': 'Transform Plate Boundaries - Geology (U.S. National Park Service)', 'link': 'https://www.nps.gov/subjects/geology/plate-tectonics-transform-plate-boundaries.htm', 'snippet': 'Feb 11, 2020 ... A transform plate boundary developed where the Pacific Plate was in contact with the North American Plate and the volcanism ceased in central\xa0...'}, {'title': 'Harlem - Wikipedia', 'link': 'https://en.wikipedia.org/wiki/Harlem', 'snippet': '... boundaries, regarding only central Harlem as part of Harlem proper. ... This section is bounded by Fifth Avenue on the east; Central Park on the south;\xa0...'}] INFO:langchain.retrievers.web_research:New URLs to load: ['https://centralpark.org/top-10-places-in-central-park/', 'https://www.centralparktours.net/blog/what-are-the-best-places-in-central-park/', 'https://en.wikipedia.org/wiki/Central_Park', 'https://www.nps.gov/subjects/geology/plate-tectonics-transform-plate-boundaries.htm', 'https://www.planetware.com/new-york-city/new-york-central-park-us-ny-central.htm', 'https://en.wikipedia.org/wiki/Harlem'] INFO:langchain.retrievers.web_research:Indexing new urls... Fetching pages: 100%|################################################################################################################################| 6/6 [00:06<00:00, 1.10s/it] 2023-10-22 17:26:45.320 Uncaught app exception Traceback (most recent call last): File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 541, in _run_script exec(code, module.__dict__) File "/Users/mostafa/Downloads/web-LLM-app/web-explorer/web_explorer.py", line 103, in <module> result = qa_chain({"question": question},callbacks=[retrieval_streamer_cb, stream_handler]) File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/langchain/chains/base.py", line 243, in __call__ raise e File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/langchain/chains/base.py", line 237, in __call__ self._call(inputs, run_manager=run_manager) File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/langchain/chains/qa_with_sources/base.py", line 141, in _call docs = self._get_docs(inputs, run_manager=_run_manager) File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/langchain/chains/qa_with_sources/retrieval.py", line 51, in _get_docs docs = self.retriever.get_relevant_documents( File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/langchain/schema/retriever.py", line 181, in get_relevant_documents raise e File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/langchain/schema/retriever.py", line 174, in get_relevant_documents result = self._get_relevant_documents( File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/langchain/retrievers/web_research.py", line 205, in _get_relevant_documents self.vectorstore.add_documents(docs) File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/langchain/vectorstores/base.py", line 104, in add_documents return self.add_texts(texts, metadatas, **kwargs) File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/langchain/vectorstores/faiss.py", line 153, in add_texts return self.__add(texts, embeddings, metadatas=metadatas, ids=ids, **kwargs) File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/langchain/vectorstores/faiss.py", line 120, in __add self.index.add(vector) File "/Users/mostafa/Downloads/web-LLM-app/venv-web/lib/python3.10/site-packages/faiss/class_wrappers.py", line 228, in replacement_add assert d == self.d AssertionError

cannot import name 'RetrievalQA' from partially initialized module 'langchain.chains' (most likely due to a circular import)

Hello

I'm getting the following error. I'm using Python 3.10.0.

Output

2023-08-02 10:22:05.807 Uncaught app exception
Traceback (most recent call last):
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script
    exec(code, module.__dict__)
  File "/home/bob/bit/myproject/web-explorer/web_explorer.py", line 2, in <module>
    from langchain.callbacks.base import BaseCallbackHandler
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/__init__.py", line 6, in <module>
    from langchain.agents import MRKLChain, ReActChain, SelfAskWithSearchChain
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/agents/__init__.py", line 2, in <module>
    from langchain.agents.agent import (
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 15, in <module>
    from langchain.agents.agent_iterator import AgentExecutorIterator
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/agents/agent_iterator.py", line 30, in <module>
    from langchain.tools import BaseTool
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/tools/__init__.py", line 60, in <module>
    from langchain.tools.powerbi.tool import (
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/tools/powerbi/tool.py", line 12, in <module>
    from langchain.chains.llm import LLMChain
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/chains/llm.py", line 18, in <module>
    from langchain.prompts.prompt import PromptTemplate
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/prompts/__init__.py", line 20, in <module>
    from langchain.prompts.loading import load_prompt
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/prompts/loading.py", line 10, in <module>
    from langchain.output_parsers.regex import RegexParser
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/output_parsers/__init__.py", line 5, in <module>
    from langchain.output_parsers.fix import OutputFixingParser
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/output_parsers/fix.py", line 5, in <module>
    from langchain.chains.llm import LLMChain
ImportError: cannot import name 'LLMChain' from partially initialized module 'langchain.chains.llm' (most likely due to a circular import) (/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/chains/llm.py)
2023-08-02 10:22:06.179 Uncaught app exception
Traceback (most recent call last):
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script
    exec(code, module.__dict__)
  File "/home/bob/bit/myproject/web-explorer/web_explorer.py", line 3, in <module>
    from langchain.chains import RetrievalQAWithSourcesChain
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/chains/__init__.py", line 16, in <module>
    from langchain.chains.api.openapi.chain import OpenAPIEndpointChain
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/__init__.py", line 6, in <module>
    from langchain.agents import MRKLChain, ReActChain, SelfAskWithSearchChain
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/agents/__init__.py", line 2, in <module>
    from langchain.agents.agent import (
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 15, in <module>
    from langchain.agents.agent_iterator import AgentExecutorIterator
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/agents/agent_iterator.py", line 30, in <module>
    from langchain.tools import BaseTool
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/tools/__init__.py", line 94, in <module>
    from langchain.tools.vectorstore.tool import (
  File "/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/tools/vectorstore/tool.py", line 12, in <module>
    from langchain.chains import RetrievalQA, RetrievalQAWithSourcesChain
ImportError: cannot import name 'RetrievalQA' from partially initialized module 'langchain.chains' (most likely due to a circular import) (/home/bob/bit/myproject/web-explorer/.venv/lib/python3.10/site-packages/langchain/chains/__init__.py)

Cant make it work when using other Search function.

Cant get rid of the GOOGLE_API_ID and GOOGLE_CSE_ID even though I am using GoogleSerperAPIWrapper(). I added random values for these both and still I am not able to make it work.

Error:

Traceback (most recent call last):
  File "c:\python\web_explorer_sample.py", line 25, in <module>
    web_retriever = WebResearchRetriever.from_llm(
  File "C:\python\env\lib\site-packages\langchain\retrievers\web_research.py", line 128, in from_llm
    return cls(
  File "C:\python\env\lib\site-packages\langchain\load\serializable.py", line 90, in __init__
    super().__init__(**kwargs)
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 7 validation errors for WebResearchRetriever
search -> aiosession
  extra fields not permitted (type=value_error.extra)
search -> gl
  extra fields not permitted (type=value_error.extra)
search -> hl
  extra fields not permitted (type=value_error.extra)
search -> result_key_for_type
  extra fields not permitted (type=value_error.extra)
search -> serper_api_key
  extra fields not permitted (type=value_error.extra)
search -> tbs
  extra fields not permitted (type=value_error.extra)
search -> type
  extra fields not permitted (type=value_error.extra)

SSLEOFError: EOF occurred in violation of protocol

Sometimes encounter traceback error but running again sometimes resolves:

SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2423)
Traceback:
File "C:\python311\Lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 552, in _run_script
exec(code, module.dict)
File "C:\Users\jeremy.arbid\python_playground\web-explorer\web_explorer.py", line 85, in
result = qa_chain({"question": question},callbacks=[retrieval_streamer_cb, stream_handler])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\langchain\chains\base.py", line 243, in call
raise e
File "C:\python311\Lib\site-packages\langchain\chains\base.py", line 237, in call
self._call(inputs, run_manager=run_manager)
File "C:\python311\Lib\site-packages\langchain\chains\qa_with_sources\base.py", line 141, in _call
docs = self._get_docs(inputs, run_manager=_run_manager)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\langchain\chains\qa_with_sources\retrieval.py", line 51, in _get_docs
docs = self.retriever.get_relevant_documents(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\langchain\schema\retriever.py", line 181, in get_relevant_documents
raise e
File "C:\python311\Lib\site-packages\langchain\schema\retriever.py", line 174, in get_relevant_documents
result = self._get_relevant_documents(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\langchain\retrievers\web_research.py", line 184, in _get_relevant_documents
search_results = self.search_tool(query, self.num_search_results)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\langchain\retrievers\web_research.py", line 154, in search_tool
result = self.search.results(query_clean, num_search_results)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\langchain\utilities\google_search.py", line 123, in results
results = self._google_search_results(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\langchain\utilities\google_search.py", line 63, in _google_search_results
res = cse.list(q=search_term, cx=self.google_cse_id, **kwargs).execute()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\googleapiclient_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\googleapiclient\http.py", line 923, in execute
resp, content = _retry_request(
^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\googleapiclient\http.py", line 222, in _retry_request
raise exception
File "C:\python311\Lib\site-packages\googleapiclient\http.py", line 191, in retry_request
resp, content = http.request(uri, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\httplib2_init.py", line 1724, in request
(response, content) = self.request(
^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\httplib2_init.py", line 1444, in _request
(response, content) = self.conn_request(conn, request_uri, method, body, headers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\site-packages\httplib2_init.py", line 1367, in _conn_request
conn.request(method, request_uri, body, headers)
File "C:\python311\Lib\http\client.py", line 1286, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\python311\Lib\http\client.py", line 1332, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\python311\Lib\http\client.py", line 1281, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\python311\Lib\http\client.py", line 1041, in _send_output
self.send(msg)
File "C:\python311\Lib\http\client.py", line 1002, in send
self.sock.sendall(data)
File "C:\python311\Lib\ssl.py", line 1241, in sendall
v = self.send(byte_view[count:])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python311\Lib\ssl.py", line 1210, in send
return self._sslobj.write(data)
^^^^^^^^^^^^^^^^^^^^^^^^

error reading API key

i tried running this streamlit app but I keep running into a validation error related to finding my openAI API key:

ValidationError: 1 validation error for OpenAIEmbeddings root Did not find openai_api_key, please add an environment variable OPENAI_API_KEY which contains it, or pass openai_api_key as a named parameter. (type=value_error)

i'm hoping this is just user error, and someone may be able to help me out? thank you

langchain_core.exceptions.OutputParserException when running the steamilt app

When trying out the web_explorer, I get the error below. I have the relevant keys. Did anyone face the same issue?

Traceback (most recent call last):
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/pydantic/v1/main.py", line 522, in parse_obj
obj = dict(obj)
TypeError: 'int' object is not iterable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/output_parsers/pydantic.py", line 25, in parse_result
return self.pydantic_object.parse_obj(json_object)
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/pydantic/v1/main.py", line 525, in parse_obj
raise ValidationError([ErrorWrapper(exc, loc=ROOT_KEY)], cls) from e
pydantic.v1.error_wrappers.ValidationError: 1 validation error for LineList
root
LineList expected dict not int (type=type_error)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/mn/PycharmProjects/web_explorer/test.py", line 126, in
result = qa_chain.invoke({"question": user_input})
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/chains/base.py", line 162, in invoke
raise e
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/chains/base.py", line 156, in invoke
self._call(inputs, run_manager=run_manager)
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/chains/qa_with_sources/base.py", line 152, in _call
docs = self._get_docs(inputs, run_manager=_run_manager)
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/chains/qa_with_sources/retrieval.py", line 49, in _get_docs
docs = self.retriever.get_relevant_documents(
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain_core/retrievers.py", line 321, in get_relevant_documents
raise e
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain_core/retrievers.py", line 314, in get_relevant_documents
result = self._get_relevant_documents(
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/retrievers/web_research.py", line 177, in _get_relevant_documents
result = self.llm_chain({"question": query})
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain_core/_api/deprecation.py", line 145, in warning_emitting_wrapper
return wrapped(*args, **kwargs)
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/chains/base.py", line 363, in call
return self.invoke(
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/chains/base.py", line 162, in invoke
raise e
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/chains/base.py", line 156, in invoke
self._call(inputs, run_manager=run_manager)
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/chains/llm.py", line 104, in _call
return self.create_outputs(response)[0]
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/chains/llm.py", line 258, in create_outputs
result = [
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/chains/llm.py", line 261, in
self.output_key: self.output_parser.parse_result(generation),
File "/Users/mn/PycharmProjects/web_explorer/venv/lib/python3.9/site-packages/langchain/output_parsers/pydantic.py", line 29, in parse_result
raise OutputParserException(msg, llm_output=json_object)
langchain_core.exceptions.OutputParserException: Failed to parse LineList from completion 1. Got: 1 validation error for LineList
root
LineList expected dict not int (type=type_error)