techthiyanes / parlai_searchengine

A search engine for ParlAI's BlenderBot project (and probably other ones as well)

Description

A web search server for ParlAI, including Blenderbot2.

Querying the server:

The server reacting correctly:

• Uses html2text to strip the markup out of the page.
• Uses beautifulsoup4 to parse the title.
• Currently only uses the googlesearch module to query Google for urls, but is coded in a modular / search engine agnostic way to allow very easily add new search engine support.

Using the googlesearch module is very slow because it parses Google search webpages instead of querying cloud webservices. This is fine for playing with the model, but makes that searcher unusable for training or large scale inference purposes. In the paper, Bing cloud services are used, matching the results over Common Crawl instead of just downloading the page.

Quick Start:

First install the requirements:

pip install -r requirements.txt

Run this command in one terminal tab:

python search_server.py serve --host 0.0.0.0:8080

[Optional] You can then test the server with

curl -X POST "http://0.0.0.0:8080" -d "q=baseball&n=1"

Then for example start Blenderbot2 in a different terminal tab:

python -m parlai interactive --model-file zoo:blenderbot2/blenderbot2_3B/model --search_server 0.0.0.0:8080

Colab

There is a jupyter notebook. Just run it. Some instances run out of memory, some don't.

Other Ways to Test the Server:

This method creates a retrieval client class instance the same way the ParlAI code would, and tries to retrieve from the server. If you have a server running, you can use this to test the server without having to load the (very large) dialog model. This will create a parlai.agents.rag.retrieve_api.SearchEngineRetriever and try to connect and send a query, and parse the answer.

python search_server.py serve --host 0.0.0.0:8080

then in a different tab

python search_server.py test_server --host 0.0.0.0:8080

Testing the parser:

python search_server.py test_parser www.some_url_of_your_choice.com/

---

This work is licensed under a Creative Commons Attribution 4.0 International License.