vida-nyu / domain_discovery_api Goto Github PK

It seems that pages without a title cannot be 'clicked on'. See geotrellis.io in screenshot

The method for selecting images for the snippet seems to have a problem.
See the screenshot below -- there are two google scholar entries for different people but with
the same picture.

Maybe, we could change the message in the bottom of the page to see "View Crawler Results", and clicking on this, would show "Crawled Data"

I did a query for R-Tree and many "tree removal" and "landscaping" services showed up. It would be really helpful to see the irrelevant and relevant terms (e.g., index, efficiency, spatial).

I created a new domain and started using the Load command. I entered
"https://www.cs.purdue.edu/people/faculty/aref/" and clicked the search button.
Nothing happened.

But if when I went to Filters and unselected "Queries", the URL was displayed in the results.

I used the newest version to test a query that did not work before:
hanan samet

Having the message "searching" is great -- it is now clear the system is working.
But this particular query took too long, so I modified it to
hanan

Then DDT kept swapping from hanan to "hanan samet". After a while it stopped 'blinking' and said "Query failed. Try Google"

At this point, I wanted to check the results for the hanan query. But when I clicked on the X to remove the query "hanan samet", the interface shows the search results, and then immediately goes back to the empty screen.

See screenshot

I queried for my name on Bing. The Filter/Queries/Juliana Freire facet says there are "62" results.

If I (local) search for juliana freire, it says 63 results.

We need a better indicator that the crawler is working and making progress. What about showing next to "Crawler is Running" the number of pages retrieved?

The labels assigned by the crawler need to be visible in DDT.
Can they be added as "Model Tags"?

Search queries sometimes do not work. I tried "hanan samet". It did not work.
I restarted the server (did control-C and ran the script again), and it still did not work.

Is this an issue parsing multi-term queries?

See a piece of the log below

MULTIFIELD TERM SEARCH
{'filter': {'or': [{'term': {'query': u'"hanan samet"'}}]}, 'query': {'bool': {'minimum_number_should_match': 0, 'should': []}}, 'fields': ['url', 'description', 'image_url', 'title', 'rank', 'x', 'y', 'tag', 'retrieved']}

172.17.0.1 - - [31/May/2017:04:46:05] "POST /getPages HTTP/1.1" 200 37 "http://localhost:8084/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

{'filter': {'or': [{'term': {'query': u'"hanan samet"'}}]}}

MULTIFIELD TERM SEARCH
{'filter': {'or': [{'term': {'query': u'"hanan samet"'}}]}, 'query': {'bool': {'minimum_number_should_match': 0, 'should': []}}, 'fields': ['url', 'description', 'image_url', 'title', 'rank', 'x', 'y', 'tag', 'retrieved']}