searx / searx

Privacy-respecting metasearch engine

Searx is no longer maintained. Thank you for your support and all your contributions.

---

Privacy-respecting, hackable metasearch engine / pronunciation sɜːks.

If you are looking for running instances, ready to use, then visit searx.space.

Otherwise jump to the user, admin and developer handbooks you will find on our homepage.

contact:

openhub // twitter // IRC: #searx @ Libera (irc.libera.chat)

Frequently asked questions

Is searx in maintenance mode?

No, searx is no longer maintained.

What is the difference between searx and SearxNG?

TL;DR: SearXNG is for users that want more features and bugs getting fixed quicker. If you prefer a minimalist software and stable experience, use searx.

SearxNG is a fork of searx, created by a former maintainer of searx. The fork was created because the majority of the maintainers at the time did not find the new proposed features privacy respecting enough. The most significant issue is with engine metrics.

Searx is built for privacy conscious users. It comes with a unique set of challenges. One of the problems we face is that users rather not report bugs, because they do not want to publicly share what engines they use or what search query triggered a problem. It is a challenge we accepted.

The new metrics feature collects more information to make engine maintenance easier. We could have had better and more error reports to benefit searx maintainers. However, we believe that the users of searx must come first, not the software. We are willing to compromise on the lack of issue reports to avoid violating the privacy of users.

Furthermore, SearxNG is under heavy refactoring and dependencies are constantly updated, even if it is unnecessary. It increases the risk of introducing regressions. In searx we strive for stability, rather than moving fast and breaking things.

Is searx for me?

Are you privacy conscious user? Then yes.

In searx we decided to double down on being privacy respecting. We are picking engine changes from SearxNG, but we are not implementing engine detailed monitoring and not adding a new UI that relies on Javascript.

If you are willing to give up some privacy respecting features, we encourage you to adopt SearxNG. Searx is targeted for privacy conscious users who run their instances locally, instead of using public instances.

Why should I use SearxNG?

SearxNG has rolling releases, dependencies updated more frequently, and engines are fixed faster. It is easy to set up your own public instance, and monitor its performance and metrics. It is simple to maintain as an instance administrator.

As a user, it provides a prettier user interface and nicer experience.

searx's Issues

stick results from same topic a little bit more together

I think the order of the results it is a little confusing, if you search for more topics like general and images.

Sticking a few results together would result in clearer look (there are 5 general results, then 10 images, then again a bunch of general results,...), which is also easier to read and more logical to the user.

Communications channel?

@asciimoo what kind of communications channels do you use for development of searx.. should we create IRC channel #searx on freenode? Sorry for opening the issue for this but I think its better than email, at least people will know.

default theme option doesn't work

If I set another default_theme in settings.yml, it doesn't change (cookies are cleared and searx were restarted)

Python 3 compatibility

Python 3 does not appear to be supported and I can't find any info concerning this. Is this on the roadmap?

cannot use python-dateutil

from dateutil import parser

if I want use dateutil, the following error occour:

ImportError: No module named dateutil

I require this package to improve publishDate extraction from the different sources. Using this package in the python shell work without problems.

thx

add embedded player to show videos directly inside searx

It would be nice to show videos directly inside searx. Probably configurable in the preferences.

To be privacy-safe and make a continuous look and feel for that, I would prefer an open source player like video.js, which also allows plugins for showing videos from youtube, vimeo,...

(probably, I would do the implementation of that functionality)

dbpedia.org

It's more an idea than an RFE.
Looking at http://dbpedia.org/page/Python or http://dbpedia.org/page/Berlin (there is json output format), it would be nice to show instant answer, or to suggest related queries.

I can't find a real search engine ( http://dbpedia.org/fct/ or http://wiki.dbpedia.org/lookup/ doesn't seem to support fuzzy). However http://dpedia.org is based on wikipedia, so the URL path are the same. May be the wikipedia engine could be linked to dbpedia ?

improve usage of duckduckgo_definitions engine

Using
https://github.com/crazedpsyc/python-duckduckgo/blob/master/duckduckgo.py

• Image gives an URL that could be added to the definition
• RelatedTopics could use to add suggestions (like the yahoo engine)
• Entity define location, musician, company, actor, politician, television, media franchise etc... could be usefull to suggest a category or at least show that information.
• Infobox gives some information like google does on the right side of the results

[doc] How to optimize searx performances?

It may be interesting to add some doc on what to do to boost performances of searx.

I followed your install doc (with Debian/Nginx) and my search queries take 6 seconds (during which the Web page doesn't show anything). Well, I've seen other instances of searx which answers nearly instantly then I certainly miss something in the configuration but I don't find it in the doc.

autocomplete : remove bang from search string

When a search for "!ddd linux", the suggested search string contains "ddd linux"

searx should ignore "!ddd" since it's a bang and not a searched word.

Extended version : the autocomplete feature should suggest all the bangs when the user type "!"

OpenSearch URL

Hi, trying to add my new public searx node to my browser’s search engines list, I noticed the link it gave in opensearch was http://localhost:8888 instead of its external name (searx.galex-713.eu). So I realized the Searx installation never asked me what was the public hostname I wanted to give to him.

Currently I just hugely fixed the problem 168s/host=url_for('index', _external=True, _scheme=scheme)/host=settings.instance_url/ on webapp.py, setting instance_url to the appropriate value in settings file, but I would like a cleaner and more portable solution.

Did anybody already had that issue?

Google Web Search API deprecated

The google, google_news, google_image engines use the Google Web Search API.
According to https://developers.google.com/web-search/docs/ the service stops the September 29, 2014 !

use wikidata.org

See :

• http://www.wikidata.org/wiki/Special:ApiSandbox
• http://une-terre.blogspot.fr/2013/03/wikidata-apidone.html

Some example (replace xml by json) :

• http://www.wikidata.org/w/api.php?action=query&list=search&format=xml&srsearch=roma&srnamespace=0&srlimit=1
• http://www.wikidata.org/w/api.php?action=wbgetentities&sites=frwiki&titles=Rome&languages=fr&format=xml

Could be usefull for :

• find link between wikipedia pages
• display a map using geodata
• etc...

May be this issue can replace the dbpedia.org issue since wikidata is up to date

ImportError: No module named 'cStringIO'

Hi,

I'm running ArchLinux ARM and I did that :

git clone [email protected]:asciimoo/searx.git && cd searx
Enter passphrase

yaourt -S python-pip python-lxml
pip install -r requirements.txt 
pip install cython git+git://github.com/surfly/gevent.git#egg=gevent
vim searx/settings.yml
Modify secret_key 
:wq!

After, this is the error :

[root@cubietruck searx]# python searx/webapp.py
Traceback (most recent call last):
  File "searx/webapp.py", line 26, in <module>
    import cStringIO
ImportError: No module named 'cStringIO'
[root@cubietruck searx]# 

The Python version is Python 3.3.4.

What could I do ?

Allow All / Block All button

It could be useful to allow a single click toggle of all the engines in the preferences page.

sorting news instead of normal ranking heuristic with publishing date

normaly, newer news are more important, so they should be ranked higher.

forcing https on big sites

I have the Idea to add a list with big websites which have https by default enabled, and rewrite urls of that websites to https, if that is not made in a previous step.

Rarely, but it happened that websites like wikipedia have no https link in the searx-results.

Probably an implementation can be done with widged-support if it would be added #60.

Here a few example sites:

*.wikipedia.org
github.com
www.openstreetmap.org

search categories resets to default when click next site

When I search for example for flowers (because there are many photos of that in the internet), and select, that I would only get photos, the selection was reset to the standard configuration when clicking on the button "next site".

Specify custom settings file

It would be nice to have an pypi release, but before that is done some sort of mechanism for specifying custom settings.py file and engines.cfg must be created. Maybe even to combine both in one .cfg file.

new install: https_rules fails

Hello,
i make a new install with all the steps in https://github.com/asciimoo/searx/wiki/Installation

After

python searx/webapp.py

i get

Traceback (most recent call last):
File "searx/webapp.py", line 40, in
from searx import settings, searx_dir
File "/usr/local/searx/searx-ve/local/lib/python2.7/site-packages/searx-0.3.1-py2.7.egg/searx/init.py", line 51, in
load_https_rules(https_rewrite_path)
File "/usr/local/searx/searx-ve/local/lib/python2.7/site-packages/searx-0.3.1-py2.7.egg/searx/https_rewrite.py", line 129, in load_https_rules
for f in listdir(rules_path)
OSError: [Errno 2] No such file or directory: '/usr/local/searx/searx-ve/local/lib/python2.7/site-packages/searx-0.3.1-py2.7.egg/searx/https_rules/'

Can somebody help?

Thanks,
Regards
henning

direct torrent download via webtorrent

Probably, we can add a feature, to download torrents direct inside the browser, using webtorrent:

https://github.com/feross/webtorrent (alpha)

adding autocomplete-function

I think autocompletion would be a nice feature.

Google API:
http://shreyaschand.com/blog/2013/01/03/google-autocomplete-api/
Wikipedia API:
https://www.mediawiki.org/wiki/API:Opensearch

Probably, storing the results would be a good idea, to prevent overhead on the server and apis. This can be done on client or server side. Furthermore, a minimum lenght of 4 characters, and force a short wait (0,5-1s) befor start a autocomplete search does also minimize the amount of querry's

make it possible to deactivate engine by default

for special search-engines like yacy, which result in more loading-time without much better result-quality, it should be possible to deactivate it by default, and let the user activate it if required.

Fetch Search Results According to Users Locale

I tried to replace someone's browser's home pagewith searx, forcing him to quit Google. It failed and asked me to switch back to Google because searx was providing english-tailored results instead of french.
It is something I experienced myself, indeed: one can easily overcome it, but it slows down the user and really hinders searx wide adoption.

I think searx should provide user-language-oriented results.

python-pybabel packet not listed in step by step installation

the packet python-pybabel is not listed in the apt-get install line. It is required to update the translation.

furthermore, a step by step instruction for updating the translation would be also fine.

add base_url support

Hi can you add the base_url support because searx doesn't works with http://domaine.com/searx

I tested with nginx and uswgi

Thanks

wrong url from yahoo engine, and category_none?

The result-url of yahoo is incorrect.

Every url contain the url of the result, and then /RS=<an id-string or so>

the news from yahoo are not affected

Example

https://searx.0x2a.tk/?q=!yh%20asdf&pageno=1&category_none

(here is the second bug, when I search !yh asdf, the category is set to none?, and does not always return results)

one of the Return-Urls:

http://de.wikipedia.org/wiki/ASDF/RS=^ADA.Ge6uhYClQvEOSvwCg8w.MD_aZU

HTTP 500 error on keyword search with Music category enabled

I've set up a basic searx installation at https://search.eliotberriot.com with Nginx and supervisor, (following the guide at https://blog.mavjs.org/setting-up-searx-with-gunicorn-and.html).

The instance seems to work : I can display the search page, and make some searches. However, I get some HTTP 500 errors with some queries. After digging a bit, it seems that searching some keywords when the Music category is enabled make searx crash.

For example, searching "autobahn" in all categories enabled, except music, works as expected. (see it live)

But doing the same query with Music category enabled raise a 500 error (example)

I've also tried on https://searx.0x2a.tk, just to be sure, and the same behaviour shows up :

• "My name is eliot" works with only General enabled, for example
• Same request crashes with music enabled

I've tried to investigate deeper, but nothing is logged (which is probably related to my server setup).

It's not a big deal for me, because I don't plan to use the Music category, but I thought you may want to know that there is a bug hiden somewhere ;)

Anyway, thank you for searx. I think it's a nice and useful project.

removing categories which are fully blocked in the preferences

When blocking all search engines of one categorie (like twitter), this category should be removed from the start and result site.

thx

Problem with yahoo & twitter engines when used from a mobile phone

When used from a mobile phone, the yahoo and the twitter engines doesn't return any results. Explanation : Yahoo and Twitter see the mobile user agent and change the result page layout.

I see two solutions :

• the yahoo / twitter are modified to support the mobile page version.
• searx doesn't send the user agent for those engine (or all engine ?)

Question : why sending the user agent is important ?

modify querrystring of tab when press search

Modifying the querrystring when searching would help to save/share search results

Broken ssl module: TypeError: __init__() got an unexpected keyword argument 'server_hostname'

This is a continuation of what I am experiencing in #92 .

Here is the complete stacktrace, for reference:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/var/www/search.example.com/searx/venv/local/lib/python2.7/site-packages/grequests.py", line 71, in send
    self.url, **merged_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 304, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 724, in _validate_conn
    conn.connect()
  File "/usr/lib/python2.7/dist-packages/urllib3/connection.py", line 237, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py", line 123, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python2.7/ssl.py", line 350, in wrap_socket
    _context=self)
TypeError: __init__() got an unexpected keyword argument 'server_hostname'
<Greenlet at 0xb65269dcL: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0xb64e508c>>(stream=False)> failed with TypeError

I am using a virtualenv on an up-to-date Debian/testing (jessie).

It seems that grequests redefined an ssl module, that want to have the same API as Python 2.7. Unfortunately, they introduced server_hostname and other niceties in Python 2.7.9 that seem to have been backported in Debian. And the wrapper is now broken.
Maybe I'm missing something, somewhere… But now, I am unable to have results for all HTTPS sites. This is quite annoying 😄

allowing searx-instance outsite the Root directory

I have installed a searx in "http://localhost/searx" to test it.

The Problem is, that all urls for css, js, images,... reference to the root-directory.

As example, the line:

<link rel="stylesheet" href="/static/css/style.css" type="text/css" media="screen" />

reference to the url "http://localhost/static/css/style.css", and not to the url "http://localhost/**searx/**static/css/style.css"

500 error when searching oe5tpo

I have typed in my amateur radio callsign oe5tpo, and then an 500 error occours, other search therms doesn't make problems

adding Widget support

Currently I'm work to get and display weather-data like google. I have code it as engine with custom template, but I think, it would an nice idea, to add support for widgets.

As widget, I would define special functions, like displaying weather data, a calculator,... if required. So, It parse the querry, and when required it display special widgets on the result list (on the top).

Integrating !bang-functionality

I would like to see a very nice feature of duckduckgo in searx: Using !bang (like "!youtube" or "!osm" in the search field) to look for a string on a specific platform, not only on categories.

Maybe later there is a way to write plugins for searx like duckduckgo has "goodies": https://duckduckgo.com/goodies

Debian Install documentation

Hi asciimoo,

I just installed searx on https://searx.laquadrature.net and since it's a Debian GNU/Linux Wheezy distribution, I had to install the following :

apt-get install build-essential python-dev libxml2-dev libxlst1-dev

also, using virtualenv is recommended since pip -r requirements.txt is installing many python packages, including binary ones.

If you can add debian-specific installation instructions somewhere (maybe in README.rst) that would be great ;) thanks !

result icon, using url instead of search-engine

Currently, we display specific icons if results come as example from the youtube engine.
Other results, which are also pointing to youtube doesn't have that icon.

parsing the hostname and displaying a icon if it match can resolve that.

Furthermore, I would recommend to specific that rules in a special template.py or template.yml, contained in the specific template folder. So every template can specific the hostnames, for which it would show a icon.

This template.py or .yml can also be used to provide additional template specific informations to searx

problem with the yahoo engine

Since few days, the URL returns by the yahoo engine are like this :
http://ri.search.yahoo.com/_ylt=A0LEV0OlOA5T5mQAJA5XNyoA;_ylu=X3oDMTEzcm42NGRyBHNlYwNzcgRwb3MDMwRjb2xvA2JmMQR2dGlkA1FCQUNLMl8x/RV=1/RE=1393527333/RO=10/RU=https%3a%2f%2fgithub.com%2fasciimoo%2fsearx/RS=%5EADAINxEbm1JliWEKuidveJ6RRwo4Mo-

allowing Strict-Transport-Security (HSTS)

For security reasons, an option to enable HSTS would be a good idea.

Unescaped html in result content

https://searx.0x2a.tk/?q=html%20span%20target&pageno=1&category_general

Improving response time : dns prefetch & tcp ssl pre-open

For each request, there is different steps :

• resolve dns
• open tcp connection
• negociate ssl
• send request
• receive response
• close connection

The first open can be optimize by prefetching dns. I guess it's a matter of 50 ms at most so... But it can be done easily by sending dns request with a timer (each hour for example).

The second and third one cost 100ms or more per request. If searx could start opening TCP connection when the user is one the search page, it could help. To implements this, a patch must done on grequests.

Of course, as chromium, this implies some useless TCP connections ( https://code.google.com/p/chromium/issues/detail?id=116982 )

dns rotation for all searx sites?

instead of a wiki page, maybe have some specific dns name (see pool.ntp.org for an example from the ntp domain) with all the currently-working searx sites?

plus maybe some monitor that kicks out sites that are offline?

showing geodata on searx (openstreetmap

Currently searx has normal results, images, music,...

Displaying a map including route-machine would be nice, and no big overhead. But I'm not sure how it could be implemented in the interface nicely. I think a seperate Map-Site would be the best idea.

I think a combination of:

• Leaflet JS Framework to display Maps
• OSM OpenStreetMap (very detailed, I'm one of million contributors there)
• Nominatim Searching in the OSM database
• OSRM Very fast and accurate Routing with OSM

would be nice, and could be extended every time.

I would make the client-side, and most of server-side of that, but I would discuss about how to make it before starting coding.

I have experience with OSM, and coding maps (http://map.ham-radio.oe5tpo.com), so I think it would be no big problem.

thx, pointhi

Add "Remove Top {number} Sites" Feature

I would find it useful, if Searx could also behave like https://millionshort.com/

Max retries exceeded with url

Hi,

Results are differents for a same request (48, 59, 0, 0, 0, 0, 23, etc). I have a "Max retries exceeded with url" error.
I'm using Nginx as a HTTP server & reverse proxy.

root@cubietruck:/var/www/searx# python searx/webapp.py
 * Running on http://127.0.0.1:8888/
 * Restarting with reloader
127.0.0.1 - - [17/Dec/2013 16:16:18] "GET / HTTP/1.0" 200 -
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/grequests.py", line 71, in send
    self.url, **merged_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='gdata.youtube.com', port=443): Max retries exceeded with url: /feeds/api/videos?alt=json&q=simon+arazi (Caused by <class 'socket.error'>: [Errno 101] Network is unreachable)
<Greenlet at 0x2066530: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x2076030>>(stream=False)> failed with ConnectionError

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/grequests.py", line 71, in send
    self.url, **merged_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='thepiratebay.sx', port=443): Max retries exceeded with url: /search/simon%20arazi/0/99/200 (Caused by <class 'gevent.dns.DNSError'>: [Errno 3] name does not exist)
<Greenlet at 0x2066350: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x206ea10>>(stream=False)> failed with ConnectionError

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/grequests.py", line 71, in send
    self.url, **merged_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='en.wikipedia.org', port=443): Max retries exceeded with url: /w/api.php?action=query&list=search&srsearch=simon+arazi&srprop=timestamp&format=json (Caused by <class 'gevent.dns.DNSError'>: [Errno 3] name does not exist)
<Greenlet at 0x2066760: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x2076db0>>(stream=False)> failed with ConnectionError

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/grequests.py", line 71, in send
    self.url, **merged_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='www.bing.com', port=80): Max retries exceeded with url: /search?q=simon+arazi&setmkt=en-us (Caused by <class 'gevent.dns.DNSError'>: [Errno 3] name does not exist)
<Greenlet at 0x20667b0: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x20df030>>(stream=False)> failed with ConnectionError

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/grequests.py", line 71, in send
    self.url, **merged_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='ajax.googleapis.com', port=443): Max retries exceeded with url: /ajax/services/search/web?v=2.0&start=0&rsz=large&safe=off&filter=off&q=simon+arazi (Caused by <class 'gevent.dns.DNSError'>: [Errno 3] name does not exist)
<Greenlet at 0x20668a0: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x20df770>>(stream=False)> failed with ConnectionError

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/grequests.py", line 71, in send
    self.url, **merged_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='thepiratebay.sx', port=443): Max retries exceeded with url: /search/simon%20arazi/0/99/100 (Caused by <class 'gevent.dns.DNSError'>: [Errno 3] name does not exist)
<Greenlet at 0x2066580: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x2076290>>(stream=False)> failed with ConnectionError

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/grequests.py", line 71, in send
    self.url, **merged_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='www.deviantart.com', port=443): Max retries exceeded with url: /search?q=simon+arazi (Caused by <class 'gevent.dns.DNSError'>: [Errno 3] name does not exist)
<Greenlet at 0x2066620: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x2076750>>(stream=False)> failed with ConnectionError

127.0.0.1 - - [17/Dec/2013 16:16:31] "POST / HTTP/1.0" 200 -

Does not work on CubieTruck

Hi,

How Searx could work on CubieTruck ?

root@cubietruck:/var/www/searx# pip install -r requirements.txt
Requirement already satisfied (use --upgrade to upgrade): flask in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): grequests in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 2))
Downloading/unpacking lxml (from -r requirements.txt (line 3))
  Downloading lxml-3.2.4.tar.gz (3.3MB): 3.3MB downloaded
  Running setup.py egg_info for package lxml
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
      warnings.warn(msg)
    Building lxml version 3.2.4.
    Building without Cython.
    ERROR: /bin/sh: 1: xslt-config: not found

    ** make sure the development packages of libxml2 and libxslt are installed **

    Using build configuration of libxslt

    warning: no previously-included files found matching '*.py'
    warning: no files found matching '*.txt' under directory 'src/lxml/tests'
Downloading/unpacking Werkzeug>=0.7 (from flask->-r requirements.txt (line 1))
  Downloading Werkzeug-0.9.4.tar.gz (1.1MB): 1.1MB downloaded
  Running setup.py egg_info for package Werkzeug

    warning: no files found matching '*' under directory 'werkzeug/debug/templates'
    warning: no files found matching '*' under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
    warning: no previously-included files matching '*.pyc' found under directory 'tests'
    warning: no previously-included files matching '*.pyo' found under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'examples'
    warning: no previously-included files matching '*.pyo' found under directory 'examples'
    no previously-included directories found matching 'docs/_build'
Downloading/unpacking Jinja2>=2.4 (from flask->-r requirements.txt (line 1))
  Downloading Jinja2-2.7.1.tar.gz (377kB): 377kB downloaded
  Running setup.py egg_info for package Jinja2

    warning: no files found matching '*' under directory 'custom_fixers'
    warning: no previously-included files matching '*' found under directory 'docs/_build'
    warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
Downloading/unpacking itsdangerous>=0.21 (from flask->-r requirements.txt (line 1))
  Downloading itsdangerous-0.23.tar.gz (46kB): 46kB downloaded
  Running setup.py egg_info for package itsdangerous

    warning: no previously-included files matching '*' found under directory 'docs/_build'
Downloading/unpacking gevent (from grequests->-r requirements.txt (line 2))
  Downloading gevent-1.0.tar.gz (1.4MB): 1.4MB downloaded
  Running setup.py egg_info for package gevent

Requirement already satisfied (use --upgrade to upgrade): requests>=1.0.0 in /usr/lib/python2.7/dist-packages (from grequests->-r requirements.txt (line 2))
Downloading/unpacking markupsafe (from Jinja2>=2.4->flask->-r requirements.txt (line 1))
  Downloading MarkupSafe-0.18.tar.gz
  Running setup.py egg_info for package markupsafe

Downloading/unpacking greenlet (from gevent->grequests->-r requirements.txt (line 2))
  Downloading greenlet-0.4.1.zip (75kB): 75kB downloaded
  Running setup.py egg_info for package greenlet

Installing collected packages: lxml, Werkzeug, Jinja2, itsdangerous, gevent, markupsafe, greenlet
  Running setup.py install for lxml
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
      warnings.warn(msg)
    Building lxml version 3.2.4.
    Building without Cython.
    ERROR: /bin/sh: 1: xslt-config: not found

    ** make sure the development packages of libxml2 and libxslt are installed **

    Using build configuration of libxslt
    building 'lxml.etree' extension
    arm-linux-gnueabihf-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/tmp/pip_build_root/lxml/src/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-armv7l-2.7/src/lxml/lxml.etree.o
    src/lxml/lxml.etree.c:8:22: fatal error: pyconfig.h: Aucun fichier ou dossier de ce type
     #include "pyconfig.h"
                          ^
    compilation terminated.
    error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1
    Complete output from command /usr/bin/python -c "import setuptools;__file__='/tmp/pip_build_root/lxml/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-IyHhlY-record/install-record.txt --single-version-externally-managed:
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'

  warnings.warn(msg)

Building lxml version 3.2.4.

Building without Cython.

ERROR: /bin/sh: 1: xslt-config: not found



** make sure the development packages of libxml2 and libxslt are installed **



Using build configuration of libxslt

running install

running build

running build_py

creating build

creating build/lib.linux-armv7l-2.7

creating build/lib.linux-armv7l-2.7/lxml

copying src/lxml/ElementInclude.py -> build/lib.linux-armv7l-2.7/lxml

copying src/lxml/doctestcompare.py -> build/lib.linux-armv7l-2.7/lxml

copying src/lxml/_elementpath.py -> build/lib.linux-armv7l-2.7/lxml

copying src/lxml/pyclasslookup.py -> build/lib.linux-armv7l-2.7/lxml

copying src/lxml/cssselect.py -> build/lib.linux-armv7l-2.7/lxml

copying src/lxml/sax.py -> build/lib.linux-armv7l-2.7/lxml

copying src/lxml/builder.py -> build/lib.linux-armv7l-2.7/lxml

copying src/lxml/__init__.py -> build/lib.linux-armv7l-2.7/lxml

copying src/lxml/usedoctest.py -> build/lib.linux-armv7l-2.7/lxml

creating build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/__init__.py -> build/lib.linux-armv7l-2.7/lxml/includes

creating build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/_html5builder.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/diff.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/clean.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/ElementSoup.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/html5parser.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/_diffcommand.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/builder.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/soupparser.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/formfill.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/defs.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/__init__.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/usedoctest.py -> build/lib.linux-armv7l-2.7/lxml/html

copying src/lxml/html/_setmixin.py -> build/lib.linux-armv7l-2.7/lxml/html

creating build/lib.linux-armv7l-2.7/lxml/isoschematron

copying src/lxml/isoschematron/__init__.py -> build/lib.linux-armv7l-2.7/lxml/isoschematron

copying src/lxml/lxml.etree.h -> build/lib.linux-armv7l-2.7/lxml

copying src/lxml/lxml.etree_api.h -> build/lib.linux-armv7l-2.7/lxml

copying src/lxml/includes/uri.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/config.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/tree.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/xslt.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/schematron.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/xpath.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/xinclude.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/relaxng.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/c14n.pxd -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/lxml-version.h -> build/lib.linux-armv7l-2.7/lxml/includes

copying src/lxml/includes/etree_defs.h -> build/lib.linux-armv7l-2.7/lxml/includes

creating build/lib.linux-armv7l-2.7/lxml/isoschematron/resources

creating build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/rng

copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/rng

creating build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/xsl

copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/xsl

copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/xsl

creating build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-armv7l-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

running build_ext

building 'lxml.etree' extension

creating build/temp.linux-armv7l-2.7

creating build/temp.linux-armv7l-2.7/src

creating build/temp.linux-armv7l-2.7/src/lxml

arm-linux-gnueabihf-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/tmp/pip_build_root/lxml/src/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-armv7l-2.7/src/lxml/lxml.etree.o

src/lxml/lxml.etree.c:8:22: fatal error: pyconfig.h: Aucun fichier ou dossier de ce type

 #include "pyconfig.h"

                      ^

compilation terminated.

error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools;__file__='/tmp/pip_build_root/lxml/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-IyHhlY-record/install-record.txt --single-version-externally-managed failed with error code 1 in /tmp/pip_build_root/lxml
Storing complete log in /root/.pip/pip.log

make test script to check if engines are parsed correct

Because the engines does not work allways wit an official api, it would be useful to make a script which check regulary if the engines return valide results.

currently the flickr engine is broken, probably blocked by flickr itself.

can't type ( in search field

It is not possible to type the open parenthesis character in the search field.
For normal request, it is most of time not really useful, but for calculation for example.

implement image-proxy (fetch and mininize images)

currently images are provided in the result-site, without data compression or ssl-encryption.

My Idea is that searx fetch the result images, minimize it if possible and store the images for a short time localy.