ayleph / mediagoblin-basicsearch

Search plugin for Gnu MediaGoblin

• 👋 Hi, I’m Andrew.
• 👀 I’m interested in industrial control systems, network security, free/open-source software, hardware hacking, and wireless systems.
• 🌱 I’m currently learning FreeCAD and Rust.
• 💞️ I’m looking to collaborate on ...
• 📫 How to reach me ...

mediagoblin-basicsearch's Issues

Queries too slow, plugin is unusable

Every search just times out. Postres logging for queries longer than 5 seconds shows a 76 second and 5x second query after a search:

2019-09-12 20:05:52 UTC LOG:  duration: 76023.803 ms  statement: SELECT count(*) AS count_1 
        FROM (SELECT core__media_entries.id AS core__media_entries_id, core__media_entries.public_id AS core__media_entries_public_id, core__media_entries.remote AS core__media_entries_remote, core__media_entries.actor AS core__media_entries_actor, core__media_entries.title AS core__media_entries_title, core__media_entries.slug AS core__media_entries_slug, core__media_entries.description AS core__media_entries_description, core__media_entries.media_type AS core__media_entries_media_type, core__media_entries.state AS core__media_entries_state, core__media_entries.license AS core__media_entries_license, core__media_entries.file_size AS core__media_entries_file_size, core__media_entries.location AS core__media_entries_location, core__media_entries.created AS core__media_entries_created, core__media_entries.updated AS core__media_entries_updated, core__media_entries.fail_error AS core__media_entries_fail_error, core__media_entries.fail_metadata AS core__media_entries_fail_metadata, core__media_entries.transcoding_progress AS core__media_entries_transcoding_progress, core__media_entries.queued_media_file AS core__media_entries_queued_media_file, core__media_entries.queued_task_id AS core__media_entries_queued_task_id, core__media_entries.media_metadata AS core__media_entries_media_metadata 
        FROM core__media_entries, core__media_tags 
        WHERE core__media_entries.state = 'processed' AND (core__media_entries.id = core__media_tags.media_entry AND core__media_tags.name ILIKE '%stallman%' OR core__media_entries.title ILIKE '%stallman%' OR core__media_entries.description ILIKE '%stallman%') ORDER BY core__media_entries.title) AS anon_1
2019-09-12 20:06:54 UTC LOG:  duration: 61389.786 ms  statement: SELECT count(*) AS count_1 
        FROM (SELECT core__media_entries.id AS core__media_entries_id, core__media_entries.public_id AS core__media_entries_public_id, core__media_entries.remote AS core__media_entries_remote, core__media_entries.actor AS core__media_entries_actor, core__media_entries.title AS core__media_entries_title, core__media_entries.slug AS core__media_entries_slug, core__media_entries.description AS core__media_entries_description, core__media_entries.media_type AS core__media_entries_media_type, core__media_entries.state AS core__media_entries_state, core__media_entries.license AS core__media_entries_license, core__media_entries.file_size AS core__media_entries_file_size, core__media_entries.location AS core__media_entries_location, core__media_entries.created AS core__media_entries_created, core__media_entries.updated AS core__media_entries_updated, core__media_entries.fail_error AS core__media_entries_fail_error, core__media_entries.fail_metadata AS core__media_entries_fail_metadata, core__media_entries.transcoding_progress AS core__media_entries_transcoding_progress, core__media_entries.queued_media_file AS core__media_entries_queued_media_file, core__media_entries.queued_task_id AS core__media_entries_queued_task_id, core__media_entries.media_metadata AS core__media_entries_media_metadata 
        FROM core__media_entries, core__media_tags 
        WHERE core__media_entries.state = 'processed' AND (core__media_entries.id = core__media_tags.media_entry AND core__media_tags.name ILIKE '%stallman%' OR core__media_entries.title ILIKE '%stallman%' OR core__media_entries.description ILIKE '%stallman%') ORDER BY core__media_entries.title 
         LIMIT 30) AS anon_1
2019-09-12 20:07:51 UTC LOG:  duration: 57949.570 ms  statement: SELECT core__media_entries.id AS core__media_entries_id, core__media_entries.public_id AS core__media_entries_public_id, core__media_entries.remote AS core__media_entries_remote, core__media_entries.actor AS core__media_entries_actor, core__media_entries.title AS core__media_entries_title, core__media_entries.slug AS core__media_entries_slug, core__media_entries.description AS core__media_entries_description, core__media_entries.media_type AS core__media_entries_media_type, core__media_entries.state AS core__media_entries_state, core__media_entries.license AS core__media_entries_license, core__media_entries.file_size AS core__media_entries_file_size, core__media_entries.location AS core__media_entries_location, core__media_entries.created AS core__media_entries_created, core__media_entries.updated AS core__media_entries_updated, core__media_entries.fail_error AS core__media_entries_fail_error, core__media_entries.fail_metadata AS core__media_entries_fail_metadata, core__media_entries.transcoding_progress AS core__media_entries_transcoding_progress, core__media_entries.queued_media_file AS core__media_entries_queued_media_file, core__media_entries.queued_task_id AS core__media_entries_queued_task_id, core__media_entries.media_metadata AS core__media_entries_media_metadata, core__locations_1.id AS core__locations_1_id, core__locations_1.name AS core__locations_1_name, core__locations_1.position AS core__locations_1_position, core__locations_1.address AS core__locations_1_address 
        FROM core__media_tags, core__media_entries LEFT OUTER JOIN core__locations AS core__locations_1 ON core__locations_1.id = core__media_entries.location 
        WHERE core__media_entries.state = 'processed' AND (core__media_entries.id = core__media_tags.media_entry AND core__media_tags.name ILIKE '%stallman%' OR core__media_entries.title ILIKE '%stallman%' OR core__media_entries.description ILIKE '%stallman%') ORDER BY core__media_entries.title 
         LIMIT 30

I don't know how to optimize things, but our database is not that big so this shouldn't happen.

Search results are in a weird order

Search only outputs 2-3 entries per page, with some entries duplicated

Server error when search query includes non-ASCII characters

Error - <type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode character u'\xf1' in position 4: ordinal not in range(128)
URL: https://goblinrefuge.com/mediagoblin/search/?query=espa%C3%B1a
File '/path/to/mediagoblin/lib/python2.7/site-packages/Paste-1.7.5.1-py2.7.egg/paste/exceptions/errormiddleware.py', line 144 in __call__
  app_iter = self.application(environ, sr_checker)
File '/path/to/mediagoblin/mediagoblin/app.py', line 342 in __call__
  return self.call_backend(environ, start_response)
File '/path/to/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.10.1-py2.7.egg/werkzeug/wsgi.py', line 591 in __call__
  return self.app(environ, start_response)
File '/path/to/mediagoblin/mediagoblin/app.py', line 276 in call_backend
  return self._finish_call_backend(request, environ, start_response)
File '/path/to/mediagoblin/mediagoblin/app.py', line 318 in _finish_call_backend
  response = controller(request)
File '/path/to/mediagoblin/mediagoblin/decorators.py', line 170 in wrapper
  return controller(request, page=page, *args, **kwargs)
File '/path/to/mediagoblin/mediagoblin/plugins/basicsearch/views.py', line 69 in search_results_view
  'form': form})
File '/path/to/mediagoblin/mediagoblin/tools/response.py', line 36 in render_to_response
  render_template(request, template, context),
File '/path/to/mediagoblin/mediagoblin/tools/template.py', line 144 in render_template
  rendered = template.render(context)
File '/path/to/mediagoblin/lib/python2.7/site-packages/Jinja2-2.7.3-py2.7.egg/jinja2/environment.py', line 969 in render
  return self.environment.handle_exception(exc_info, True)
File '/path/to/mediagoblin/lib/python2.7/site-packages/Jinja2-2.7.3-py2.7.egg/jinja2/environment.py', line 742 in handle_exception
  reraise(exc_type, exc_value, tb)
File '/path/to/mediagoblin/mediagoblin/plugins/basicsearch/templates/mediagoblin/plugins/basicsearch/results.html', line 21 in top-level template code 
  {% from "mediagoblin/utils/object_gallery.html" import object_gallery %}
File '/var/lib/mediagoblin/templates/mediagoblin/base.html', line 67 in top-level template code 
  {% block mediagoblin_body %}
File '/var/lib/mediagoblin/templates/mediagoblin/base.html', line 207 in block "mediagoblin_body"
  {% block mediagoblin_content %}
File '/path/to/mediagoblin/mediagoblin/plugins/basicsearch/templates/mediagoblin/plugins/basicsearch/results.html', line 33 in block "mediagoblin_content"
  {{ object_gallery(request, media_entries, pagination) }}
File '/path/to/mediagoblin/mediagoblin/templates/mediagoblin/utils/object_gallery.html', line 68 in template
  {{ render_pagination(request, pagination) }}
File '/path/to/mediagoblin/mediagoblin/templates/mediagoblin/utils/pagination.html', line 42 in template
  {% set next_url = pagination.get_page_url_explicit(
File '/path/to/mediagoblin/mediagoblin/tools/pagination.py', line 110 in get_page_url_explicit
  base_url, urllib.urlencode(new_get_params))
File '/usr/lib64/python2.7/urllib.py', line 1347 in urlencode
  v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 4: ordinal not in range(128)

Media without tags aren't included in search results

Merging in the code to search tags seems to have prevented untagged media from being included in search results.