Fresh install of the bakerydemo using pyenv generates 'KeyError at / <wagtail.images.models.Filter object at 0x10953ef40>' about bakerydemo

[ababic]

While I don't think we should be going to any lengths to support unstable builds, this issue did highlight that the code wasn't very easy to follow in it's current state, and could benefit from clearer exception handling (a limitation of the ThreadPoolExecutor.map() approach). As a result, I have submitted: wagtail/wagtail#11832

[gasman]

Hi @birchcode - thanks for the report. Can you share the full traceback of the error, please? (From the 'traceback' section of the error page, "Switch to copy-and-paste view")

[birchcode]

the full traceback of the error

Request Method: GET
Request URL: http://127.0.0.1:8000/

Django Version: 4.2.10
Python Version: 3.8.10
Installed Applications:
['bakerydemo.base',
 'bakerydemo.blog',
 'bakerydemo.breads',
 'bakerydemo.locations',
 'bakerydemo.recipes',
 'bakerydemo.search',
 'wagtail.embeds',
 'wagtail.sites',
 'wagtail.users',
 'wagtail.snippets',
 'wagtail.documents',
 'wagtail.images',
 'wagtail.search',
 'wagtail.admin',
 'wagtail.api.v2',
 'wagtail.locales',
 'wagtail.contrib.forms',
 'wagtail.contrib.redirects',
 'wagtail.contrib.routable_page',
 'wagtail.contrib.table_block',
 'wagtail.contrib.typed_table_block',
 'wagtail.contrib.search_promotions',
 'wagtail.contrib.settings',
 'wagtail.contrib.simple_translation',
 'wagtail.contrib.styleguide',
 'wagtail',
 'rest_framework',
 'modelcluster',
 'taggit',
 'wagtailfontawesomesvg',
 'debug_toolbar',
 'django_extensions',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.sitemaps']
Installed Middleware:
['debug_toolbar.middleware.DebugToolbarMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'wagtail.contrib.redirects.middleware.RedirectMiddleware']


Template error:
In template /Users/rmp/projects/bakerydemo/bakerydemo/templates/base/home_page.html, error at line 8
   format-jpeg|fill-1920x900
   1 : {% extends "base.html" %}
   2 : {% load wagtailimages_tags wagtailcore_tags %}
   3 : 
   4 : {% block content %}
   5 :     <div class="homepage">
   6 : 
   7 :         <div class="container-fluid hero">
   8 :              {% picture page.image format-{avif,webp,jpeg} fill-{800x650,1920x900} sizes="100vw" class="hero-image" alt="" %} 
   9 :             <div class="hero-gradient-mask"></div>
   10 :             <div class="container">
   11 :                 <div class="row">
   12 :                     <div class="col-md-6 col-md-offset-1 col-lg-5 home-hero">
   13 :                         <h1>{{ page.title }}</h1>
   14 :                         <p class="lead">{{ page.hero_text }}</p>
   15 :                         {% if page.hero_cta_link %}
   16 :                             <a href="{% pageurl page.hero_cta_link %}" class="hero-cta-link">
   17 :                                 {{ page.hero_cta }}
   18 :                             </a>


Traceback (most recent call last):
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/core/handlers/base.py", line 220, in _get_response
    response = response.render()
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/response.py", line 114, in render
    self.content = self.rendered_content
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/response.py", line 92, in rendered_content
    return template.render(context, self._request)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/base.py", line 175, in render
    return self._render(context)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/test/utils.py", line 112, in instrumented_test_render
    return self.nodelist.render(context)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/loader_tags.py", line 157, in render
    return compiled_parent._render(context)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/test/utils.py", line 112, in instrumented_test_render
    return self.nodelist.render(context)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/loader_tags.py", line 63, in render
    result = block.nodelist.render(context)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/base.py", line 1005, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/base.py", line 1005, in <listcomp>
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/django/template/base.py", line 966, in render_annotated
    return self.render(context)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/wagtail/images/templatetags/wagtailimages_tags.py", line 203, in render
    renditions = get_renditions_or_not_found(
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/wagtail/images/shortcuts.py", line 35, in get_renditions_or_not_found
    return image.get_renditions(*specs)
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/wagtail/images/models.py", line 533, in get_renditions
    for filter, rendition in self.create_renditions(*not_found).items():
  File "/Users/rmp/.virtualenvs/wagtailbakerydemo/lib/python3.8/site-packages/wagtail/images/models.py", line 693, in create_renditions
    filter = filter_map[existing.filter_spec]

Exception Type: KeyError at /
Exception Value: 'format-jpeg|fill-1920x900'

[NKETIAH53]

Hi @birchcode is this issue resolved for you?

[carlotm]

I am getting the same error

[gasman]

@birchcode @carlotm Thanks for confirming. Does the error go away if you refresh the page, or does it happen consistently?

@ababic Any thoughts on what might be going on here? I can't replicate this locally, and for it to reach this line and fail, the filter spec would have to be returned in the "identify potential clashes" query, but not be in filter_map. But filter_map should contain everything in the filters list, so that doesn't make sense... My only hunch is that maybe it isn't safe to have multiple threads appending to to_create, and it's mixing up those results with other invocations of create_renditions - and we should be returning the Rendition instances and picking up the return value of executor.map instead. Not sure how we verify that, though...

@zerolab In https://github.com/wagtail/wagtail/pull/9550/files#r1246522485, you said: "if we change to source=self.file I had to reload a number of times until all renditions were generated as oppposed to all in one go with this implementation". Do you happen to remember what the "not all renditions being generated" failure case looked like - was it an error like this, by any chance?

[zerolab]

@gasman no 500 like this. Just one or two renditions not being generated, if I recall correctly

[carlotm]

@gasman it's consistent for me.
BTW I am not using pyenv

[ababic]

@birchcode Could I just check that you are using a virtual env of some kind and freshly installing all of the requirements? I think I've seen something like this after upgrading wagtail, but having an out-of-date version of pillow

[ababic]

@gasman I'm pretty sure this is what's happening...

1. generate_rendition_file() is silently failing whilst being run by the threadpool
2. Because of 1, the to_create list is empty
3. Because of 2, the lookup_q created to find clashes doesn't actually apply any filtering
4. Because the current rendition queryset isn't being filtered, the line in question is looking in filter_map for any old value, hence the KeyError

[gasman]

Hopefully addressed now in wagtail/wagtail#11832, which will be included in the upcoming Wagtail 6.1 release.