"Oops! 500" on package 'marble' about fedora-packages

[ralphbean]

Looks like marble never got indexed.

I checked the cronjob log in /var/log/fedoracommunity/cron-sync-package-index.log and it was getting Permission denied: '/var/cache/fedoracommunity/packages/tmp/var/cache/fedoracommunity/packages/tmp/rawhide-source/repomd.xml.old.tmp'

The dir was owned by root but it should have been owned by apache. I issued a chown; let's see if that fixes it. (FWIW, no new packages were being indexed because of this).

[bitlord]

ok.
Not sure how often that cronjob runs, but still returns error "500".

[ralphbean]

@bitlord, well, the cronjob ran but it didn't pick it up. I'm trying to force it now.

[ralphbean]

Learned new stuff: the issue is a UnicodeDecodeError coming from the template details.mak.

It is dying because marble has a little non-ascii character in its description:
http://pkgs.fedoraproject.org/cgit/marble.git/tree/marble.spec#n88

This is fedora-packages' fault. It should be able to handle this.

[ralphbean]

Doh. I'm unable to reproduce this locally on F20. Will have to debug a fix in the staging environment.

[lmacken]

A bunch of packages are hitting this now, it seems.

Module details_mak:28 in render_body
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 853: ordinal not in range(128)

[lmacken]

The package description that it's choking on in fedoracommunity/widgets/package/templates/details.mak is:

u'Marble is a Virtual Globe and World Atlas that you can use to learn more\\nabout Earth: You can pan and zoom around and you can look up places and\\nroads. A mouse click on a place label will provide th
e respective Wikipedia\\narticle.\\n\\nOf course it\\'s also possible to measure distances between locations or watch\\nthe current cloud cover. Marble offers different thematic maps: A classroom-\\nstyle topographic map, a satellite view,
 street map, earth at night and\\ntemperature and precipitation maps. All maps include a custom map key, so it\\ncan also be used as an educational tool for use in class-rooms. For\\neducational purposes you can also change date and time a
nd watch how the\\nstarry sky and the twilight zone on the map change.\\n\\nIn opposite to other virtual globes Marble also features multiple\\nprojections: Choose between a Flat Map ("Plate carr\\xe9"), Mercator or the Globe.'

Which is fetched from Xapian in fedoracommunity/widgets/package/overview.py. I'm guessing we're not properly decoding the description in our indexer.

[techtonik]

I am looking at https://github.com/fedora-infra/fedora-packages/blob/develop/fedoracommunity/widgets/package/overview.py and can't find where description data is passed to template.

[ralphbean]

I think it comes from self.description which gets set in here: https://github.com/fedora-infra/fedora-packages/blob/develop/fedoracommunity/widgets/package/overview.py#L28-L46

[techtonik]

So, result comes from https://github.com/fedora-infra/fedora-packages/blob/develop/fedoracommunity/widgets/package/overview.py#L31 which reads some kind of document here - https://github.com/fedora-infra/fedora-packages/blob/develop/fedoracommunity/connectors/xapianconnector.py#L159, but doesn't decode it. To push this further, we need a binary dump of contents returned by .get_data() call and accurate representation of parsed JSON.

It may happen that there is an error in Python documentation of https://docs.python.org/2/library/json.html#json.loads and json.loads never tries to decode utf-8 to unicode leaving everything as-is.