theprojecter / whazzup Goto Github PK

What it says.

The tokenizer will sometimes create tokens like "berlusconi'", which isn't 
right. Need to also look for other bad tokenizations.

It should be possible to add sites simply by uploading an OPML file into 
Whazzup.

This should keep the impact of checking feeds fairly low.

In the item report, that is.

We don't need to recheck bad feeds every time we check feeds.

For one thing, need to flush the log file.

We need to break up the bigger tasks more so they don't run all at once and hog 
the queue.

u'\u0441\u0431, 13 feb 2010 15:51:30 +0300'

This from a Russian blog.

Now we rerun the scheduled tasks every time the queue is restarted. Not good. 
Instead, move scheduling out to actual cron, and use send.py to start the tasks.

What it says.

Use send.py to inject the tasks in the queue.

This is one of the biggest obstacles to scaling. Need to analyze it to see 
where it actually spends the time. If it turns out that parsing the text is the 
time-waster then we should consider caching the word vector, for example by 
marshalling the hash into a database field.

That is, feeds which haven't been updated in a long time should be visually 
indicated somehow.

Error message is "'ascii' codec can't decode byte 0xe5 in position 111: ordinal 
not in range(128)"

We can do this easily by doing it async. CheckFeed adds a feed-checking task to 
an internal in-memory queue. A sub-thread handles these tasks, and stores 
parsed feeds as in memory object structures. Then it sends a ProcessFeed 
message to the queue, which does the actual processing. Should be trivial to 
implement.

Need to stop repeating the same boilerplate stuff in all templates.

Basically, serve the list with N extra rows. For each vote, send an async 
message to the server (basically the same as now), then hide that row, and 
unhide another row.

If the recalculation of story points as well as downloading of new stories were 
done in a queue-based approach it would make the code a lot more amenable to 
deployment in a cloud-type/service-type environment.

The current memory footprint is excessive, mainly because both the original 
summary text for a post is stored in memory, and also the text vector.

Or perhaps not? Given that all feeds are public now. Need to consider this more 
carefully.

What it says.

We have a feed in the system now that uses https.

Basically, aggregate the scores for posts across all users (simple SQL sum?) 
and show a couple of pages of that.

If a vote request causes an error, the error display appears. If the next vote 
request succeeds, the error display remains.

This will make it much easier to implement issue 18.

It's worth a shot, because this operation is just way too slow right now.

If the message is already in the queue there's no point in adding it again. 
This could prevent some kinds of issues.

what i see now: 
25 feed items on one page

what i'd like: 
to configure f.ex. one of 10, 15, 20, 25
(motivation: other feature request on marking all on one page as read)

Need a statistics screen showing basic information about the amount of data in 
the system.

Stories often get a datetime before the present, and so are listed at "60 
minutes", then after a good while they bounce down to "n minutes" and increase 
normally after that. Presumably this is a timezone issue somewhere.

The UFP would probably be a far better alternative than the RSS library 
currently in whazzup.

At the moment whazzup needs to know in advance what the format of the feed it 
is loading is. This restriction needs to be removed by auto-detecting the feed 
format.

Remove feeds which never load after a given grace period. Should perhaps notify 
subscribers by email?

What it says. One user has asked for this.

This date format occurs in one of the feeds and is causing errors, so support 
it.

At the moment we can only run a single dbqueue instance, as we might otherwise 
get contention over the gdbm files. Need to consider how we can get around this.

Right now we're using an unholy hack to throw web.py off its stride and stop, 
causing Python to stop. We don't respect Ctrl-C, either.

At the moment the interaction between whazzup.py and the rest of the system is 
a real mix-up of different kinds of calls. Need to build a proper abstraction 
that makes this separation cleaner. Ideally it should make it possible to swap 
in and out different queueing mechanisms.

The URIs change, which causes the feed identifier to also change, and the code 
cannot handle this at the moment. A better solution is to assign IDs 
automatically and use those everywhere.

It should be fairly easy to get this to run on Google AppEngine, which would 
make it possible to set it up as a free service for anyone to use. Would be 
cool to try it out.

what i see now:
a list of 25 feed items, where i can mark one item as read at a time.

what i'd like: 
(in addition to fewer/ configurable number of items pr page, )
ability to mark all items on a page as read in one operation.

Occasionally DB connections break, and when they do they never get repaired 
again. Need to come up with some way to detect and fix.

Sometimes, when reading a feed, the worker thread hangs for several minutes 
waiting for the server to time out. This means the entire worker thread hangs 
for the same time, blocking all updates. Should redo the code so that it times 
out more quickly.

One way would be to dump stats to HTML (for viewing through the web interface) 
at intervals, and to keep comparable stats for several such intervals.

Some possible measures: number of tasks in the queue, time to process a task, 
wait time for a task in the queue before it's finally processed. Showing max, 
min, average, and right now values for these measures might work.

We can handle this by parsing the HTML to find the link to the RSS/Atom feed 
and subscribing to that instead.

Don't allow a feed with the same URL as a previous feed to be added to the 
database.

What it says.

Turns out SysV IPC message queues suck big time, so we need to change over to 
Gearman.

Now that we have OPML we should be good citizens and also support export to OPML