srspear / scifitropes Goto Github PK

Some of these need to be doublechecked to see if they are an index and should be removed. May also need to create a set of urls always filtered out (like 'Administrivia' etc, but also for tracking urls who may be structured in an expected way but contain the wrong content).

Currently only has a few works associated with it; in fact, it has a page for each medium. There are a few ways to do this:

• Go ahead and develop custom script to crawl specifically through "AI Is a Crapshoot" pages
• Go through the main scifi works db, count up how many times it occurs there and add and then dedupe those titles from the works/works_count
• See if there is an accurate way to detect pages with a) unique structures like this and b) create a generalized script to address all such pages