bookieio / breadability Goto Github PK

[D 120827 20:17:42 existing:67] Q1 getting content for a6bb837b68038a http://themeforest.net/item/themeology-portfolio-and-blog-theme/full_screen_preview/127873

Exception in thread Thread-1325:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in *bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 484, in run
self.__target(_self.__args, _self.__kwargs)
File "scripts/readability/existing.py", line 68, in fetch_content
read = ReadUrl.parse(url)
File "/home/bmark.us/0.5/bookie/lib/readable.py", line 176, in parse
if not document.readable:
File "/home/bmark.us/0.5/lib/python2.6/site-packages/breadability/utils.py", line 55, in __get
value = self.fget(inst)
File "/home/bmark.us/0.5/lib/python2.6/site-packages/breadability/readable.py", line 457, in readable
return tounicode(self._readable)
File "/home/bmark.us/0.5/lib/python2.6/site-packages/breadability/utils.py", line 55, in get
value = self.fget(inst)
File "/home/bmark.us/0.5/lib/python2.6/site-packages/breadability/readable.py", line 482, in _readable
doc = build_base_document(updated_winner.node, self.fragment)
File "/home/bmark.us/0.5/lib/python2.6/site-packages/breadability/readable.py", line 93, in build_base_document
if html.tag == 'body':
AttributeError: 'NoneType' object has no attribute 'tag'

Leaving this open as we hope to keep adding sites that fail to parse and bug fixes that correct it.

http://shrinik.blogspot.com/2012/05/brief-introduction-of-test-automation.html

012-06-29T16:55:50+00:00 app[web.1]: File "/app/bookie_parser/handlers/init.py", line 131, in _readable_content
2012-06-29T16:55:50+00:00 app[web.1]: value = self.fget(inst)
2012-06-29T16:55:50+00:00 app[web.1]: File "/app/.heroku/venv/lib/python2.7/site-packages/breadability/readable.py", line 426, in readable
2012-06-29T16:55:50+00:00 app[web.1]: return tounicode(self._readable)
2012-06-29T16:55:50+00:00 app[web.1]: [n.drop_tree() for n in self._should_drop]
2012-06-29T16:55:50+00:00 app[web.1]: value = self.fget(inst)
2012-06-29T16:55:50+00:00 app[web.1]: File "/app/.heroku/venv/lib/python2.7/site-packages/breadability/utils.py", line 55, in get
2012-06-29T16:55:50+00:00 app[web.1]: File "/app/.heroku/venv/lib/python2.7/site-packages/breadability/readable.py", line 436, in _readable
2012-06-29T16:55:50+00:00 app[web.1]: File "/app/.heroku/venv/lib/python2.7/site-packages/lxml/html/init.py", line 169, in drop_tree
2012-06-29T16:55:50+00:00 app[web.1]: assert parent is not None

http://ramblings.timgolden.me.uk/2012/05/06/the-joys-of-open-source/

The heading bits should be removed

The comment blocks at the end should be removed.

http://pythonhosted.org/juju-deployer/

ends up with no content. We should check if the page is html and if we can't find any candidates maybe default to the content as a whole.

http://sportsillustrated.cnn.com/2012/football/nfl/06/15/saints.bounty.investigation.ap/index.html?sct=hp_t2_a3&eref=sihp

Are you planning to issue a Python 3.7 compatible release in a near future?

I am the Debian maintainer of breadability. For now we are based on 0.1.20, and I'm not sure how compatible this version is with recent Python 3 releases.

Thanks for your work.

Blatant rip off of ticket from buriy/python-readability#43

While attempting to use breadability to parse this page: http://bgr.com/2013/10/08/iphone-6-specs-display-resolution-size/

With something like:

url = 'http://bgr.com/2013/10/08/iphone-6-specs-display-resolution-size/'
content = requests.get(url).content
post = readable.Article(content, url)

I end up with post.dom == None.

I traced this down to OriginalDocument.dom where the character encoding is strictly enforced which results in failure if there is even a single invalid character.

I propose that either the decoding be switched to errors='ignore' in this block:

    @cached_property
    def dom(self):
        """Parsed HTML document from the input."""
        html = self._html
        if not isinstance(html, unicode):
            encoding = determine_encoding(html)
            html = html.decode(encoding)

        html = convert_breaks_to_paragraphs(html)
        document = build_document(html, self._url)

        return document

Or that a parameter be added to Article, OriginalDocument to relax the requirement.

I set up an example on runnable: http://runnable.com/Uw-D11n75cQLKkI_/breadability-parse-failure-for-python

I'll gladly do this and submit a pull request - let me know how you would prefer it to be implemented.

page_unicode = page.decode(enc, 'replace')

looks like enc came out to be none in one query.

I installed it successfully with pip in a clean virtualenv (with --no-site-packages).
I didn't run it because it uses argparse, but argparse isn't present in requirements.txt. Please add argparse to the requirements.
Python 2.6, Debian squeeze.

The following link under bmark.us is missing the section headings:

View readable content http://www.theilluminerdy.com/2014/08/20/gumshoe-adventure-master-class/

See http://www.wired.com/wiredenterprise/2012/10/ff-inside-google-data-center/all/

The images are loaded as required. You can check for the lazy-data for the real url and pull that into play as the adjusted src attribute.

http://sweetshark.livejournal.com/11564.html

http://www.itnews.com.au/News/355204,why-paypal-chose-openstack.aspx shows the wrong content

@miso-belica pointed out that the tests for Bookie are failing. That's part of the issue. The other part is that Bookie is not currently able to build because certain packages have aged out and are no longer available. I don't remember the specifics at the moment so I can't elaborate more without re-trying the build process. What I remember though is I got stuck while trying to modernize the packages with later versions.

Bookmarked http://www.businessinsider.com/tech-ceos-favorite-productivity-hacks-2013-8?op=1 and clicked on the "View Readable Content" button.

Results are here: https://bmark.us/bmark/readable/885acc8f0d9299

Not all of the images made it across. Wondering what happened.

breadability ships a number of documents for testing that are possibly restricted because they are copyrighted and not BSD-licensed.

Would it be possible to use alternatives, or possibly fetch them during the test process so they don't have to be distributed?

This makes it harder to package breadability for Debian.

There are a lot of nodes that should be removed during the prep_article phase. It seems only a few nodes are getting hit in the iter() loop there while it should be each node from the winning dom element down.

Not sure why this is, but it's preventing us from getting really clean docs.

http://www.nytimes.com/2013/01/25/sports/25iht-sumo25.html?_r=2&

The first paragraph is cut out. The html has multiple articleBody css classes and we could be using that to help keep the content.

Want to try to keep things consistent for ease of development.

Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in *bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(_self.__args, _self.__kwargs)
File "scripts/readability/existing.py", line 65, in fetch_content
read = ReadUrl.parse(url)
File "/home/rharding/src/bookie/bookie/lib/readable.py", line 171, in parse
if not document.readable:
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/utils.py", line 55, in __get
value = self.fget(inst)
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/readable.py", line 426, in readable
return tounicode(self._readable)
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/utils.py", line 55, in get
value = self.fget(inst)
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/readable.py", line 431, in _readable
if self.candidates:
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/utils.py", line 55, in get
value = self.fget(inst)
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/readable.py", line 419, in candidates
doc = self.doc
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/utils.py", line 55, in get
value = self.fget(inst)
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/readable.py", line 409, in doc
doc = self.orig.html
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/utils.py", line 55, in get
value = self.fget(inst)
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/document.py", line 93, in html
return self._parse(self.orig_html)
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/document.py", line 80, in _parse
doc = build_doc(html)
File "/home/rharding/src/bookie/local/lib/python2.7/site-packages/breadability/document.py", line 54, in build_doc
page_unicode = page.decode(enc, 'replace')
TypeError: decode() argument 1 must be string, not None