zinking / bgweb Goto Github PK

What steps will reproduce the problem?
1. some stupid titles will cause the table not display correctly
2. some stupid titles are very very long, should be limited
3.

What is the expected output? What do you see instead?
stupid titles like this:
         厦门大学2010年研究生迎新晚会10月17日晚七点隆重上演，让我们.....
stupid titles which contain html tags :
         <a>what a ,<!--...

Please use labels and text to provide additional information.
title html should be unescaped properly
title length should be limited.

What steps will reproduce the problem?
1. some sites with status code 3 will not be spidered
2. caused by historic reasons
3.

What is the expected output? What do you see instead?
all sites correctly displayed.

Please use labels and text to provide additional information.
all parsing engines are turning to regular expression parsers
these problems will be fixed.

What steps will reproduce the problem?
1. parsing a gb2312 encoded page
2. render the page using corresponding GBK
3. generally OK, but still not function ok on some special characters.

What is the expected output? What do you see instead?
no more errors on the detail page
currently 5.2% error occured during requests.


Please use labels and text to provide additional information.
seen a GBK character filter package on mico-blog project
use this filter to filter all encoding and convert encoding all to UTF-8

What steps will reproduce the problem?
1. I guess the script size is too large now
2.
3.

What is the expected output? What do you see instead?


Please use labels and text to provide additional information.

Combining combined-en.js...
  Running yuicompressor... 72791 bytes
Combining combined-content.css...
  Running yuicompressor... 14959 bytes
Combining combined-content.js...
  Running yuicompressor... 114784 byte
Combining combined-frame-style.css...
  Running yuicompressor... 9070 bytes
Combining combined-jquery-toolkit.js..
  Running yuicompressor... 96676 bytes
Combining content-xlayout.css...
  Running yuicompressor... 14411 bytes

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?
should be shortned

Please use labels and text to provide additional information.

1. Ad System
2. Ad design
3. debug views

4. A rewrite of current verison using nonrel

What steps will reproduce the problem?
1. buaa parse configuration
2. harvestlink
3. no content actually generated

What is the expected output? What do you see instead?
failed actually but engine did not rpoduce error log

DEBUG    2010-10-19 22:39:37,743 bbs_parser.py:238] Successfully parsing 
school:buaa costing 309 milliseconds;

What steps will reproduce the problem?
1. the log errors report the ZJU site structure changed
2. but the site is keeping update, merely temporarily unavailable
3.
not reproduced

What is the expected output? What do you see instead?

could not open ZJU....

failed to parse bbs by RE parser; schoolname= zju
failed to parse required content SITE structure changed; schoolname= zju




Please use labels and text to provide additional information.

What steps will reproduce the problem?
1. croning links from various bbses
2. not reproducing all the time, because some bbs sytem go down casually
3. generally are deadline exceeded error. some casual request aborted.

What is the expected output? What do you see instead?
no more cron failures, links can be updated in time.
29% cron failures at current.

Please use labels and text to provide additional information.
find algorithms to test the health of target server, and harvestlinks 
accordingly

What steps will reproduce the problem?
1. failed to login to specific mblog
2.
3.

What is the expected output? What do you see instead?
AttributeError: 'Microblog' object has no attribute 'sinahttp'

Please use labels and text to provide additional information.

  File "/base/data/home/apps/bbstop10/2.344140536702112663/pageharvest/bbs_parser.py", line 123, in login_to_sina_microblog
    self.sinacookie = Cookie.SimpleCookie(self.sinahttp.headers.get('set-cookie', ''));
AttributeError: 'Microblog' object has no attribute 'sinahttp'

What steps will reproduce the problem?
1. parsing zju pic board probably
2.
3.

What is the expected output? What do you see instead?
do not understand where is the problem

Traceback (most recent call last):
  File "F:\WorkSpace\zihigh\imgwall\pageharvest\img_parser.py", li
    tn = self.saveParsedImagePage( id,purl,title,c,cc);
  File "F:\WorkSpace\zihigh\imgwall\pageharvest\img_parser.py", li
    (imglist, tn ) = self.parseImageConifgedPage( c, cc, purl );

Please use labels and text to provide additional information.