xiaoleiy / letlink-crawler Goto Github PK

Web Server: Tomcat
OS: Ubuntu Linux server
Techs: jQuery, JS, Ajax, css, monitoring tools
Additional struts action classes should also be developed to react to the web 
client.

This issue item will be my TODO list, which will contain web-crawler related 
and other tech related tasks together with schedules.

Initial Idea on the DB Design:

+------------------------------------+
| domains                         |
| frontierReportMonitor       |
| fullharvests                     |
| global_crawler_trap_expressions |
| global_crawler_trap_lists  |
| harvestdefinitions            |
| historyinfo                      |
| jobs                               |
| partialharvests                |
| runningJobsHistory         |
| runningJobsMonitor         |
| schedules                       |
| schemaversions              |
| seedlists                         |
+-------------------------------------+

domains:        id, name, alias, desc, isCrawling (if being crawled, cannot modify 
it)
excluded_url: id, name, domain_id, desc, isActive
seedslist:      id, name, desc, seeds(a list of domains)
cron:
crawler:            id, name, crawler_id, domain_id, excluded_url, cron, 
max_bytes_download, max_image_download, max_time_seconds, max_threads, 
activated, desc

job:                id, name, crawler_id, start_time, end_time, submit_time, priority, 
seedlist_id, status, crawling_error, crawling_error_details, 
image_upload_error, image_upload_error_details, bak_col01, bak_col02, bak_col03

image_info(raw image information):  id, filename (same as image name in located 
web page), storage_path, image_url, page_url, pixels, size, format
image_stat(statistics data):    id, crawler_id, job_id, seedlist_id, domain_id, 
crawled_time