ishtartang / lofterspider Goto Github PK
View Code? Open in Web Editor NEWlofter的爬虫,爬所有点过的喜欢/推荐/tag、爬取个人主页。
lofter的爬虫,爬所有点过的喜欢/推荐/tag、爬取个人主页。
我在用通用模板的时候碰到了这个lxml.etree.SerialisationError: IO_ENCODER
,
搜了一下,在parse_template.py
的第35行的etree.tostring(temp)
里面加了一个encoding='utf-8'
解决了问题:
open("test.html", "w", encoding="utf-8").write(etree.tostring(parse,encoding='utf-8').decode("utf-8"))
因为太短了就不提merge了
Traceback (most recent call last):
File "D:\PythonFiles\lofterSpider-master\l10_blogs_txt.py", line 111, in
archives_info = save_files(blog_urls)
File "D:\PythonFiles\lofterSpider-master\l10_blogs_txt.py", line 77, in save_files
time_and_title = get_time_and_title(blog_url, author_page_parse)
File "D:\PythonFiles\lofterSpider-master\l10_blogs_txt.py", line 24, in get_time_and_title
author_id = author_page_parse.xpath("//body/iframe[@id='control_frame']/@src")[0].split("blogId=")[1]
IndexError: list index out of range
l4.py下载lofter博主所有图片,下载完删除所有归档信息,如果lofter博主有更新,得重新“全部”下载了?
我看程序可以单独设置时间段,可以保存当前是时间点吗,以后有博主有更新,从上次时间点下载。
Traceback (most recent call last):
File "F:\lofterSpider-master\l9_author_txt.py", line 472, in
run(author_url, get_comm, additional_break, start_time, end_time, chapter_merge_title, additional_chapter_index)
File "F:\llofterSpider-master\l9_author_txt.py", line 273, in run
author_id = author_page_parse.xpath("//body//iframe[@id='control_frame']/@src")[0].split("blogId=")[1]
IndexError: list index out of range
更新了也不行,无法下载
链接 https://weitayiji.lofter.com/
报错
Traceback (most recent call last):
File "E:\py\lofterSpider-master\l9_author_txt.py", line 472, in
run(author_url, get_comm, additional_break, start_time, end_time, chapter_merge_title, additional_chapter_index)
File "E:\py\lofterSpider-master\l9_author_txt.py", line 273, in run
author_id = author_page_parse.xpath("//body/iframe[@id='control_frame']/@src")[0].split("blogId=")[1]
IndexError: list index out of range
认真地拜读了您的这个项目
您为每一个功能都详细地解说,做了辛勤的工作,再此首先感谢
我下午试用了本项目的python脚本
发现当爬取某一个博主的所有作品时,如果其作品数非常多的话
本脚本就会没爬取到,结果是没有爬到照片
我试了另一个作者的作品
发现其脚本能够克服以上问题
在此贴出出处希望可以给您提供参考,改进本项目
https://github.com/Litreily/capturer
https://www.litreily.top/2018/03/17/lofter/
因为我觉得您的项目提供了更为丰富的功能
前文的作者只提供了lofter-图片这一功能
而您的项目提供了文字、图片的功能
一旦前文的问题得以改进
一定是一个更好的项目
所以才敢大胆的跟您提出以上想法
前文提到的那个项目,只能实现lofter的图片保存
如果您能克服爬取丢失的bug,而且又能覆盖文字、图片、视频
那么就会成为关于lofter最全面的项目了
这也是我发这个issue的原因和动机
最后再次感谢您做出的辛勤工作
您好,可能说的不太清楚,但是有些文章是包含图片和文字的,是否能够将收录模式更改成一篇文章是一个文件夹,里面包括图片和txt呢?我们正在进行资源站建设所以量比较大……就用了您的爬虫啦,非常感谢。有些太太授权之后也无法提供所有的原文了,所以图片和 txt 分开收录,到时候拼在一起也挺麻烦的。不知道您是否有时间进行更新?
https://msy60048.lofter.com/view
目前遇到困难的是收录这位太太的文章,您可以看一下。
顺便,好兆头我也磕一口!
你好,谢谢写了这个!
我发现在下载作者博客图片以后图片的顺序不是原博客作者贴的顺序,请问有办法解决嘛?
下载的是 https://manon09.lofter.com/ 里的所有图片,用了l4_author_img.py
。
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.