ogslp / kuaishou-crawler Goto Github PK

View Code? Open in Web Editor NEW

179.0 179.0 65.0 12.64 MB

As you can see, a kuaishou crawler

License: MIT License

Python 100.00%

crawler kuaishou kuaishou-crawler

kuaishou-crawler's People

Contributors

Stargazers

Watchers

Forkers

yqbboy github2017luo barnett2010 aliweiya sirwgame qq809736555 lanchaohao superman3125 freedom2022 wemecan rip-github congtoubyby chengzhangz thunderlm gregoriusxu biubiuww taozi00 wx7614140 tradaniel usbiz mochp psxl1066 fgyfn linxiyifan lpzi1024 honeybeb shiyongde sxyseo moliqing2023 yfshy mlzboy collectbot jfmo sayhello023 mfcer110 c1a1o1 supervisoredis gasbarroni8 boshi18 liaol devenlu cursorhu zengbybet zhoghua123 ttraveller2055 849319996 yang-xiansen woshangfenglove zhangsai521314 stone009 xlizzard zhenyizhao jiyuefeng leisurelyrn g-vision aihacker qzone49 wangmiemie heoolaya ziyuanguole igooglevip aprilrain0505 ctkqiang muskgit

kuaishou-crawler's Issues

亲测可用无水印，不过did是F12查到的，浏览器url没有did字段，方法如下：

网页打开要下载的快手用户的任意视频，F12 -> Application栏 -> Cookies项展开，可以看到did字段的值是 web_xxxxx，用户id去手机APP看

一直显示list index out of range，重新设置cookies也没用

设置了did值，用了刚登陆的cookies，依然提示list index out of range，想不到方法解决了，有人和我一样吗？

Traceback (most recent call last):
File "ks.py", line 28, in
File "ks.py", line 22, in crawl
File "lib\crawler.py", line 73, in crawl
File "lib\crawler.py", line 113, in __crawl_user
File "lib\crawler.py", line 167, in __crawl_work
AttributeError: 'NoneType' object has no attribute 'group'
[932] Failed to execute script ks

ConnectionError: HTTPConnectionPool(host='tx2.a.yximgs.com', port=80): Max retries exceeded with url: /ufile/atlas/ODgyMDg0MjAzXzE4NTU0ODA0ODE4XzE1NzExNDM1ODMxOTE=_0.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000019CDE5F1D60>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))

抓取某一主题下的前一百位的用户信息，网页版有时候会跳出滑块验证码，请问有办法绕过吗？

问个可能和本项目不相干的问题，请多多指教，谢谢！

比如说我要抓取“手工”主题下的前一百位的用户信息
https://live.kuaishou.com/search/author?keyword=%E6%89%8B%E5%B7%A5&page=1
网页版有时候会跳出滑块验证码，请问有办法绕过吗？

运行报错AttributeError: 'NoneType' object has no attribute 'group'

开始爬取用户 xxx，保存在目录 data/xxx/
共有21个作品
Traceback (most recent call last):
File "D:/develop-py/workspace/kuaishou-crawler-master/crawl.py", line 23, in
crawl()
File "D:/develop-py/workspace/kuaishou-crawler-master/crawl.py", line 19, in crawl
crawler.crawl()
File "D:\develop-py\workspace\kuaishou-crawler-master\lib\crawler.py", line 73, in crawl
self.__crawl_user(uid)
File "D:\develop-py\workspace\kuaishou-crawler-master\lib\crawler.py", line 113, in __crawl_user
self.__crawl_work(dir, works[j], j + 1)
File "D:\develop-py\workspace\kuaishou-crawler-master\lib\crawler.py", line 167, in __crawl_work
v_url = re.search(pattern, html).group(1)+".mp4"
AttributeError: 'NoneType' object has no attribute 'group'

建议web cookie did和 mobile cookie did分开

经过我多天的测试，web did跟mobile did很容易混淆写入代码里面，建议分开配置。

运行.exe输入完主页id后闪退了

获取视频源的时候快手要求验证

访问w_url = WORK_URL + work['id']的时候快手返回的是请验证。

exe 版本会闪退

首先感谢作者。
但是使用过程中发现，下载到 150 多个视频的时候，会自动闪退

首先非常感谢分享这个软件，最近使用发现了两个问题。1.当用户作品数量超过1000时，软件只能显示并爬取到999。2.当爬取到后半阶段时，只能保存图片，而视频作品信息能显示出来，但是爬取不到，显示未找到视频。

自从v0.4.0版本之后无法下载视频

自从v0.4.0版本之后无法下载视频，全部都显示未找到视频，其他版本正常

当快手号为纯数字的时候获取uid失败

关于v0.5.0输入did和用户id之后闪退的问题

打开ks.exe之后，大概下载了60个视频，软件闪退，之后重新打开输入did和用户id，回车之后闪退无法下载，在确保软件完整的情况下，切换过did，尝试过管理员运行以及将ks.exe放在非c盘目录，结果均无效，以下是错误截图:https://s1.ax1x.com/2020/08/11/aXNBSe.png

快手限制下载视频的数量

当爬取数量达到几十个视频之后，就会出现下面的报错，等了很久之后才能继续下载。而且尝试过使用其他ip来继续请求也没用

| kuaishou-crawler (v0.5.0 20-08-06)
| 本程序由oGsLP提供, www.github.com/oGsLP/kuaishou-crawler, 喜欢的话可以给个star >_<

准备开始爬取，共有1个用户...

{"data":{"privateFeeds":{"pcursor":"","list":[],"__typename":"PCProfileFeeds"}}}

[]
Traceback (most recent call last):
File "D:/python_project/test1/crawlers/main.py", line 26, in
main()
File "D:/python_project/test1/crawlers/main.py", line 20, in main
kuaishou.crawler_kuaishou.main()
File "D:\python_project\test1\crawlers\kuaishou\crawler_kuaishou.py", line 30, in main
crawl(param_did,data_dir)
File "D:\python_project\test1\crawlers\kuaishou\crawler_kuaishou.py", line 15, in crawl
crawler.crawl()
File "D:\python_project\test1\crawlers\kuaishou\lib\crawler.py", line 81, in crawl
self.__crawl_user(uid)
File "D:\python_project\test1\crawlers\kuaishou\lib\crawler.py", line 106, in __crawl_user
if works[0]['id'] is None:
IndexError: list index out of range

exe执行总是报错

报错如下，隔几天运行时，一开始可以正常下载，大概下载几十个之后就会报错，然后再次运行一直都是这个报错，需要隔几天后再次运行，如此反复。
Traceback (most recent call last):
File "ks.py", line 28, in
File "ks.py", line 22, in crawl
File "lib\crawler.py", line 73, in crawl
File "lib\crawler.py", line 113, in __crawl_user
File "lib\crawler.py", line 167, in __crawl_work
AttributeError: 'NoneType' object has no attribute 'group'
[19088] Failed to execute script ks

运行报错：AttributeError: 'NoneType' object has no attribute 'group'

Traceback (most recent call last):
File "C:/mypythonfile/car_info/driving_attention_video/kuaishou-crawler/ks.py", line 28, in
crawl()
File "C:/mypythonfile/car_info/driving_attention_video/kuaishou-crawler/ks.py", line 22, in crawl
crawler.crawl()
File "C:\mypythonfile\car_info\driving_attention_video\kuaishou-crawler\lib\crawler.py", line 81, in crawl
self.__crawl_user(uid)
File "C:\mypythonfile\car_info\driving_attention_video\kuaishou-crawler\lib\crawler.py", line 122, in __crawl_user
self.__crawl_work(dir, works[j], j + 1)
File "C:\mypythonfile\car_info\driving_attention_video\kuaishou-crawler\lib\crawler.py", line 178, in __crawl_work
v_url = re.search(pattern, html).group(1)+".mp4"
AttributeError: 'NoneType' object has no attribute 'group'

第一次可以运行，但是下载了十几个视频之后就一直报错

用户cookie中的did值

作者你好我按你的方法获取了did值然后输入了用户的uid
但是出现闪退的情况
我觉得可能我获取错了
可以仔细的给我讲解一下
预先输入本用户cookie中的did值：该填什么吗？