aoaostar / alidrive-uploader Goto Github PK
View Code? Open in Web Editor NEW阿里云盘上传工具
阿里云盘上传工具
希望增加上传完成自动删除本地文件的功能
如题
阿里云CCP(基于OSS)应该是不支持多线程上传的,必须按顺序上传吧.
OSS貌似可以多线程上传
Client.py文件中,第77行判断是否有后台运行参数时,如果存在,立刻删除会导致后面数组访问越界,原始代码如下。
77 def init_command_line_parameter(self):
78 for k in range(len(sys.argv)):
79 if sys.argv[k] == '--resident' or sys.argv[k] == '-r':
80 DATA['config']['RESIDENT'] = True
81 del sys.argv[k]
报错情况为:
Traceback (most recent call last):
File "/root/aliyundrive-uploader-master/main.py", line 26, in
client.init_command_line_parameter()
File "/root/aliyundrive-uploader-master/Client.py", line 77, in init_command_line_parameter
if sys.argv[k] == '--resident' or sys.argv[k] == '-r':
IndexError: list index out of range
修改建议,增加标识位,for循环后再删除,如
label = -1
for k in range(len(sys.argv)):
if sys.argv[k] == '--resident' or sys.argv[k] == '-r':
DATA['config']['RESIDENT'] = True
label = k
if label>-1:
del sys.argv[label]
无法结束当前正在上传文件的任务,用kill之后,重新上传还会执行继续上次未完成的任务。
请问如何正确地终止正在上传的任务?
系统:CenOS
检索目录中
目录id获取成功60898fee4524df8614d74fa6a9df689e35b4ea0a
正在上传: 1%|1 | 53.0M/6.84G [00:30<1:06:53, 1.82MB/s]
Traceback (most recent call last):
File "/root/aliyundrive-uploader/main.py", line 137, in
if res.result():
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 438, in result
return self.__get_result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
raise self._exception
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/root/aliyundrive-uploader/main.py", line 61, in upload_file
drive.upload(upload_url)
File "/root/aliyundrive-uploader/AliyunDrive.py", line 117, in upload
res.raise_for_status()
File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://bj29.cn-beijing.data.alicloudccp.com/xYBXOEkb%2F69848%2F608991ae51a2863ccd5b466b8bf84ab711c83d46%2F608991ae11f21bbc22964fbbaf08e0f2fca9d24f?Expires=1619632062&OSSAccessKeyId=LTAIsE5mAn2F493Q&Signature=IUxxWsQ36QnVxA0W%2FnwWESNHsfs%3D&partNumber=1&uploadId=658982D766A74E5AB6BB7D64A564C38C
我一个1G的文件从Ubuntu 20.04服务器上传到阿里云网盘老是出错,每次都有重新开始传,错误提示:
pipes-presentation-20210423.mp4正在校检文件中,耗时与文件大小有关
文件名:11-pipes-presentation-20210423.mp4
hash:4817bcc8425ac8be2264ff379887a64b0f46e661
文件大小:1134460273
文件路径:youtube/11-pipes-presentation-20210423.mp4
检索目录中
目录id获取成功6073f804da645896ab184012afdfd4e0c7fe9ab3
Traceback (most recent call last):
File "main.py", line 137, in
if res.result():
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
raise self._exception
File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "main.py", line 61, in upload_file
drive.upload(upload_url)
File "/root/aliyundrive-uploader/AliyunDrive.py", line 117, in upload
res.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://bj29.cn-beijing.data.alicloudccp.com/4oW5G1mH%2F27564%2F6088ab6af27e6944c74c4b0aa509838b6182e85d%2F6088ab6a58520b3afd2b480a8a7b020fa7df8982?Expires=1619573114&OSSAccessKeyId=LTAIsE5mAn2F493Q&Signature=xEewGJ4QSlF1Ir8u7%2Bd0biuKyp4%3D&partNumber=1&uploadId=BD00C9F4C9CF4E2B8C71C5620236BB42
有windows 版本的使用说明吗?
使用python3 main.py --resident或者python3 main.py -r,程序还是过了一两秒自动结束了,并没有常驻,这是什么原因呢?
文件名太长,会无法完整显示,上传进度也会不见,
发现上传到阿里云盘里的文件重复了,而且两个文件名还是一样的,按理说一个文件夹应该不能两个完全一样文件名的文件,作者大大能看看这是哪里的问题吗?
我这要改成这样才能正常跑config = json.loads(f.read().decode())
Error: 上传地址已失效正在获取新的上传地址
一直在获取新的token但是无法上传
find = db.table('task').where('filepath=? and realpath=?', (tmp['filepath'], tmp['realpath'],)).find()
if find:
print_info('【%s】已存在任务队列中,跳过' % tmp['filepath'])
我理解是去查询数据库,该文件是否已经上传了,如果上传过了就直接跳过上传。但是如果这样,文件内容修改了也不上传了吗?不太合理吧
另外如果不是常驻,main.py结束的时候会调用suicide函数,但问题是这个退出函数为什么返回1而不是返回0?返回1会导致bash中该执行结果为失败,影响后续脚本的执行
阿里云上传重复文件会自动在后面加日期,希望加一个功能可以把源文件覆盖掉而不是加日期
本地目录中我只想上传特定文件名,如只想上传rar文件,应该在那儿增加限制文件后缀的代码来控制上传文件呀?谢谢
经常会上传到一半,因为断开了链接,就停止了上传
特此留言,表示膜拜
尝试传送一个有70000+文件夹的文件夹,由于会一下把所有文件打开于是too many open files。
另一个问题是,递归创建70000多个文件夹也会失败。但是失败时打印下面的backtrace
正在获取AccessToken
正在生成目录
上传完毕!共计70690个文件,失败文件个数:0个panic: (*logrus.Entry) 0xc0028c1260
goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0028c11f0, 0x0, {0xc000470360, 0x53})
/home/x/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:259 +0x43e
github.com/sirupsen/logrus.(*Entry).Log(0xc0028c11f0, 0x0, {0xc002e17b00?, 0x4?, 0xc002a9ebb0?})
/home/x/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:293 +0x4f
github.com/sirupsen/logrus.(*Logger).Log(0xc0000c0000, 0x0, {0xc002e17b00, 0x1, 0x1})
/home/x/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:198 +0x65
github.com/sirupsen/logrus.(*Logger).Panic(...)
/home/x/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:247
github.com/sirupsen/logrus.Panic(...)
/home/x/go/pkg/mod/github.com/sirupsen/[email protected]/exported.go:129
alidrive_uploader/bootstrap.TreeFolders(0xc0010c3d40, {0x7ffc23388187?, 0xc000284528?}, 0xfe?)
/home/x/Project/alidrive-uploader/bootstrap/drive.go:25 +0x2f8
alidrive_uploader/bootstrap.Run()
/home/x/Project/alidrive-uploader/bootstrap/run.go:87 +0x8d8
main.main()
/home/x/Project/alidrive-uploader/main.go:6 +0x17
win10 编译版本
Traceback (most recent call last):
File "main.py", line 51, in
File "common.py", line 161, in print_info
File "common.py", line 208, in log
UnicodeEncodeError: 'gbk' codec can't encode character '\u200b' in position 69: illegal multibyte sequence
[4044] Failed to execute script 'main' due to unhandled exception!
ACCESS_TOKEN
管理服务Aliyundrive
当前阶段ACCESS_TOKEN
约每2h刷新一次,而本项目关于ACCESS_TOKEN
的管理方案似乎并不科学。
大概看了一下源码,本人认为本项目的ACCESS_TOKEN
处于欠初始化状态。配置文件中只要求开发者(首次)手动获取DRIVE_ID
和REFRESH_TOKEN
(然而完全可以手动初始化ACCESS_TOKEN
因为它们就隔了一行,如README中的截图所示),于是程序运行后(无论ACCESS_TOKEN
是否真的失效),都会在上传任务正式开始前被重新更新一次,并且在后续长段时间内子线程通过强竞争的方式进行”倒计时600s被动更新ACCESS_TOKEN
“的任务。
这会引发一系列难以应付的问题,check_auth()被放置在如“新建网盘目录”等多个核心业务当中。当用户配置多线程下载大量(指向目录的)小文件时,每个“分发线程”都会执行一次ACCESS_TOKEN
更新请求。
此外,ThreadPoolExecutor(max_workers=None, thread_name_prefix='', initializer=None, initargs=())
在 3.5 版本后,如果 max_workers 为 None 或没有指定,将默认为机器处理器的个数而非1;在3.8版本后则默认启动至少5个核心线程min(32, os.cpu_count() + 4)
,所以我们可能消耗了成倍的资源的去”计算“更新时间,程序也会在某个时间点频繁更新权限出现“绊脚”的情况。
使用通信共享内存,而不是使用内存通信。
本项目中写满了子线程的竞争行为,这在批量传输小文件时将带来极大的性能损耗。可以使用gevent
等优秀的协程框架处理密集I/O业务,在虚拟线程阻塞时让出资源,也能更加方便地管理事务回滚以及标记异常实例。
请问能不能跳过验证?直接上传
现在上传文件前都需要检查文件哈希值,如果云端存在该文件,就会秒传。
我会打包并加密上传的照片,因此每个文件必定是独一无二的,因此希望直接跳过哈希计算,直接上传。
由于每次会上传数百G的文件,浪费CPU也浪费电,浪费煤炭,产生二氧化碳,污染环境。。。(喂扯远了。。)
希望能稍微增加一个配置选项。谢谢!
用了宝塔版本的,也用了原版的,都是刚开始有上传,过一会就失败了。小文件能上传成功,大点的视频就不行了。
{'code': 'InvalidParameter.ContentHash', 'message': 'The input parameter content_hash is not valid. file content hash is not matched with content_hash in complete file request'}
{'code': 'ForbiddenNoPermission.File', 'message': 'No Permission to access resource File'}
普通用户会提示权限不足,File "sqlite.py", line 321, in write_lock
with(open(self.__LOCK, 'w+'))as f:
PermissionError: [Errno 13] Permission denied:可是里面的文件所有者以及权限我都修改了,依然这样
首先感谢作者 提供这么好的开源插件,辛苦了 谢谢!
就是提个小建议,在上传命令基础上加一个删除本地文件的参数就更完美了,比如:
python3 main.py -d D:\下载\需要转移的\人人商城
d参数:上次后删除本地文件。因为已经上次到网盘本地存在的价值不大。
正在上传【0138 - 五一假期带孩子出去玩怎么拍美美的亲子照?.mp4】: 100MB [00:00, ?B/s]
{'code': 'InvalidParameter.ContentHash', 'message': 'The input parameter content_hash is not valid. file content hash is not matched with content_hash in complete file request'}
手动改了几次 refresh_token ,但还是一直提示失效
文件名:13.mp4
hash:57C27320D8996AB1F332298B5140687BC668A6
文件大小:529901882
文件路径:/root/13.mp4
=================================================
检索目录中
AccessToken已失效,尝试刷新AccessToken中
refresh_token已经失效
无法刷新AccessToken,准备退出
{'code': 'AccessTokenInvalid', 'message': 'AccessToken is invalid. ErrValidateTokenFailed'}`
能否添加一个排除参数,想github ignore文件一样的功能。
{'code': 'InvalidParameter.FileId', 'message': 'The input parameter file_id is not valid. for cpp path domain file_id is required'}
{'drive_id': 'drive_id', 'file_id': None, 'upload_id': None}
debian 10
之前在另外一个账户可以正常上传,现在我更改另外一个网盘的token和id,弹出如下错误:
{'code': 'ForbiddenNoPermission.File', 'message': 'No Permission to access resource File'}
我原以为这是权限问题,因此使用sudo运行main.py,又出现这个错误:
Traceback (most recent call last):
File "/root/.aliup/main.py", line 12, in <module>
from concurrent.futures import ThreadPoolExecutor
ImportError: No module named concurrent.futures
好奇怪呀..`~~
要设置的参数也不多,希望能直接在命令行中全部设置,而不需要配置文件config.yaml。
或者像aria2一样,所有参数既能在命令行给出,也能在配置文件给出。
比如,./alidrive --id xxx --token xxxx --root-dir -t 3 scr-path des-path
这样一行搞定。
请问使用
python main.py -r D:\123 后不见上传文件
提示:当前无任务,等待新的任务队列中
这个任务来自何方?数据库里的任务内容格式如何写
100G的文件上传显示这个
The input parameter part_info_list is not valid. part count can not over 10000
检索目录中
AccessToken已失效,尝试刷新AccessToken中
refresh_token已经失效
refresh_token已经失效
Traceback (most recent call last):
File "main.py", line 105, in
File "main.py", line 63, in distribute_thread
File "main.py", line 52, in thread
File "Client.py", line 167, in upload_file
File "AliyunDrive.py", line 291, in get_parent_folder_id
File "AliyunDrive.py", line 275, in create_folder
File "AliyunDrive.py", line 334, in check_auth
File "AliyunDrive.py", line 115, in token_refresh
File "AliyunDrive.py", line 109, in token_refresh
File "requests/models.py", line 900, in json
File "json/init.py", line 346, in loads
File "json/decoder.py", line 337, in decode
File "json/decoder.py", line 355, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
[562590] Failed to execute script 'main' due to unhandled exception!
root@Server-4bae148c-dd03-4cfe-ae72-f2f61e41420d:/usr/local/src/ali#
root@Server-4bae148c-dd03-4cfe-ae72-f2f61e41420d:/usr/local/src/ali#
drive_id 怎么获取啊,你下面给你命令没用了
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.