tencentyun / cos-python-sdk-v5

Qcloud COSv5 SDK

介绍

腾讯云COSV5Python SDK, 目前可以支持Python2.6与Python2.7以及Python3.x。

安装指南

使用pip安装 :

pip install -U cos-python-sdk-v5

手动安装:

python setup.py install

使用方法

使用python sdk，参照 https://github.com/tencentyun/cos-python-sdk-v5/blob/master/demo/demo.py

cos最新可用地域，参照 https://cloud.tencent.com/document/product/436/6224

python sdk 快速入门，参照 https://cloud.tencent.com/document/product/436/12269

python sdk 接口文档，参照 https://cloud.tencent.com/document/product/436/12270

cos-python-sdk-v5's Issues

为什么readme和demo.py的配置和参数里面去掉了appid?

为什么readme和demo.py的配置和参数里面去掉了appid?
我还以为升级了，结果不带appid就说请求无效。
真的不要坑人啊。

使用pip install -U cos-python-sdk-v5安装完成后出现import错误

使用上述命令安装完成后导入时出现下面错误：


这是什么问题呢

get_auth() error in latest version

Python Version: 2.7.18
SDK Version: 1.9.15

---

if headers contains unicode key, get_auth() will throw error TypeError: descriptor 'lower' requires a 'str' object but received a 'unicode'.

Code:

    sign_str = COS_CLIENT.get_auth(Method=method,
                                   Bucket=COS_BUCKET,
                                   Key=path,
                                   Headers={u'Content-Length': 10607}, # Cause error
                                   Params=params,
                                   Expired=expird)

Log:

---

Old version(1.6.5) works fine.

批量上传没有么？

for 循环调用 put_object_from_local_file 会不会太傻了点。

Please support anonymous mode in client

分段上传的返回结果中，Content-Length为0

代码如下

...
upload_response = self.cos_client.upload_file_from_buffer(
                Bucket=bucket_name, Key=key, Body=file, MAXThread=os.cpu_count(),
                ContentType=file.content_type, ContentDisposition=f'attachment; filename="{file.filename}"')
...

得到的返回结果(upload_response)中，Content-Length值为0

import CosS3Client 报错

安装：
C:\Python27>python -m pip install cos-python-sdk-v5
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already satisfied: cos-python-sdk-v5 in c:\python27\lib\site-packages (1.9.0)
Requirement already satisfied: dicttoxml in c:\python27\lib\site-packages (from cos-python-sdk-v5) (1.7.4)
Requirement already satisfied: six in c:\python27\lib\site-packages (from cos-python-sdk-v5) (1.15.0)
Requirement already satisfied: requests>=2.8 in c:\python27\lib\site-packages (from cos-python-sdk-v5) (2.25.0)
Requirement already satisfied: pycryptodome in c:\python27\lib\site-packages (from cos-python-sdk-v5) (3.9.9)
Requirement already satisfied: crcmod in c:\python27\lib\site-packages (from cos-python-sdk-v5) (1.7)
Requirement already satisfied: certifi>=2017.4.17 in c:\python27\lib\site-packages (from requests>=2.8->cos-python-sdk-v5) (2020.11.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\python27\lib\site-packages (from requests>=2.8->cos-python-sdk-v5) (1.26.2)
Requirement already satisfied: idna<3,>=2.5 in c:\python27\lib\site-packages (from requests>=2.8->cos-python-sdk-v5) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in c:\python27\lib\site-packages (from requests>=2.8->cos-python-sdk-v5) (3.0.4)

报错信息 ：
from qcloud_cos import CosS3Client
ImportError: cannot import name CosS3Client

url参数中出现 / 字符时，签名错误，q-signature not matched

例如 list_objects 接口，Prefix 参数为"aaa/aaa"时，报错。
猜测原因为auth需要全小写，而 / 转义之后为 %2F 导致。

No Module Named cos_client

按说明安装后提示找不到cos_client模块

安装命令：pip install -U cos-python-sdk-v5

安装完成后显示安装成功，导入时提示找不到对应的模块：

$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from cos_client import CosS3Client
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named cos_client

UnicodeDecodeError: 'gbk' codec can't decode byte 0xae in position 360: illegal multibyte sequence

When trying pip install -U cos-python-sdk-v5 on Windows 10 with python 3.7, the cmd gave the following error.

ERROR: Command errored out with exit status 1:
     command: 'd:\xxx\venv\scripts\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\xxx\\AppData\\Local\\Temp\\pip-install-93tth04c\\cos-python-sdk-v5\\setup.py'"'"'; __file__='"'"'C:\\Users\\xxx\\AppData\\Local\\Temp\\pip-install-93tth04c\\cos-python-sdk-v5\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\xxx\AppData\Local\Temp\pip-pip-egg-info-xbraocxr'
         cwd: C:\Users\xxx\AppData\Local\Temp\pip-install-93tth04c\cos-python-sdk-v5\
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\xxx\AppData\Local\Temp\pip-install-93tth04c\cos-python-sdk-v5\setup.py", line 25, in <module>
        long_description=long_description(),
      File "C:\Users\xxx\AppData\Local\Temp\pip-install-93tth04c\cos-python-sdk-v5\setup.py", line 14, in long_description
        return fileobj.read()
    UnicodeDecodeError: 'gbk' codec can't decode byte 0xae in position 360: illegal multibyte sequence
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

看完SDK和接口，发现不支持创建目录

不太确定是不是漏看了，所以确认下。

通篇没有找到关于 bucket 如何创建一个目录的操作。

download_file work with SimpleThreadPool keep reporting: [WARNING ] Connection pool is full, discarding connection

Issue:
when you use batch download reference code from doc with default value something like:
config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token)
client = CosS3Client(config)
pool = SimpleThreadPool()
pool.add_task(client.download_file, test_bucket, file_cos_key, localName)
the requests lib will keep reporting:
[WARNING ] Connection pool is full, discarding connection

according to the doc: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#customizing-pool-behavior
it will not impact/block the connections if param block is not set to block=True, BUT it will impact the connection performance so per the doc words:
If youare making many requests to the same host simultaneously it might improveperformance to increase this number.

Possible RC:
Due to num_threads param of SimpleThreadPool default value is 5 and MAXThread param of ResumableDownLoader default value is 5.
So if Im not wrong the PoolMaxSize param of CosConfig default value should be 25.
BUT now the PoolMaxSize param of CosConfig default value is 10, so it will keep alerting: [WARNING ] Connection pool is full, discarding connection and maybe also impacting the connection performance but honestly I didn't feel any obviously slow-down. BUT it indeed increase a lot of useless log prints...

Possible Solution
Set CosConfig class's PoolMaxSize default value to 25.

how to delete a bucket which is not empty

Once I tried to delete a bucket, it returned 'The bucket you tried to delete is not empty'.

So I have to scan all the keys and delete all the bucket objects, finally I can delete the bucket ?

Is there a param that I can pass to delete_bucket() method, such as Recursive=True ?

使用 object_exists 时预期的 404 错误，在日志中仍然打出 error

我们这里有个需求会对日志 error 进行监控并上报，但是在调用 object_exists 方法时，会在日志中记录 error。翻看代码发现是由于 send_request 中特殊处理了。如下

        if res.status_code >= 400:  # 所有的4XX,5XX都认为是COSServiceError
            if method == 'HEAD' and res.status_code == 404:   # Head 需要处理
                info = dict()
                info['code'] = 'NoSuchResource'
                info['message'] = 'The Resource You Head Not Exist'
                info['resource'] = url
                if 'x-cos-request-id' in res.headers:
                    info['requestid'] = res.headers['x-cos-request-id']
                if 'x-cos-trace-id' in res.headers:
                    info['traceid'] = res.headers['x-cos-trace-id']
                logger.error(info)
                raise CosServiceError(method, info, res.status_code)
            else:
                msg = res.text
                if msg == u'':  # 服务器没有返回Error Body时 给出头部的信息
                    msg = res.headers
                logger.error(msg)
                raise CosServiceError(method, msg, res.status_code)

但实际 head 404 这个是预期范围内的，代表文件不存在，而且，已经 raise CosServiceError 到上层，这个地方记录的日志等级过高，能否改为 warn 或者 info。

为什么没有创建文件夹的接口？

如题

python版本 没有临时凭证的用法?

ImportError: No module named urllib.parse

# python 
Python 2.6.6 (r266:84292, Dec  7 2011, 20:48:22) 
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from qcloud_cos import CosConfig 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "qcloud_cos/__init__.py", line 1, in <module>
    from .cos_client import CosS3Client
  File "qcloud_cos/cos_client.py", line 14, in <module>
    from six.moves.urllib.parse import quote, unquote
ImportError: No module named urllib.parse
>>> 

预签名URL未处理临时秘钥使用的token

各个SDK函数对CosS3Client的实现不太一致，希望对获取预签名URL的相关函数，如

cos-python-sdk-v5/qcloud_cos/cos_client.py

Line 484 in 718ab50

def get_presigned_url(self, Bucket, Key, Method, Expired=300, Params={}, Headers={}):

添加类似于

cos-python-sdk-v5/qcloud_cos/cos_client.py

Lines 235 to 236 in 718ab50

	if self._conf._token is not None:
	kwargs['headers']['x-cos-security-token'] = self._conf._token

的token支持，不然使用临时密钥还需要自己手动添加，很麻烦

def getFileResponse(filekey):
    tmpkey = object_get_tmpkey(filekey)
    config = CosConfig(
        Region='ap-shanghai',
        SecretId=tmpkey['credentials']['tmpSecretId'],
        SecretKey=tmpkey['credentials']['tmpSecretKey'],
        Token=tmpkey['credentials']['sessionToken'],
        Scheme='http'
    )
    client = CosS3Client(config)
    return client.get_presigned_url(
        Bucket='xxxxx-xxxxxxxxxx',
        Key=filekey,
        Method='GET',
    ) + '&x-cos-security-token=' + tmpkey['credentials']['sessionToken']

[FOR CSP]client.put_bucket_cors(), 参数MaxAgeSecond可传入string类型

put_bucket_cors方法MaxAgeSecond参数可传入string类型的值，传入参数的数据类型SDK中是否要做判断？

[FOR CSP]client.complete_multipart_upload()返回值类型为CaseInsensitiveDict

虽然对SDK使用没有太大影响，但是SDK文档给出的返回值类型是Dict

在Jupyter 上使用 的 时候会让print 函数失效

我在Jupyter 上使用， from qcloud_cos import CosConfig， 然后我的print函数就失效了。 是不是你们做了重定向？

windows上使用upload_file

请问代码如何确定文件上传成功呢？

如题，谢谢

Missing http Host header

url = self._conf.uri(bucket=Bucket, path=Key) host = self._conf.uri(bucket=Bucket) host = host.strip('http://') headers['Host'] = host
I have to write a subclass and add host header for all most every request.
Hope to fix this bug~

有没有帮助开发cos-python-sdk 的文档？

既然是开源， 想必是想让大家一起参与，如果能有开发指引最好了，（不是指sdk的使用文档）

logger.warn DeprecationWarning: The 'warn' method is deprecated

lib/python3.9/site-packages/qcloud_cos/cos_client.py:335: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    logger.warn(info)

https://github.com/tencentyun/cos-python-sdk-v5/search?q=.warn

如何获取bucket下所有的objects方法(前提不知道key的具体名称)

腾讯官网的方法不完整，只是有请求实例，但是如何调取的压根没有说明
response = client.list_objects(
Bucket='test01-123456789',
Prefix='string',
Delimiter='/',
Marker='string',
MaxKeys=100,
EncodingType='url'
)

旁友 你参数写错了

config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token) # 获取配置对象
===》
config = CosConfig(Region=region, Secret_id=secret_id, Secret_key=secret_key, Token=token) # 获取配置对象

[bug] "You are denied by bucket referer rule" when get_object

when "HotLink Protection" feature is enabled, objects cannot be fetched with the domain of <Bucket>.cos.<Region>.myqcloud.com, so method get_object will fail as a result.

one solution is to add a header "Referer: a.refer.with.valid.domain".
in Java SDK v4 it will be

getFileLocalRequest.setReferer("http://xxx.com");

everything works as is.

but there is no similar interface in the v5 sdk, and add arbitrary header to cos request is forbidden.
so I have to hack the code like this:

    from qcloud_cos import cos_comm
    cos_comm.maplist["Referer"] = "Referer"
    cos_client.get_object(
        Bucket=cos_bucket,
        Key=key,
        Referer="http://xxx.com"
    )

安装报错

Collecting cos-python-sdk-v5
Downloading https://pypi.doubanio.com/packages/30/95/cc594ff07c7663a38d8ba3393cfe38657b2cb9f2b5762f5db634bf4d60c3/cos-python-sdk-v5-1.8.7.tar.gz (41kB)
100% |████████████████████████████████| 51kB 855kB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\hank\AppData\Local\Temp\pip-install-cejqcd0p\cos-python-sdk-v5\setup.py", line 25, in
long_description=long_description(),
File "C:\Users\hank\AppData\Local\Temp\pip-install-cejqcd0p\cos-python-sdk-v5\setup.py", line 14, in long_description
return fileobj.read()
UnicodeDecodeError: 'gbk' codec can't decode byte 0xae in position 360: illegal multibyte sequence

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in C:\Users\hank\AppData\Local\Temp\pip-install-cejqcd0p\cos-python-sdk-v5\

安装sdk时报错，windows10,python=3.7(Mac系统下正常)

how can i put object with tagging?

demo.py中的ETag存在KeyError

当前采用python 2.7的时候为etags，没有大写字母。

文档里单词写错了。。。

client写成clieent了
这么看来你们文档里的代码肯定没跑过。。。。

原文节选

# 9. 列出分片
response = clieent.list_parts(
    Bucket='test01-123456789',
    Key='mutilpartfile.txt',
    UploadId=uploadid
)
lst = response['Part'] # list_parts最大数量为1000

windows上使用upload_file()函数出现content-length缺少的问题

MissingContentLength You must provide the Content-Length HTTP header. NWFlYWE4ZTZfNWJhYTk0MGFfMWU3N182MzE2ZDA= OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODc0OWRkZjk0ZDM1NmI1M2E 2MTRlY2MzZDhmNmI5MWI1OTY4OGQ5OWY4YWFhNjAzOTkyNDJhZmQyOTk1YWVmOWFlNWE0MDE1NTNhODY 3MTdhNDNlZTk2ZThhZmMxMDJkZWM=

同样的代码，在mac上运行可以正常使用，在window上就就缺少content-length

提交了一个python3 支持的版本

所有方法和接口都没变，只是做了python2 到python3 的兼容性处理。初步使用无bug。有bug欢迎提issue。

此版本只支持python3.

#42
https://github.com/xfdywy/qcloud_cos_py3

Python 2.7.16 AttributeError: 'CosS3Client' object has no attribute 'get_object_url'

python2.7 好像并未支持 client.get_object_url
python3.8.5 调用正常。

python3语法bug

cos_client.py文件第2497行
if data == ""在python3下当数据读取完时，data为b''，而b'' == “”返回False

如何关闭sdk自带的log

因为我自己有log了，所以不想让sdk再打印，怎样关闭呢？

这东西下载了怎么用？怎么设置ak sk之类？

README希望详细点

禁用aksk后，接口仍然概率性可用

禁用主账号aksk后的15分钟内，使用sdk调用list_objects、object_exists等接口，仍然可以概率性调用成功，10次调用5、6次可以调用成功。

Please add support for PAC type proxy and ntlm authorization

Currently sdk only support fixed IP proxy and username/pwd authorization.
however in some global etnerprise, use a dynamic PAC list as proxy setting.
so please add support for this scenario, thanks.

sample code:
from requests_ntlm3 import HttpNtlmAuth
from pypac import PACSession, get_pac

auth=HttpNtlmAuth("proxyUserName","proxyPassword")
pac = get_pac(url="http://pacserver.mycompany.com/config.pac")
s = PACSession(pac)
s.auth =auth
r = s.post(config["http://a.com/sample/,json =pload)

请教一下，怎么删除bucket内的文件夹

如题

Missing required header for this request: Appid

这个错误是什么意思？？？

如何实现图片缩放功能？

https://cloud.tencent.com/document/product/460/36540

client.get_object 是否支持imageMogr2的传参呢？试了一下没有试出来，没有相关的文档

请求签名中 StartTimestamp 的实现减了 60

cos-python-sdk-v5/qcloud_cos/cos_auth.py

Line 72 in 49dc6d4

sign_time = "{bg_time};{ed_time}".format(bg_time=start_sign_time - 60, ed_time=start_sign_time + self._expire)

根据文档 https://cloud.tencent.com/document/product/436/7778#.E6.AD.A5.E9.AA.A41.EF.BC.9A.E7.94.9F.E6.88.90-keytime ，KeyTime 中的 StartTimestamp 应为当前时间戳，而代码中的实现却是当前时间戳减了 60；但是其他语言的 SDK 并没有减 60

请支持Python3

如题，Python3已经被大规模使用，请提供支持。

涉及到bucket的接口在构造bucket时的问题

cos_comm.py的format_bucket函数中,最后对bucket格式的判断为<bucket_name>-< appID>,但实际应为< appID>-<bucket_name>,所以在传入正确的bucket之后,bucket会被转换成< appID>-<bucket_name>-< appID>,导致请求失败.
另:Python SDK的文档中也有部分描述为<bucket_name>-< appID>.

请问中文文件如何处理？API 报错

D:\workspace\blog\source\tools\2020-01-08-╞Σ╦√╙δ╞Σ╦ⁿ╡─╟°▒≡.md
<type 'str'>
Traceback (most recent call last):
  File "Tencent_cos.py", line 121, in <module>
    main(sys.argv)
  File "Tencent_cos.py", line 117, in main
    cosoper.upload_traversal("", client)
  File "Tencent_cos.py", line 91, in upload_traversal
    MAXThread = maxthread
  File "D:\env_released_1.1.3\env\tools\Python27\lib\site-packages\qcloud_cos\cos_client.py", line 2850, in upload_file
    rt = self.put_object(Bucket=Bucket, Key=Key, Body=fp, EnableMD5=EnableMD5, **kwargs)
  File "D:\env_released_1.1.3\env\tools\Python27\lib\site-packages\qcloud_cos\cos_client.py", line 303, in put_object
    url = self._conf.uri(bucket=Bucket, path=Key)
  File "D:\env_released_1.1.3\env\tools\Python27\lib\site-packages\qcloud_cos\cos_client.py", line 121, in uri
    path = to_unicode(path)
  File "D:\env_released_1.1.3\env\tools\Python27\lib\site-packages\qcloud_cos\cos_comm.py", line 69, in to_unicode
    raise CosClientError('your bytes strings can not be decoded in utf8, utf8 support only!')
qcloud_cos.cos_exception.CosClientError: your bytes strings can not be decoded in utf8, utf8 support only!

请问如何列举出一个文件夹下的所有内容

最大数量为1k

但是数量超过了1k，如何处理？

[FOR CSP]client.copy_object(...) raise CosServiceError: 404 NoSuchBucket

cos-python-sdk-v5 在TCE-COS(CSP)上进行copy_object操作时，出现404 NoSuchBucket的错误。


实际是存在这个bucket的