alibaba / acm-sdk-python

Alibaba ACM SDK for Python

User Guide

Introduction

Python SDK for ACM.

Features

1. Get/Publish/Remove config from ACM server use REST API.
2. Watch config changes from server.
3. Auto failover on server failure.
4. TLS supported.
5. Address server supported.
6. Both Alibaba Cloud ACM and Stand-alone deployment supported.

Supported Python：

1. Python 2.6
2. Python 2.7
3. Python 3.3
4. Python 3.4
5. Python 3.5
6. Python 3.6
7. Python 3.7

Supported ACM version

1. ACM 1.0

Change Logs

Installation

For Python 2.7 and above:

pip install acm-sdk-python

For Python 2.6:

# install setuptools first:
wget https://pypi.io/packages/source/s/setuptools/setuptools-33.1.1.zip
unzip setuptools-33.1.1.zip
cd setuptools-33.1.1 && sudo python setup.py install

# if setuptools already exists:
sudo easy_install acm-sdk-python

Getting Started

import acm

ENDPOINT = "acm.aliyun.com:8080"
NAMESPACE = "**********"
AK = "**********"
SK = "**********"

# get config
client = acm.ACMClient(ENDPOINT, NAMESPACE, AK, SK)
data_id = "com.alibaba.cloud.acm:sample-app.properties"
group = "group"
print(client.get(data_id, group))

# add watch
import time
client.add_watcher(data_id, group, lambda x:print("config change detected: " + x))
time.sleep(5) # wait for config changes

Configuration

client = ACMClient(endpoint, namespace, ak, sk)

• endpoint - required - ACM server address.
• namespace - Namespace. | default: DEFAULT_TENANT
• ak - AccessKey For Alibaba Cloud ACM. | default: None
• sk - SecretKey For Alibaba Cloud ACM. | default: None

Extra Options

Extra option can be set by set_options, as following:

client.set_options({key}={value})

Configurable options are:

• default_timeout - Default timeout for get config from server in seconds.
• tls_enabled - Whether to use https.
• auth_enabled - Whether to use auth features.
• cai_enabled - Whether to use address server.
• pulling_timeout - Long polling timeout in seconds.
• pulling_config_size - Max config items number listened by one polling process.
• callback_thread_num - Concurrency for invoking callback.
• failover_base - Dir to store failover config files.
• snapshot_base - Dir to store snapshot config files.
• app_name - Client app identifier.
• no_snapshot - To disable default snapshot behavior, this can be overridden by param no_snapshot in get method.

API Reference

Get Config

ACMClient.get(data_id, group, timeout, no_snapshot)

• param data_id Data id.

• param group Group, use DEFAULT_GROUP if no group specified.

• param timeout Timeout for requesting server in seconds.

• param no_snapshot Whether to use local snapshot while server is unavailable.

• return W Get value of one config item following priority:

• Step 1 - Get from local failover dir(default: ${cwd}/acm/data).

  • Failover dir can be manually copied from snapshot dir(default: ${cwd}/acm/snapshot) in advance.
  • This helps to suppress the effect of known server failure.

• Step 2 - Get from one server until value is got or all servers tried.

  • Content will be save to snapshot dir after got from server.

• Step 3 - Get from snapshot dir.

Add Watchers

ACMClient.add_watchers(data_id, group, cb_list)

• param data_id Data id.
• param group Group, use DEFAULT_GROUP if no group specified.
• param cb_list List of callback functions to add.
• return

Add watchers to a specified config item.

• Once changes or deletion of the item happened, callback functions will be invoked.
• If the item is already exists in server, callback functions will be invoked for once.
• Multiple callbacks on one item is allowed and all callback functions are invoked concurrently by threading.Thread.
• Callback functions are invoked from current process.

Remove Watcher

ACMClient.remove_watcher(data_id, group, cb, remove_all)

• param data_id Data id.
• param group Group, use "DEFAULT_GROUP" if no group specified.
• param cb Callback function to delete.
• param remove_all Whether to remove all occurrence of the callback or just once.
• return

Remove watcher from specified key.

List All Config

ACMClient.list_all(group, prefix)

• param group Only dataIds with group match shall be returned, default is None.
• param group only dataIds startswith prefix shall be returned, default is None Case sensitive.
• return List of data items.

Get all config items of current namespace, with dataId and group information only.

• Warning: If there are lots of config in namespace, this function may cost some time.

Publish Config

ACMClient.publish(data_id, group, content, timeout)

• param data_id Data id.
• param group Group, use "DEFAULT_GROUP" if no group specified.
• param content Config value.
• param timeout Timeout for requesting server in seconds.
• return True if success or an exception will be raised.

Publish one data item to ACM.

• If the data key is not exist, create one first.
• If the data key is exist, update to the content specified.
• Content can not be set to None, if there is need to delete config item, use function remove instead.

Remove Config

ACMClient.remove(data_id, group, timeout)

• param data_id Data id.
• param group Group, use "DEFAULT_GROUP" if no group specified.
• param timeout Timeout for requesting server in seconds.
• return True if success or an exception will be raised.

Remove one data item from ACM.

Debugging Mode

Debugging mode if useful for getting more detailed log on console.

Debugging mode can be set by:

ACMClient.set_debugging()
# only effective within the current process

CLI Tool

A CLI Tool is along with python SDK to make convenient access and management of config items in ACM server.

You can use acm {subcommand} directly after installation, sub commands available are as following:

    add                 add a namespace
    use                 switch to a namespace
    current             show current endpoint and namespace
    show                show all endpoints and namespaces
    list                get list of dataIds
    pull                get one config content
    push                push one config
    export              export dataIds to local files
    import              import files to ACM server

Use acm -h to see the detailed manual.

Data Security Options

ACM allows you to encrypt data along with Key Management Service, service provided by Alibaba Cloud (also known as KMS).

To use this feature, you can follow these steps:

1. Install KMS SDK by pip install aliyun-python-sdk-kms.
2. Name your data_id with a cipher- prefix.
3. Get and filling all the needed configuration to ACMClient, info needed are: region_id, kms_ak, kms_secret, key_id.
4. Just make API calls and SDK will process data encrypt & decrypt automatically.

Example:

c = acm.ACMClient(ENDPOINT, NAMESPACE, AK, SK)
c.set_options(kms_enabled=True, kms_ak=KMS_AK, kms_secret=KMS_SECRET, region_id=REGION_ID, key_id=KEY_ID)

# publish an encrypted config item.
c.publish("cipher-dataId", None, "plainText")

# get the content of an encrypted config item.
c.get("cipher-dataId", None)

Use RAM

It is a best practice to use RAM instead of hard coded AccessKey and SecretKey at client side, because it's much more safe and simple.

Example:

ENDPOINT = "acm.aliyun.com"
NAMESPACE = "9ca*****c1e"
RAM_ROLE_NAME = "ECS-STS-KMS-ACM"
REGION_ID = "cn-shanghai"
KEY_ID="192d****dc"

# use RAM role name for configuration.
a=acm.ACMClient(ENDPOINT, NAMESPACE, ram_role_name=RAM_ROLE_NAME)
a.set_options(kms_enabled=True, region_id=REGION_ID, key_id=KEY_ID)

# call API like the same as before.
a.list_all()
a.get('cipher-dataId','DEFAULT_GROUP')

For more information, refer to this document.

Other Resources

• Alibaba Cloud ACM homepage: https://www.aliyun.com/product/acm

acm-sdk-python's Issues

Bug in AcmClient with ram_role_name as auth

When I'm trying to do following:

import acm

ENDPOINT = ''
NAMESPACE = ''
RAM_ROLE = ''
DATA_ID = ''
GROUP = ''

c = acm.ACMClient(ENDPOINT, NAMESPACE, AK, SK)
c.set_options(kms_enabled=True, region_id="cn-beijing")
print(c.get(DATA_ID, GROUP))

I get following error:

    print(c.get(DATA_ID, GROUP))
  File "/usr/local/lib/python3.8/dist-packages/acm/client.py", line 397, in get
    return self.decrypt(content)
  File "/usr/local/lib/python3.8/dist-packages/acm/client.py", line 897, in decrypt
    resp = json.loads(self.kms_client.do_action_with_exception(req).decode("utf8"))
  File "/usr/local/lib/python3.8/dist-packages/aliyunsdkcore/client.py", line 473, in do_action_with_exception
    status, headers, body, exception = self._implementation_of_do_action(acs_request)
  File "/usr/local/lib/python3.8/dist-packages/aliyunsdkcore/client.py", line 308, in _implementation_of_do_action
    return self._handle_retry_and_timeout(endpoint, request, signer)
  File "/usr/local/lib/python3.8/dist-packages/aliyunsdkcore/client.py", line 376, in _handle_retry_and_timeout
    status, headers, body, exception = self._handle_single_request(endpoint,
  File "/usr/local/lib/python3.8/dist-packages/aliyunsdkcore/client.py", line 398, in _handle_single_request
    http_response = self._make_http_response(endpoint, request, read_timeout, connect_timeout,
  File "/usr/local/lib/python3.8/dist-packages/aliyunsdkcore/client.py", line 264, in _make_http_response
    header, url = signer.sign(self._region_id, request)
  File "/usr/local/lib/python3.8/dist-packages/aliyunsdkcore/auth/signers/ecs_ram_role_signer.py", line 42, in sign
    self._check_session_credential()
  File "/usr/local/lib/python3.8/dist-packages/acm/client.py", line 74, in _check_session_credential_patch
    expiration = self._expiration if isinstance(self._expiration, (float, int)) \
AttributeError: 'EcsRamRoleSigner' object has no attribute '_expiration'

bug 啊，这明明只会返回两个数据，怎么用三个参数去接收

pypi release版本高于github仓库代码版本

pypi上发布的最新版本是0.4.11，但是当前项目的代码还是0.4.10，是不是有一些代码没有合上来？辛苦对齐一下吧。

How can I add a new namespace

How can use the " acm add" to add a new namespace?It always says you don't have the right for the new namespace.

MissingParameter The parameter KeyId needed but no provided

I get this error when publish a data item.

script:

import acm
from os import environ

ENDPOINT = "acm.aliyun.com"
NAMESPACE = environ['NAMESPACE']
AK = environ['ALI_MY_AK']
SK = environ['ALI_MY_SK']


c = acm.ACMClient(ENDPOINT, NAMESPACE, AK, SK)
c.set_options(kms_enabled=True, region_id="cn-shanghai")
c.publish("cipher-dataId", None, "plainText")

output:

ERROR:aliyunsdkcore.client:ServerException occurred. Host:kms.cn-shanghai.aliyuncs.com SDK-Version:2.13.5 ServerException:HTTP Status: 400 Error:MissingParameter The parameter  KeyId  needed but no provided. RequestID: 44250351-8679-4a5b-8099-4140fd4b291a
Traceback (most recent call last):
  File "deploy/fetch_env.py", line 15, in <module>
    c.publish("cipher-dataId", None, "plainText")
  File "/home/mymusise/pro/trade-server/env/lib/python3.7/site-packages/acm/client.py", line 360, in publish
    content = self.encrypt(content)
  File "/home/mymusise/pro/trade-server/env/lib/python3.7/site-packages/acm/client.py", line 885, in encrypt
    resp = json.loads(self.kms_client.do_action_with_exception(req).decode("utf8"))
  File "/home/mymusise/pro/trade-server/env/lib/python3.7/site-packages/aliyunsdkcore/client.py", line 456, in do_action_with_exception
    raise exception
aliyunsdkcore.acs_exception.exceptions.ServerException: HTTP Status: 400 Error:MissingParameter The parameter  KeyId  needed but no provided. RequestID: 44250351-8679-4a5b-8099-4140fd4b291a

How to publish a new data and make sure the type is "JSON" not "TEXT"?

How to publish a new data and make sure the type is "JSON" not "TEXT"?

解密kms 128加密的配置报错，解密KMS加密的配置正常

看了Java acm sdk 的代码发现使用kms server解密的是encryptedDataKey ，不是配置的密文。
python那边在解密kms 128的配置的时候应该没有对encryptedDataKey进行解密。
见下面Java代码

ModuleNotFoundError: No module named 'fcntl'

python 3.6 使用
import acm

ENDPOINT = "acm.aliyun.com:8080"
NAMESPACE = "base-dev"
AK = "××××××××××"
SK = "×××××××××××××××××"

get config

client = acm.ACMClient(ENDPOINT, NAMESPACE, AK, SK)
data_id = "cn.jfhealthcare.devops:opssampler.ini"
group = "DEFAULT_GROUP"

add watch

#import time
#client.add_watcher(data_id, group, lambda x:print("config change detected: " + x))
#time.sleep(5) # wait for config changes

if(name=='main'):
print(client.get(data_id, group))
#client.list_all()

请补充endpoint配置端口的文档说明

endpoint相关的配置完整的结构应该是 domain:port, 这个只在示例代码中有展示，在官方文档中却没有说明应该使用的正确的端口信息。 希望官方文档中能把端口信息做准确的说明.

https://help.aliyun.com/document_detail/60137.htm?spm=a2c4g.11186623.2.19.2a6958bcfR52ir#section-3ug-f75-fff

ACMClient.list()出错：[list] exception the JSON object must be str, not 'bytes' occur

467 resp = self._do_sync_req("/diamond-server/basestone.do", None, params, None, self.default_timeout)
468 d = resp.read()
469 return json.loads(d)

需改为return json.loads(d.decode("utf-8")) 。

watcher with gevent will throw BlockingIOError:

bug recurrence:
acm watcher script with code
"from gevent import monkey; monkey.patch_all()" or
gunicorn start service with "-k gevent"

Traceback (most recent call last):
File "/root/anaconda3/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/root/anaconda3/lib/python3.6/site-packages/gunicorn/workers/gthread.py", line 104, in init_process
super(ThreadWorker, self).init_process()
File "/root/anaconda3/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/root/anaconda3/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/root/anaconda3/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/root/anaconda3/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/root/anaconda3/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/root/anaconda3/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
import(module)
File "xxxxxxxxx", line 32, in
xxxxxxxxxxxx
File "xxxxxxxxx", line 12, in init
self.client.add_watcher(xxxxxxxxxxx)
File "/root/anaconda3/lib/python3.6/site-packages/acm/commons.py", line 10, in synced_func
return func(*args, **kws)
File "/root/anaconda3/lib/python3.6/site-packages/acm/client.py", line 557, in add_watcher
self.add_watchers(data_id, group, [cb])
File "/root/anaconda3/lib/python3.6/site-packages/acm/commons.py", line 10, in synced_func
return func(*args, **kws)
File "/root/anaconda3/lib/python3.6/site-packages/acm/client.py", line 602, in add_watchers
key_list = self.process_mgr.list()
File "/root/anaconda3/lib/python3.6/multiprocessing/managers.py", line 662, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/root/anaconda3/lib/python3.6/multiprocessing/managers.py", line 554, in _create
conn = self._Client(self._address, authkey=self._authkey)
File "/root/anaconda3/lib/python3.6/multiprocessing/connection.py", line 493, in Client
answer_challenge(c, authkey)
File "/root/anaconda3/lib/python3.6/multiprocessing/connection.py", line 732, in answer_challenge
message = connection.recv_bytes(256) # reject large message
File "/root/anaconda3/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/root/anaconda3/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/root/anaconda3/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
BlockingIOError: [Errno 11] Resource temporarily unavailable

不设置环境变量DIAMOND_SERVER_IPS导致异常exception 'NoneType' object has no attribute 'split' occur

ERROR:root:[get-config] exception 'NoneType' object has no attribute 'split' occur
Traceback (most recent call last):
File "/Users/alsc/work/.pyodps/lib/python3.9/site-packages/acm/client.py", line 439, in get_raw
resp = self._do_sync_req("/diamond-server/config.co", None, params, None, timeout or self.default_timeout)
File "/Users/alsc/work/.pyodps/lib/python3.9/site-packages/acm/client.py", line 663, in _do_sync_req
server_info = self.get_server()
File "/Users/alsc/work/.pyodps/lib/python3.9/site-packages/acm/client.py", line 284, in get_server
server_list = get_server_list(self.endpoint, 443 if self.tls_enabled else 8080, self.cai_enabled,
File "/Users/alsc/work/.pyodps/lib/python3.9/site-packages/acm/server.py", line 43, in get_server_list
for server_addr in server_addresses.split(","):
AttributeError: 'NoneType' object has no attribute 'split'

有些环境下，不需要设置DIAMOND_SERVER_IPS，但目前版本，必须设置，否则会抛异常中断后续流程，获取不到data；

建议加个server_addresses的NoneType判断，再进行server_addresses.split(",")

def get_server_list(endpoint, default_port=8080, cai_enabled=True, unit_name=None):
server_list = list()

import os
server_addresses = os.getenv('DIAMOND_SERVER_IPS')
for server_addr in server_addresses.split(","):
    if server_addr is not "" and parse_nacos_server_addr(server_addr):
        server_list.append(parse_nacos_server_addr(server_addr.strip()))
if len(server_list) > 0:
    return server_list