azure / azure-storage-python

Microsoft Azure Storage Library for Python

Microsoft Azure Storage SDK for Python (Deprecated)

This project will be in [Community Support](https://azure.github.io/azure-sdk/policies_support.html#package-lifecycle) until 13 September 2024. After this date the project and associated client libraries will be retired permanently. For more details on the retirement and alternatives to using this project, visit [Retirement notice: The legacy Azure Storage Python client libraries will be retired on 13 September 2024](https://azure.microsoft.com/updates/retirement-notice-the-legacy-azure-storage-python-client-libraries-will-be-retired-on-13-september-2024/).

---

NEWS!! azure-storage-blob v12 is GA now!

Visit https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage to learn more.

Note: the current repo is for azure-storage-blob<=2.1.0, upgrading to v12.0.0 could break you current code.

This project provides a client library in Python that makes it easy to consume Microsoft Azure Storage services. For documentation please see the Microsoft Azure Python Developer Center and our API Reference (also available on readthedocs).

If you are looking for the Service Bus or Azure Management libraries, please visit https://github.com/Azure/azure-sdk-for-python.

Compatibility

IMPORTANT: If you have an earlier version of the azure package (version < 1.0), you should uninstall it before installing this package.

You can check the version using pip:

pip freeze

If you see azure==0.11.0 (or any version below 1.0), uninstall it first then install it again:

pip uninstall azure
pip install azure

If you are upgrading from a version older than 0.30.0, see the upgrade doc, the usage samples in the samples directory, and the ChangeLog and BreakingChanges.

If you are encountering problems installing azure storage on Azure Web Apps, upgrading pip might help.

IMPORTANT: If you have an earlier version of the azure-storage package (version <= 0.36.0), you should uninstall it before installing the new split packages.

You can check the version using pip:

pip freeze

If you see azure-storage==0.36.0 (or any version below 0.36.0), uninstall it first:

pip uninstall azure-storage

Features

• Blob
  • Create/Read/Update/Delete Containers
  • Create/Read/Update/Delete Blobs
  • Advanced Blob Operations
• Queue
  • Create/Delete Queues
  • Insert/Peek Queue Messages
  • Advanced Queue Operations
• Files
  • Create/Update/Delete Shares
  • Create/Update/Delete Directories
  • Create/Read/Update/Delete Files
  • Advanced File Operations

Getting Started

Download

The Azure Storage SDK for Python is composed of 5 packages:

• azure-storage-blob
  • Contains the blob service APIs.
• azure-storage-file
  • Contains the file service APIs.
• azure-storage-queue
  • Contains the queue service APIs.
• azure-storage-common
  • Contains common code shared by blob, file and queue.
• azure-storage-nspkg
  • Owns the azure.storage namespace, user should not use this directly.

Note: prior to and including version 0.36.0, there used to be a single package (azure-storage) containing all services. It is no longer supported, and users should install the 3 before-mentioned service packages individually, depending on the need. In addition, the table package is no longer releasing under the azure-storage namespace, please refer to cosmosdb.

Option 1: Via PyPi

To install via the Python Package Index (PyPI), type:

pip install azure-storage-blob
pip install azure-storage-file
pip install azure-storage-queue

Option 2: Source Via Git

To get the source code of the SDK via git just type:

git clone git://github.com/Azure/azure-storage-python.git

cd ./azure-storage-python/azure-storage-nspkg
python setup.py install

cd ../azure-storage-common
python setup.py install

cd ../azure-storage-blob
python setup.py install

Replace azure-storage-blob with azure-storage-file or azure-storage-queue, to install the other services.

Option 3: Source Zip

Download a zip of the code via GitHub or PyPi. Then follow the same instructions in option 2.

Minimum Requirements

• Python 2.7, 3.3-3.7.
• See setup.py for dependencies

Usage

To use this SDK to call Microsoft Azure storage services, you need to first create an account.

Logging

To make debugging easier, it is recommended to turn on logging for the logger named 'azure.storage'. Here are two example configurations:

# Basic configuration: configure the root logger, including 'azure.storage'
logging.basicConfig(format='%(asctime)s %(name)-20s %(levelname)-5s %(message)s', level=logging.INFO)

# More advanced configuration allowing more control
logger = logging.getLogger('azure.storage')
handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s %(name)-20s %(levelname)-5s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)

Here is how we use the logging levels, it is recommended to use INFO:

• DEBUG: log strings to sign
• INFO: log outgoing requests and responses, as well as retry attempts
• WARNING: not used
• ERROR: log calls that still failed after all the retries

Code Sample

See the samples directory for blob, queue, and file usage samples.

Need Help?

Be sure to check out the Microsoft Azure Developer Forums on MSDN or the Developer Forums on Stack Overflow if you have trouble with the provided code.

Contribute Code or Provide Feedback

If you would like to become an active contributor to this project, please follow the instructions provided in Azure Projects Contribution Guidelines. You can find more details for contributing in the CONTRIBUTING.md doc.

If you encounter any bugs with the library, please file an issue in the Issues section of the project.

Learn More

• Python Developer Center
• Azure Storage Service
• Azure Storage Team Blog
• API Reference

azure-storage-python's Issues

Support for blob type 'Append'

Description:

See http://blogs.msdn.com/b/windowsazurestorage/archive/2015/04/13/introducing-azure-storage-append-blob.aspx

Original Issue: Azure/azure-sdk-for-python#342

Comments (0):

Bug with uploading large files using `put_block_blob_from_path` and specifying content-md5

I believe there's a bug in uploading large files using put_block_blob_from_path and specifying md5 hash of the blob. What I noticed is that when uploading large block blobs, in the code 1st an empty block blob is created (line 1077) which I don't think is needed. Since a user is specifying content-md5 of the file and an empty blob is created by the SDK, their md5 hashes will not match and the user will get an error.

Please see this Stack Overflow thread for more details: http://stackoverflow.com/questions/34914261/python-azure-blob-storage-md5-check-fails-on-blob-upload-using-put-block-blob-fr.

Timeout issue copying blobs within a storage account

Started experiencing an issue where a copy_blob command to the API will timeout when attempting to copy a blob within the same storage account. The error generated is:

HTTPSConnectionPool(host='storageaccountname.blob.core.windows.net', port=443): Read timed out.

Attempting to copy the same blob to a different storage account works fine.

Attempting to copy the same blob within the storage account using the azure-cli also works fine. It takes a while for the command to return, but it seems like the azure-cli just has a longer timeout setting.

commit_batch to respect 100 entity limit

Description:

Rather than leave it up to the user to decipher this* cryptic error message and implement the 100 entity limitation on batching themselves, it would be great if the library could handle this.

WindowsAzureBatchOperationError('99:One of the request inputs is not valid.RequestId:******Time:2015-06-30T20:19:29.9051959Z',)

Original Issue: Azure/azure-sdk-for-python#402

Comments (1):

@huguesv commented on 2015-06-30 21:08:08:

That's an excellent suggestion!

Incompatibility with "requests" 2.9.0, just released

My code worked perfectly so far, but when requests ungraded to 2.9.0, I get this exception:

  File "/usr/local/lib/python3.5/site-packages/azure/storage/blob/blobservice.py", line 1228, in put_block_blob_from_bytes
    x_ms_lease_id)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/blob/blobservice.py", line 883, in put_blob
    self._perform_request(request)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/storageclient.py", line 171, in _perform_request
    resp = self._filter(request)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/storageclient.py", line 160, in _perform_request_worker
    return self._httpclient.perform_request(request)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/_http/httpclient.py", line 181, in perform_request
    self.send_request_body(connection, request.body)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/_http/httpclient.py", line 143, in send_request_body
    connection.send(request_body)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/_http/requestsclient.py", line 81, in send
    self.response = self.session.request(self.method, self.uri, data=request_body, headers=self.headers, timeout=self.timeout)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 454, in request
    prep = self.prepare_request(req)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 388, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/local/lib/python3.5/site-packages/requests/models.py", line 296, in prepare
    self.prepare_body(data, files, json)
  File "/usr/local/lib/python3.5/site-packages/requests/models.py", line 447, in prepare_body
    body = self._encode_params(data)
  File "/usr/local/lib/python3.5/site-packages/requests/models.py", line 84, in _encode_params
    return to_native_string(data)
  File "/usr/local/lib/python3.5/site-packages/requests/utils.py", line 700, in to_native_string
    out = string.decode(encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)

querying table storage is extremely slow due to 1,000 entites limit

I am trying to count how many rows I have and it takes ages. Much slower than Excel. According to the web, there is a way to query more than 1,000 entities at a time via AsTableServiceQuery but I cannot find it's Python equivalent.
Currently I am doing something like:

field='field name'
data = table_service.query_entities('DetectionAlertsExportTableUnsent', select=field)
while hasattr(data, 'x_ms_continuation'):
    more_count += 1
    logging.debug('getting more data for the %d time', more_count)
    next_partition_key = data.x_ms_continuation['nextpartitionkey']
    next_row_key = data.x_ms_continuation['nextrowkey']
    more_data = table_service.query_entities('DetectionAlertsExportTableUnsent', select=field,
                                             next_partition_key=next_partition_key, next_row_key=next_row_key)

Storage queue service should automatically encode/decode base 64

Description:

The other SDKs (.NET, Node.js) do that, so if you want to interop with them, it's currently the user's responsibility to do the encoding/decoding when using the Python SDK.

We should make it transparent and automatically encode/decode base64 when sending/receiving queue messages.

This would be a breaking change.

If we don't do this, we should fix the docs at http://azure.microsoft.com/en-us/documentation/articles/storage-python-how-to-use-queue-storage to show the encoding/decoding code required on the client's side.

Original Issue: Azure/azure-sdk-for-python#150

Comments (2):

@cheif commented on 2014-04-25 15:01:51:

I totally agree, the main idea of a SDK is to abstract away the implementation-details of the underlying resource, such as the transport using base64.

I think the best way to break the API is to bundle the breaking changes into a new major release (that would be 1.0.0 in this case), I think that there are some other things that also could be have a better abstraction (e.g. that StorageQueue.get_messages() should be a generator an not take a num_messages argument, that is also a leaky abstraction) or could be more pythonic (e.g. that QueueMessage-objects should be able to delete themselves, without using a StorageQueue-object, and a Blob-object being able to read their own data)

EDIT:
I realize that the "more-pythonic" changes can be implemented without breaking the API, I'd be happy to create a PR if others also think it's a good idea?

@saschagottfried commented on 2014-05-28 09:22:53:

I had this issue this week with version 0.8.0 while developing code that receives queue messages transporting just a GUID that was inserted by .NET code. Since the .NET client did not insert XML or JSON into queue message I did not recognize the data as a base64 string. After checking .NET code is using usual code to insert queues messages, I eventually found MSDN information regarding base64 encoding/decoding of queue message.

I expected this topic to be in discussion, so I am happy that package developers themselves make that an issue.

base64 decoding was not a big deal. The real problem at least for me was that I, as a developer, was not aware of that need to decode. Help developer to be aware that there are conditions where message.message_text is a base64 encoded string and therefore has to be decoded. And yes, update your docs as a first step towards a more pythonic Message Queue API. Otherwise you may risk to reduce Python developer satisfaction. That's not what you want since Azure rocks.

@cheif
just do it and make a pull request. Open source is meant for contributions.

Content-MD5 header does not validate properly for blob uploads in chunks

Description:

I would like to use MD5 validation to ensure my data survives transport when I upload blobs to Azure storage. Per the documentation, I supply a content-MD5 header along with my upload, so that the service validates the incoming file against this header.

This works great for small files. But once I try to upload files that require chunked uploads -- e.g. I'm currently trying to upload a 1GB file -- the validation is always unsuccessful. I'm using BlobService.put_block_blob_from_path(...) for the upload.

For reference, here is how I calculate the MD5 locally and pass it along with my upload.

# md5 generation for validating uploads
import hashlib
def md5_for_file(f, block_size=2**20):
    md5 = hashlib.md5()
    while True:
        data = f.read(block_size)
        if not data:
            break
        md5.update(data)
    return md5.digest().encode('base64')[:-1] 

# do the uploads, with file path p
for p in paths:
    blob_name = path_leaf(p)
    original_md5 = md5_for_file(open(p, 'rb'))
    blob_service.put_block_blob_from_path(storage_container_name, blob_name, p, content_md5=original_md5) # specify md5 to do an md5 check automatically

I get a WindowsAzureError when I run the above, due to the MD5 not validating properly:

Traceback (most recent call last):
  File "azureUpload.py", line 44, in <module>
    blob_service.put_block_blob_from_path(storage_container_name, blob_name, p, content_md5=original_md5) # specify md5 to do an md5 check automatically
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/blobservice.py", line 764, in put_block_blob_from_path
    progress_callback)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/blobservice.py", line 861, in put_block_blob_from_file
    x_ms_lease_id)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/blobservice.py", line 696, in put_blob
    self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 150, in _perform_request
    _storage_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/__init__.py", line 889, in _storage_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 931, in _general_error_handler
    http_error.respbody.decode('utf-8'))
azure.WindowsAzureError

The issue

My local MD5 computation isn't the issue -- I've checked it against Azure with small uploads, and it produces the same result as md5sum for my 1GB file.

How should big files be uploaded with MD5 validation? It seems that the API should be changed so that MD5 is computed for each chunk.

How to reproduce

Create a >64MB file using dd if=/dev/urandom of=65M.random bs=1M count=65, then upload it with the code above. You get the same traceback as I did.

But if you chop 20MB off that using dd if=/dev/urandom of=45M.random bs=1M count=45, we won't trigger chunking, and the upload succeeds, along with MD5 validation.

If you remove MD5 validation (don't specify content_md5 parameter), both uploads succeed.

Thus, the issue lies with MD5 validation with files that trigger chunking.

Original Issue: Azure/azure-sdk-for-python#264

Comments (2):

@huguesv commented on 2015-01-05 15:57:16:

Thanks for letting us know about this issue.

The documentation for Put Block List REST API says:
"Note that this header is associated with the request content, and not with the content of the blob itself."

So passing in the MD5 of the whole blob to Put Block List is definitely the wrong thing to do.

We should calculate the MD5 for each chunk and pass it to Put Block (Put Blob in the case of no chunking), if the user wants validation. The content_md5 parameter should be changed to a Boolean, matching the behavior of the useTransactionalMD5 parameter used in Node.js SDK:
https://github.com/Azure/azure-storage-node/blob/master/lib/services/blob/blobservice.js

All Page Blob and Block Blob APIs should be reviewed for this, both upload and download.

@maximz commented on 2015-01-06 18:44:13:

@huguesv Thanks for the update. Let me know if there's any further way I can help. (For now, I've just disabled MD5 verification in my code)

Upload large files (>195 GB)

Hi all,

According to the documentation:
https://msdn.microsoft.com/en-us/library/azure/ee691964.aspx
The maximum size of a blob is 195GB, my question is:
What should be the approach if we need to upload files bigger than 195GB

Thanks,

_common_serialization.py missing import of DEV_ACCOUNT_NAME

It should have a:

from .constants import DEV_ACCOUNT_NAME

so that _update_request_uri_query_local_storage works against the storage emulator.

Allow 'UseDevelopmentStorage=true' in the storage connection string

Description:

Right now the only way to run in storage emulator is to pass None for both account key/name, and set the EMULATED env variable. We should allow setting that in the connection string, like node.js storage sdk.

Original Issue: Azure/azure-sdk-for-python#375

Comments (0):

listing a "folder" under container

How can I list files under a specific regex expression?
If not possible how can I at least get the names of just one level of name (e.g. if path is CONTAINER/top1/bottom, CONTAINER/top2/bottom I would like to get only top1 and top2 rather than listing all the blobs under the container).
I know I can give a prefix to list_blobs but than won't do for what I need above, wouldn't it?!

Return x-ms-request-id from storage service query_entities

Description:

For diagnostics, in _convert_response_to_feeds, add something like this:

if 'x-ms-request-id' in name:
            setattr(feeds, 'x_ms_request_id', value)

Original Issue: Azure/azure-sdk-for-python#175

Comments (0):

get_container_acl always returns an empty azure.storage.models.SignedIdentifiers

I can list all containers from an account, and try to get their acl, but it always return an empty SignedIdentifiers object.

from azure.storage.blob import BlobService
bs = BlobService('account', 'private_key')

for c in bs.list_containers():
    container_acl = bs.get_container_acl(c.name)

container_acl is always an empty object.

azure.storage.QueueMessage considered harmful

Description:

This week I wrote code addressing Azure Storage Blobs, Tables and Queues. Great stuff. Since I ran into errors while removing message that were dequeued more often than my limit, I looked into the details of azure.storage.QueueMessage. The implementation was surprising me...

azure.storage.QueueMessage is just a bunch of strings. I was really surprised that this class (obviously nothing more than a Data Transfer Object) ships with an SDK that is marked as 'production/stable/ on PyPI.

class QueueMessage(WindowsAzureData):

''' Queue message class. '''

def __init__(self):
    self.message_id = u''
    self.insertion_time = u''
    self.expiration_time = u''
    self.pop_receipt = u''
    self.time_next_visible = u''
    self.dequeue_count = u''
    self.message_text = u''

I would expect members having names containing 'time' to have type DateTime and members having names indicating an integer (dequeue_count) to be an Integer. Instances of this class are just prone to error. That's what happened to me.

Focused Python developers may expect a standard-level pythonic API coming from Microsoft that of course will hide transport implementations details as well that are discussed in this issue [https://github.com/Azure/azure-sdk-for-python/issues/150].

Can anybody comment this or explain why it is like that or even has to be like that? Any suggestions how to contribute on that important topic without breaking APIs are welcome.

Original Issue: Azure/azure-sdk-for-python#156

Comments (0):

Low BLOB_MAX_CHUNK_DATA_SIZE leads to performance hits for max_connections=1

Description:

When using azure.storage.blobservice.get_blob_to_file, the algorithm will download a 64MB (or smaller) file in one call REST call.

However larger files are chunked in 4MB blocks. For a 65MB file this would result in 17 calls, which significantly decreases performance. For my application, changing _BLOB_MAX_CHUNK_DATA_SIZE to be the same as _BLOB_MAX_DATA_SIZE (64MB) improved performance drastically.

Not sure of all use cases to know if this will impact other applications, but if it cannot be hard coded to 64MB, perhaps allowing a chunk_size argument would help.

Original Issue: Azure/azure-sdk-for-python#337

Comments (1):

@huguesv commented on 2015-04-14 22:33:03:

Thanks for letting us know about this. Looking at the node storage sdk, it appears that download chunk size may not be limited to 4MB, unless the md5 is requested. I'll do some testing.

AzureHttpException not defined in _chunking.py

AzureHttpException not defined in _chunking.py

To reproduce, try downloading a very large blob (triggering chunking behavior).

Support storage client side encryption

Description:

http://blogs.msdn.com/b/windowsazurestorage/archive/2015/04/29/getting-started-with-client-side-encryption-for-microsoft-azure-storage.aspx

Original Issue: Azure/azure-sdk-for-python#386

Comments (0):

409 Conflict in copy blob should include the x-ms-copy-id

Currently when you get a 409 copy error because the target of a blobservice copy_blob API call already has a "async" copy to that target in progress, there is nothing useful inside the message. I can't tell why it failed (My code didn't know someone else did an async copy to that particular target blob). When calling copy_blob I also can't predict whether it will be synchronous or asynchronous, so I don't know when my blobs might enter the "async copy" state where all other copy operations will fail until I explicitly abort the copy...

As a small fix, would be nice if the 409 Conflict which shows up as an exception contained the "x-ms-copy-id" which would make it so my code could intelligently handle that failure. Currently all my synchronous copy code has to explicitly catch the exceptions for scenarios they never could have caused, do a get_blob_properties() call, pull out the 'x-ms-copy-id', abort the copy in progress and then retry the copy. Note that there is no timing guarantee on things like the get_blob_properties and the "cancel copy", so my subsequent calls may (depending on timing) produce a new 409 Conflict error[1]...

[1] https://msdn.microsoft.com/en-us/library/azure/jj159098.aspx "Trying to abort a copy that has completed or failed results in 409 Conflict."

PUBLIC NOTICE: Azure Storage Python Revamp

The Azure Storage team is taking ownership of the storage runtime library which is present in this repository and released as the azure-storage package on pypi. As promised in the package change announcement, there will be many new storage features coming soon.

The first round of changes will be the largest and will bring the library up to the newest service version. As it currently stands, the library supports very few features which were released after 2012. To best support all the new features released on the service in the past 3 years, we will be making breaking changes to bring the Python storage library up to date. Some of these changes include append blob support, file service support, table batch refactors, replacing AtomPub with JSON in table, account SAS, improved response parsing and many other small features and API updates.

All of this is being done in the dev branch and we will be maintaining ChangeLog and BreakingChanges files so you can see a summary of what’s happening. Note that this branch should not be considered stable!

After bringing the library up to the latest version we will do a preview release. Then we will begin adding additional improvements. These may include items such as built-in MD5 calculation, automatically following continuation tokens on list APIs, improved queue message encoding support, and many others, in addition to improvements the community suggests.

We’d love to hear from you!

Thanks,
The Azure Storage Team

Make Entity subclass of dict

We should integrate this change:
Azure/azure-sdk-for-python#411

We already allow passing dicts or Entity objects into azure storage, but for queries, we always return Entity objects. This change lets the user access the entity objects as dicts.

RowKey with ~ fails with error message for a bad auth header

table_service.insert_or_replace_entity('table', 'partition', 'row~key') fails.

The issue is that row keys cannot have a ~ (U+007E) character.

I am unsure where to report the issue with the documentation but ~ is not called out as a character limitation in the list restricted for row keys: https://msdn.microsoft.com/library/azure/dd179338.aspx

This is the error that is provided and is misleading:

azure\storage\_common_error.py", line 82, in _general_error_handler
    raise AzureHttpError(message, http_error.status)
azure.common.AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
<?xml version="1.0" encoding="utf-8"?><m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><m:code>AuthenticationFailed</m:code><m:message xml:lang="en-US">Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:3fc9641b-0002-0104-2e2a-26f9c9000000
Time:2015-11-23T20:07:19.6220380Z</m:message></m:error>

Django FileField/ImageField custom storage for azure blob

Description:

It's possible to customize the loading/saving of files for FileField/ImageField, and it would be nice to have a default implementation which stores it in azure blob storage.

Original Issue: Azure/azure-sdk-for-python#361

Comments (0):

azure.storage.StorageServiceProperties.Cors &(.CorsProperties) is missing

Description:

Original Issue: Azure/azure-sdk-for-python#191

Comments (3):

@huguesv commented on 2014-08-18 16:54:32:

http://msdn.microsoft.com/en-us/library/azure/dn535601.aspx

@ahmetalpbalkan commented on 2014-08-18 16:57:51:

I know the request but I couldn't create Cors or CorsProperties instances because there are no such classes, therefore I don't know what to pass to set_blob_service_properties() , could you provide an example please?

@huguesv commented on 2014-08-18 18:00:14:

Sorry, that link was just for reference, for when someone gets to implementing this feature.

Adding support for CORS requires changing the target x-ms-version to '2013-08-15', which turns this in a much bigger task than if the version stayed the same. In every revision, there are potential breaking changes in the XML format sent/received to/from Azure.

From what I can see, the way metrics are represented in 2013-08-15 has changed: http://msdn.microsoft.com/en-us/library/azure/hh452235.aspx.

There are potentially more, we'll need to review the docs and run all the tests to find out.

To adapt StorageServiceProperties to return the new CORS properties, you'll need to make the data classes match the XML returned by Azure. When there are lists involved, like in this case, it's a little more complex. You can see an example of object models with lists in azure.servicemanagement.AffinityGroup, which contains StorageServices and HostedServices.

How to handle errors?

How to efficiently handle error from this sdk? which are the common Exceptions this sdk may raise. the docs doesn't have any information on this. Please advice.

Python 2.7.10 Mac OS X 10.10 - binascii.Error TypeError: Incorrect padding

I have the following code:

 key = 'my_storage_account_key'
 blob_service = BlobService(account_name='my_storage_account', account_key='key')
 container = u'test'
 containers = blob_service.list_containers()

which results in the following error:

 ---------------------------------------------------------------------------
 TypeError                                 Traceback (most recent call last)
 <ipython-input-15-5ecff0813140> in <module>()
       3 
       4 container = u'test'
 ----> 5 containers = blob_service.list_containers()
       6 if container not in containers:
       7     try:

 /Users/emh5y7u/Documents/MIDS/W205/W205env/lib/python2.7/site-    packages/azure/storage/blob/blobservice.pyc in list_containers(self, prefix, marker, maxresults, include)
     291             request, self.use_local_storage)
     292         request.headers = _update_storage_blob_header(
 --> 293             request, self.authentication)
     294         response = self._perform_request(request)
     295 

 /Users/emh5y7u/Documents/MIDS/W205/W205env/lib/python2.7/site-packages/azure/storage/blob/_serialization.pyc in _update_storage_blob_header(request, authentication)
      41     request.headers.append(
      42         ('Content-Type', 'application/octet-stream Charset=UTF-8'))
 ---> 43     authentication.sign_request(request)
      44 
      45     return request.headers

 /Users/emh5y7u/Documents/MIDS/W205/W205env/lib/python2.7/site-packages/azure/storage/auth.pyc in sign_request(self, request)
      68             self._get_canonicalized_resource_query(request)
      69 
 ---> 70         self._add_authorization_header(request, string_to_sign)
      71 
      72     def _get_canonicalized_resource_query(self, request):

 /Users/emh5y7u/Documents/MIDS/W205/W205env/lib/python2.7/site-packages/azure/storage/auth.pyc in _add_authorization_header(self, request, string_to_sign)
      47 
      48     def _add_authorization_header(self, request, string_to_sign):
 ---> 49         signature = _sign_string(self.account_key, string_to_sign)
      50         auth_string = 'SharedKey ' + self.account_name + ':' + signature
      51         request.headers.append(('Authorization', auth_string))

 /Users/emh5y7u/Documents/MIDS/W205/W205env/lib/python2.7/site-packages/azure/storage/_common_conversion.pyc in _sign_string(key, string_to_sign, key_is_base64)
      81 def _sign_string(key, string_to_sign, key_is_base64=True):
      82     if key_is_base64:
 ---> 83         key = _decode_base64_to_bytes(key)
      84     else:
      85         if isinstance(key, _unicode_type):

 /Users/emh5y7u/Documents/MIDS/W205/W205env/lib/python2.7/site-packages/azure/storage/_common_conversion.pyc in _decode_base64_to_bytes(data)
      71     if isinstance(data, _unicode_type):
      72         data = data.encode('utf-8')
 ---> 73     return base64.b64decode(data)
      74 
      75 

  /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/base64.pyc in b64decode(s, altchars)
      74     except binascii.Error, msg:
      75         # Transform this exception for consistency
 ---> 76         raise TypeError(msg)
      77 
      78 

 TypeError: Incorrect padding

I've copied the key correctly, I've tried base64 encoding the key but I can't seem to establish a connection to the storage account. I'm not sure what else to try...

Failed to create a table in Azure Stack environment because of "azure.common.AzureHttpError: JSON format is not supported"

Hi,

I tried to create a container and table in Azure Stack environment. And creating the container succeeded, but creating the table failed.

This is the script:

from azure.storage.blob import BlobService
from azure.storage.table import TableService

storage_account_name = "xxx"
storage_access_key = "xxx"
service_host_base = "azurestack.local"
blob_service = BlobService(
    account_name=storage_account_name,
    account_key=storage_access_key,
    host_base='.blob.{0}'.format(service_host_base)
)
blob_service.create_container('foo1')

table_service = TableService(
    account_name=storage_account_name,
    account_key=storage_access_key,
    host_base='.table.{0}'.format(service_host_base)
)
table_service.create_table('bar1')

This is the error:

Traceback (most recent call last):
  File "test_setup_devbox.py", line 37, in <module>
    table_service.create_table('bar1')
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/table/tableservice.py", line 281, in create_table
    _dont_fail_on_exist(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/_common_error.py", line 65, in _dont_fail_on_exist
    raise error
azure.common.AzureHttpError: JSON format is not supported.
<?xml version="1.0" encoding="utf-8"?><m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><m:code>JsonFormatNotSupported</m:code><m:message xml:lang="en-US">JSON format is not supported.
RequestId:4e4069c0-20fa-45fa-b328-9facf35d03c1
Time:2016-02-02T02:21:49.7169355Z</m:message><m:innererror><m:message>JSON format is not supported.</m:message><m:type>Microsoft.Cis.Services.Nephos.Table.Service.Protocols.Rest.TableServiceProtocolException</m:type><m:stacktrace>   at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.EndPerformOperation(IAsyncResult ar)&#xD;
   at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.&lt;ProcessImpl&gt;d__4.MoveNext()</m:stacktrace><m:internalexception><m:message>JSON format is not supported.</m:message><m:type>Microsoft.Cis.Services.Nephos.Table.Service.Protocols.Rest.TableServiceProtocolException</m:type><m:stacktrace>   at Microsoft.ObjectStorage.FrontEnd.Odl.TableProvider.ODataTableManager.EndPerformOperation(IAsyncResult ar)&#xD;
   at Microsoft.Cis.Services.Nephos.Table.Service.Protocols.Rest.TableProtocolHead.&lt;TableAndRowOperationImpl&gt;d__0.MoveNext()&#xD;
   at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin)</m:stacktrace><m:internalexception><m:message>JSON format is not supported.</m:message><m:type>Microsoft.Cis.Services.Nephos.Table.Service.Protocols.Rest.TableServiceProtocolException</m:type><m:stacktrace>   at Microsoft.ObjectStorage.FrontEnd.Odl.TableProvider.ODataMediaTypeUtils.ParseAcceptHeader(String acceptHeader, String formatOption, ODataVersion serviceVersion, String&amp; supportedMediaType, ODataFormat&amp; supportedFormat)&#xD;
   at Microsoft.ObjectStorage.FrontEnd.Odl.TableProvider.Handlers.RequestHandler.Init()&#xD;
   at Microsoft.ObjectStorage.FrontEnd.Odl.TableProvider.Handlers.RequestHandler..ctor(IODataRequestMessage requestMessage, IODataResponseMessage responseMessage)&#xD;
   at Microsoft.ObjectStorage.FrontEnd.Odl.TableProvider.Handlers.RootRequestHandler..ctor(IODataRequestMessage requestMessage, IODataResponseMessage responseMessage, ITableClient tableClient)&#xD;
   at Microsoft.ObjectStorage.FrontEnd.Odl.TableProvider.ODataTableManager.BeginOdlProcess(IAccountIdentifier identifier, TableProtocolHead tableProtocolHead, RequestStartedCallback requestStartedCallback, CheckPermissionDelegate checkPermissionCallback, QueryRowCommandPropertiesAvailableCallback queryRowCommandPropertiesAvailableCallback, Dictionary`2 continuationToken, ContinuationTokenAvailableCallback continuationTokenAvailableCallback, Boolean operationIsConditional, AsyncCallback callback, Object state)&#xD;
   at Microsoft.ObjectStorage.FrontEnd.Odl.TableProvider.ODataTableManager.&lt;PerformOperationImpl&gt;d__0.MoveNext()&#xD;
   at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin)</m:stacktrace></m:internalexception></m:internalexception></m:innererror></m:error>

Thanks

Need document to use storage emulater

Description:

This version document is no description to use local storage emulater.
Need some description that environment variable("EMULATED") and set None to account_name and account_key.

Original Issue: Azure/azure-sdk-for-python#388

Comments (0):

High Level functions

I was looking at https://azure.microsoft.com/en-us/documentation/articles/storage-python-how-to-use-blob-storage/#list-the-blobs-in-a-container and was wondering if the azure storage sdk can provide a similar functionality.

Something close too the following example?

class AzureListBlobsIterator(collections.Iterable):

    def __init__(self, service, container_name, prefix=None, **kwargs):
        self.service = service
        self.container_name = container_name
        self.prefix = prefix
        self.kwargs = kwargs

    def __iter__(self):
        marker = None
        while True:
            batch = self.service.list_blobs(self.container_name, self.prefix, marker=marker, **self.kwargs)

            yield from batch
            if not batch.next_marker:
                break
            marker = batch.next_marker

class AzureListPrefixesIterator(collections.Iterable):

    def __init__(self, service, container_name, prefix=None, **kwargs):
        self.service = service
        self.container_name = container_name
        self.prefix = prefix
        self.kwargs = kwargs

    def __iter__(self):
        marker = None
        while True:
            batch = self.service.list_blobs(self.container_name, self.prefix, marker=marker, **self.kwargs)

            yield from batch.prefixes
            if not batch.next_marker:
                break
            marker = batch.next_marker

_ERROR_STORAGE_MISSING_INFO message too specific

if you make a queue (and likely other things) without a key it will tell you that you failed to provide the account name. This error comes up for more cases than that as evidenced by this. Consider broadening the error to represent the conditions it is thrown for.

CODE:
queue.QueueService("account", "")

RESULT:

    qs = queue.QueueService(account_name, account_key)
  File "\site-packages\azure\storage\queue\queueservice.py"
, line 134, in __init__
    raise ValueError(_ERROR_STORAGE_MISSING_INFO)
ValueError: You need to provide an account name
Press any key to continue . . .

Table name validation

Description:

It seems that you are unable to use punctuation in table names, although it isn't clear what is and isn't allowed.

>>> ts = TableService()
>>> ts.create_table("foo")
True
>>> ts.create_table("routing_table")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\azure\storage\tableservice.py", line 135, in create_table
    _dont_fail_on_exist(e)
  File "C:\Python27\lib\site-packages\azure\__init__.py", line 453, in _dont_fail_on_exist
    raise error
azure.WindowsAzureError: Unknown error (Bad Request)
>>> ts.create_table("routingtable")
True

It would be nice if some sort of client-side validation was done, if possible.

Original Issue: Azure/azure-sdk-for-python#52

Comments (3):

@slyons commented on 2012-06-27 21:43:31:

On further review it looks like it's not just punctuation that causes it, but I'm not quite sure what the cause is.

>>> target = "routing_table"
>>> for fn in funcs:
...     try:
...             print fn, fn(target)
...             ts.create_table(fn(target))
...     except Exception, e:
...             print repr(e)
...     else:
...             print fn, "passed!"
...
<function b64encode at 0x0000000001FCC7B8> cm91dGluZ190YWJsZQ==
WindowsAzureError(u'Unknown error (Bad Request)',)
<function urlsafe_b64encode at 0x0000000001FCC978> cm91dGluZ190YWJsZQ==
WindowsAzureError(u'Unknown error (Bad Request)',)
<function b32encode at 0x0000000001FCCA58> OJXXK5DJNZTV65DBMJWGK===
WindowsAzureError(u'Unknown error (Bad Request)',)
<function b16encode at 0x0000000001FCCB38> 726F7574696E675F7461626C65
WindowsAzureError(u'Unknown error (Bad Request)',)

>>> target="routingtable"
>>> for fn in funcs:
...     try:
...             print fn, fn(target)
...             ts.create_table(fn(target))
...     except Exception, e:
...             print repr(e)
...     else:
...             print fn, "passed!"
...
<function b64encode at 0x0000000001FCC7B8> cm91dGluZ3RhYmxl
False
<function b64encode at 0x0000000001FCC7B8> passed!
<function urlsafe_b64encode at 0x0000000001FCC978> cm91dGluZ3RhYmxl
False
<function urlsafe_b64encode at 0x0000000001FCC978> passed!
<function b32encode at 0x0000000001FCCA58> OJXXK5DJNZTXIYLCNRSQ====
WindowsAzureError(u'Unknown error (Bad Request)',)
<function b16encode at 0x0000000001FCCB38> 726F7574696E677461626C65
WindowsAzureError(u'Unknown error (Bad Request)',)

@weidongh commented on 2012-07-05 16:44:36:

Table names may contain only alphanumeric characters and cannot begin with a numeric character. They are case-insensitive and must be from 3 to 63 characters long.
Please refer to http://msdn.microsoft.com/en-us/library/windowsazure/dd179338.aspx.

We will add comments for the requirement of table name in next push.

@mrhwick commented on 2012-10-22 16:06:38:

The regular expression defines the rules most concisely.

These rules are also described by the regular expression "^[A-Za-z][A-Za-z0-9]{2,62}$".

Documentation on ReadTheDocs is missing all the module and method reference information

The table service query_entities method is defined here:

azure-storage-python/azure/storage/table/tableservice.py

Line 385 in 09e9f18

def query_entities(self, table_name, filter=None, select=None, top=None,

However, it does not show up in the ReadTheDocs documentation, either when doing a search:

http://azure-storage.readthedocs.org/en/latest/search.html?q=query_entities&check_keywords=yes&area=default

Or when looking at the page about the tables:

http://azure-storage.readthedocs.org/en/latest/table.html

Or when looking at the module page in the index:

http://azure-storage.readthedocs.org/en/latest/ref/azure.storage.table.tableservice.html

In fact, it seems like all the module pages, which are presumably the official API reference for what each method does and what each argument means, are blank. Presumably this isn't intentional.

Azure.Storage Python 2.6 compatibility

Description:

Hi there,

If 2.6 supported? The thing is CentOS features this one python version out-of-box.
We make an installable software for Linux. We use it to communicate with azure prior installing anything on the server.

At the moment, a run of our code produces whenever we do 'import azure.storage':
File "/usr/lib/python2.6/site-packages/azure/storage/init.py", line 1197
name.lower() : value for name, value in request.headers if value
SyntaxError: invalid syntax

Is there any means to fix that one incompatibility?
Thanks,
Vlad

Original Issue: Azure/azure-sdk-for-python#395

Comments (1):

@crwilcox commented on 2015-06-24 00:18:12:

The issue is that Python 2.6 does not support dictionary comprehensions.

We don't officially support 2.6 and don't make changes to address issues in 2.6. That being said, this one is rather simple. I would have little issue taking a pull request that addresses this

The way to fix this is to remove the dict comp and replace it with dict().

{
    name.lower() : value for name, value in request.headers if value
}

to

dict((name.lower(), value) for name, value in request.headers if value)

Again, you are welcome to patch this on your own or submit a pull request for us to take.

Thanks!

BlobService class blobs with "question marks" in the path issue

Moved from Azure/azure-sdk-for-python#494

Hi guys,

We are using Azure storage with one of our clients, and we are using the python sdk.

When we yield over the container_name with list_blobs method we get blobs with "question marks".
After that if we try to get the same blob thru the get_blob_to_file (or get_blob) method,
We get "Not found (The specified blob does not exist.)" exception.

I see that the issue is in the "_update_request_uri_query" method.
You guys expect a query string after "question marks" but you allowing blobs with "question marks" in the path...

For example:
if my path is: "test_path/test name?.txt"
I see that in the request the path is "//test_path/test name"

B.T.W:
I see that in the C# sdk it was fixed
http://stackoverflow.com/questions/12242120/azure-blob-reference-to-file-with-question-mark-in-the-name

please advice,

Thanks,
Lior
[email protected]

BlobService.get_page_ranges should support timeout parameter

https://msdn.microsoft.com/en-us/library/azure/ee691973.aspx

A timeout parameter is supported by the API, but not implemented in the SDK. The default seems very short (~ 25 seconds).

New release?

Is a new release planned that includes a fix for the critical #69 issue? Specifically, when can the fix be expected to appear in PyPI https://pypi.python.org/pypi/azure-storage/0.20.1

Broken source install from pypi

Running into an issue when installing from the zip on pypi.

(bar) $ pip freeze
You are using pip version 7.1.0, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
wheel==0.24.0
(bar) $ curl -L -C - https://pypi.python.org/packages/source/a/azure-storage/azure-storage-0.20.0.zip#md5=1d67ece73fae17fe29d5d9d0321aa601 -o azure-storage-0.20.0.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 90371  100 90371    0     0  79364      0  0:00:01  0:00:01 --:--:-- 79412
(bar) $ ls
azure-storage-0.20.0.zip
(bar) $ unzip azure-storage-0.20.0.zip
<snip>
(bar) $ cd azure-storage-0.20.0
(bar) $ ls -al azure
total 0
drwxr-xr-x   3 usernamehere staff  102 Aug 31 17:41 .
drwxr-xr-x   9 usernamehere staff  306 Aug 31 17:41 ..
drwxr-xr-x  20 usernamehere staff  680 Aug 31 17:41 storage
(bar) $ python setup.py install
<snip>
(bar) $ ipython
In [1]: from azure.storage.blob import BlobService
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-7f43c9cb0958> in <module>()
----> 1 from azure.storage.blob import BlobService

ImportError: No module named azure.storage.blob

The __init__.py is missing from the pypi release, though the github zip has it.

$ tree
.
├── MANIFEST.in
├── PKG-INFO
├── README.rst
├── azure
│   └── storage
│       ├── __init__.py
│       ├── _common_conversion.py
│       ├── _common_error.py
│       ├── _common_models.py
│       ├── _common_serialization.py
│       ├── _http
│       │   ├── __init__.py
│       │   ├── batchclient.py
│       │   ├── httpclient.py
│       │   └── requestsclient.py
│       ├── _serialization.py
│       ├── auth.py
│       ├── blob
│       │   ├── __init__.py
│       │   ├── _chunking.py
│       │   ├── _serialization.py
│       │   ├── blobservice.py
│       │   └── models.py
│       ├── cloudstorageaccount.py
│       ├── connection.py
│       ├── constants.py
│       ├── file
│       │   ├── __init__.py
│       │   ├── _chunking.py
│       │   ├── _serialization.py
│       │   ├── fileservice.py
│       │   └── models.py
│       ├── models.py
│       ├── queue
│       │   ├── __init__.py
│       │   ├── _serialization.py
│       │   ├── models.py
│       │   └── queueservice.py
│       ├── sharedaccesssignature.py
│       ├── storageclient.py
│       └── table
│           ├── __init__.py
│           ├── _serialization.py
│           ├── models.py
│           └── tableservice.py
├── azure_storage.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   ├── not-zip-safe
│   ├── requires.txt
│   └── top_level.txt
├── setup.cfg
└── setup.py

BlobService.set_container_acl optional param not documented

Description:

signed_identifiers is also optional, but not documented as so.

Original Issue: Azure/azure-sdk-for-python#185

Comments (0):

Putting message in queue silently fails to add content with Python 3.4

Using queue_service.put_message('queue', 'message') fails using Python 3.4
Instead, it places an empty message in the queue.

edit:
This is using the example from: http://azure-storage.readthedocs.org/en/latest/queue.html?highlight=put_message

Support File Service REST API

Description:

http://msdn.microsoft.com/en-us/library/azure/dn167006.aspx

Original Issue: Azure/azure-sdk-for-python#252

Comments (0):

Installing the storage libraries on Linux fails

Installing the storage libraries on Linux fails because init.py files for some packages cannot be found (see output below). One issue can be that there are some Windows-type new lines left in the file and those cannot be recognized on Linux. The same libraries work on Windows.

atvmadmin@atvmlnx03:~$ sudo easy_install3 azure-storage
Searching for azure-storage
Reading https://pypi.python.org/simple/azure-storage/
Best match: azure-storage 0.20.2
Downloading https://pypi.python.org/packages/source/a/azure-storage/azure-storage-0.20.2.zip#md5=19266cb50c9e6f31a463ca0909e0cc5a
Processing azure-storage-0.20.2.zip
Writing /tmp/easy_install-w41be_ss/azure-storage-0.20.2/setup.cfg
Running azure-storage-0.20.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-w41be_ss/azure-storage-0.20.2/egg-dist-tmp-v8dun7lk
package init file 'azure/init.py' not found (or not a regular file)
warning: no previously-included files found matching 'azure/init.py'
Adding azure-storage 0.20.2 to easy-install.pth file

Installed /usr/local/lib/python3.4/dist-packages/azure_storage-0.20.2-py3.4.egg
Processing dependencies for azure-storage
Searching for python-dateutil
Reading https://pypi.python.org/simple/python-dateutil/
Reading http://labix.org/python-dateutil
Reading https://dateutil.readthedocs.org
Best match: python-dateutil 2.4.2
Downloading https://pypi.python.org/packages/source/p/python-dateutil/python-dateutil-2.4.2.tar.gz#md5=4ef68e1c485b09e9f034e10473e5add2
Processing python-dateutil-2.4.2.tar.gz
Writing /tmp/easy_install-xbk2zb/python-dateutil-2.4.2/setup.cfg
Running python-dateutil-2.4.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-xbk2zb/python-dateutil-2.4.2/egg-dist-tmp-ugpjir_o
Adding python-dateutil 2.4.2 to easy-install.pth file

Installed /usr/local/lib/python3.4/dist-packages/python_dateutil-2.4.2-py3.4.egg
Searching for azure-common
Reading https://pypi.python.org/simple/azure-common/
Best match: azure-common 1.0.0
Downloading https://pypi.python.org/packages/source/a/azure-common/azure-common-1.0.0.zip#md5=936f523f8f0e11780fc0b8a87d56d390
Processing azure-common-1.0.0.zip
Writing /tmp/easy_install-khjsvnqk/azure-common-1.0.0/setup.cfg
Running azure-common-1.0.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-khjsvnqk/azure-common-1.0.0/egg-dist-tmp-843ee6_p
package init file 'azure/init.py' not found (or not a regular file)
warning: no previously-included files found matching 'azure/init.py'
Adding azure-common 1.0.0 to easy-install.pth file

Installed /usr/local/lib/python3.4/dist-packages/azure_common-1.0.0-py3.4.egg
Searching for azure-nspkg
Reading https://pypi.python.org/simple/azure-nspkg/
Best match: azure-nspkg 1.0.0
Downloading https://pypi.python.org/packages/source/a/azure-nspkg/azure-nspkg-1.0.0.zip#md5=cb046beb44c3efdb8bf1688af6c011ca
Processing azure-nspkg-1.0.0.zip
Writing /tmp/easy_install-iq0amqy_/azure-nspkg-1.0.0/setup.cfg
Running azure-nspkg-1.0.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-iq0amqy_/azure-nspkg-1.0.0/egg-dist-tmp-4s2vd4bg
Adding azure-nspkg 1.0.0 to easy-install.pth file

Installed /usr/local/lib/python3.4/dist-packages/azure_nspkg-1.0.0-py3.4.egg
Searching for six>=1.5
Reading https://pypi.python.org/simple/six/
Best match: six 1.10.0
Downloading https://pypi.python.org/packages/source/s/six/six-1.10.0.tar.gz#md5=34eed507548117b2ab523ab14b2f8b55
Processing six-1.10.0.tar.gz
Writing /tmp/easy_install-o2cza5g_/six-1.10.0/setup.cfg
Running six-1.10.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-o2cza5g_/six-1.10.0/egg-dist-tmp-in03ps3n
no previously-included directories found matching 'documentation/_build'
zip_safe flag not set; analyzing archive contents...
pycache.six.cpython-34: module references path
Adding six 1.10.0 to easy-install.pth file

Installed /usr/local/lib/python3.4/dist-packages/six-1.10.0-py3.4.egg
Finished processing dependencies for azure-storage
atvmadmin@atvmlnx03:~$ python3
Python 3.4.0 (default, Jun 19 2015, 14:20:21)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.

import azure.common
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named 'azure.common'
import azure.storage
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named 'azure.storage'

Cannot delete snapshots - Authentication Error

I'm running a VM with infinitely leased storage from OS blobs. The following works:

from azure.storage.blob import BlobService
blob_service = BlobService(AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_ACCESS_KEY)
blobs = [b for b in blob_service.list_blobs(CONTAINER, include='snapshots')]
for b in blobs: print b.name + "|" + b.snapshot

There is a minor anomaly though, any blobs[i].snapshot returns an empty string u'' (Python 2.7). I'm certain that but one blob was created from the Azure portal by capturing it.

Now, deleting such a captured blob fails:

blob_service.delete_blob('vhds', blobs[1].name, snapshot=blobs[1].snapshot)
AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

<Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:
Time:2016-02-07T04:19:22.2515532Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request &apos;Thp/816hcXlHBbAt0Xne6iCJK5y87kD1nmCCOQZH7hY=&apos; is not the same as any computed signature. Server used following string to sign: &apos;DELETE
x-ms-date:Sun, 07 Feb 2016 04:19:22 GMT
x-ms-version:2014-02-14
snapshot:&apos;.</AuthenticationErrorDetail></Error>

The same authentication error is thrown when I try the indirect way via shared access policy and the respective token, but even earlier (listing the blobs):

blob_service = BlobService(AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_ACCESS_KEY)
ap = AccessPolicy(
    expiry='2016-10-12',
    permission=BlobSharedAccessPermissions.WRITE,
)
sas_token = blob_service.generate_shared_access_signature(
    container_name='vhds',
    blob_name=blobs[1].name,
    shared_access_policy=SharedAccessPolicy(ap),
)
blob_service = BlobService(AZURE_STORAGE_ACCOUNT, sas_token=sas_token)
blobs = [b for b in blob_service.list_blobs('vhds', include='snapshots')]
^-- same exception as above

It's crucial to delete old snapshots and I haven't yet a way around this problem. Any ideas where the requests are misformed or the root cause of the failure is?

Using JSON payload

Description:

Since Azure storage now supports JSON, it might be beneficial to parse JSON vs XML using python (haven't bench-marked)

Original Issue: Azure/azure-sdk-for-python#134

Comments (1):

@huguesv commented on 2015-03-05 21:17:00:

Storage and Service Bus now use ElementTree for xml deserialization. This, along with some other changes, have improved the performance of deserialization significantly (in the coming 0.10 release).
Azure/azure-sdk-for-python#254

Storage table insert_entity should have an echo_content option

Description:

Controls whether the service returns the entity that was added or not.

It's the 'prefer' header: https://msdn.microsoft.com/en-us/library/azure/dd179433.aspx

Original Issue: Azure/azure-sdk-for-python#381

Comments (4):

@coffenbacher commented on 2015-07-03 16:19:22:

Yes, just came here to add this issue! It definitely should =) I'm getting hammered with those responses.

@coffenbacher commented on 2015-07-03 17:55:37:

I spent quite some time trying to implement this myself (for batch requests specifically), but no luck. I keep getting 201 responses with the echo, even though my requests look (I think) exactly like the examples here (https://msdn.microsoft.com/en-us/library/azure/dd894038.aspx).

--changeset_dc541eaa-21a8-11e5-a70f-0242ac110b34
Content-Type: application/http
Content-Transfer-Encoding: binary

POST http://xxx.table.core.windows.net/xxxHTTP/1.1
Content-ID: 1
Content-Type: application/atom+xml;type=entry
Prefer: return-no-content
Content-Length: 732

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<entry xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom" >
<title /><updated>2015-07-03T17:28:03.937876+00:00</updated><author><name /></author><id />
<content type="application/xml">
    <m:properties xml:space="preserve">xxx</m:properties></content></entry>

returns 201, along with the associated body echo

ipdb> p responses[0].status
201

@huguesv commented on 2015-07-08 01:37:41:

Yeah, sometimes it's a lot harder than it should be :(
Is it behaving the same in non-batch mode?

@emgerner-msft commented on 2015-07-08 16:53:22:

Could you provide not just the headers for the changeset entry, but for the batch operation as a whole? If you see the document on Prefer Headers, the request service version must be above 2013-08-15 and the request OData data service version above 3.0 which would be one of the first things to check.

futures package not being installed on Python 2.7

Since we're now distributing a wheel, which is marked as universal between python 2 and 3, and the wheel was produced from python 3.4, this means that the wheel doesn't have the dependency on futures, which is conditional in setup.py to python 2.x.

I guess this means we'll need to produce 2 different wheels, one for 2.x, one for 3.x.

BlobService.put_blob needs timeout parameter

Description:

It seems like sometime my socket connections from a Linux Azure VM to Azure Storage are stalling. Can we add a timeout parameter to put_blob like some of the other functions?

Traceback below:

Unexpected upload error: <class 'socket.error'>
Traceback (most recent call last):
  File "GetHistory.py", line 81, in getAndUploadHist
    _bs.put_blob('thistory',filename,sz ,x_ms_blob_type='BlockBlob')
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/blobservice.py", line 693, in put_blob
    self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 143, in _perform_request
    resp = self._filter(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 132, in _perform_request_worker
    return self._httpclient.perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/http/httpclient.py", line 190, in perform_request
    self.send_request_body(connection, request.body)
  File "/usr/local/lib/python2.7/dist-packages/azure/http/httpclient.py", line 173, in send_request_body
    connection.send(request_body)
  File "/usr/lib/python2.7/httplib.py", line 805, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 104] Connection reset by peer

Original Issue: Azure/azure-sdk-for-python#153

Comments (0):

put_blob results in [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')] error

I have a pretty simple scenario in which I am trying to write a set of blobs into Blob Storage ala:

blobService = BlobService(account_name=os.environ["STORAGE_ACCOUNT"], account_key=os.environ["STORAGE_ACCESS_KEY"])
blobService.create_container(CONTAINER, x_ms_blob_public_access='container')

....

for resultSet in collectedResultSets:
    container = str(resultSet[0])
    json = str(build_json(resultSet))
    print container
    blobService.put_blob(CONTAINER, container, json, "BlockBlob")

This intermittently results in the following error:

Traceback (most recent call last):
File "", line 5, in
File "/Library/Python/2.7/site-packages/azure/storage/blob/blobservice.py", line 883, in put_blob
self._perform_request(request)
File "/Library/Python/2.7/site-packages/azure/storage/storageclient.py", line 171, in _perform_request
resp = self._filter(request)
File "/Library/Python/2.7/site-packages/azure/storage/storageclient.py", line 160, in _perform_request_worker
return self._httpclient.perform_request(request)
File "/Library/Python/2.7/site-packages/azure/storage/_http/httpclient.py", line 181, in perform_request
self.send_request_body(connection, request.body)
File "/Library/Python/2.7/site-packages/azure/storage/_http/httpclient.py", line 143, in send_request_body
connection.send(request_body)
File "/Library/Python/2.7/site-packages/azure/storage/_http/requestsclient.py", line 81, in send
self.response = self.session.request(self.method, self.uri, data=request_body, headers=self.headers, timeout=self.timeout)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, *_send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, *_kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/Library/Python/2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/Library/Python/2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1001, in request
self._send_request(method, url, body, headers)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1035, in _send_request
self.endheaders(body)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 997, in endheaders
self._send_output(message_body)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 850, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 826, in send
self.sock.sendall(data)
File "/Library/Python/2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 220, in sendall
sent = self._send_until_done(data[total_sent:total_sent+SSL_WRITE_BLOCKSIZE])
File "/Library/Python/2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 206, in _send_until_done
return self.connection.send(data)
File "/Library/Python/2.7/site-packages/OpenSSL/SSL.py", line 1271, in send
self._raise_ssl_error(self._ssl, result)
File "/Library/Python/2.7/site-packages/OpenSSL/SSL.py", line 1187, in _raise_ssl_error
_raise_current_error()
File "/Library/Python/2.7/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]

Can't insert entity with chinese string property

[1] If I use the codes. it gets error response.
Code:
entity = azure.storage.table.Entity()
entity.SiteCode = '232633627068'
entity.SiteName = azure.storage.table.EntityProperty('Edm.String', '蘋果新聞'.encode('utf-16'))
table.insert_entity('sites', entity)

Response:
azure.common.AzureHttpError: Bad Request

InvalidInputOne of the request inputs is not valid.

RequestId:b87b16db-0002-004b-4b15-40a58e000000
Time:2015-12-26T19:44:59.4595166Z

[2] If I use the codes. it gets success response.
Code:
entity = azure.storage.table.Entity()
entity.SiteCode = '232633627068'
table.insert_entity('sites', entity)

Shared Access Signatures don't work with Premium Storage Accounts

I'm getting the following error message when trying to copy a blob between Premium Storage accounts:

<?xml version="1.0" encoding="utf-8"?>
<Error><Code>CannotVerifyCopySource</Code><Message>Could not verify the copy source within the specified time.
RequestId:
Time:2015-09-02T00:14:20.1883717Z</Message></Error>

Using version 0.11.1 of the SDK. I've verified that the same blob copy logic works with regular Storage Accounts.

Getting error on uploading file using put_page_blob_from_path

Description:

Hi,
I got following error while uploading file as page blob-

azure/storage/blobservice.py", line 1318, in put_page_blob_from_path
retry_wait)
IOError: close() called during concurrent operation on the same file object.

Thanks,
Vaibhav

Original Issue: Azure/azure-sdk-for-python#341

Comments (1):

@huguesv commented on 2015-04-14 23:08:56:

Hi! Is this something you can repro consistently? Could you include the code I can use to repro it, including the size of the file you are trying to upload?