Comments (10)
I'm guessing that the auth error is because you're sending an empty snapshot which is not valid. In future versions this will be handled better, but at the moment if you don't send snapshot unless it's actually populated I think you'll see results that make more sense.
Another issue with this code is that list_blobs does not follow continuation tokens so you may be missing pages of results. This is again something that will be made automatic in newer versions, but for the moment you'll need to implement that code and pass the marker parameter.
include='snapshots' is the correct way to specify including snapshots, so if you've tried the two above steps and still are not seeing expected results, please update your code and we can keep debugging. That being said, we do have a test for this functionality. Note this test does not show how to follow continuation tokens, though there is a test nearby that does if you want to see.
from azure-storage-python.
Thanks for the well laid out suggestion. I amended the code to work with continuation tokens.
blob_service = BlobService(AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_ACCESS_KEY)
blobs = blob_service.list_blobs(CONTAINER, include='snapshots')
while True:
for b in blobs:
print b.name + " | " + b.snapshot
if not blobs.next_marker: break
blobs = blob_service.list_blobs(CONTAINER, None, blobs.next_marker, include='snapshots')
None of the results yield a snapshot, meaning all snapshot fields are zero length strings. This either means there is a bug in the API retrieving snapshots or a bug when creating snapshots from Azure portal in that it doesn't tag them as snapshots. I've created the latest on a few days ago.
What other possibilities are there to get rid of orphan snapshots in a timely manner?
from azure-storage-python.
What other possibilities are there to get rid of orphan snapshots in a timely manner?
Maybe this is the confusion? You cannot delete a blob without deleting its snapshots. If you check out the delete_blob API, if the blob has snapshots you must specify the x_ms_delete_snapshots option as include or only, meaning delete the blob and all of its snapshots or delete just the snapshots. You can also delete a specific snapshot by specifying the 'snapshot' parameter. However, note that you cannot delete just the base blob and not it's snapshots. Thus, no orphaned snapshots.
Is above the confusion? If not, can you include more info on exactly how you're creating a blob and snapshots?
from azure-storage-python.
Thx for following up!
I try to delete snapshots only as outlined in the original issue. However, the snapshot parameter set to a zero length string because this is what the API returns. This led me to the conclusion it is either a bug while retrieving snapshot names/timestamps or during creation. I created them via the UI in the Azure portal by clicking "capture VM".
Oh and sorry, I actually tried it with x_ms_delete_snapshots="only" or "include" to no avail.
blob_service.delete_blob('vhds', blobs[1].name, snapshot=blobs[1].snapshot, x_ms_delete_snapshots="only")
from azure-storage-python.
Ah, we got misdirected with the whole snapshot/auth error thing in the first place. Per this blog the format of VM names and their captures is as follows:
"Once you have a virtual machine set up and configured as you want, you can capture the instance as a VM Image. During the capture process, all relevant properties of the virtual machine and disks are stored and copies of the backing VHD page blob(s) are made. The copy of each VHD is stored in the same storage account and container as the original VHD being copied. If you are interested in finding the copied VHDs in Azure Storage, search for page blobs with the following naming convention: for the OS VHD, we use -os-YYYY-MM-DD<-ZZ> and for the copied data VHDs, we use -datadisk--YYYY-MM-DD(-ZZ). The date is when the VM Image was captured and the -ZZ is a number, added only if there is a collision, to make the name unique."
Is this what you see?
from azure-storage-python.
Partially. All the blobs with the snapshots in them are on the same VHD storage. So this expected behavior. However, querying with the API returns an empty string. To infer a string manually is unclear because one blob's name is for example 'backup1-os-2015-12-28.vhd' with a last modified date of '12/27/2015 6:22 PM'. Since per the blog post "the -ZZ is a number, added only if there is a collision" and "for the OS VHD, we use -os-YYYY-MM-DD<-ZZ>", I was tempted to delete the snapshot with two string identifiers:
blob_service.delete_blob('vhds', blobs[1].name, snapshot="backup1-os-2015-12-28.vhd",
x_ms_delete_snapshots="only")
blob_service.delete_blob('vhds', blobs[1].name, snapshot="backup1-os-2015-12-27.vhd", x_ms_delete_snapshots="only")
Both of them threw this error:
AzureHttpError: Value for one of the query parameters specified in the request URI is invalid.
<?xml version="1.0" encoding="utf-8"?>
<Error><Code>InvalidQueryParameterValue</Code><Message>Value for one of the query parameters specified in the request URI is invalid.
RequestId:
Time:2016-02-10T00:28:53.4610242Z</Message><QueryParameterName>snapshot</QueryParameterName><QueryParameterValue>backup1-os-2015-12-27.vhd</QueryParameterValue></Error>
It seems to me, that snapshot identifiers are buried somewhere in the system while the API doesn't return them or they weren't stored at the time of creation.
from azure-storage-python.
VM captures are not snapshots. :) Per the blog, the blob name has the date in it. So, that you're not getting snapshots back from list_blobs and that deleting them isn't working makes perfect sense. Check the output of list and the naming format and you should see that many of the blobs have the capture format the blog lists. Then you can delete them without any of the snapshot tooling.
from azure-storage-python.
Is this working for you?
from azure-storage-python.
Thank you for follow-up. Indeed all of my blobs I wanted to delete were non-snapshot blobs how the API sees them (despite ambiguity also in the blog post). However, none of them could be selectively deleted from vhds containers on a running and even stopped machine due to lease locks on every blob in the vhds (not only the currently used ones). I think this inconvenience is somewhat common. I concluded to deprovision and tear the machine down was fastest method going forward.
On 23.02.2016, at 14:06, Emily Gerner [email protected] wrote:
Is this working for you?
—
Reply to this email directly or view it on GitHub #112 (comment).
from azure-storage-python.
Okay, sounds like you figured it out then. I'll close the issue.
FYI: If they're storage leases and you absolutely need to break them manually you can use the break_blob_lease method (in 0.30.0) or in older versions, lease_blob with the lease action set to break. That being said, your method is probably safer and more recommended, especially on active machines.
from azure-storage-python.
Related Issues (20)
- new feature request: adding unix style globbing tool to list files on azure blobs
- https://github.com/Azure/azure-storage-python has no package.swift manifest
- get md5 for blob HOT 1
- BlobClient, BlobServiceClient, ContainerClient cannot be imported HOT 1
- `exists` method for the async blob client leaks file descriptors
- Large File Upload Max_Conncurrenct HOT 1
- How to check if blob is directory
- Please fix mypy issues HOT 5
- ModuleNotFoundError: No module named 'azure.storage.cloudstorageaccount' HOT 2
- Not able to upload bigger files into azure blob container HOT 1
- Cannot import name 'RecoveryServicesBackupClient' from 'azure.mgmt.recoveryservicesbackup' HOT 3
- new version cryptography dependency crashes app
- Streaming blob to client side ?
- Include millisecond component (%f) in datetime serialization
- This repo is missing important files
- How to handle timeout for Uploading a blob in Azure Storage using Python SDK? HOT 1
- ModuleNotFoundError: No module named 'azure.storage'
- problem with python get_file_properties HOT 2
- Create symlink via azure.storage.file FileService
- I'm getting Azure.Storage.Blob can not import in Azure ML notebook, what is the reason?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-storage-python.