Comments (17)
If I put together a PR that provides both AES-CBC and AES-GCM client side encryption, would there be appetite to adding it to the library?
from boto3.
Hi,
I've been facing somehow the same issue, and I found a lib that provides compatibility with the Ruby SDK:
https://github.com/boldfield/s3-encryption
I've not been able to test thoroughly but I believe it could help. I'm on a project that requires client side encryption but is unfortunately built around the original boto. I'll subscribe this issue and if I come to use boto3 for this purpose I'd be glad to help.
from boto3.
@Caligatio Could you maybe share it?
from boto3.
Isn't there a direct way to get decrypted from S3 by passing in KMS Key Id value?
from boto3.
At the moment there is no such functionality in Boto 3, however future hand-written customizations could provide something similar.
from boto3.
Seems like this should be pretty doable. Started working on it for client side with AWS KMS, but ran into some issues with decrypting the envelope key. Will update once it is working. Here is current code snippet:
import base64
from Crypto.Cipher import AES
encrypted = bucket.get_key(object_key)
metadata = encrypted.metadata
envelope_key = base64.b64decode(metadata['x-amz-key-v2'])
envelope_iv = base64.b64decode(metadata['x-amz-iv'])
encryption_key = kms.decrypt(CiphertextBlob=envelope_key)
from boto3.
Got it working with KMS CMK key. Works with client side sdk for ruby and java. Used the following link as a starting point:
http://stackoverflow.com/questions/29784535/how-to-decrypt-aws-ruby-client-side-encryption-in-python
Same code as previous comment for decrypting envelope key with one tweak:
import base64
import json
from Crypto.Cipher import AES
encrypted = bucket.get_key(object_key)
metadata = encrypted.metadata
envelope_key = base64.b64decode(metadata['x-amz-key-v2'])
envelope_iv = base64.b64decode(metadata['x-amz-iv'])
encrypt_ctx = json.loads(metadata['x-amz-matdesc'])
encryption_key = kms.decrypt(CiphertextBlob=envelope_key,EncryptionContext=encrypt_ctx)
from boto3.
I'm trying to do the kms.decrypt thing and failing. @ty-dev does the above work for you?
from boto3.
Hey Tedder,
Here is the code I got working to decrypt Java KMS client side uploads. Its hacky but was enough to prove out the solution. I did the encryption side for both python and dotnet as well if needed.
from __future__ import print_function
import boto
import boto3
import tempfile
import base64
import json
import Crypto
from Crypto.Cipher import AES
from Crypto import Random
import os, random, struct
# decrypt_file method from: http://eli.thegreenplace.net/2010/06/25/aes-encryption-of-files-in-python-with-pycrypto
def decrypt_file(key, in_filename, iv, original_size, out_filename=None, chunksize=16*1024):
if not out_filename:
out_filename = 'tempfile.png'
with open(in_filename, 'rb') as infile:
decryptor = AES.new(key, AES.MODE_CBC, iv)
with open(out_filename, 'wb') as outfile:
while True:
chunk = infile.read(chunksize)
if len(chunk) == 0:
break
outfile.write(decryptor.decrypt(chunk))
outfile.truncate(original_size)
REGION = '***'
BUCKET = '***'
s3_key = '***.png'
filename = '***.png'
s3 = boto3.client('s3')
kms = boto3.client('kms')
# download encrypted object from S3
encrypted = s3.get_object(Bucket=BUCKET,Key=s3_key)
# get object metadata from encrypted object and decode base64 strings
metadata = encrypted['Metadata']
envelope_key = base64.b64decode(metadata['x-amz-key-v2'])
envelope_iv = base64.b64decode(metadata['x-amz-iv'])
encrypt_ctx = json.loads(metadata['x-amz-matdesc'])
original_size = metadata['x-amz-unencrypted-content-length']
# use AWS KMS to decrtyp envelop key (envelop key is used to encrypt object data)
envelope_key_decrypt = kms.decrypt(CiphertextBlob=envelope_key,EncryptionContext=encrypt_ctx)
print(envelope_key_decrypt)
# download encrypted object directly to file (could also just write existing encrypted object to file)
s3.download_file(BUCKET, s3_key, filename)
# decrypt file
decrypt_file(envelope_key_decrypt['Plaintext'],filename,envelope_iv, int(original_size))
from boto3.
Brilliant. I just rewrote my code slightly and have it working. The main issue was the EncryptionContext- both what it needed to be, and that it needed to be json.parsed. It also means s3-encryption is unnecessary.
If you use download_file
in your script, I'd suggest changing get_object
to head_object
since it isn't necessary.
from boto3.
Okay, I just reversed it- this code does a "put". It's my proof-of-concept code, so it needs some refactoring and such. I verified it works by pulling an object through the Java SDK, but please let me know if you test it independently.
https://github.com/tedder/s3-client-side-encryption/blob/master/put.py
from boto3.
Fixed an edge case with padding in my put.py, so make sure to update if you happen to have grabbed my older version.
I'd love to inject this into s3 similar to how s3transfer works but I need someone more fluent in boto3.
from boto3.
Checking in on this issue. I haven't seen traffic on it for a while. Does it make sense to revive it?
from boto3.
I know that one of the reasons it might be hard to include it to boto3 is PyCrypto dependency. Maybe it could be made as separate package / optional dependency then?
from boto3.
@mariusgrigaitis One could use cryptography and avoid PyCrypto altogether. I have a proof-of-concept that uses it and it works just fine.
from boto3.
No there is not.
from boto3.
Hey Tedder,
Here is the code I got working to decrypt Java KMS client side uploads. Its hacky but was enough to prove out the solution. I did the encryption side for both python and dotnet as well if needed.
from __future__ import print_function import boto import boto3 import tempfile import base64 import json import Crypto from Crypto.Cipher import AES from Crypto import Random import os, random, struct # decrypt_file method from: http://eli.thegreenplace.net/2010/06/25/aes-encryption-of-files-in-python-with-pycrypto def decrypt_file(key, in_filename, iv, original_size, out_filename=None, chunksize=16*1024): if not out_filename: out_filename = 'tempfile.png' with open(in_filename, 'rb') as infile: decryptor = AES.new(key, AES.MODE_CBC, iv) with open(out_filename, 'wb') as outfile: while True: chunk = infile.read(chunksize) if len(chunk) == 0: break outfile.write(decryptor.decrypt(chunk)) outfile.truncate(original_size) REGION = '***' BUCKET = '***' s3_key = '***.png' filename = '***.png' s3 = boto3.client('s3') kms = boto3.client('kms') # download encrypted object from S3 encrypted = s3.get_object(Bucket=BUCKET,Key=s3_key) # get object metadata from encrypted object and decode base64 strings metadata = encrypted['Metadata'] envelope_key = base64.b64decode(metadata['x-amz-key-v2']) envelope_iv = base64.b64decode(metadata['x-amz-iv']) encrypt_ctx = json.loads(metadata['x-amz-matdesc']) original_size = metadata['x-amz-unencrypted-content-length'] # use AWS KMS to decrtyp envelop key (envelop key is used to encrypt object data) envelope_key_decrypt = kms.decrypt(CiphertextBlob=envelope_key,EncryptionContext=encrypt_ctx) print(envelope_key_decrypt) # download encrypted object directly to file (could also just write existing encrypted object to file) s3.download_file(BUCKET, s3_key, filename) # decrypt file decrypt_file(envelope_key_decrypt['Plaintext'],filename,envelope_iv, int(original_size))
Hi,
you can send me the encryption side for python ?
from boto3.
Related Issues (20)
- Collaborate with AWS SDK Code Examples team
- ECS ListContainerInstances (1.34.150) - incorrect 'MaxItems' details HOT 2
- AWS Lambda - "list_coverage" filter on `lambdaFunctionRuntime` not working HOT 8
- list_protections doc does not state an optional field. HOT 3
- Converse API skips 'required' tool spec parameters HOT 2
- Dangling Parameter - should be removed from SNS Docs HOT 1
- Image corruption when uploading to S3 using Boto3 in Lambda HOT 2
- Use IAM ID instead of username HOT 1
- Example for MultipartUpload HOT 5
- Error writing records to Timestream HOT 2
- Bedrock - Add way to identify which models an account currently has access to HOT 3
- Inconsistent description of `type` response for `describe_directories` HOT 2
- Bedrock invoke_agent boto3 does not have knowledgeBaseConfigurations HOT 6
- Allow to define client_config when using boto3.setup_default_session HOT 1
- Default Value for `NextToken` in `SageMaker.Client.list_model_packages` HOT 5
- why MissingFields error is occured HOT 2
- How to invoke an API gateway rest api endpoint with credentials HOT 1
- create_budget 'CostFilters' doesn't filter cost data with tag and service HOT 3
- CognitoIdentityProvider object has no attribute 'confirm' HOT 5
- boto3 fails to submit Batch job with env var in eks pod properties override with a string value of ISO date HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from boto3.