thumbor-community / aws Goto Github PK

View Code? Open in Web Editor NEW

This project forked from willtrking/thumbor_aws

155.0 155.0 71.0 247 KB

Thumbor AWS extensions

License: MIT License

Python 98.86% Makefile 1.14%

aws's People

Stargazers

Watchers

Forkers

eventmobi bladrak seichner gitter-badger guilhermef andrew-a-dev voxmedia abaldwin1 pgr0ss vmadmax aoqfonseca vladimirfilonov raphaelfruneaux pedrommone sayeghr eduherraiz lostdragon nzme fanhero christianjgreen kevinmusker phoet komoot carmi miracle2k leafly-com pida42 calla987 kmk-online massipasquesi thewhit tsauzeau babylone-repo nbolender ypcrts ahmedmashour codywilbourn bopo philipbjorge kkopachev narashin gendigbadig juliandbt thehappydinoa bassrock tideorz ahuangpod internetyi tnftechindia inalbisdev orbitvu neighborhoods giftpyk93 eitanshapiro gmax76 mrlesh gauravthapar amanagr bagipriyank mvanbrummen ca7dem0n msgpo fdintino dominik-bln ckan719 gcavalcante8808 mrecl dannychi logart thenounproject bniwredyc

aws's Issues

Fix thumbor required versions (in future releases)

As you can't guarantee that a tc_aws version will be compatible with all future Thumbor releases you should probably don't just use a greater then requirement, such as >=5.2. I currenly have the problem that this installs thumbor 6.0 with the tc_aws 2.0 release which somehow breaks (at least when keeping configs identical). To avoid this in the future it would be nice to pin the the dependency to specific version or at least just a range of versions that are guaranteed to work together with the tc_aws release.

Thanks for the consideration.

The result storage (S3) is not very helpfull for signed keys and safe urls

Everytime a page loads, my backend creates the images with different signed keys. Thumbor analyses these images and creates thumbnails on-the-fly. The results of thumbor all get stored in the result storage (S3 storage) as different images, because of the changing safe url thumbor key and the different amazon s3 signing key. It would be nice if it could detect and omit those keys.

How to use this loader?

Hi,

Where can I find instructions on how to use this loader with thumbor? We have used the version by willtrking successfully before with private buckets on Amazon. With his version, we just enter the key and secret in the thumbor config file.

I've followed the getting started guide on Boto 3, which I guess you are using instead of raw strings in the thumbor config?

Still I get this error when trying to retreive an image using the same url we used before:

I'm not a python developer, so any help would be greatly appreciated :)

i got this error, is there something configure error

tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 282, in wrapper
yielded = next(result)
File "/usr/local/lib/python2.7/dist-packages/thumbor/handlers/imaging.py", line 31, in check_image
exists = yield gen.maybe_future(self.context.modules.storage.exists(kw['image'][:self.context.config.MAX_ID_LENGTH]))
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 473, in wrapper
future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 232, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 461, in wrapper
result = f(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/tc_aws/aws/storage.py", line 107, in exists
self.storage.get(file_abspath, callback=return_data)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 473, in wrapper
future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 232, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 461, in wrapper
result = f(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/tc_aws/aws/bucket.py", line 44, in get
Key=self._clean_key(path),
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 97, in call
return self._make_api_call(operation_name=self.operation, api_params=kwargs, callback=callback)
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 58, in _make_api_call
request_dict = self.client._convert_to_request_dict(api_params, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 598, in _convert_to_request_dict
params=api_params, model=operation_model, context=context)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/utils.py", line 937, in redirect_from_cache
context['signing'] = {'bucket': bucket}
TypeError: 'NoneType' object does not support item assignment

S3 Storage breaking smart detectors?

Hello, I am getting these errors:

ubuntu@ip-10-0-255-97:~$ thumbor -c /etc/thumbor-config.conf  -l debug
libdc1394 error: Failed to initialize libdc1394
2016-05-10 00:19:59 root:DEBUG thumbor running at 0.0.0.0:8888
2016-05-10 00:20:06 thumbor:DEBUG Cleaning key: gfr/source-assets/img/2016/05/09/9524d1f3-027e-4335-97c1-b12bc462780f.jpg
2016-05-10 00:20:06 thumbor:DEBUG Cleansed key: gfr/source-assets/img/2016/05/09/9524d1f3-027e-4335-97c1-b12bc462780f.jpg
2016-05-10 00:20:06 thumbor:DEBUG Cleaning key: gfr/source-assets/img/2016/05/09/9524d1f3-027e-4335-97c1-b12bc462780f.jpg
2016-05-10 00:20:06 thumbor:DEBUG Cleansed key: gfr/source-assets/img/2016/05/09/9524d1f3-027e-4335-97c1-b12bc462780f.jpg
2016-05-10 00:20:06 thumbor:DEBUG METRICS: inc: storage.hit:1
2016-05-10 00:20:06 thumbor:DEBUG Cleaning key: gfr/source-assets/img/2016/05/09/9524d1f3-027e-4335-97c1-b12bc462780f.detectors.txt
2016-05-10 00:20:06 thumbor:DEBUG Cleansed key: gfr/source-assets/img/2016/05/09/9524d1f3-027e-4335-97c1-b12bc462780f.detectors.txt
2016-05-10 00:31:41 botocore.utils:DEBUG URI updated to: https://<bucketpath>/img/2016/05/09/9524d1f3-027e-4335-97c1-b12bc462780f.detectors.txt
2016-05-10 00:31:41 botocore.auth:DEBUG Calculating signature using hmacv1 auth.
2016-05-10 00:31:41 botocore.auth:DEBUG HTTP request method: GET
2016-05-10 00:20:06 tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1017, in run
    yielded = self.gen.send(value)
  File "/usr/local/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 177, in get_image
    self.filters_runner.apply_filters(thumbor.filters.PHASE_AFTER_LOAD, transform)
  File "/usr/local/lib/python2.7/dist-packages/thumbor/filters/__init__.py", line 81, in apply_filters
    callback()
  File "/usr/local/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 175, in transform
    self.context.transformer.transform(after_transform_cb)
  File "/usr/local/lib/python2.7/dist-packages/thumbor/transformer.py", line 102, in transform
    self.smart_detect()
  File "/usr/local/lib/python2.7/dist-packages/thumbor/transformer.py", line 151, in smart_detect
    self.do_smart_detection().result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 233, in result
    self._check_done()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 311, in _check_done
    raise Exception("DummyFuture does not support blocking for results")
Exception: DummyFuture does not support blocking for results
2016-05-10 00:30:46 botocore.parsers:DEBUG Response headers: <tornado.httputil.HTTPHeaders object at 0x7fc4c9a11050>
2016-05-10 00:30:46 botocore.parsers:DEBUG Response body:
<botocore.response.StreamingBody object at 0x7fc4c9c72110>

I removed all lines with botocore (except those last few which are somewhat interesting in that they happen around the exception) or x-amz-security on them, but there were no errors from botocore, it was properly getting the files and putting them ( I can see the results on the bucket itself).

If I disable all detectors no error messages pop up (but of course smart detection doesn't do diddly), if I switch to the default file storage (literally the only change is commenting the STORAGE = 'tc_aws.storages.s3_storage' line) then everything works perfectly.

I will try to have more feedback tomorrow but if you guys could point me to where in the code I can look for issues I'd appreciate it. I suspect it has something to do with the recent async changes, specifically aws.storage.AwsStorage.put_detector_data being changed to a Future as a side effect of aws.storage.AwsStorage.set being changed. It's the only big difference I can see with thumbor.storages.file_storage.

P.S. This is the content of one of the .detectors.txt in S3:

[{"origin": "Face Detection", "height": 49, "width": 49, "y": 51.120000000000005, "x": 106, "z": 2401}, {"origin": "Face Detection", "height": 47, "width": 47, "y": 53.36, "x": 186, "z": 2209}]

Parameter validation failed using result_storage

I'm testing this thumbor_aws fork trying to include it my docker-thumbor image:
APSL/docker-thumbor#4

The case is, is working when put a image at s3 storage, but when try to put the same with the result_storage I get the exception:

thumbor_1 | 2016-01-15 16:30:03 thumbor:ERROR ERROR: Traceback (most recent call last):
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado/web.py", line 1401, in _stack_context_handle_exception
thumbor_1 |     raise_exc_info((type, value, traceback))
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado/stack_context.py", line 314, in wrapped
thumbor_1 |     ret = fn(*args, **kwargs)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/thumbor/handlers/__init__.py", line 343, in save_to_result_storage
thumbor_1 |     context.modules.result_storage.put(results)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tc_aws/result_storages/s3_storage.py", line 35, in put
thumbor_1 |     self.set(bytes, path)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tc_aws/aws/storage.py", line 80, in set
thumbor_1 |     callback=self._handle_error,
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado/concurrent.py", line 473, in wrapper
thumbor_1 |     future.result()
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado/concurrent.py", line 232, in result
thumbor_1 |     raise_exc_info(self._exc_info)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado/concurrent.py", line 461, in wrapper
thumbor_1 |     result = f(*args, **kwargs)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tc_aws/aws/bucket.py", line 98, in put
thumbor_1 |     session.call(**args)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado_botocore/base.py", line 97, in call
thumbor_1 |     return self._make_api_call(operation_name=self.operation, api_params=kwargs, callback=callback)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado_botocore/base.py", line 58, in _make_api_call
thumbor_1 |     request_dict = self.client._convert_to_request_dict(api_params, operation_model)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 369, in _convert_to_request_dict
thumbor_1 |     api_params, operation_model)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/botocore/validate.py", line 273, in serialize_to_request
thumbor_1 |     raise ParamValidationError(report=report.generate_report())
thumbor_1 | ParamValidationError: Parameter validation failed:
thumbor_1 | Invalid type for parameter Metadata, value: <tornado.httputil.HTTPHeaders object at 0x7fefb14277d0>, type: <class 'tornado.httputil.HTTPHeaders'>, valid types: <type 'dict'>

I tried but I can't see the problem.

Use key instead of url

I think this line is supposed to use key instead of url:

aws/tc_aws/loaders/s3_loader.py

Line 30 in 7499028

file_key = bucket_loader.get_key(url)

It looks like the url still includes the bucket name, but the key looks correct.

RESULT_STORAGE_EXPIRATION_SECONDS

First of all, thank you guys for your great work!

The documentation of thumbor distinguish between expiration of storage and result storage:

RESULT_STORAGE_EXPIRATION_SECONDS
STORAGE_EXPIRATION_SECONDS

In comparison tc_aws does not distinguish between and considers STORAGE_EXPIRATION_SECONDS for the result storage too.

Using sha1 hashes as filenames

Would be great to have an option to use digest hash as filenames, as it was with willtrking's version.

As per aws best practices it is not performance efficient with the current schema, when object is stored with its original filename.

Fix documentation

Technical doc is mis-formatted, we should have a generation script & pass on the code to fix it.

Metadata

Hi,

I could not get the metadata (always times out), example : server.url/unsafe/meta/images/filename.jpg or server.url/unsafe/meta/fit-in/0x0/images/filename.jpg.

Is this an issue, or I have to change my configuration?

Thanks.

Frankfurt location not working

Frankfurt as a location is not wokring, until you set in boto the host to "s3.eu-central-1.amazonaws.com" or use boto sigv4.

Look at this issue: boto/boto#2916

Retry on HTTP 5xx errors.

Sometimes S3 calls fail with a 5xx error code. According to the S3 documentation these errors can happen and we can retry the request: it should work eventually!

I'm thinking specifically about fetching images from S3.

Would it be possible for the s3_loader to retry on 5xx error?

License

willtrking#26

AttributeError: Context instance has no attribute 'request'

Hi there,

Receiving the following error when trying to use the tc_aws.storages.s3_storage storage backend with Thumbor with AUTO_WEBP = True configured.

2015-10-23 04:45:39 tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
  File "/srv/virtualenv/thumbor/local/lib/python2.7/site-packages/tornado/gen.py", line 230, in wrapper
    yielded = next(result)
  File "/srv/virtualenv/thumbor/local/lib/python2.7/site-packages/thumbor/handlers/imaging.py", line 32, in check_image
    exists = yield gen.maybe_future(self.context.modules.storage.exists(kw['image'][:self.context.config.MAX_ID_LENGTH]))
  File "/srv/virtualenv/thumbor/local/lib/python2.7/site-packages/tornado/concurrent.py", line 455, in wrapper
    future.result()
  File "/srv/virtualenv/thumbor/local/lib/python2.7/site-packages/tornado/concurrent.py", line 215, in result
    raise_exc_info(self._exc_info)
  File "/srv/virtualenv/thumbor/local/lib/python2.7/site-packages/tornado/concurrent.py", line 443, in wrapper
    result = f(*args, **kwargs)
  File "/srv/virtualenv/thumbor/local/lib/python2.7/site-packages/tc_aws/aws/storage.py", line 102, in exists
    file_abspath = self._normalize_path(path)
  File "/srv/virtualenv/thumbor/local/lib/python2.7/site-packages/tc_aws/aws/storage.py", line 284, in _normalize_path
    if self.is_auto_webp:
  File "/srv/virtualenv/thumbor/local/lib/python2.7/site-packages/tc_aws/aws/storage.py", line 30, in is_auto_webp
    return self.context.config.AUTO_WEBP and self.context.request.accepts_webp
AttributeError: Context instance has no attribute 'request'

Running tc-aws==2.0.7 and thumbor==5.2.1 on Ubuntu 14.04.2.

Anyone else seeing this?

thumbor_aws does not return a 404 when it fails to find an image in the s3 bucket.

Hi there,

I notice in the logs that when tc_aws fails to find an image in the bucket it gets a 404 from the S3 bucket, but it does not propagate that 404 out to the http request to thumbor.

Am I doing something wrong?

Handle multiprocessing

As seen in #14

There is a high chance the aws loader is blocking the i/o loop. Anything that isn't made for Tornado tends to block the loop ( like the redis loader, but it's so fast we don't see the blocking ).
One way to go around this would be to use the multiprocessing module and queue requests to a separate process pool that will fetch the images from AWS and pass them back to the main process using queues.
Might require a little bit of refactoring but it would probably help with concurrency.
http://tornado.readthedocs.org/en/latest/process.html might help with that.
https://gist.github.com/FZambia/5756470

@masom

RESULT_STORAGE_AWS_STORAGE_ROOT_PATH has no effect

See 1df8fdf#commitcomment-11754847

The specified key does not exist.

Hi,

I'm getting the below error

2016-12-05 22:32:31 thumbor:WARNING ERROR retrieving image from S3 https://s3-eu-west-1.amazonaws.com/myapp-thumbor/test.jpg: {'ResponseMetadata': {'HTTPStatusCode': 404, 'HostId': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=', 'RequestId': 'xxxxxxxxxxxxxxxxx'}, 'Error': {'Message': 'The specified key does not exist.', 'Code': 'NoSuchKey', 'Key': 'https:/s3-eu-west-1.amazonaws.com/myapp-thumbor/test.jpg'}}
2016-12-05 22:32:31 tornado.access:WARNING 404 GET /unsafe/300x200/test.jpg (127.0.0.1) 214.86ms

My config
TC_AWS_LOADER_BUCKET='myapp-thumbor' #S3 bucket for loader
TC_AWS_LOADER_ROOT_PATH='https://s3-eu-west-1.amazonaws.com/myapp-thumbor' # S3 path prefix for Loader bucket
LOADER = 'tc_aws.loaders.s3_loader'

This image is public and I can access it: https:/s3-eu-west-1.amazonaws.com/myapp-thumbor/test.jpg
(you can't access because myapp is not real). I guess the key should be "test.jpg"... What am I doing wrong?

Thanks very much

AWS credentials not cached - rising number of network connections

Hi,
it seems that the AWS credentials which are fetched with boto through the instance meta are not cached inside the plugin at the moment. Since boto itself will not cache the credentials they will be fetched on every call to S3.
The latter will cause a constantly rising number of network connections waiting for credentials (probably due to API rate limiting) which causes thumbor to use 100% CPU. The whole thing goes on until the OS finally runs out of file descriptors and goes down.
A workaround is to use a .boto file instead of the credentials from the instance's role.

DELETE not functionnal

maybe I missed something but DELETE does not remove images on my S3 bucket

$ pip freeze 
[...]
boto==2.3.0
botocore==1.3.4
tornado-botocore==1.0.2
tc-aws==2.0.10
thumbor==5.2.1
[...]

$ grep DELETE /etc/thumbor.conf 
UPLOAD_DELETE_ALLOWED = True 

$ curl -vXDELETE localhost:8885/image/d0dd18ed96fd453293a42226726de7a4/image.jpg
* About to connect() to localhost port 8885 (#0)
*   Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8885 (#0)
> DELETE /image/d0dd18ed96fd453293a42226726de7a4/image.jpg HTTP/1.1
> User-Agent: curl/7.26.0
> Host: localhost:8885
> Accept: */*
> 
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 204 No Content
< Date: Thu, 31 Dec 2015 14:48:13 GMT
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
< Server: TornadoServer/4.3
< 
* Connection #0 to host localhost left intact
* Closing connection #0

on the thumbor log side (debug) :

2015-12-31 15:48:13 botocore.credentials:DEBUG Skipping environment variable credential check because profile name was explicitly set.
2015-12-31 15:48:13 botocore.credentials:DEBUG Looking for credentials via: env
2015-12-31 15:48:13 botocore.credentials:DEBUG Looking for credentials via: shared-credentials-file
2015-12-31 15:48:13 botocore.credentials:DEBUG Looking for credentials via: config-file
2015-12-31 15:48:13 botocore.credentials:DEBUG Looking for credentials via: ec2-credentials-file
2015-12-31 15:48:13 botocore.credentials:DEBUG Looking for credentials via: boto-config
2015-12-31 15:48:13 botocore.credentials:DEBUG Looking for credentials via: iam-role
2015-12-31 15:48:13 botocore.vendored.requests.packages.urllib3.connectionpool:INFO Starting new HTTP connection (1): 169.254.169.254
2015-12-31 15:48:13 botocore.vendored.requests.packages.urllib3.connectionpool:DEBUG "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 63
2015-12-31 15:48:13 botocore.vendored.requests.packages.urllib3.connectionpool:INFO Starting new HTTP connection (1): 169.254.169.254
2015-12-31 15:48:13 botocore.vendored.requests.packages.urllib3.connectionpool:DEBUG "GET /latest/meta-data/iam/security-credentials/components-services-demo-thumbor-ThumborAccessRole-XXXXXXXXXXXXXXXX HTTP/1.1" 200 930
2015-12-31 15:48:13 botocore.credentials:INFO Found credentials from IAM Role: components-services-demo-thumbor-ThumborAccessRole-XXXXXXXXXXXXXXXX
2015-12-31 15:48:13 botocore.client:DEBUG Registering retry handlers for service: s3
2015-12-31 15:48:13 botocore.hooks:DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x226a938>
2015-12-31 15:48:13 botocore.hooks:DEBUG Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x22578c0>
2015-12-31 15:48:13 botocore.endpoint:DEBUG Setting s3 timeout as (60, 60)
2015-12-31 15:48:13 botocore.hooks:DEBUG Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x22aaed8>
2015-12-31 15:48:13 botocore.hooks:DEBUG Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x22aae60>
2015-12-31 15:48:13 botocore.hooks:DEBUG Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x22ad230>
2015-12-31 15:48:13 tornado_botocore.base:DEBUG Making request for <botocore.model.OperationModel object at 0x2885590> (verify_ssl=True) with params: {'body': '', 'url': u'https://s3-us-west-1.amazonaws.com/xxxxxxxxxxxxbucketxxxxxxxxxx/demo/storage/d0dd18ed96fd453293a42226726de7a4', 'headers': {'User-Agent': 'Botocore/1.3.4 Python/2.7.3 Linux/3.2.0-4-amd64'}, 'query_string': {}, 'url_path': u'/xxxxxxxxxxxxbucketxxxxxxxxxx/demo/storage/d0dd18ed96fd453293a42226726de7a4', 'method': u'GET'}
2015-12-31 15:48:13 botocore.hooks:DEBUG Event request-created.s3.GetObject: calling handler <bound method S3._sign_request of <botocore.client.S3 object at 0x288a950>>
2015-12-31 15:48:13 botocore.hooks:DEBUG Event before-sign.s3.GetObject: calling handler <function fix_s3_host at 0x20fb488>
2015-12-31 15:48:13 botocore.utils:DEBUG Checking for DNS compatible bucket for: https://s3-us-west-1.amazonaws.com/xxxxxxxxxxxxbucketxxxxxxxxxx/demo/storage/d0dd18ed96fd453293a42226726de7a4
2015-12-31 15:48:13 botocore.utils:DEBUG URI updated to: https://viadeo-images-us-west-1.s3.amazonaws.com/demo/storage/d0dd18ed96fd453293a42226726de7a4
2015-12-31 15:48:13 botocore.auth:DEBUG Calculating signature using hmacv1 auth.
2015-12-31 15:48:13 botocore.auth:DEBUG HTTP request method: GET
2015-12-31 15:48:13 botocore.auth:DEBUG StringToSign:
GET


Thu, 31 Dec 2015 14:48:13 GMT
x-amz-security-token:AQoDYXdzENf//////////XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
/xxxxxxxxxxxxbucketxxxxxxxxxx/demo/storage/d0dd18ed96fd453293a42226726de7a4
2015-12-31 15:48:13 botocore.parsers:DEBUG Response headers: <tornado.httputil.HTTPHeaders object at 0x2629710>
2015-12-31 15:48:13 botocore.parsers:DEBUG Response body:
<botocore.response.StreamingBody object at 0x2194990>
<tc_aws.storages.s3_storage.Storage object at 0x24a1890>
2015-12-31 15:48:13 tornado.access:INFO 204 DELETE /image/d0dd18ed96fd453293a42226726de7a4/image.jpg (127.0.0.1) 156.78ms

And file still there ...

plus, I try to add print 'tc_aws.aws.storage.remove:', path in /usr/local/lib/python2.7/dist-packages/tc_aws/aws/storage.py remove method.
And nothing appear ... /o
(at least adding print self.context.modules.storage in /usr/local/lib/python2.7/dist-packages/thumbor/handlers/image_resource.py shows <tc_aws.storages.s3_storage.Storage object at 0x24a1890> in logs ... )

TypeError: argument of type 'NoneType' is not iterable

Hi
not sure if this is thumbor_aws problem, but ...
I'm trying to store unsafe results to s3. If i enable RESULT_STORAGE_STORES_UNSAFE, thumbor crashes:

2016-01-14 12:36:06 tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 63, in execute_image_operations
    result = yield gen.maybe_future(self.context.modules.result_storage.get())
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 314, in wrapped
    ret = fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 469, in run_callback
    callback(future.result())
  File "/usr/local/lib/python2.7/dist-packages/tc_aws/result_storages/s3_storage.py", line 49, in return_result
    if self._get_error(key):
  File "/usr/local/lib/python2.7/dist-packages/tc_aws/aws/storage.py", line 236, in _get_error
    return response['Error']['Message'] if 'Error' in response else None

Source storage (tc_aws.storages.s3_storage) is working fine. My configuration:

ALLOWED_SOURCES = ["whitelistedurls"]
ALLOW_UNSAFE_URL = "True"
HTTP_LOADER_FOLLOW_REDIRECTS = "True"
MAX_AGE = 2678400
RESULT_STORAGE_STORES_UNSAFE = "True"
RESULT_STORAGE = "tc_aws.result_storages.s3_storage"
STORAGE = "tc_aws.storages.s3_storage"
TC_AWS_REGION = "eu-west-1"
TC_AWS_RESULT_STORAGE_BUCKET = "my-thumbor-bucket"
TC_AWS_RESULT_STORAGE_ROOT_PATH = "result"
TC_AWS_STORAGE_BUCKET = "my-thumbor-bucket"
TC_AWS_STORAGE_ROOT_PATH = "storage"
TC_AWS_STORAGE_RRS = "False"
TC_AWS_STORAGE_SSE = "False"

Is it possible to configure a custom host in version 2.0.0?

In version 1.3.2 the following can be configured:

# Optional additional configuration for the Boto-Client used to access S3.
# see http://boto.readthedocs.org/en/latest/ref/s3.html?highlight=boto.s3.connection.s3connection#boto.s3.connection.S3Connection
# for all available config options
BOTO_CONFIG = {
    'host': 'fakes3.local.dev',
    'is_secure': False
}

What would be the way to configure this in version 2.0.0?

Implement Expiration parameter

Hello,

In default configuration of Storage in thumbor project. I can configure EXPIRATION. It doesn't seem to be implemented in AWS Storage.

################################# File Storage #################################
## Expiration in seconds for the images in the File Storage. Defaults to one
## month
## Defaults to: 2592000
#STORAGE_EXPIRATION_SECONDS = 2592000

################################ Result Storage ################################
## Expiration in seconds of generated images in the result storage
## Defaults to: 0
#RESULT_STORAGE_EXPIRATION_SECONDS = 0

Image Urls without filename do not store in S3

thumbor/unsafe/400x0/filters:format(png)/https://d3ix816x6wuc0d.cloudfront.net/i/372852/

That URL creates an empty directory in S3 rather than storing the image when using storages.s3_storage

Removing the slash, does work -- thumbor/unsafe/400x0/filters:format(png)/https://d3ix816x6wuc0d.cloudfront.net/i/372852

Are result storage uploads async?

I'm wondering if I configure result storage to use S3 implementation, these uploads are going to be async or they will block the thread?

Size marking

Hi!

I am using - https://github.com/APSL/docker-thumbor. Is there a way to configure Thumbor to mark size of image upon resize request? e.g.:

http://thumbor:9000/unsafe/100x/s3-eu-central-1.amazonaws.com/grok7-dev/orionjs/15d0654f-88d7-4b31-ad5f-7ffb434e9d1b.jpg

Result:

https://s3-eu-central-1.amazonaws.com/thumbor/grok7-dev/orionjs/100x/15d0654f-88d7-4b31-ad5f-7ffb434e9d1b.jpg

https://s3-eu-central-1.amazonaws.com/thumbor/grok7-dev/orionjs/15d0654f-88d7-4b31-ad5f-7ffb434e9d1bx100.jpg

latest botocore does not allow private s3 urls

using the pre-signed-loader is currently broken for private S3-urls because of an issue in botocore.

/cc @Bladrak

Uniformize options names

It would be nice at some point to move all config values to AWS_ or S3_ prefix to prevent collisions.

Migrate tests to nose

S3_ALLOWED_BUCKETS not working properly in conjunction with watermark

As discussed with Bladrak I am opening this issue in respects of the S3 loader not working properly.

Please see the below config and logs.

S3_LOADER_BUCKET = 'my-bucket-xxx'
S3_ALLOWED_BUCKETS = ['my-bucket-xxx']
AWS_ENABLE_HTTP_LOADER=True

2015-09-28 13:51:13 thumbor:DEBUG STATSD: storage.miss:1|c
2015-09-28 13:51:13 tornado.access:WARNING 404 GET /unsafe/fit-in/525x650/filters:watermark(/my-bucket-xxx/watermark.png,0,0,0)/my-bucket-xxx/APAP131C78CE.jpg (127.0.0.1) 5.05ms

Error when trying to get image from bucket

Hi,
i have this image and i can normally get it whit my thumbor service here http://thumborservice-weflymate.rhcloud.com/unsafe/https://s3-eu-west-1.amazonaws.com/3cube/thumbor/storage/035de4821bbb4a109a8cfcb2bf4af3a08f9ee298.
As you can see the image is on aws s3 and i have configured my thumbor instance to access this s3 instance, but if i try to get image directly from instance (url) i get the following error:

2016-11-17 14:08:31 thumbor:ERROR ERROR: Traceback (most recent call last):
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/thumbor/handlers/__init__.py", line 142, in get_image
    self.context.request.image_url
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/gen.py", line 1024, in run
    yielded = self.gen.send(value)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/thumbor/handlers/__init__.py", line 516, in _fetch
    loader_result = yield self.context.modules.loader.load(self.context, url)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 478, in wrapper
    future.result()
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 466, in wrapper
    result = f(*args, **kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tc_aws/loaders/s3_loader.py", line 49, in load
    bucket_loader.get(key, callback=handle_data)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 478, in wrapper
    future.result()
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado/concurrent.py", line 466, in wrapper
    result = f(*args, **kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tc_aws/aws/bucket.py", line 48, in get
    Key=self._clean_key(path),
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado_botocore/base.py", line 97, in call
    return self._make_api_call(operation_name=self.operation, api_params=kwargs, callback=callback)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/tornado_botocore/base.py", line 58, in _make_api_call
    request_dict = self.client._convert_to_request_dict(api_params, operation_model)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/client.py", line 563, in _convert_to_request_dict
    params=api_params, model=operation_model, context=context)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/var/lib/openshift/58107eca2d5271d5a10000da/app-root/data/lib/python2.7/site-packages/botocore/utils.py", line 947, in redirect_from_cache
    context['signing'] = {'bucket': bucket}
TypeError: 'NoneType' object does not support item assignment

2016-11-17 14:08:31 thumbor:ERROR [BaseHandler] get_image failed for url `035de4821bbb4a109a8cfcb2bf4af3a08f9ee298`. error: `'NoneType' object does not support item assignment`
2016-11-17 14:08:31 tornado.access:ERROR 500 GET /unsafe/100x100/035de4821bbb4a109a8cfcb2bf4af3a08f9ee298 (127.3.209.129) 1395.57ms

Below there a part of my thumbor.conf file:

## The loader thumbor should use to load the original image. This must be the
## full name of a python module (python must be able to import it)
## Defaults to: thumbor.loaders.http_loader
#LOADER = 'thumbor.loaders.http_loader'
#LOADER = 'tc_aws.loaders.s3_loader'
LOADER = 'tc_aws.loaders.s3_loader'

## The file storage thumbor should use to store original images. This must be the
## full name of a python module (python must be able to import it)
## Defaults to: thumbor.storages.file_storage
STORAGE = 'thumbor_aws.storages.s3_storage'
AWS_ACCESS_KEY = 'KEY'
AWS_SECRET_KEY = 'SECRET'
STORAGE_BUCKET = '3cube'

## The result storage thumbor should use to store generated images. This must be
## the full name of a python module (python must be able to import it)
## Defaults to: None
RESULT_STORAGE = 'thumbor_aws.result_storages.s3_storage'
RESULT_STORAGE_BUCKET = '3cube'

TC_AWS_REGION='eu-west-1' # AWS Region

TC_AWS_STORAGE_BUCKET= '3cube' # S3 bucket for Storage
TC_AWS_STORAGE_ROOT_PATH='' # S3 path prefix for Storage bucket

TC_AWS_LOADER_BUCKET='3cube' #S3 bucket for loader
TC_AWS_LOADER_ROOT_PATH='' # S3 path prefix for Loader bucket

TC_AWS_RESULT_STORAGE_BUCKET= '3cube' # S3 bucket for result Storage
TC_AWS_RESULT_STORAGE_ROOT_PATH='' # S3 path prefix for Result storage bucket

# put data into S3 using the Server Side Encryption functionality to
# encrypt data at rest in S3
# https://aws.amazon.com/about-aws/whats-new/2011/10/04/amazon-s3-announces-server-side-encryption-support/
TC_AWS_STORAGE_SSE=False

# put data into S3 with Reduced Redundancy
# https://aws.amazon.com/about-aws/whats-new/2010/05/19/announcing-amazon-s3-reduced-redundancy-storage/
TC_AWS_STORAGE_RRS=False


# Enable HTTP Loader as well?
# This would allow you to load watermarks in over your images dynamically through a URI
# E.g.
# http://your-thumbor.com/unsafe/filters:watermark(http://example.com/watermark.png,0,0,50)/s3_bucket/photo.jpg
TC_AWS_ENABLE_HTTP_LOADER=True

TC_AWS_ALLOWED_BUCKETS=False # List of allowed bucket to be requested
TC_AWS_STORE_METADATA=False # Store result with metadata (for instance content-type)

## The imaging engine thumbor should use to perform image operations. This must
## be the full name of a python module (python must be able to import it)
## Defaults to: thumbor.engines.pil
ENGINE = 'thumbor.engines.pil'

Thanks for help ;)

Install documentation

pip install tc_aws

RESULT_STORAGE_AWS_STORAGE_ROOT_PATH have impact on tc_aws.storages.s3_storage

following #4 setting RESULT_STORAGE_AWS_STORAGE_ROOT_PATH other than '/' make tc_aws.storages.s3_storage fail at getting original keys

This should only impact on tc_aws.result_storages.s3_storage

We should fine a way to merge those code base and avoid duplication.

Using Smart/face detection dos not work

2016-12-13 02:27:55 tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1024, in run
yielded = self.gen.send(value)
File "/usr/local/lib/python2.7/dist-packages/thumbor/handlers/init.py", line 211, in get_image
self.filters_runner.apply_filters(thumbor.filters.PHASE_AFTER_LOAD, transform)
File "/usr/local/lib/python2.7/dist-packages/thumbor/filters/init.py", line 81, in apply_filters
callback()
File "/usr/local/lib/python2.7/dist-packages/thumbor/handlers/init.py", line 209, in transform
self.context.transformer.transform(after_transform_cb)
File "/usr/local/lib/python2.7/dist-packages/thumbor/transformer.py", line 102, in transform
self.smart_detect()
File "/usr/local/lib/python2.7/dist-packages/thumbor/transformer.py", line 151, in smart_detect
self.do_smart_detection().result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 238, in result
self._check_done()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 316, in _check_done
raise Exception("DummyFuture does not support blocking for results")
Exception: DummyFuture does not support blocking for results
2016-12-13 02:27:55 thumbor:WARNING [AwsStorage] s3 key not found at 234e6915e9c52302.29448476.detectors.txt

use thumbor_aws with s3 compatible server

Hi.

we are begining with thumbor and would like use this plugin to save result images on a bucket on our s3 compatible cluster.
Is possible to modify the url of the server hosting the bucket ?

using s3cmd the param is host_base

regards

S3 connection never re-used?

I've just noticed that get_connection method from connection.py never reuses an S3 connection, since module level connection variable is never assigned to anything but None. I wonder if that's by design, and what are possible drawbacks of keeping a reference to an S3 connection once it was initialized?

Cover S3 Loader in tests

Currently, the S3 loader is not covered by the tests. We should cover it somehow (this will most likely require us to fully load up thumbor to do so).

versionId

Any planned support for AWS S3 object versionId? I didn't see anything regarding this in the doc or code.

Instance loader or storage cache

All images are currently loaded from S3, which is basically the source. It would be useful to be able to specify some kind of local storage (e.g. local cache) in order to reduce the amount of requests to AWS S3. We're currently mitigating this issue with a local nginx proxy with a defined cache size.

Variables	Default	Description
AWS_LOCALCACHE_STORAGE	thumbor.storages.file_storage
AWS_LOCALCACHE_STORAGE_EXPIRATION_SECONDS	0	Determine the time in seconds that a file is considered to be expired. 0 disables expiration.
AWS_LOCALCACHE_STORAGE_SIZE	1024	Size of the local cache. Oldest files will be deleted first.

Thumbor - tc_aws - result_storage error

I'm trying install Thumbor with tc_aws and I have a problem with result_storage.

The image can not be saved in result bucket and Thumbor return error that cannot be debugged.

Someone can help me?

Thanks,

Fred

console.out.txt
thumbor.cfg.txt

Remotecv fails using tc_aws.s3_loader

I'm trying to configure remotecv in order to use the tc_aws.loaders.s3_loader

I can see this error logged:

ERROR:pyres.worker:(Job{Detect} | remotecv.pyres_tasks.DetectTask | [u'all', u'path/to/image/s3/image.jpg', u'path/to/image/s3/image.jpg']) failed: 'module' object has no attribute 'load_sync'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pyres/worker.py", line 253, in process
    return job.perform()
  File "/usr/local/lib/python2.7/site-packages/pyres/job.py", line 55, in perform
    return payload_class.perform(*args)
  File "/usr/local/lib/python2.7/site-packages/remotecv/pyres_tasks.py", line 11, in perform
    image_data = config.loader.load_sync(image_path)
AttributeError: 'module' object has no attribute 'load_sync'

Versions:

argparse==1.2.1
envtpl==0.4.1
boto==2.38.0
dateutils==0.6.6
numpy==1.9.2
pyremotecv==0.5.0
remotecv==2.1.0
opencv-engine==1.0.1
redis==2.4.12
thumbor==5.2.1
tc-aws==2.1.1
tc-core==0.3.0
tc-shortener==0.2.2
raven==5.10.1

S3 path using "?" not functionnal since thumbor 6

For legacy reason we used image id such as xxxxxx?ts=123456
Using Thumbor 1.5.2 it works well storing s3 object just like s3://.../xxxxxx?ts=123456

since Thumbor 6 and tc-aws corresponding version, those image are not available any more, tc-aws is looking for xxxxxx (removing paramater from id)

I tried using encoded URL with "xxxxxx%3Fts=123456" id in URL but now of course tc-aws looks for s3://.../xxxxxx%3Fts%3D123456 (that does not exists either)

Any clue to handle this ?
Add some retry with a special toggle in config maybe ?

tc_aws performance compared to http_loader

Hi all!

@phoet and i are testing the tc_aws-backend on a bunch of Heroku instances with quite a lot of traffic. Before we used the HTTP-Loader. I attached Thumbor's reponse time graph, which shows a significant increase from 19.00, when we switched from http loader to tc_aws (the orange line is the median, jumping from ~300ms to ~450ms.

We wondered if anybody has an idea on possible reasons? What could be the major difference between tc_aws and http-loader requesting an S3-file? Looks a bit like tc_aws runs into some kind of resource limit...?
We will move back to the http loader for now, but still intend to switch to tc_aws if we can fix the performance problem and integrate metrics on the s3 connection times.

S3 storage does not work

Hi,

I set thumbor to use S3 also for the storage, but I'm getting the following error:

2016-12-06 11:23:12 thumbor:ERROR ERROR: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/thumbor-6.2.0-py2.7-linux-x86_64.egg/thumbor/handlers/__init__.py", line 142, in get_image
    self.context.request.image_url
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/gen.py", line 1024, in run
    yielded = self.gen.send(value)
  File "/usr/local/lib/python2.7/site-packages/thumbor-6.2.0-py2.7-linux-x86_64.egg/thumbor/handlers/__init__.py", line 571, in _fetch
    storage.put(url, fetch_result.buffer)
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 478, in wrapper
    future.result()
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 466, in wrapper
    result = f(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/tc_aws/storages/s3_storage.py", line 40, in put
    self.set(bytes, self._normalize_path(path), callback=once_written)
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 478, in wrapper
    future.result()
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 466, in wrapper
    result = f(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/tc_aws/aws/storage.py", line 80, in set
    callback=callback,
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 478, in wrapper
    future.result()
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 466, in wrapper
    result = f(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/tc_aws/aws/bucket.py", line 106, in put
    session.call(**args)
  File "build/bdist.linux-x86_64/egg/tornado_botocore/base.py", line 97, in call
    return self._make_api_call(operation_name=self.operation, api_params=kwargs, callback=callback)
  File "build/bdist.linux-x86_64/egg/tornado_botocore/base.py", line 60, in _make_api_call
    operation_model=operation_model, request_dict=request_dict, callback=callback)
  File "build/bdist.linux-x86_64/egg/tornado_botocore/base.py", line 54, in _make_request
    request_dict=request_dict, operation_model=operation_model, callback=callback)
  File "build/bdist.linux-x86_64/egg/tornado_botocore/base.py", line 40, in _send_request
    validate_cert=False)
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/httpclient.py", line 435, in __init__
    self.body = body
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/httpclient.py", line 480, in body
    self._body = utf8(value)
  File "/usr/local/lib/python2.7/site-packages/tornado-4.4.2-py2.7-linux-x86_64.egg/tornado/escape.py", line 198, in utf8
    "Expected bytes, unicode, or None; got %r" % type(value)
TypeError: Expected bytes, unicode, or None; got <type 'instance'>

2016-12-06 11:23:12 thumbor:ERROR [BaseHandler] get_image failed for url `IMG_3314_mod1.JPG`. error: `Expected bytes, unicode, or None; got <type 'instance'>`
2016-12-06 11:23:12 tornado.access:ERROR 500 GET /unsafe/300x200/IMG_3314_mod1.JPG (127.0.0.1) 219.57ms

Here my configurations:
STORAGE = 'tc_aws.storages.s3_storage'
TC_AWS_REGION='eu-west-1' # AWS Region
TC_AWS_STORAGE_BUCKET='myapp-thumbor' # S3 bucket for Storage
TC_AWS_STORAGE_ROOT_PATH='storage' # S3 path prefix for Storage bucket
TC_AWS_LOADER_BUCKET='myapp-thumbor' #S3 bucket for loader
TC_AWS_LOADER_ROOT_PATH='loader' # S3 path prefix for Loader bucket

If I use the default storage it works correctly. What am I doing wrong?
Thanks very much!

Sentry logging doesn't work with `LOADER=tc_aws.loaders.s3_loader`

Error with LOADER=thumbor.loaders.http_loader:

2016-11-10 13:01:05 thumbor:ERROR ERROR: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/thumbor/handlers/__init__.py", line 142, in get_image
    self.context.request.image_url
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python2.7/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python2.7/site-packages/thumbor/handlers/__init__.py", line 516, in _fetch
    loader_result = yield self.context.modules.loader.load(self.context, url)
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python2.7/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/site-packages/tornado/stack_context.py", line 314, in wrapped
    ret = fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/thumbor/loaders/http_loader.py", line 57, in return_contents
    raise Exception('test')
Exception: test

Same error with LOADER=tc_aws.loaders.s3_loader:

2016-11-10 13:00:35 tornado.application:ERROR Uncaught exception
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/tornado/http1connection.py", line 238, in _read_message
    delegate.finish()
  File "/usr/local/lib/python2.7/site-packages/tornado/http1connection.py", line 666, in finish
    return self._delegate.finish()
  File "/usr/local/lib/python2.7/site-packages/tornado/simple_httpclient.py", line 512, in finish
    self.headers["Location"])
  File "/usr/local/lib/python2.7/site-packages/tornado/httputil.py", line 215, in __getitem__
    return self._dict[_normalized_headers[name]]
KeyError: 'Location'
2016-11-10 13:00:35 thumbor:WARNING ERROR retrieving image from S3 bvjzPct.jpg: {'ResponseMetadata': {'HTTPStatusCode': 599, 'HostId': '', 'RequestId': ''}, 'Error': {'Message': '', 'Code': '599'}}
2016-11-10 13:00:35 tornado.access:ERROR 502 GET /unsafe/300x300/i.imgur.com/bvjzPct.jpg (172.17.0.1) 5505.94ms

I'm guessing that it has something to do with https://github.com/thumbor-community/aws/blob/master/tc_aws/loaders/s3_loader.py#L82 logger.warn(.... Maybe should be at least logger.error?

It would be best if s3_loader doesn't affect to Thumbor's logging features.

Can't load images using s3_loader

I'm using https://github.com/APSL/docker-thumbor with tc_aws.loaders.s3_loader, but I can't seem to get it to work. The error I'm getting is:

2016-11-22 09:26:57 thumbor:ERROR [BaseHandler] get_image failed for url `bucket-name/design-example-1.jpg`. error: `Parameter validation failed:
Invalid bucket name "": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"`
2016-11-22 09:26:57 tornado.access:ERROR 500 GET /PbhAZkbZdM0Q5AWPztSfaFdalNY=/fit-in/500x500/bucket-name/design-example-1.jpg (172.31.1.220) 39.45ms

which seems to imply that it's failing to extract the bucket name from my url... is the url I'm generating wrong or is this a bug/configuration problem?

Thanks

S3 Results storage fails when using gifv

Not entirely sure if this is Thumbor core or AWS plugins domain but Thumbor fails to load GIFv results from S3. The first request works perfectly and regular images like jpegs work as expected.

An example request:
curl -I "http://localhost:8000/unsafe/filters:gifv(mp4)/https://somedomain.com/big.gif"

First response:

2017-01-05 06:19:15 tornado.access:INFO 200 HEAD /unsafe/filters:gifv(mp4)//https://somedomain.com/big.gif (172.17.0.1) 10220.31ms

All subsequent responses:

2017-01-05 06:28:47 tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 1024, in run
    yielded = self.gen.send(value)
  File "/usr/local/lib/python2.7/site-packages/thumbor/handlers/__init__.py", line 117, in execute_image_operations
    self.context.request.engine.load(buffer, EXTENSION.get(mime, '.jpg'))
  File "/usr/local/lib/python2.7/site-packages/thumbor/engines/pil.py", line 312, in load
    super(Engine, self).load(buffer, self.extension)
  File "/usr/local/lib/python2.7/site-packages/thumbor/engines/__init__.py", line 162, in load
    image_or_frames = self.create_image(buffer)
  File "/usr/local/lib/python2.7/site-packages/thumbor/engines/pil.py", line 70, in create_image
    img = Image.open(BytesIO(buffer))
  File "/usr/local/lib/python2.7/site-packages/PIL/Image.py", line 2319, in open
    % (filename if filename else fp))
IOError: cannot identify image file <_io.BytesIO object at 0x7fd77b692710>

Example config here:

DETECTORS=['thumbor.detectors.face_detector','thumbor.detectors.profile_detector','thumbor.detectors.feature_detector']
FILTERS=['thumbor.filters.format','thumbor.filters.extract_focal','thumbor.filters.no_upscale']
HTTP_LOADER_REQUEST_TIMEOUT=120
MAX_AGE=31557600
OPTIMIZERS=['thumbor.optimizers.gifv']
STORAGE=thumbor.storages.no_storage
THUMBOR_WORKER_COUNT=1
USE_GIFSICLE_ENGINE=True
WEBP_QUALITY=90
LOG_PARAMETER=-l info
AWS_ACCESS_KEY_ID=AN_ID
AWS_SECRET_ACCESS_KEY=A_KEY
TC_AWS_REGION=A_REGION
TC_AWS_ENDPOINT=AN_ENDPOINT
TC_AWS_RESULT_STORAGE_BUCKET=A_BUCKET
TC_AWS_RESULT_STORAGE_ROOT_PATH=A_DIRECTORY
RESULT_STORAGE=tc_aws.result_storages.s3_storage
RESULT_STORAGE_STORES_UNSAFE=True

I'm using the APSL docker image which you can easily run with:
docker run -p 8000:8000 --env-file .env "apsl/thumbor"

^ You just need to create a .env file, adding valid credentials for an S3 bucket. Unsafe is optional.

Note that the transformed result (an mp4) is in fact sitting in S3, it did store correctly. It just fails to recognize the format upon retrieval.

NoCredentialsError: Unable to locate credentials

I keep getting "NoCredentialsError: Unable to locate credentials" even if i thought i provided AWS_access_key and aws_secret_key in configure file.

could you help me?

2016-01-27 12:58:52 thumbor:ERROR ERROR: Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/thumbor/handlers/init.py", line 102, in get_image
self.context.request.image_url
File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 1008, in run
value = future.result()
File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 232, in result
raise_exc_info(self._exc_info)
File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 1017, in run
yielded = self.gen.send(value)
File "/usr/lib64/python2.7/site-packages/thumbor/handlers/init.py", line 442, in _fetch
loader_result = yield self.context.modules.loader.load(self.context, url)
File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 473, in wrapper
future.result()
File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 232, in result
raise_exc_info(self._exc_info)
File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 461, in wrapper
result = f(_args, *_kwargs)
File "/usr/lib/python2.7/site-packages/tc_aws/loaders/s3_loader.py", line 48, in load
bucket_loader.get(key, callback=handle_data)
File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 473, in wrapper
future.result()
File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 232, in result
raise_exc_info(self._exc_info)
File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 461, in wrapper
result = f(_args, *_kwargs)
File "/usr/lib/python2.7/site-packages/tc_aws/aws/bucket.py", line 43, in get
Key=path,
File "/usr/lib/python2.7/site-packages/tornado_botocore/base.py", line 97, in call
return self._make_api_call(operation_name=self.operation, api_params=kwargs, callback=callback)
File "/usr/lib/python2.7/site-packages/tornado_botocore/base.py", line 60, in _make_api_call
operation_model=operation_model, request_dict=request_dict, callback=callback)
File "/usr/lib/python2.7/site-packages/tornado_botocore/base.py", line 54, in _make_request
request_dict=request_dict, operation_model=operation_model, callback=callback)
File "/usr/lib/python2.7/site-packages/tornado_botocore/base.py", line 32, in _send_request
request = self.endpoint.create_request(request_dict, operation_model)
File "/usr/lib/python2.7/site-packages/botocore/endpoint.py", line 120, in create_request
operation_name=operation_model.name)
File "/usr/lib/python2.7/site-packages/botocore/hooks.py", line 226, in emit
return self._emit(event_name, kwargs)
File "/usr/lib/python2.7/site-packages/botocore/hooks.py", line 209, in _emit
response = handler(**kwargs)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 384, in _sign_request
self._request_signer.sign(operation_name, request)
File "/usr/lib/python2.7/site-packages/botocore/signers.py", line 110, in sign
signer.add_auth(request=request)
File "/usr/lib/python2.7/site-packages/botocore/auth.py", line 621, in add_auth
raise NoCredentialsError

thumbor-community / aws Goto Github PK

aws's People

Stargazers

Watchers

Forkers

aws's Issues

Recommend Projects

Recommend Topics

Recommend Org