aws-solutions / serverless-image-handler Goto Github PK

View Code? Open in Web Editor NEW

1.3K 58.0 522.0 8.28 MB

A solution to dynamically handle images on the fly, utilizing SharpJS

License: Apache License 2.0

Shell 0.72% CSS 0.19% HTML 2.64% JavaScript 1.35% TypeScript 95.10%

serverless-image-handler's Introduction

Serverless Image Handler | 🚧 Feature request | 🐛 Bug Report | ❓ General Question

Note: If you want to use the solution without building from source, navigate to Solution Landing Page.

Table of Content

Solution Overview
Architecture Diagram
AWS CDK and Solutions Constructs
Customizing the Solution
- Prerequisites for Customization
Collection of operational metrics
External Contributors
License

Solution Overview

The Serverless Image Handler solution helps to embed images on websites and mobile applications to drive user engagement. It uses Sharp to provide high-speed image processing without sacrificing image quality. To minimize costs of image optimization, manipulation, and processing, this solution automates version control and provides flexible storage and compute options for file reprocessing.

This solution automatically deploys and configures a serverless architecture optimized for dynamic image manipulation. Images can be rendered and returned spontaneously. For example, an image can be resized based on different screen sizes by adding code on a website that leverages this solution to resize the image before being sent to the screen using the image. It uses Amazon CloudFront for global content delivery and Amazon Simple Storage Service (Amazon S3) for reliable and durable cloud storage.

For more information and a detailed deployment guide, visit the Serverless Image Handler solution page.

Architecture Diagram

The AWS CloudFormation template deploys an Amazon CloudFront distribution, Amazon API Gateway REST API, and an AWS Lambda function. Amazon CloudFront provides a caching layer to reduce the cost of image processing and the latency of subsequent image delivery. The Amazon API Gateway provides endpoint resources and triggers the AWS Lambda function. The AWS Lambda function retrieves the image from the customer's Amazon Simple Storage Service (Amazon S3) bucket and uses Sharp to return a modified version of the image to the API Gateway. Additionally, the solution generates a CloudFront domain name that provides cached access to the image handler API.

AWS CDK and Solutions Constructs

AWS Cloud Development Kit (AWS CDK) and AWS Solutions Constructs make it easier to consistently create well-architected infrastructure applications. All AWS Solutions Constructs are reviewed by AWS and use best practices established by the AWS Well-Architected Framework. This solution uses the following AWS Solutions Constructs:

In addition to the AWS Solutions Constructs, the solution uses AWS CDK directly to create infrastructure resources.

Customizing the Solution

Prerequisites for Customization

AWS Command Line Interface
Node.js 20.x or later

1. Clone the repository

git clone https://github.com/aws-solutions/serverless-image-handler.git
cd serverless-image-handler
export MAIN_DIRECTORY=$PWD

2. Unit Test

After making changes, run unit tests to make sure added customization passes the tests:

cd $MAIN_DIRECTORY/deployment
chmod +x run-unit-tests.sh && ./run-unit-tests.sh

3. Build and Deploy

cd $MAIN_DIRECTORY/source/constructs
npm run clean:install
overrideWarningsEnabled=false npx cdk bootstrap --profile <PROFILE_NAME>
overrideWarningsEnabled=false npx cdk deploy\
 --parameters DeployDemoUIParameter=Yes\
  --parameters SourceBucketsParameter=<MY_BUCKET>\
   --profile <PROFILE_NAME>

Note:

MY_BUCKET: name of an existing bucket in your account
PROFILE_NAME: name of an AWS CLI profile that has appropriate credentials for deploying in your preferred region

Collection of operational metrics

This solution collects anonymous operational metrics to help AWS improve the quality and features of the solution. For more information, including how to disable this capability, please see the implementation guide.

External Contributors

@leviwilson for #117
@rpong for #130
@harriswong for #138
@ganey for #139
@browniebroke for #151, #152
@john-shaffer for #158
@toredash for #174, #195
@lith-imad for #194
@pch for #227
@atrope for #201, #202
@bretto36 for #182
@makoncline for #255
@frankenbubble for #302
@guidev for #309
@njtmead for #276
@StaymanHou for #320
@alenpaulvarghese for #392
@Fjool for #489
@fvsnippets for #373, #380
@ccchapman for #490
[@bennet-esyoil][https://github.com/bennet-esyoil] for #521
[@vaniyokk][https://github.com/vaniyokk] for #511

License

serverless-image-handler's People

Contributors

Stargazers

Watchers

Forkers

richardwooding zapimoveis phound-inc scottk212 arahayrabedian robinbowes medizzy gridl bananabandy garylam junheon myyellowshoe irshadsyed alexbilbie marcieltorres decafdev hyandell valurhrafn forestryio thattimc corymcan zatarra maxgutman pakinwet sydsvenskan i17c blalor bassrock northwoodspd alixaxel bentterp mdtomo ahmadpriatama zattoo rxfork omrisiri fkei prolificinteractive brokerageengine liqinsg micke thebookpeople rama3i chrisfu valery-barysok survivant rah003 balinea angelmoreno87 dhardy92 lowlevel96 roverdotcom zakdoek hamatoyogi simllll timkelty ehlertij juliankmazo nbcu-digitaldistribution ipernet dwtcourses thanakij j2d2 snappr phantomk bhoule eszechet80 peterkuiper ckbk sbambach shotishu jbye curtisconway ventaspopgit kevinferm fishbrain stackpoet esplorio morristech obsicn repinel lastessa doteshopping hayesry ivawzh funliday helehelehele venveo ferclaverino gorkamb alexzelenuyk leviwilson copyhackers flaivy pxsonny jodevsa markerio innovationlove sshong19 ecaron

serverless-image-handler's Issues

How long will this project be supported

Hi,

Love this project. Is there any information on how long this project will be supported?

Thank you!

webp fix doesn't work

I checked the code source/image-handler/thumbor.conf. The webp related code has been uncommented out. However it's set to false:
AUTO_WEBP = False

According to https://github.com/thumbor/thumbor/wiki/Configuration#auto_webp, the value should be set to TRUE.

setting as subfolder as root in the cloudfront distribution

Hi i got the Serverless Image Handler up and running, all good. I pointed it to my already existing bucket "MyBucket", and i can do image rescaling and stuff when placing images into that bucket.

However we have all our images in a subfolder to that bucket, called "cloudfront_assets". So after assigning my CNAME to the new cloudfront distribution, i am stuck with having to reference my images like this:

https:///cloudfront_assets/image.jpg

instead of

https:///image.jpg

I tried editing the cloudfront disitrbutions origin settings, and set "Origin Path" from /image to things like /cloudfront_assets or /image/cloudfront_assets.

It fixed the path issue, so i didnt have to write the "/cloudfront_assets/" before the image, but regardless of what i set, the image rescaling stopped working.

What is the correct way to solve my issue?

Please help, currently stuck at the moment Set the log level to debug in the lambda function in order to see whats happening, but it only says its getting "access denied" as far as i can tell

Getting http status code 502 instead of 404 for missing images

hi
I am trying to access the file via cloudfront and that file does not exists in S3 bucket. I am getting 502 error code instead of 404 in my browser.

Is it the right expected response code from this service ?

Also if my original image not found, I want to return noImageFound.jpeg as a response. Is there anyway to do it ?

Readme is not up to date

Readme file is not up to date, compared to the official documentation

As you can see

The documentation locked down setuptools and virtualenv version

$ sudo pip install setuptools==39.0.1
$ sudo pip install virtualenv==15.2.0

Moreover, this command line is not the same ./build-s3-dist.sh mybucket v1.0

Receiving Pip Import Errors for Req

trying to run the build script on a fresh amazon linux ec2 instance and receiving numerous errors..

Installing setuptools, pip, wheel...done.
source env/bin/activate
pip install /home/ec2-user/serverless-image-handler/deployment/../source/image-handler-custom-resource/. --target=/home/ec2-user/serverless-image-handler/deployment/dist/env/lib/python2.7/site-packages/
Processing /home/ec2-user/serverless-image-handler/source/image-handler-custom-resource
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-Il0EeT/setup.py", line 4, in <module>
        from pip.req import parse_requirements
    ImportError: No module named req

I cant seem to resolve this one.. I saw something about needing to specify pip._internal.req and i am locking the pip version to 9.0.3 but even when its above this ie. the latest, same error... not sure what to try next?

Requirements for building the lambda function

I'm having issues being able to successfully build the ./build-s3-dist.sh source-bucket-base-name. The README doesn't specify what the base OS requirements are for building the solution, but given yum is mentioned I assumed that it was built on an Amazon Linux AMI or the like. To get around this I'm working on a Dockerfile that bases the image off of the amazonlinux based image.

`pip`

The first issue I ran into is that the latest version of pip does not expose req, so for pip > version 10 it needs to say pip._internal.req rather than pip.req. Here is what I've updated both setup.py to contain to get around it.

try: # for pip >= 10
    from pip._internal.req import parse_requirements
except ImportError: # for pip <= 9.0.3
    from pip.req import parse_requirements

However, when tornado_botocore is included, it suffers from the same problem:

Collecting tornado_botocore==1.0.2 (from image-handler==2.0)
  Downloading https://files.pythonhosted.org/packages/35/35/65434dd70a524c9bffc8c24c3c549573887c741f82bf33b8dc2bed696d88/tornado-botocore-1.0.2.tar.gz
    Complete output from command python setup.py egg_info:
    WARNING:root:It looks like some requirements are missing.
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-RY9v9C/tornado-botocore/setup.py", line 35, in <module>
        from pip.req import parse_requirements
    ImportError: No module named req

To get around this, I had to update 'tornado_botocore==1.3.2' as well as 'botocore==1.8' as those versions have updates around the pip issue.

`pycurl`

The next issue that I found was when installing pycurl. Here is the output:

Collecting pycurl<7.44.0,>=7.19.0 (from thumbor->thumbor-plugins==0.2.1->-r source/image-handler/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz (214kB)
    100% |################################| 215kB 17.4MB/s
    Complete output from command python setup.py egg_info:
    Using curl-config (libcurl 7.51.0)
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-a6Ym8A/pycurl/setup.py", line 913, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/pip-install-a6Ym8A/pycurl/setup.py", line 582, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/pip-install-a6Ym8A/pycurl/setup.py", line 99, in __init__
        self.configure()
      File "/tmp/pip-install-a6Ym8A/pycurl/setup.py", line 316, in configure_unix
        specify the SSL backend manually.''')
    __main__.ConfigurationError: Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using. Please see PycURL documentation for how to specify the SSL backend manually.

So it'd seem that the amazonlinux base image isn't setup to be able to build pycurl. In order to get around that issue, I had to do the following:

yum install -y python27-devel openssl-devel

Success? ✅

...or so I thought. I was at least able to build the ZIP and deploy the function, however when I went to convert an image I got a 502. Here are a couple of things from the CloudWatch logs:

13:29:27 [WARNING] It looks like some requirements are missing.
13:29:27 [ERROR] start_thumbor error: cannot import name Botocore

So, the bump for botocore seemed to have broken some things with tc_aws, so in order to get past that I had to use 'tc_aws==6.2.10' per thumbor-community/aws#100.

Where I'm At

After bumping tc_aws and deploying the latest built package, now all I'm relegated is a 502 still and simply a message that says:

[WARNING] It looks like some requirements are missing.

At this point, I'm at a loss at what to look into, so any help would be greatly appreciated to point me in the right direction as to what I might be doing wrong. I'll probably try loading an actual AWS AMI that Lambda is using and see if I get the same results or not from there.

Ultimately, all I'm trying to do is bump thumbor to 6.5.2 but I've thus far been unsuccessful at even building a package that I can start with.

Here is the Dockerfile that I'm currently attempting this with:

FROM amazonlinux:2017.03.1.20170812
MAINTAINER Levi Wilson <[email protected]>

# lock yum to the same repository version
RUN sed -i 's/releasever=.*/releasever=2017.03/g' /etc/yum.conf

# base requirements
RUN yum install yum-utils zip -y && \
    yum-config-manager --enable epel && \
    yum install git libpng-devel libcurl-devel gcc python27-devel libjpeg-devel -y

# enable epel on Amazon Linux 2
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

# pip
RUN alias sudo='env PATH=$PATH' && \
    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python get-pip.py && rm get-pip.py && \
    pip install --upgrade setuptools && \
    pip install --upgrade virtualenv

# pycurl
RUN yum install -y nss-devel
ENV PYCURL_SSL_LIBRARY=nss

# ImageMagick
RUN yum install -y ImageMagick-devel

URL rewrite not working for us - adding unsafe to the url?

Hi,

We are using the serverless image resizer in a new project and also looking to bring it into an old project. However, we want to use a similar URL to how we used to do image resizing on the server which was like.

/AUS/image001.jpg?h=400&w=400

In the image resizer for the serverless URL this needs to be...

/fit-in/400x400/AUS/image001.jpg

We have come up with the following Regexp for Python as per the docs https://docs.aws.amazon.com/solutions/latest/serverless-image-handler/appendix-b.html

[(r'^([\w\./]+)\?h=([\d]+)&w=([\d]+)$', r'/fit-in/\2x\3\1'),(r'^([\w\./]+)\?w=([\d]+)&h=([\d]+)$', r'/fit-in/\3x\2\1')]

However, this is never resizing the image and in the cloudwatch logs we are getting logs like the following.

[INFO] 2018-09-20T08:40:47.149Z dd22ee1e-bcb0-11e8-8346-6fbe28a7c414 200 GET /unsafe/AUS/image001.jpg (0.0.0.0) 961.66ms

The URL in the log obviously doesn't have the '/fit-in/400x400' and '/unsafe' is added to the URL.

If anyone could help here that would be fantastic. The lambda function was deployed using the cloud formation template and has not been modified. We have specified in the environment variables for the lambda function that rewrite is enabled and the rewrite pattern above.

Thanks

Handling Large Lambda zip file sizes when deploying

I made a small change to the thumbor.conf file (added a security key and set ALLOW_UNSAFE_URLS to false), and then used the build-s3-dist.sh script to zip up the files and then uploaded them to S3.

However, when I attempt to redeploy the Serverless Image Handler with CloudFormation, I get an error for the Lambda being too big. On inspection of the zip file, I noticed that the package is about 75mb zipped/289mb unzipped, far above the 50mb/250mb limits mentioned for Lambdas.

I am wondering if I missed step somewhere before zipping, or if I need to change some dependencies somewhere. However since I didn't really change much aside from the 2 lines in the thumbor.conf file, I'm not sure why it got so big.

One additional note, I did make a slight change in the deploy script to downgrade pip pip install "pip==9.0.3", since the source files for this solution don't appear to support the default pip solution installed (18).

My OS is macOS High Sierra, 10.13.6, if that helps.

Rewrite Feature - Ability to restrict requests on the given patterns

My main use case for it is to restrict user from making their own image optimization request aside from what we require for the app. The rewrite is awesome and it gives you the ability to hide your S3 path to your users. However, it doesn't stop any request when you're using the standard way (see sample url below).

https://somecodehere.cloudfront.net/fit-in/300x300/inquiry/img/aws-logo.png

It opens for malicious usage that might incur unnecessary cost. Is there a way we can prevent direct request aside from the declared set of patterns?

Thanks.

Serverless?

Why an example of serverless app runs server?

https://github.com/awslabs/serverless-image-handler/blob/6a30bd785999bea6ba9dfb402bf550b2dd8f2f8f/source/image-handler/lambda_function.py#L85-L155

Large files cause "body size too long" error

Transforming a large (9.8MB) gif causes a body size too long error. This seems related to the 6MB body payload size, as described in the Lambda Limits page. I think this issue will face image formats other than .gif, that's just what I've seen it pop up on. Log screenshot below.

Any advice for how to work around this?

At the moment, I have an onError function on my <img /> tags that replaces the src with the original non-resized URL. But - this occurs in the worst case scenario, where we have huge files - so it's really not desirable.

Create a serverless configuration

To enable easier local debugging of issues so that we can contribute back to this project. We need a way to simplify the development process of making a change and testing it quick.

https://serverless.com/ (serverless.yml) can give us that.

support svg

svg does not seem to be supported on the Serverless Image Handler template. I get the following error:

convert_svg_to_png failed cairosvg not
imported (if you want svg conversion to png please install cairosvg)

Resized version is over-saturated compared to original when source is CMYK (example sources)

Example CMYK original: http://basethree.s3.eu-west-1.amazonaws.com/4663079141376000
Overly-saturated: https://d1095j5a3invzg.cloudfront.net/800x800/4663079141376000
More live examples can be seen on:
https://sw2.base3.pro/7/marine-acquisition/our-fleet
I'm using the latest template (arn:aws:cloudformation:eu-west-1:280125953110:stack/latest-basethree/05005440-98cf-11e8-8169-500c423e34d2)
I wondered if there are filters I can apply that would fix this but can't find anything.

modifying Thumbor conf value on Mac OS X (RESPECT_ORIENTATION = True)

I need to change the Thumbor to not rotate images by default by changing this config value
RESPECT_ORIENTATION = True to avoid the case like the attached image (basically all landscape mobile photos)

(I'm befuddled why RESPECT_ORIENTATION is False by default? Why would we want Thumbor to rotate an source image w/out explicitly asking for it?)

I'm on Mac OS X (High Sierra) - checked out repo, set up Python w/ virtualenv, ran ./build-s3-dist.sh {{bucketname...actually doesn't matter}} and all compiled well locally - then I uploaded the generated file from dist/serverless-image-handler.zip into an s3 bucket (needed to do that rather than direct upload since > 50MB) and then uploaded from the associated Lamda.

Excited to see my images not rotated...but got an internal server error trying to render an image and looking at CloudWatch see the following:

Unable to import module 'image_handler/lambda_function': /var/task/PIL/_imaging.so: invalid ELF header

Almost sure this is b/c I'm compiling in an environment that (OS X) that doesn't match that of the Lamda. (@arahayrabedian apparently had more luck: #11 - guessing he was on Linux?)

Any hints? Do I need to spin up an EC2 Linux or use Docker?

This is a lot of pain for a 1 line change! I wish Thumbor config files be adjusted from Lamda environment variables, but I believe they're build time so not possible.

Can't get AUTO_WEBP to work

First of all thank you guys for an awesome solution!

I'm however struggling to enable the auto webp feature.
I'm more than likely just not understanding the configuration instructions properly.
Would you know what i'm missing here?

Non-string environment variables cause crash.

This was first mentioned in #25, but that original issue had been resolved, so I am repeated this related, but separate issue here.

Thanks to @stevemorad for pointing out that environment variables can be used to set Thumbor conf values. I found that this works for string values, but not for int values, e.g. QUALITY.

I get an error when I set an environment variable with key QUALITY and value 95.

Here is the error I'm receiving.

And here is where the error originates from in PIL

Any suggestions on how to work around this? Or do I have to just manually set the value in thumbor.conf as you describe above?

PNG transparency lost in resized version

Original: http://basethree.s3.eu-west-1.amazonaws.com/5472046692171776
Resized: https://d1095j5a3invzg.cloudfront.net/100x100/5472046692171776
Using latest template

CORS support

After deploying the stack, I'm trying to figure out how to enable CORS so that assets coming back from the Lambda function can be used in the Canvas element.

Would be helpful if a parameter could be defined so that when someone runs this template, they can choose YES or NO for adding CORS support.

Front-end calls with the Image Handler Lambda

I apologize if my question is ignorant. I've been trying to figure out how to use the handler on the front end in a static app served from an AWS s3 bucket. If I lay the endpoint and the image names open in the browser for calls made by an unauthenticated user (which is a common scenario as there are not-authorized routes in the app with lots of images processed on-the-fly with the handler), the end-point is an easy prey for a malicious user.

I'd configure the resource to use safe URLs but I still don't see how this can prevent malicious calls: what is the way not to lay the endpoint and the images names open and be able to use the handler endpoint in src="" tags? Wether I call <handler-endpoint>/<options>/<img-name>, or <handler-endpoint>/<secret_key>/<options>/<img-name>, the logic of how to get a safe image URL, that is protected against abuse is not clear to me.

Making calls to /meta/ doesn't work

Getting an error JSON file when requesting the meta data for a transformation.
Documentation on this is here: http://thumbor.readthedocs.io/en/latest/usage.html#metadata-endpoint

Sample response I'm getting:

Failed to create resource with CF template.

Hello there,

For some reason when I try to load the cloudformation template, it's failing at "DeployingUI" "Custom::LoadLambda".

Here is the stack creation log I have below. Am I missing some permissions problem? If anyone can shed me some light I'd be very appreciated. Thank you!

DELETE_IN_PROGRESS	AWS::Logs::LogGroup	ImageHandlerLogGroup
01:42:49 UTC+0800	DELETE_IN_PROGRESS	AWS::Lambda::Permission	ImageHandlerPermission
01:42:49 UTC+0800	DELETE_IN_PROGRESS	Custom::LoadLambda	DeployingUI
01:42:46 UTC+0800	ROLLBACK_IN_PROGRESS	AWS::CloudFormation::Stack	test-image
Physical ID:arn:aws:cloudformation:ap-southeast-1:334446083852:stack/test-image/b1847d30-b455-11e8-a483-500c336e0c1e
Client Request Token:Console-CreateStack-049dd5e4-b261-4dd5-bcd7-27f1932efb1b
01:42:45 UTC+0800	CREATE_FAILED	Custom::LoadLambda	DeployingUI
Physical ID:original-images-bucket-name/serverless-image-handler-ui/
Client Request Token:Console-CreateStack-049dd5e4-b261-4dd5-bcd7-27f1932efb1b
01:42:45 UTC+0800	CREATE_IN_PROGRESS	Custom::LoadLambda	DeployingUI
01:42:40 UTC+0800	CREATE_IN_PROGRESS	Custom::LoadLambda	DeployingUI

Optimize JPEGs

The images created by this service can sometimes be optimized 10% or more by tools like kraken.io or mozjpeg.

I think it would be worthwile to include and enable such optimizations by default.

I see that Thumbor has built in support for jpegtran, but it's not enabeld in thumbor.conf (and the binary is probably missing).

Could you add this feature? It would make this a much more viable alternative to commercial services like imgix.

Setting ALLOW_UNSAFE_URL=False doesn't work as expected

This line doesn't work correctly:

https://github.com/awslabs/serverless-image-handler/blob/master/source/image-handler/lambda_function.py#L176-L177

    if config.ALLOW_UNSAFE_URL:
        http_path = '/unsafe' + http_path

It seems that all config values are treated as strings, so ALLOW_UNSAFE_URL=False functions exactly the same as ALLOW_UNSAFE_URL=True.

The way to not all unsafe URLs is to set ALLOW_UNSAFE_URL= ie. set it empty.

Weird redirects

Hello, we are just strated to use serverless-image-handler, but we are having some weird behaviour:
some images are causing 301 redirects, e.g:
http://mrp-listings.myrealpage.com/fit-in/800x0/3/9/1/61373193/3d7498457b6b796f85cd9ca6aa14e23c.jpeg -> https://hadvqdih86.execute-api.us-east-1.amazonaws.com/image/fit-in/800x0/3/9/1/61373193/3d7498457b6b796f85cd9ca6aa14e23c.jpeg
http://mrp-listings.myrealpage.com/fit-in/800x600/9/1/0/19/112c1b67f8a2a132cf3d88dcce6958d1.jpg -> https://hadvqdih86.execute-api.us-east-1.amazonaws.com/image/fit-in/800x600/9/1/0/19/112c1b67f8a2a132cf3d88dcce6958d1.jpg

while the others are working ok without redirect:
http://mrp-listings.myrealpage.com/fit-in/800x600/8/8/9/19757988/00355da2004c00fc7633116588229baa.jpg

I thought it might be caused by file extension - jpg vs jpeg, but then found an example of jpg file redirecting as well. Could you please advise what might be causing those redirects and how to fix it?

Thank you!

Allowed Bucket Name Patterns

The allowed pattern does not allow periods to exist in bucket names - even though periods are valid in S3 bucket creation. Perhaps this could be changed to [a-zA-Z][a-zA-Z0-9-.]* to allow periods.

https://github.com/awslabs/serverless-image-handler/blob/master/deployment/serverless-image-handler.template#L12

Timeout upon requesting a full-fit-in on a large JPEG

See this JPEG:
https://s3-us-west-2.amazonaws.com/app.local.getstencil.com/upld71ad1ebb

When requesting a larger version through the AWS Solution (via the full-fit-in option), I get a timeout:
https://d32q3izcmhqbti.cloudfront.net/full-fit-in/3200x5120/upld71ad1ebb

However when requesting through Thumbor's Thumborize service, it works:
http://thumbor.thumborize.me/unsafe/full-fit-in/3200x5120/filters:format(png)/https://s3-us-west-2.amazonaws.com/app.local.getstencil.com/upld71ad1ebb

It does take some time (for me just now, it took 62.1 seconds, and resulted in an image that is 19mb in filesize and 70,891,520 pixels).

Smart crop not working on V3

After updated to V3 I noticed that the smart crop is not working anymore.
Using the same image on V2 it works fine.

This is the error message I get from cloudwatch:

[ERROR] 2018-10-03T15:36:21.533Z 14155d40-c722-11e8-8e9c-0989d2d55e75 Future exception was never retrieved: Traceback (most recent call last): File "/var/task/tornado/gen.py", line 1107, in run yielded = self.gen.throw(*exc_info) File "/var/task/thumbor/transformer.py", line 152, in smart_detect yield self.do_smart_detection() File "/var/task/tornado/gen.py", line 1099, in run value = future.result() File "/var/task/tornado/concurrent.py", line 260, in result raise_exc_info(self._exc_info) File "/var/task/tornado/gen.py", line 1113, in run yielded = self.gen.send(value) File "/var/task/thumbor/transformer.py", line 180, in do_smart_detection detectors[0](self.context, index=0, detectors=detectors).detect(self.after_smart_detect) File "/var/task/thumbor_rekognition/__init__.py", line 25, in detect raw_bytes = self.read_jpeg_bytes() File "/var/task/thumbor_rekognition/__init__.py", line 45, in read_jpeg_bytes self.context.modules.engine.image.save(bio, 'JPEG') File "/var/task/PIL/Image.py", line 1935, in save save_handler(self, fp, filename) File "/var/task/PIL/JpegImagePlugin.py", line 622, in _save raise IOError("cannot write mode %s as JPEG" % im.mode) IOError: cannot write mode RGBA as JPEG

Output images not stored

In the Thumbor documentation, I can read the following

Images are stored in whatever path is specified in the RESULT_STORAGE_FILE_STORAGE_ROOT_PATH, and consequently retrieved from the same path.

By default, the file system result storage keeps images forever. You are allowed to specify an expiration, though, using the RESULT_STORAGE_EXPIRATION_SECONDS configuration. Again, as the name implies, it specifies the number of seconds with which files expire.

I cannot see any output in my bucket after having resized an image. However, the very first request on an image resize is really slower than the subsequent requests for the same file. Thus, it seems that the output is saved somewhere. In the case of a serverless application, I was wondering where are the output stored ?

Feature Request: Face detection cropping

This is by far the easiest way to get Thumbor up and running, but it would be nice to see a way to set it up with automatic face cropping.

Automatic WEBP support

currently lacking this feature. Requires to check Accept header and add proxy Vary back from thumbor.

How to update Thumbor and Python Pillow

Thanks for all your work on this @stevemorad
Really appreciate it.

Unfort, it seems like there are some issues in Thumbor / Python Pillow that prevent this solution from being 100% ready for a production environment. Specifically, it doesn't work properly for a number of PNGs that have transparent backgrounds (which is quite a common use-case).

Here is the ticket I opened with Thumbor:
thumbor/thumbor#987

My question is: when Thumbor and Pillow are fixed/patched, what's the best process to update the Lambda function? Can I simply download Thumbor/Pillow exactly how they're released, download the Lambda package, replace the existing folders with the updated releases, and then re-upload?

CCITT Group 4 TIFF Images / Pillow and AWS Lambda

I was looking into incorporating this amazing project into something I am working on. Out of the gate, everything worked pretty smoothly for a lot of my use cases but I DID notice that for some reason I get 502 from the lambda function whenever I try to do anything with a TIFF images that has Group 4 compression on it.

For TIFF images with other compressions (like LZW) things seemed to be fine, but the CCITT Fax 4 compression.

To validate that PIL (well, Pillow) could even do this, I took one of the images that had troubles in the lambda function and ran it through the following:

from PIL import Image;
image = Image.open("some_file.tiff")
image.convert("RGB")
image.save("some_file.jpg", "JPEG")

And everything checked out. After that, I had two 🤔 👍

it might've been because I have Pillow 5.2.0 that it worked and the project has an older version
something with perhaps libtiff when it runs in lambda?

I wasn't sure what to look into next, so I thought I would check here to see what I'm doing wrong or how to even go about validating my assumptions.

Also, and pardon my ignorance, but is this the repo that actually builds the lambda function? Because I ran the shell script and it didn't produce that as an output. Either that or there's an assumption that you're building it on some lambda equivalent version of Linux before you do so? This is the first time I've messed with it so I wasn't sure.

Thanks in advance.

Why including CloudFront if APIGateway already provides it by default ?

At least that's the information we have on this forum thread:

https://forums.aws.amazon.com/thread.jspa?threadID=195290

Cannot convert to png

I try to convert an image from JPEG to a PNG. I get this in the logs:

16:16:06
[PNGQUANT] running: cat /tmp/tmpbT3FLJ | /var/task/pngquant --speed 3 --quality=80 - > /tmp/tmpORRq75

16:16:06
/bin/sh: cat: command not found

16:16:06
error: Read error (libpng failed)

16:16:06
error: cannot decode image from stdin

default cloudformation config results in low cloudfront caching rate

It seems the default setup that is generated by the cloudformation template for this project includes a default value for the Cloudfront distribution that has this setting: "Cache Based on Selected Request Headers" set to use the "Accept" header. I believe this is set on line 433 in deployment/serverless-image-handler.template:

"ForwardedValues": {
"Headers" : ["Accept"],
"QueryString": "false",
"Cookies": { "Forward": "none" }
},

Wouldn't this result in a much lower Cloudfront cache hit rate and thus more Lambda invocations? Since each browser can use a different Accept value in their request headers each browser version could potentially generate a different version of the same cached image.

Removing this from our Cloudfront distribution made our Cloudfront cache hit rate jump up to 75-80% from under 50% previously, and significantly lowered the number of Lambda invocation and our overall cost.

Unless this Accept header is needed I think it should be removed from the template as it needlessly increases the cost of using this solution.

Multiple s3 buckets

Is it possible to make this work with multiple s3 buckets? I'm using multiple cloud front distributions backed by it's own s3 bucket. It would be great to re-use the lambda by passing the s3 bucket in a header for instance.

Cannot request a GIF image with 2-frames

See this GIF:
http://s3-us-west-2.amazonaws.com/app.dev.getstencil.com/upld22353ae3

When requested through this AWS Solution, I receive an error. See here:
https://d1th8vjf1eu5eu.cloudfront.net/upld22353ae3

Not that this works fine when requested through Thumbor's Thumborize service:
http://thumbor.thumborize.me/unsafe/http://s3-us-west-2.amazonaws.com/app.dev.getstencil.com/upld22353ae3

I have not tested other GIFs extensively. It's possible this is related to an issue with frames, since this GIF has two frames (which are however not animated; just the same frame twice).

global name 'response' is not defined

https://github.com/awslabs/serverless-image-handler/blob/3b8198edd6a46ec9d63c6c0973af5880989e7918/source/image-handler/lambda_function.py#L216

Assuming the above line should be

return response_formater(status_code=thumbor_response.status_code)

can't bypass non-image file

We have deployed serverless-image-handler and it works well with the images. But our cloudfront linked S3 also store pdf and txt files. When we try to access those pdf and txt files through the cloudfront url, we get internal errors.
How to make the serverless handler simply return the requested pdf and txt files through the cloudfront and S3?

Secure and unsafe URLs do not work as expected

Secure URLs in version 3 are not secure.
- The new behaviour of v3 that changes the safe URLs from /<hash>/<path> to /<security_key>/<path> is incorrect and completely defeats the purpose of the safe URL. The security key cannot appear in the public url to an image, only the hash can appear in a public url.
- See #54 (comment)
- See the official docs which support this: https://github.com/thumbor/thumbor/wiki/security
It is impossible to disable unsafe URLs using the ENV vars (in v2 or v3)

Below are examples from my deployments for each version to demonstrate the problem.
All urls were requested for the first time for these tests, so CloudFront cache was empty for each one.
Screenshots of the Lambda config for the v2 and v3 deployments are shown below.

v2

With hash: loads (good):

https://d3f24ahusl0vhu.cloudfront.net/UpwsFH4gS8sPRTzw6Adc0sHKGwQ=/onfLoZx7d

With no hash or unsafe: error (good)

https://d3f24ahusl0vhu.cloudfront.net/onfLoZx7d

With unsafe: loads (bad)

https://d3f24ahusl0vhu.cloudfront.net/unsafe/onfLoZx7d

v3

With hash: error (bad)

https://d18aelkez1p3a0.cloudfront.net/5jel0cJtBXmMLI1Rupih8M6eiPc=/8GKv8lk32m

With no hash or unsafe: error (good)

https://d18aelkez1p3a0.cloudfront.net/8GKv8lk32m

With unsafe: loads (bad)

https://d18aelkez1p3a0.cloudfront.net/unsafe/8GKv8lk32m

With secret key in URL: loads (bad)

https://d18aelkez1p3a0.cloudfront.net/<redacted>/8GKv8lk32m

v2

v3

Gifsicle not working on AWS Lambda

How to get gifsicle working, I have compiled the binary but it still fails gives cannot fine gifsicle in the path error.

Even normal gif images are failing

Only work in S3 bucket in US-East region

Tried to install this stack with S3 US-East region and it work as expected. However, it doesn't work with other S3 region.

For example, when install this stack with S3 bucket in singapore. The output SolutionUI url will generate like this:
https://s3.ap-south-1.amazonaws.com/[bucker-name]/serverless-image-handler-ui/index.html

Which will get this error when browsing it:

After changing the url to
https://s3-ap-southeast-1.amazonaws.com/[bucker-name]/serverless-image-handler-ui/index.html

Then it can show the index page but all images links from the left hand side are invalid too.

I need to manually changing the images url to https://s3-ap-southeast-1.amazonaws.com in order to show those images. However, I still got loading when processing image.

ERROR pnqquant path '/var/task/pngquant' is not accessible

Hi,
I changed the Thumbor config file (PNGQUANT_QUALITY=80 to PNGQUANT_QUALITY=90) and uploaded my zip again to my AWS Lambda function. But now I got the following CloudFront Error message:

ERROR pnqquant path '/var/task/pngquant' is not accessible

Did I need to change anything else on AWS or in my config or is the error somewhere else?

Thanks in advance

Implementing Thumbor Configuration Options

(Note: Not sure if this should be posted in https://github.com/awslabs/serverless-image-handler or https://github.com/awslabs/serverless-image-resizing)

As can be seen here (https://github.com/thumbor/thumbor/wiki/Configuration) there are a number of configuration options available. There are a few in particular I'm interested in.

Instead of manually adding an option to the CloudFlormation setup flow prompting me to choose a value for each one, would it be possible for me to specify a JSON string that contains values for the different configuration options, and then your script take those into consideration?

At the moment, I don't have a good grasp on how to manually update these configuration options, and require the ability to have this script respect the image orientation, otherwise issues like this happen:

Thanks

Why reload file in memory and then encode it again?

https://github.com/awslabs/serverless-image-handler/blob/master/source/image-handler/lambda_function.py#L203
This seems redundant, because thumbor done work encoding and maybe even optimizing output image, so re-encoding might increase content size and introduce visual defects due to rounding errors, etc.

Specific images crash server

Photos taken from Samsung phones contain exif that crashes lambda ( and return 502 )

[ERROR]	2018-08-29T18:39:12.842Z	d28fb4a3-abba-11e8-b4de-ad0a1e83e58b	Future exception was never retrieved: Traceback (most recent call last):
File "/var/task/tornado/gen.py", line 1113, in run
yielded = self.gen.send(value)
File "/var/task/thumbor/handlers/__init__.py", line 224, in get_image
self.filters_runner.apply_filters(thumbor.filters.PHASE_AFTER_LOAD, transform)
File "/var/task/thumbor/filters/__init__.py", line 81, in apply_filters
callback()
File "/var/task/thumbor/handlers/__init__.py", line 222, in transform
self.context.transformer.transform(self.after_transform)
File "/var/task/thumbor/transformer.py", line 100, in transform
self.engine.reorientate()
File "/var/task/thumbor/engines/__init__.py", line 295, in reorientate
self.exif = piexif.dump(exif_dict)
File "/var/task/piexif/_dump.py", line 68, in dump
exif_set = _dict_to_bytes(exif_ifd, "Exif", zeroth_length)
File "/var/task/piexif/_dump.py", line 341, in _dict_to_bytes
'{0} in {1} IFD. Got as {2}.'.format(key, ifd, type(ifd_dict[key]))
ValueError: "dump" got wrong type of exif value.
41729 in Exif IFD. Got as <type 'int'>.

[ERROR]	2018-08-29T18:40:47.940Z	0b91364d-abbb-11e8-ba46-29a2d9212dd4	call_thumbor error: tornado server unavailable, proceeding with tornado server restart

[ERROR]	2018-08-29T18:40:47.942Z	0b91364d-abbb-11e8-ba46-29a2d9212dd4	lambda_handler trace: Traceback (most recent call last):
File "/var/task/image_handler/lambda_function.py", line 280, in lambda_handler
result = call_thumbor(event)
File "/var/task/image_handler/lambda_function.py", line 236, in call_thumbor
thumbor_down, session = is_thumbor_down()
ValueError: too many values to unpack

Examining 'bad' photo exif showing some malformed properties:

// image exif
Maker Unknown
Unknown _0x0001 0100Unknown _0x0002 73728Unknown _0x000cN/AUnknown _0x0010 0.0009344655263Unknown _0x0040 N/AUnknown _0x0050 1Unknown _0x0100 N/A

Adding extra processing to strip out exif before upload would solve the issue, but perhaps would be better to fix it on image handler or thumbor side ?

actual image:

Is it better to create issue in thumbor repo?

any way to modify RESPECT_ORIENTATION?

Thumbor's RESPECT_ORIENTATION configuration is super useful when you don't want EXIF data sent but do want the photos to come out correctly rotated, it's a simple configuration flag - is there any way to get that enabled or support added for it?

Thanks!