willnorris / imageproxy

A caching, resizing image proxy written in Go

imageproxy

imageproxy is a caching image proxy server written in go. It features:

• basic image adjustments like resizing, cropping, and rotation
• access control using allowed hosts list or request signing (HMAC-SHA256)
• support for jpeg, png, webp (decode only), tiff, and gif image formats (including animated gifs)
• caching in-memory, on disk, or with Amazon S3, Google Cloud Storage, Azure Storage, or Redis
• easy deployment, since it's pure go

Personally, I use it primarily to dynamically resize images hosted on my own site (read more in this post). But you can also enable request signing and use it as an SSL proxy for remote images, similar to atmos/camo but with additional image adjustment options.

I aim to keep imageproxy compatible with the two most recent major go releases. I also keep track of the minimum go version that still works (currently go1.18), but that might change at any time. You can see the go versions that are tested against in .github/workflows/tests.yml.

URL Structure

imageproxy URLs are of the form http://localhost/{options}/{remote_url}.

Options

Options are available for cropping, resizing, rotation, flipping, and digital signatures among a few others. Options for are specified as a comma delimited list of parameters, which can be supplied in any order. Duplicate parameters overwrite previous values.

See the full list of available options at https://godoc.org/willnorris.com/go/imageproxy#ParseOptions.

Remote URL

The URL of the original image to load is specified as the remainder of the path, without any encoding. For example, http://localhost/200/https://willnorris.com/logo.jpg.

In order to optimize caching, it is recommended that URLs not contain query strings.

Examples

The following live examples demonstrate setting different options on this source image, which measures 1024 by 678 pixels.

Options	Meaning	Image
200x	200px wide, proportional height
x0.15	15% original height, proportional width
100x150	100 by 150 pixels, cropping as needed
100	100px square, cropping as needed
150,fit	scale to fit 150px square, no cropping
100,r90	100px square, rotated 90 degrees
100,fv,fh	100px square, flipped horizontal and vertical
200x,q60	200px wide, proportional height, 60% quality
200x,png	200px wide, converted to PNG format
cx175,cw400,ch300,100x	crop to 400x300px starting at (175,0), scale to 100px wide

The smart crop feature can best be seen by comparing crops of this source image, with and without smart crop enabled.

Options	Meaning	Image
150x300	150x300px, standard crop
150x300,sc	150x300px, smart crop

Transformation also works on animated gifs. Here is this source image resized to 200px square and rotated 270 degrees:

Getting Started

Install the package using:

go install willnorris.com/go/imageproxy/cmd/imageproxy@latest

Once installed, ensure $GOPATH/bin is in your $PATH, then run the proxy using:

imageproxy

This will start the proxy on port 8080, without any caching and with no allowed host list (meaning any remote URL can be proxied). Test this by navigating to http://localhost:8080/500/https://octodex.github.com/images/codercat.jpg and you should see a 500px square coder octocat.

Cache

By default, the imageproxy command does not cache responses, but caching can be enabled using the -cache flag. It supports the following values:

• memory - uses an in-memory LRU cache. By default, this is limited to 100mb. To customize the size of the cache or the max age for cached items, use the format memory:size:age where size is measured in mb and age is a duration. For example, memory:200:4h will create a 200mb cache that will cache items no longer than 4 hours.

• directory on local disk (e.g. /tmp/imageproxy) - will cache images on disk

• s3 URL (e.g. s3://region/bucket-name/optional-path-prefix) - will cache images on Amazon S3. This requires either an IAM role and instance profile with access to your your bucket or AWS_ACCESS_KEY_ID and AWS_SECRET_KEY environmental variables be set. (Additional methods of loading credentials are documented in the aws-sdk-go session package).

  Additional configuration options (further documented here) may be specified as URL query string parameters, which are mostly useful when working with s3-compatible services:

  • "endpoint" - specify an alternate API endpoint
  • "disableSSL" - set to "1" to disable SSL when calling the API
  • "s3ForcePathStyle" - set to "1" to force the request to use path-style addressing

  For example, when working with minio, which doesn't use regions, provide a dummy region value and custom endpoint value:

  s3://fake-region/bucket/folder?endpoint=minio:9000&disableSSL=1&s3ForcePathStyle=1
  

  Similarly, for Digital Ocean Spaces, provide a dummy region value and the appropriate endpoint for your space:

  s3://fake-region/bucket/folder?endpoint=sfo2.digitaloceanspaces.com
  

• gcs URL (e.g. gcs://bucket-name/optional-path-prefix) - will cache images on Google Cloud Storage. Authentication is documented in Google's Application Default Credentials docs.

• azure URL (e.g. azure://container-name/) - will cache images on Azure Storage. This requires AZURESTORAGE_ACCOUNT_NAME and AZURESTORAGE_ACCESS_KEY environment variables to bet set.

• redis URL (e.g. redis://hostname/) - will cache images on the specified redis host. The full URL syntax is defined by the redis URI registration. Rather than specify password in the URI, use the REDIS_PASSWORD environment variable.

For example, to cache files on disk in the /tmp/imageproxy directory:

imageproxy -cache /tmp/imageproxy

Reload the codercat URL, and then inspect the contents of /tmp/imageproxy. Within the subdirectories, there should be two files, one for the original full-size codercat image, and one for the resized 500px version.

Multiple caches can be specified by separating them by spaces or by repeating the -cache flag multiple times. The caches will be created in a tiered fashion. Typically this is used to put a smaller and faster in-memory cache in front of a larger but slower on-disk cache. For example, the following will first check an in-memory cache for an image, followed by a gcs bucket:

imageproxy -cache memory -cache gcs://my-bucket/

Allowed Referrer List

You can limit images to only be accessible for certain hosts in the HTTP referrer header, which can help prevent others from hotlinking to images. It can be enabled by running:

imageproxy  -referrers example.com

Reload the codercat URL, and you should now get an error message. You can specify multiple hosts as a comma separated list, or prefix a host value with *. to allow all sub-domains as well.

Allowed and Denied Hosts List

You can limit the remote hosts that the proxy will fetch images from using the allowHosts and denyHosts flags. This is useful, for example, for locking the proxy down to your own hosts to prevent others from abusing it. Of course if you want to support fetching from any host, leave off these flags.

Try it out by running:

imageproxy -allowHosts example.com

Reload the codercat URL, and you should now get an error message. Alternately, try running:

imageproxy -denyHosts octodex.github.com

Reloading the codercat URL will still return an error message.

You can specify multiple hosts as a comma separated list to either flag, or prefix a host value with *. to allow or deny all sub-domains. You can also specify a netblock in CIDR notation (127.0.0.0/8) -- this is useful for blocking reserved ranges like 127.0.0.0/8, 192.168.0.0/16, etc.

If a host matches both an allowed and denied host, the request will be denied.

Allowed Content-Type List

You can limit what content types can be proxied by using the contentTypes flag. By default, this is set to image/*, meaning that imageproxy will process any image types. You can specify multiple content types as a comma separated list, and suffix values with * to perform a wildcard match. Set the flag to an empty string to proxy all requests, regardless of content type.

Signed Requests

Instead of an allowed host list, you can require that requests be signed. This is useful in preventing abuse when you don't have just a static list of hosts you want to allow. Signatures are generated using HMAC-SHA256 against the remote URL, and url-safe base64 encoding the result:

base64urlencode(hmac.New(sha256, <key>).digest(<remote_url>))

The HMAC key is specified using the signatureKey flag. If this flag begins with an "@", the remainder of the value is interpreted as a file on disk which contains the HMAC key.

Try it out by running:

imageproxy -signatureKey "secretkey"

Reload the codercat URL, and you should see an error message. Now load a signed codercat URL (which contains the signature option) and verify that it loads properly.

Some simple code samples for generating signatures in various languages can be found in docs/url-signing.md. Multiple valid signature keys may be provided to support key rotation by repeating the signatureKey flag multiple times, or by providing a space-separated list of keys. To use a key with a literal space character, load the key from a file using the "@" prefix documented above.

If both a whiltelist and signatureKey are specified, requests can match either. In other words, requests that match one of the allowed hosts don't necessarily need to be signed, though they can be.

Default Base URL

Typically, remote images to be proxied are specified as absolute URLs. However, if you commonly proxy images from a single source, you can provide a base URL and then specify remote images relative to that base. Try it out by running:

imageproxy -baseURL https://octodex.github.com/

Then load the codercat image, specified as a URL relative to that base: http://localhost:8080/500/images/codercat.jpg. Note that this is not an effective method to mask the true source of the images being proxied; it is trivial to discover the base URL being used. Even when a base URL is specified, you can always provide the absolute URL of the image to be proxied.

Scaling beyond original size

By default, the imageproxy won't scale images beyond their original size. However, you can use the scaleUp command-line flag to allow this to happen:

imageproxy -scaleUp true

WebP and TIFF support

Imageproxy can proxy remote webp images, but they will be served in either jpeg or png format (this is because the golang webp library only supports webp decoding) if any transformation is requested. If no format is specified, imageproxy will use jpeg by default. If no transformation is requested (for example, if you are just using imageproxy as an SSL proxy) then the original webp image will be served as-is without any format conversion.

Because so few browsers support tiff images, they will be converted to jpeg by default if any transformation is requested. To force encoding as tiff, pass the "tiff" option. Like webp, tiff images will be served as-is without any format conversion if no transformation is requested.

Run imageproxy -help for a complete list of flags the command accepts. If you want to use a different caching implementation, it's probably easiest to just make a copy of cmd/imageproxy/main.go and customize it to fit your needs... it's a very simple command.

Environment Variables

All configuration flags have equivalent environment variables of the form IMAGEPROXY_$NAME. For example, an on-disk cache could be configured by calling

IMAGEPROXY_CACHE="/tmp/imageproxy" imageproxy

Deploying

In most cases, you can follow the normal procedure for building a deploying any go application. For example:

• go build willnorris.com/go/imageproxy/cmd/imageproxy
• copy resulting binary to /usr/local/bin
• copy etc/imageproxy.service to /lib/systemd/system and enable using systemctl.

Instructions have been contributed below for running on other platforms, but I don't have much experience with them personally.

Heroku

It's easy to vendorize the dependencies with Godep and deploy to Heroku. Take a look at this GitHub repo (make sure you use the heroku branch).

AWS Elastic Beanstalk

O’Reilly Media set up a repository with everything you need to deploy imageproxy to Elastic Beanstalk. Just follow the instructions in the README.

Docker

A docker image is available at ghcr.io/willnorris/imageproxy.

You can run it by

docker run -p 8080:8080 ghcr.io/willnorris/imageproxy -addr 0.0.0.0:8080

Or in your Dockerfile:

ENTRYPOINT ["/app/imageproxy", "-addr 0.0.0.0:8080"]

If running imageproxy inside docker with a bind-mounted on-disk cache, make sure the container is running as a user that has write permission to the mounted host directory. See more details in #198.

Note that all configuration options can be set using environment variables, which is often the preferred approach for containers.

nginx

Use the proxy_pass directive to send requests to your imageproxy instance. For example, to run imageproxy at the path "/api/imageproxy/", set:

  location /api/imageproxy/ {
    proxy_pass http://localhost:4593/;
  }

Depending on other directives you may have in your nginx config, you might need to alter the precedence order by setting:

  location ^~ /api/imageproxy/ {
    proxy_pass http://localhost:4593/;
  }

Clients

• Ruby

License

imageproxy is copyright its respective authors. All of my personal work on imageproxy through 2020 (which accounts for the majority of the code) is copyright Google, my employer at the time. It is available under the Apache 2.0 License.

imageproxy's Issues

Docs upgrade to support deployment

As a Golang newbie deploying the app is pretty tricky.
It would sure be useful to have some directions on that.
I managed to deploy the app in EC2 after an hour of experiments and could add my notes on the process, but directions from the maker itself would probably be more valuable.

Support for SSL

If i'm correct that there seems to be no way to activate SSL support. I tried setting the --addr to localhost:443 but there seems to be nowhere to point to ssl certificates to use.

This can be done by using a nginx-proxy, but it would be great If there was native support for SSL connections.

Log files rotation

Hi,

Would it be possible to have a log rotate option (it's generally done with a signal who also reload the configuration) ? I would like to rotate the log files every days at midnight, without restarting the service.

From what I've read, the log file is actually rotated whis this code from glog project:

// MaxSize is the maximum size of a log file in bytes.
var MaxSize uint64 = 1024 * 1024 * 1800

func (sb *syncBuffer) Write(p []byte) (n int, err error) {
  if sb.nbytes+uint64(len(p)) >= MaxSize {
    if err := sb.rotateFile(time.Now()); err != nil {
      sb.logger.exit(err)
    }
}
...

Would be nice too to change the MaxSize value ...

allow in-memory cache to limit max cache size

Add support for base URL

It would be great if one could add a BASE_URL environment variable that gets added to each request.

A call like this:

http://proxy/{options}/folder/myimage.jpg

Would get proxied like this:

{BASE_URL}/folder/myimage.jpg

That would allow us to have much nicer URL's. Would you be interested in a pull request if I add this?

s3 cache returns AWSAccessKeyID error

I'm getting an error from AWS S3 that has been very tricky to debug:

2016/07/11 22:42:35 s3util.Create failed: unwanted http status 403: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId></AWSAccessKeyId><RequestId>3BA092B6EBA3975B</RequestId><HostId>pDi5vYDyJszinDnRtboHuATtH80kOv67PXjvI08j/pYIMaWjsIw8ggHQ9t86ta4NI5o+Q0vFzNs=</HostId></Error>"

Inside the same docker container with the environment variables, set this command works:

root@4cd96af5dc13:/go/src/app# aws s3api put-object --bucket image-cache --key hello --debug

Here's my startup command:

/go/bin/imageproxy -addr 0.0.0.0:8080 -cache s3://s3-us-west-2.amazonaws.com/image-cache

I setup an IAM user and set the 2 required environment variables. The user has full S3 access, I attached every S3-related policy I could find.

Tried all the fixes suggested in #58

add support for animated gifs

Example code (under Apache 2.0 license): https://github.com/dpup/go-scratch/blob/master/gif-resize/gif-resize.go

add etag support

we support etags for fetching remote images, but not when serving images.

Heroku deploy guide?

Are there some instructions or a basic guide on deploying this service to Heroku?

cache and serve gzipped resource if client requests it

I'm wondering if it would be useful to support gzip compression if the Accept-Encoding: gzip header is set. This would probably impact the caching logic, as imageproxy would need to store both the uncompressed and compressed version in cache, and serve it based on the header.

I'm mounting imageproxy as a custom origin for a Cloudfront distritution, and it works great. But Cloudfront will automatically cache different version of the same image based on the Accept-Encoding: gzip header, so if imageproxy supported it, it would just work.

Thoughts? I'd be happy to do some development on it.

Scaling images up

Hi again.

Another question: We are in a situation where we actually want our images to scale up if they are too small. We have a very particular design, and it breaks the grid if an image is requested to be 1000px, but the original source is 500px and it stays that way.

Would you consider a scaleup=true flag that allowed scaling up images?

add support for URL signing

Instead of just URL whitelisting, @teodor-pripoae has been working on support for URL signing, which would allow proxying from any remote host without risk of the proxy being abused. Beginning work can be seen at teodor-pripoae/imageproxy@willnorris:master...0b31315. I think the implementation can be simplified a bit, but starting this issue to track this feature.

S3

I'm having trouble configuring S3 caching. Can you confirm I am using the correct syntax ? I have verified my S3 permissions and connected from this instance using the AWS SDK. My access key and security key are setup via aws configure.

Start Command
imageproxy/bin/imageproxy -cache s3://s3.amazonaws.com/zone-dropbox/output/cm/image-cache/ -addr 10.138.107.125:8080

Error
2016/04/11 22:00:46 s3util.Create failed: unwanted http status 403: "\nInvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.AEAC7AA329BBE037tEBB1ELYxnC8PD5lT1JHhO02so4ZWyB3Vw+tIfQl2GPbs0398kR+S+7VFhoJy7Wzc5azjN/I0N0=

better support of caching headers

notably, we don't do anything directly with the Cache-Control. The httpcache library handles respecting upstream directives in terms of what gets cached, but we don't set any Cache-Control directives on downstream responses.

@shepik, I see that you rewrote part of the httpcache library in 781938a, but I haven't yet looked closely at what you changed. Were these custom changes for something you're doing, or are these things that should be considered for inclusion upstream?

also /cc @jelinden who added Cache-Control support in e41beef for cloudfront. I'm not sure that we want to hard-code a max-age, as opposed to respecting the max-age from upstream, but I'd be curious to hear what cloudfront needs in this regard.

Crop mode - art directed crop

I don't think there are any existing go packages to do this level of intelligent cropping, so this is more of a "wouldn't it be nice" feature request. But it would sure be cool if imageproxy could do intelligent art-directed cropping as described in the following articles:

• https://www.smashingmagazine.com/2016/02/automatically-art-directed-responsive-images-go/
• http://cloudinary.com/blog/automatically_art_directed_responsive_images

A good first step for this would simply be to support #18 and #19, which would allow manually defining these various crops.

A way to limit options to some values

I have N configs for images and don't want to allow any other options except the whitelisted.
I see two solutions:

• hmac options in this case a legitimate user (server) can generate any request URL.
• provide a whitelist to imageproxy either as config or a func type Whitelist func(ops Options) bool

What do you think?

more flexibility with size option

Currently, size must appear at the beginning of the options, and must be present (or at least a leading comma must be present). This should be more flexible. For example, the following should work:

https://s.wjn.me/r=90/https://willnorris.com/content/uploads/2013/12/small-things.jpg
https://s.wjn.me/fv,100x200/https://willnorris.com/content/uploads/2013/12/small-things.jpg

Default image for HTTP errors

Would it be possible to specify a default image to return in the event of an HTTP error, or invalid content type?

expand cache implementations and simplify command line options

To date, I've pushed back on adding alternate caching implementations to the core project. That stems largely from the fact that this was originally developed just for my own use, and the only caching I use is the filesystem. The core imageproxy package knows nothing about cache implementations, only cmd/imageproxy does, so adding support for other cache backends just involved changing that package. That seemed relatively simple enough that I've just instructed people to do that. Plus, I didn't want to bring along the weight of a bunch of additional storage libraries that weren't going to get used in most cases.

Anyway, all that being said, I think I've changed my mind on this issue. The increase to the binary size should be minimal in the grand scheme of things, and having support for common backends (namely, s3 and redis) out of the box is worth the trade off.

Cache implementations:

• #40 adds support for s3 using https://github.com/sourcegraph/s3cache
• some fork somewhere added support for redis, I'll have to track that down
• memcache would probably be easy to add, but I'm not sure there's any demand
• https://github.com/gregjones/httpcache added leveldb support some time ago, so that would be easy to add if desirable

---

Additionally, I'd like to simplify the cache-related command line options, particularly in light of adding support for more backends. I'm thinking about having a single -cache flag that takes a URL (see s3 examples in #40), or a few special values (namely, "memory" for the in-memory cache). By default, there would be no caching (that would be a change from the behavior today).

The -cacheSize flag would stick around, and would be passed to the cache backends that support a max size.

The -cacheDir flag would probably stick around for a while for compatibility, but it would be deprecated in favor of -cache.

Auto expiry of items in the disk cache.

If using the disk cache, once the disk cache hits 100MB of usage the http server will lock up and not serve new requests because the disk cache is full.

Log new image sizes (and options) being written to cache

This would be great to keep track of what image sizes (and options) are being used by the image proxy. Maybe this would be a flag when starting the service since not everyone are interested by this info.

Crop mode - set rectangle of crop

It would be awesome if the crop mode could take coordinates for where to crop the image. Something like: cropx (x coord), cropy (y coord), cropw (width of crop) and crop) (height of crop). This would ofc only be valid if the request is in crop mode.

This would make integration with libs like jcrop (and others) really easy.

Cannot operate on an image

We tried to operate on this url, all operations (resize, crop, rotate) failed, please help
https://photos.hotelbeds.com/giata/bigger/13/139735/139735a_hb_a_001.jpg

optimize images

whenever possible, optimize images using something like pngcrush, jpegoptim, etc.

http: panic serving [IP address]: runtime error: invalid memory address or nil pointer dereference goroutine 6

Go code compiles just fine on a RHEL 6.4 box, but any hit trying to grab something from the host fails with the subject line's error.

Here's the stack trace:

[running]:
net/http.func·009()
    /usr/local/go/src/pkg/net/http/server.go:1093 +0xae
runtime.panic(0x6738e0, 0x98e988)
    /usr/local/go/src/pkg/runtime/panic.c:248 +0x106
github.com/willnorris/imageproxy/proxy.(*Proxy).ServeHTTP(0xc2100513c0, 0x7f7d1fb948f8, 0xc21000f5a0, 0xc210063340)
    /root/gocode/src/github.com/willnorris/imageproxy/proxy/proxy.go:78 +0x247
net/http.serverHandler.ServeHTTP(0xc21001e460, 0x7f7d1fb948f8, 0xc21000f5a0, 0xc210063340)
    /usr/local/go/src/pkg/net/http/server.go:1597 +0x16e
net/http.(*conn).serve(0xc210062480)
    /usr/local/go/src/pkg/net/http/server.go:1167 +0x7b7
created by net/http.(*Server).Serve
    /usr/local/go/src/pkg/net/http/server.go:1644 +0x28b

Entrypoint '/gopath/bin/imageproxy' does not exist

Getting an error when running:

ERROR: Cannot start container 985dd2dab5a0401a8ec8835638632e4d6bc45b50b6c2c029e04cd43584e9954f: [8] System error: exec: "/gopath/bin/imageproxy": stat /gopath/bin/imageproxy: no such file or directory

Further investigating I found the ENTRYPOINT written in Dockerfile '/gopath/bin/imageproxy' does not exist in the image.

Blacklisting private ip addresses/hosts

Hi there,
I'd like to blacklist private ip addresses or hosts, so it won't request it self and other services running in the same private network.
Does it support some kind of blacklist?
I tried this and it worked: http://localhost:8080/http://172.17.0.1/html/ being 172.17.0.1 the ip address of the host running docker.
Thanks!

Crop mode - set gravity of crop

It would be great if it was possible to set the gravity for the crop.
Something like Top, Bottom, Left, Right or Center(default).

Might be an option like: cropt (top), cropb (bottom), cropl (left) and cropr (right) that is only valid if the image is being cropped.

add CORS support

If imageproxy is being run behind a web server, then this of course could be configured there. If not, then an option to set CORS headers on responses would be helpful (it looks like a handful of downstream forks have done this: b6b3e33, aa8b461). Rather than setting this to "*", we may want to make it configurable?

more info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS

/cc @bnmnetp @danisla

How can I install it at my server via FTP?

Any possible mean to install it at my server via FTP?

Add a content type check

We should use http.DetectContentType to verify the content type and return an error if not an image.

See PR #42 for request background.

Disk cache grows beyond the limit

I've done a load test on my installation with this configuration (there is some options from my fork):

/usr/bin/imageproxy -addr=localhost:8980 -cacheDir=/data/imageproxy/cache -cacheSize=1024 -log_dir=/data/imageproxy/log -filetypes=png,PNG,jpg,JPG,jpeg,JPEG,gif,GIF -signatureKey=xxx -logrotatecompatible=true

In front of that there is a nginx with a disk cache (who doesn't work the same than imageproxy cache as imageproxy still make a head request to the original server).

I've requested lot of different files and the imageproxy cache is now 3,2GiB with 56197 files. That's far beyond of the 1014MB given on the command line. Am I missing something ?

Since I already have a cache with nginx, it is possible to add an option to disable imageproxy cache ?

Rotation exif data lost after resize

Does imageproxy preserve EXIF data? When users upload portrait photos under some conditions that haven't been able to nail down, they are rotated incorrectly after imageproxy has processed them, even though they render correctly in the browser natively.

I seem to be having the same problem as described here: http://stackoverflow.com/questions/13971491/captured-photo-automatically-rotated-during-upload-in-ios-6-0-or-iphone

Any ideas how to solve? Thanks!

Bug with transparent animated gifs

When I resize a transparent animated gif the result appears to keep the last frame until the loop is done.

Example:

Source: http://media1.giphy.com/media/TGxuKUoMjea4w/giphy.gif

Resized: https://j.gifs.com/zp5P6O.gif

Placing behind a reverse proxy

In front of all our services I put nginx and use a location directive called /resize to proxy traffic to imageproxy. However I cannot get this to work because I get this error:

invalid request URL: malformed URL "/resize/50x/http://example.com/image.png": must provide absolute remote URL

I've looked into changing the code in data.go but have been unsuccessful in my attempts, maybe you could point me in the right direction or add a flag to support a URI path prefix?

add support for jpeg quality option

adding this as a placeholder for @cubabit's work in his add-quality-option branch (at least, I assume you intended to contribute this. Note also the relevant changes I made in 0980ea6)

skip transform if only non-transformation options set

In #63 (comment), I noted how the same image was oriented properly on one instance of imageproxy, but not on another. The underlying problem was that my server required a request signature to be present because of the whitelist I have on my server. This signature on the Options struct caused this check for empty options to return false, and therefore the image was decoded and re-encoded. However, the only option set was the signature itself, so there was no need to attempt to transform the image. The same is true for Fit, Quality, and ScaleUp. If only those options are set, then no actual transformation has been requested and the decoding and be skipped.

invalid request URL: malformed URL "http://[someurl]/": too few path segments

I'm trying out imageproxy via safari on Mac OS X 10.10.5, Go version is go version go1.6.2 darwin/amd64

If I set the system proxy to localhost:8080 and attempt to load http://news.bbc.co.uk in Safari, the browser shows:

invalid request URL: malformed URL "http://news.bbc.co.uk/": too few path segments

The terminal where I ran imageproxy from shows a lot of output like the following:

Alexs-Mac-Pro:~ alex$ /Users/alex/Documents/go/bin/imageproxy -cache  ~/Documents/imagecache/
imageproxy (version HEAD) listening on localhost:8080
2016/06/27 17:29:25 http: panic serving 127.0.0.1:57575: runtime error: slice bounds out of range
goroutine 21 [running]:
net/http.(*conn).serve.func1(0xc820094200)
    /usr/local/go/src/net/http/server.go:1389 +0xc1
panic(0x437100, 0xc820010090)
    /usr/local/go/src/runtime/panic.go:443 +0x4e9
willnorris.com/go/imageproxy.NewRequest(0xc82010e000, 0x0, 0xc820073440, 0x0, 0x0)
    /Users/alex/Documents/go/src/willnorris.com/go/imageproxy/data.go:242 +0x88f
willnorris.com/go/imageproxy.(*Proxy).ServeHTTP(0xc82006a620, 0xa49dc0, 0xc820075ba0, 0xc82010e000)
    /Users/alex/Documents/go/src/willnorris.com/go/imageproxy/imageproxy.go:101 +0x26a
net/http.serverHandler.ServeHTTP(0xc820094180, 0xa49dc0, 0xc820075ba0, 0xc82010e000)
    /usr/local/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc820094200)
    /usr/local/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e

I get exactly the same results using Chrome too.

nginx

I'm currently trying to set up this great tool, using nginx, however, i must be too stupid.

I've installed imageproxy and started it:

imageproxy -addr=127.0.0.1:8147

I've created a simple nginx config:

server {
    listen 80;
    listen [::]:80;

    server_name     whatever-you-name-it.tld;

    access_log      /var/log/nginx/imageproxy.access.log;
    error_log       /var/log/nginx/imageproxy.error.log;

    location / {
        proxy_pass  http://127.0.0.1:8147;
        proxy_redirect off;
        break;
    }
}

Restarted nginx and done. But connecting to it via browser fails with "Empty response". So i've tried a Curl request to check, what's the issue:

curl -o ~/test.jpg --user-agent "Mozilla 5.0" -H "Host: whatever-you-name-it.tld" 127.0.0.1:8080/https://willnorris.com/logo.jpg

It worked. I had the test.jpg in the folder and it was not malformed, broken, etc.

So, the issue is not your script. However, i would like to ask if you have any idea, what's wrong on my end.

Display default 404 image if not loaded

In the case of using the proxy URL in an email, it would be nice to show a default image if the external image 404s. While this may be possible to set on a server, I don't think it's possible to set a 404 image within an email.

Currently the service returns the 404 content (even HTML!) of the external site when requesting a bad image. Is there any way around this? Happy to submit a PR to add this functionality.

How to configure for docker cloud: service can't be reached

Sorry if this is a stupid question... I deployed the docker image to docker cloud. It works inside the remote terminal (localhost:8080), but the service is unreachable outside the container.

My docker-compose file looks like:

imageproxy:
  image: 'willnorris/imageproxy:latest'
  ports:
    - '8080:8080'

I'm assuming the internal web service is blocking connections but I am not savvy enough to know how to configure it.

Thank you for an otherwise awesome service and I look forward to be being able to deploy it to docker cloud once I have figured this out.

use a trie for diskv cache

diskv supports specifying a TransformFunction for defining where on disk a certain key should be stored. By default, it stores everything in a single directory. To prevent issues with maximum number of files in a large cache, we would use a trie (or similar structure) to break keys up into a couple of levels of sub-directories. Basically, I'm just thinking of something along the lines of:

func(s string) []string { return []string{s[0:2], s[2:4]} }

(see original report at #45 (comment))

cache transformed images

Currently, only the original image is cached, and transformations are performed on each request. Each requested image transformation should be cached so that it doesn't have to be redone each time.

Cache issue.

Hi,

I'm currently running imageproxy on a box with single core CPU and 768mb RAM. It is the only process aside nginx, both ran as docker containers.

After a few hours on a medium load (I don't have monitoring yet, but I can say about several hundred RPM) the server starts to return 500. The only way to make it work again is to restart the docker container.

The log looks like this:

2015/03/23 23:02:12 http: panic serving 172.17.42.1:42247: 7112 bytes still won't fit in the cache! (max 104857600 bytes)
goroutine 1583 [running]:
net/http.func·011()
    /usr/src/go/src/pkg/net/http/server.go:1100 +0xb7
runtime.panic(0x662420, 0xc20836da70)
    /usr/src/go/src/pkg/runtime/panic.c:248 +0x18d
io/ioutil.func·002()
    /usr/src/go/src/pkg/io/ioutil/ioutil.go:30 +0xf8
runtime.panic(0x662420, 0xc20836da70)
    /usr/src/go/src/pkg/runtime/panic.c:248 +0x18d
github.com/peterbourgon/diskv.(*Diskv).ensureCacheSpaceFor(0xc208054280, 0x1bc8, 0x0, 0x0)
    /go/src/github.com/peterbourgon/diskv/diskv.go:471 +0x366
github.com/peterbourgon/diskv.(*Diskv).cacheWithLock(0xc208054280, 0xc208389260, 0x20, 0xc208a1a000, 0x1bc8, 0x3dc8, 0x0, 0x0)
    /go/src/github.com/peterbourgon/diskv/diskv.go:390 +0x61
github.com/peterbourgon/diskv.(*Diskv).cacheWithoutLock(0xc208054280, 0xc208389260, 0x20, 0xc208a1a000, 0x1bc8, 0x3dc8, 0x0, 0x0)
    /go/src/github.com/peterbourgon/diskv/diskv.go:414 +0xba
github.com/peterbourgon/diskv.(*siphon).Read(0xc20836aff0, 0xc20d0c5bc8, 0x238, 0x238, 0xdc8, 0x0, 0x0)
    /go/src/github.com/peterbourgon/diskv/diskv.go:270 +0x20e
io/ioutil.(*nopCloser).Read(0xc20836da40, 0xc20d0c5bc8, 0x238, 0x238, 0xdc8, 0x0, 0x0)
    <autogenerated>:4 +0x7b
bytes.(*Buffer).ReadFrom(0xc20d377030, 0x7f9e8ee13ec0, 0xc20836da40, 0x1bc8, 0x0, 0x0)
    /usr/src/go/src/pkg/bytes/buffer.go:169 +0x20e
io/ioutil.readAll(0x7f9e8ee13ec0, 0xc20836da40, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/src/go/src/pkg/io/ioutil/ioutil.go:33 +0x1ae
io/ioutil.ReadAll(0x7f9e8ee13ec0, 0xc20836da40, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/src/go/src/pkg/io/ioutil/ioutil.go:42 +0x6c
github.com/peterbourgon/diskv.(*Diskv).Read(0xc208054280, 0xc208389260, 0x20, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/peterbourgon/diskv/diskv.go:174 +0x196
github.com/gregjones/httpcache/diskcache.(*Cache).Get(0xc20803a060, 0xc208389260, 0x20, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/gregjones/httpcache/diskcache/diskcache.go:22 +0x95
github.com/gregjones/httpcache.CachedResponse(0x7f9e8ee12998, 0xc20803a060, 0xc20cbde680, 0x0, 0x0, 0x0)
    /go/src/github.com/gregjones/httpcache/httpcache.go:48 +0x8e
github.com/gregjones/httpcache.(*Transport).RoundTrip(0xc208024ba0, 0xc20cbde680, 0xc9, 0x0, 0x0)
    /go/src/github.com/gregjones/httpcache/httpcache.go:140 +0x1a5
net/http.send(0xc20cbde4e0, 0x7f9e8ee12a30, 0xc208024ba0, 0xc20cbde5b0, 0x0, 0x0)
    /usr/src/go/src/pkg/net/http/client.go:195 +0x43d
net/http.(*Client).send(0xc208024b40, 0xc20cbde4e0, 0x6d, 0x0, 0x0)
    /usr/src/go/src/pkg/net/http/client.go:118 +0x15b
net/http.(*Client).doFollowingRedirects(0xc208024b40, 0xc20cbde4e0, 0x7c3ea8, 0x0, 0x0, 0x0)
    /usr/src/go/src/pkg/net/http/client.go:343 +0x97f
net/http.(*Client).Get(0xc208024b40, 0xc20d376bd0, 0x6d, 0x1, 0x0, 0x0)
    /usr/src/go/src/pkg/net/http/client.go:275 +0xaf
github.com/teodor-pripoae/imageproxy.(*Proxy).ServeHTTP(0xc20803e1c0, 0x7f9e8ee13cc0, 0xc2082fdf40, 0xc20cbde0d0)
    /go/src/github.com/teodor-pripoae/imageproxy/imageproxy.go:99 +0x733
net/http.serverHandler.ServeHTTP(0xc2080040c0, 0x7f9e8ee13cc0, 0xc2082fdf40, 0xc20cbde0d0)
    /usr/src/go/src/pkg/net/http/server.go:1673 +0x19f
net/http.(*conn).serve(0xc208c92d80)
    /usr/src/go/src/pkg/net/http/server.go:1174 +0xa7e
created by net/http.(*Server).Serve
    /usr/src/go/src/pkg/net/http/server.go:1721 +0x313

allow running behind an http.ServeMux

currently, an imageproxy.Proxy can't be run behind an http.ServeMux because the mux will canonicalize URL paths, collapsing the double forward slash in the remote URL. This is why the CLI registers the proxy as the only handler on the http.Server. I don't have any great ideas for how to fix this, but it would be really nice if an imageproxy.Proxy could be deployed as part of a larger binary that uses a ServeMux.

Possible solutions:

• encode the remote URL, similar to what atmos/camo does. I really don't want to do this
• internally, detect the presence of "http:/host" and re-add the second forward slash. This could work pretty well for the scheme, but wouldn't help if the remote path included an intentional double slash (which does seem unlikely)
• allow scheme-less URLs. I really don't want to do this either, plus it brings up the question of whether we assume https or http.

add Dockerfile

I don't run imageproxy inside docker myself, but it appears that at least some people do. Adding a dockerfile should be pretty simple.

/cc @sevki @etcinit

Add support for client hint headers

Client Hints are defined by draft-grigorik-http-client-hints and are supported in the recently-released-to-stable Chrome 46. They allow user agents to provide additional hints regarding how the content is going to be rendered. Particularly interesting for imageproxy:

• DPR (Device Pixel Ratio) - could be used to override the requested size options to accomodate higher DPR values
• Width - override requested width, since the client hint indicates the width at which the image is actually going to be displayed
• Downlink - could be used to change the default quality setting for JPEG images (?)

We may want to hide this behind a flag, at least for the time being, just to make sure it doesn't cause any problems. We'll also need to make sure and set an appropriate Vary header.

Further Reading:

• https://developers.google.com/web/updates/2015/09/automating-resource-selection-with-client-hints
• https://github.com/igrigorik/http-client-hints

add file-based cache implementation

Old go versions dont support ssl properly

might want to serve the binary on plain http because some old version of go (1.2) does not support some ssl, had to manually update to 1.4 to download it

what is the effect of the parameter inactive in nginx.conf

inactive : set the time of the cache to update not visited
description : Actually,it's impossible that the nginx server is not visited enough time. After the update on the real server, the cache on the nginx server would not be update in time. I don't know how should be useful. I have tried the purge, which is not convenient