atmos / camo

:lock: an http proxy to route images through SSL

camo

Camo is all about making insecure assets look secure. This is an SSL image proxy to prevent mixed content warnings on secure pages served from GitHub.

We want to allow people to keep embedding images in comments/issues/READMEs.

There's more info on the GitHub blog.

Using a shared key, proxy URLs are authenticated with hmac so we can bust caches/ban/rate limit if needed.

Camo currently runs on node version 0.10.29 at GitHub on heroku.

Features

• Max size for proxied images
• Follow redirects to a certain depth
• Restricts proxied images content-types to a whitelist
• Forward images regardless of HTTP status code

At GitHub we render markdown and replace all of the src attributes on the img tags with the appropriate URL to hit the proxies. There's example code for creating URLs in the tests.

URL Formats

Camo supports two distinct URL formats:

http://example.org/<digest>?url=<image-url>
http://example.org/<digest>/<image-url>

The <digest> is a 40 character hex encoded HMAC digest generated with a shared secret key and the unescaped <image-url> value. The <image-url> is the absolute URL locating an image. In the first format, the <image-url> should be URL escaped aggressively to ensure the original value isn't mangled in transit. In the second format, each byte of the <image-url> should be hex encoded such that the resulting value includes only characters [0-9a-f].

Configuration

Camo is configured through environment variables.

• PORT: The port number Camo should listen on. (default: 8081)
• CAMO_HEADER_VIA: The string for Camo to include in the Via and User-Agent headers it sends in requests to origin servers. (default: Camo Asset Proxy <version>)
• CAMO_KEY: A shared key consisting of a random string, used to generate the HMAC digest.
• CAMO_LENGTH_LIMIT: The maximum Content-Length Camo will proxy. (default: 5242880)
• CAMO_LOGGING_ENABLED: The logging level used for reporting debug or error information. Options are debug and disabled. (default: disabled)
• CAMO_MAX_REDIRECTS: The maximum number of redirects Camo will follow while fetching an image. (default: 4)
• CAMO_SOCKET_TIMEOUT: The maximum number of seconds Camo will wait before giving up on fetching an image. (default: 10)
• CAMO_TIMING_ALLOW_ORIGIN: The string for Camo to include in the Timing-Allow-Origin header it sends in responses to clients. The header is omitted if this environment variable is not set. (default: not set)
• CAMO_HOSTNAME: The Camo-Host header value that Camo will send. (default: unknown)
• CAMO_KEEP_ALIVE: Whether or not to enable keep-alive session. (default: false)

Testing Functionality

Bundle Everything

% rake bundle

Start the server

% coffee server.coffee

In another shell

% rake

Debugging

To see the full URL restclient is hitting etc, try this.

% RESTCLIENT_LOG=stdout rake

Deployment

You should run this on heroku.

To enable useful line numbers in stacktraces you probably want to compile the server.coffee file to native javascript when deploying.

% coffee -c server.coffee
% /usr/bin/env PORT=9090 CAMO_KEY="<my application key>" node server.js

Docker

A Dockerfile is included, you can build and run it with:

docker build -t camo .
docker run --env CAMO_KEY=YOUR_KEY -t camo

Examples

• Ruby - https://github.com/ankane/camo
• PHP - https://github.com/willwashburn/Phpamo

camo's Issues

Assets incompletely or not downloading

Camo seems to be incompletely downloading images (causing tearing and artifacts), or downright failing to download, and not re-attempting.

The example image on the Camo homepage has artifacts at the bottom. The example image of my project returns a 404.

What up?

Not Found (404) from Facebook CDN

I'm requesting the following URL through a Camo instance hosted on Heroku:

https://fbcdn-sphotos-e-a.akamaihd.net/hphotos-ak-xat1/v/t1.0-9/10646643_808192525973942_1181037437419058314_n.jpg?oh=463a2dcc9d3c7f54af0944b3738f4746&oe=570EBF6E&__gda__=1459390641_5aae331b8049ceb34f809db3cd097a47

The proxied URL has the following syntax:

https://camo.goldenline.io/bf676fa0a574eef65f18f1e5afc55e1cf12f6b4b/68747470733a2f2f666263646e2d7370686f746f732d652d612e616b616d616968642e6e65742f6870686f746f732d616b2d786174312f762f74312e302d392f31303634363634335f3830383139323532353937333934325f313138313033373433373431393035383331345f6e2e6a70673f6f683d343633613264636339643363376635346166303934346233373338663437343626616d703b6f653d353730454246364526616d703b5f5f6764615f5f3d313435393339303634315f3561616533333162383034396365623334663830396462336364303937613437

Unfortunately this results in a 404 error from Facebook's CDN.

Heroku logs:

2016-01-19T09:35:04.940187+00:00 heroku[router]: at=info method=GET path="/bf676fa0a574eef65f18f1e5afc55e1cf12f6b4b/68747470733a2f2f666263646e2d7370686f746f732d652d612e616b616d616968642e6e65742f6870686f746f732d616b2d786174312f762f74312e302d392f31303634363634335f3830383139323532353937333934325f313138313033373433373431393035383331345f6e2e6a70673f6f683d343633613264636339643363376635346166303934346233373338663437343626616d703b6f653d353730454246364526616d703b5f5f6764615f5f3d313435393339303634315f3561616533333162383034396365623334663830396462336364303937613437" host=camo.goldenline.io request_id=6efe3c06-b62d-48a7-88ee-81b67452529f fwd="217.153.60.153,162.158.89.187" dyno=web.1 connect=1ms service=66ms status=404 bytes=442
2016-01-19T09:35:04.876421+00:00 app[web.1]: --------------------------------------------
2016-01-19T09:35:04.939444+00:00 app[web.1]: { server: 'AkamaiGHost',
2016-01-19T09:35:04.939447+00:00 app[web.1]:   'mime-version': '1.0',
2016-01-19T09:35:04.939448+00:00 app[web.1]:   'content-type': 'text/html',
2016-01-19T09:35:04.939449+00:00 app[web.1]:   'content-length': '176',
2016-01-19T09:35:04.939450+00:00 app[web.1]:   expires: 'Tue, 19 Jan 2016 09:35:04 GMT',
2016-01-19T09:35:04.939450+00:00 app[web.1]:   date: 'Tue, 19 Jan 2016 09:35:04 GMT',
2016-01-19T09:35:04.939451+00:00 app[web.1]:   connection: 'close',
2016-01-19T09:35:04.939452+00:00 app[web.1]:   'last-modified': 'Tue, 1 Jan 2008 00:00:00 GMT' }
2016-01-19T09:35:04.939515+00:00 app[web.1]: --------------------------------------------
2016-01-19T09:35:04.939643+00:00 app[web.1]: [2016-01-19T09:35:04.939Z] Non-Image content-type returned 'text/html': https://fbcdn-sphotos-e-a.akamaihd.net/hphotos-ak-xat1/v/t1.0-9/10646643_808192525973942_1181037437419058314_n.jpg?oh=463a2dcc9d3c7f54af0944b3738f4746&oe=570EBF6E&__gda__=1459390641_5aae331b8049ceb34f809db3cd097a47
2016-01-19T09:35:04.877126+00:00 app[web.1]: { type: 'path',
2016-01-19T09:35:04.877132+00:00 app[web.1]:   url: '/bf676fa0a574eef65f18f1e5afc55e1cf12f6b4b/68747470733a2f2f666263646e2d7370686f746f732d652d612e616b616d616968642e6e65742f6870686f746f732d616b2d786174312f762f74312e302d392f31303634363634335f3830383139323532353937333934325f313138313033373433373431393035383331345f6e2e6a70673f6f683d343633613264636339643363376635346166303934346233373338663437343626616d703b6f653d353730454246364526616d703b5f5f6764615f5f3d313435393339303634315f3561616533333162383034396365623334663830396462336364303937613437',
2016-01-19T09:35:04.877133+00:00 app[web.1]:   headers: 
2016-01-19T09:35:04.877134+00:00 app[web.1]:    { host: 'camo.goldenline.io',
2016-01-19T09:35:04.877135+00:00 app[web.1]:      connection: 'close',
2016-01-19T09:35:04.877136+00:00 app[web.1]:      'accept-encoding': 'gzip',
2016-01-19T09:35:04.877136+00:00 app[web.1]:      'cf-ipcountry': 'PL',
2016-01-19T09:35:04.877137+00:00 app[web.1]:      'x-forwarded-for': '217.153.60.153, 162.158.89.187',
2016-01-19T09:35:04.877138+00:00 app[web.1]:      'cf-ray': '26718c272967238a-FRA',
2016-01-19T09:35:04.877138+00:00 app[web.1]:      'x-forwarded-proto': 'http',
2016-01-19T09:35:04.877139+00:00 app[web.1]:      'cf-visitor': '{"scheme":"https"}',
2016-01-19T09:35:04.877140+00:00 app[web.1]:      'cache-control': 'max-age=0',
2016-01-19T09:35:04.877140+00:00 app[web.1]:      accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
2016-01-19T09:35:04.877141+00:00 app[web.1]:      'upgrade-insecure-requests': '1',
2016-01-19T09:35:04.877143+00:00 app[web.1]:      'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36',
2016-01-19T09:35:04.877143+00:00 app[web.1]:      'accept-language': 'pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4',
2016-01-19T09:35:04.877144+00:00 app[web.1]:      via: '2.0 cloudflare, 1.1 vegur',
2016-01-19T09:35:04.877144+00:00 app[web.1]:      'cf-connecting-ip': '217.153.60.153',
2016-01-19T09:35:04.877145+00:00 app[web.1]:      'x-request-id': '6efe3c06-b62d-48a7-88ee-81b67452529f',
2016-01-19T09:35:04.877146+00:00 app[web.1]:      'x-forwarded-port': '80',
2016-01-19T09:35:04.877146+00:00 app[web.1]:      'connect-time': '1',
2016-01-19T09:35:04.877147+00:00 app[web.1]:      'x-request-start': '1453196104871',
2016-01-19T09:35:04.877148+00:00 app[web.1]:      'total-route-time': '0' },
2016-01-19T09:35:04.877149+00:00 app[web.1]:   dest: 'https://fbcdn-sphotos-e-a.akamaihd.net/hphotos-ak-xat1/v/t1.0-9/10646643_808192525973942_1181037437419058314_n.jpg?oh=463a2dcc9d3c7f54af0944b3738f4746&oe=570EBF6E&__gda__=1459390641_5aae331b8049ceb34f809db3cd097a47',
2016-01-19T09:35:04.877150+00:00 app[web.1]:   digest: 'bf676fa0a574eef65f18f1e5afc55e1cf12f6b4b' }
2016-01-19T09:35:04.877223+00:00 app[web.1]: --------------------------------------------
2016-01-19T09:35:04.877503+00:00 app[web.1]: --------------------------------------------
2016-01-19T09:35:04.877830+00:00 app[web.1]: { Via: 'Camo Asset Proxy 2.3.0',
2016-01-19T09:35:04.877832+00:00 app[web.1]:   'User-Agent': 'Camo Asset Proxy 2.3.0',
2016-01-19T09:35:04.877833+00:00 app[web.1]:   Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
2016-01-19T09:35:04.877834+00:00 app[web.1]:   'Accept-Encoding': 'gzip',
2016-01-19T09:35:04.877834+00:00 app[web.1]:   'X-Frame-Options': 'deny',
2016-01-19T09:35:04.877835+00:00 app[web.1]:   'X-XSS-Protection': '1; mode=block',
2016-01-19T09:35:04.877836+00:00 app[web.1]:   'X-Content-Type-Options': 'nosniff',
2016-01-19T09:35:04.877837+00:00 app[web.1]:   'Content-Security-Policy': 'default-src \'none\'; img-src data:; style-src \'unsafe-inline\'',
2016-01-19T09:35:04.877837+00:00 app[web.1]:   host: 'fbcdn-sphotos-e-a.akamaihd.net' }
2016-01-19T09:35:04.877917+00:00 app[web.1]: --------------------------------------------
2016-01-19T09:35:04.939221+00:00 app[web.1]: --------------------------------------------

Requesting the same URL from the same Heroku instance via cURL works:

$ curl -XGET -I "https://fbcdn-sphotos-e-a.akamaihd.net/hphotos-ak-xat1/v/t1.0-9/10646643_808192525973942_1181037437419058314_n.jpg?oh=463a2dcc9d3c7f54af0944b3738f4746&oe=570EBF6E&__gda__=1459390641_5aae331b8049ceb34f809db3cd097a47"
HTTP/1.1 200 OK
Last-Modified: Sun, 03 Jan 2016 09:46:08 GMT
Content-Type: image/jpeg
X-Cache-Ts: 1451816268
Timing-Allow-Origin: *
Access-Control-Allow-Origin: *
Content-Length: 38426
Date: Tue, 19 Jan 2016 09:41:50 GMT
Connection: keep-alive
Cache-Control: max-age=1209600

Any ideas?

Forward image when valid Content-Type & body

Some services return a 404 or 5xx with an image. This can be used e.g. for default / fallback images and it's totally legitimate to do so.

I think camo should proxy all valid images regardless of the HTTP status – this is what Browsers do as well (or intend to, see chromium-464515).

I'd be happy to make a PR if you agree.

Service url??

Hi, I am new to this technology. I want to show the http:// images on my https:// website. Is there somewhere, I can use as a service?

Should follow redirections but not cache against the redirected path

I'm working on a flexible badge service, Umbo, that allows user to update the badge with any values.

Example: https://github.com/uzyn/umbo-weather/blob/master/README.md

To not reinvent the wheel, I'm relying on the great service, shields.io, for badge generation (for example: https://img.shields.io/badge/Singapore%2C%20Singapore-27.47%C2%B0C-yellow.svg)

For the example at umbo-weather, there's a background process that updates the value every 5 minutes.

The badge URL for Singapore weather is http://umbo.zynesis.com/singaporeio8gg0ok8ckwcck8.svg and it redirects to a different shields.io badge URL based on the value.

(Try going to http://umbo.zynesis.com/singaporeio8gg0ok8ckwcck8.svg in 5 minutes interval or so)

Camo, however, returns the final redirected URL and caches against that, of which the value never changes.

It would be great if Camo caches the original requesting URL instead. If not, what do you recommend that I should be doing for such service?

SyntaxError: In server.coffee, octal literals '0600' must be prefixed with '0o' on line 184

Don't know much about Node or coffee, so I don't know what is going on. Do you know what this means?

jong$ coffee server.coffee
SyntaxError: In server.coffee, octal literals '0600' must be prefixed with '0o' on line 184
    at SyntaxError (unknown source)
    at Lexer.error (/usr/local/lib/coffee-script/lib/coffee-script/lexer.js:606:13)
    at Lexer.numberToken (/usr/local/lib/coffee-script/lib/coffee-script/lexer.js:122:14)
    at Lexer.tokenize (/usr/local/lib/coffee-script/lib/coffee-script/lexer.js:31:159)
    at Object.compile (/usr/local/lib/coffee-script/lib/coffee-script/coffee-script.js:41:32)
    at /usr/local/lib/coffee-script/lib/coffee-script/command.js:141:33
    at /usr/local/lib/coffee-script/lib/coffee-script/command.js:111:18
    at [object Object].<anonymous> (fs.js:115:5)
    at [object Object].emit (events.js:64:17)
    at afterRead (fs.js:1111:12)

I'm on OS X 10.7. Do you know what's going on?

HTTP_PROXY not being used

This app does not seem to use HTTP_PROXY and HTTPS_PROXY environment variables. And as such is useless behind a proxy.

I didn't neither find a config option to set HTTP proxy manually.

how to work around this?

I'm working on a service that has png badges that users can put in their git hub repos, https://feedopensource.com

Of course there are many other services which depend on this feature also.

The recent deployment of camo for https images has broken this service, however. So, I understand that setting content-type: image/png and cache-control: no-cache should make the proxy download a fresh image each time (the necessary behavior for badges to work)

However, since the URLs are cached there are many images which are hidden behind a cache but they should not be. Images like this are now stuck.

How can I get the badges to work again?
I can't figure out why my badge is 404ing

https://github-camo.global.ssl.fastly.net/73996164d000f0a65b18c888566e2ea469bf02fc/68747470733a2f2f666565646f70656e736f757263652e636f6d2f697465726174696f6e2f646f6d696e6963746172722f666565646f70656e736f757263652f31386f42455639684664444d45637439597332625341754559346269324b534d48692e706e673f727272727223312e32

https://feedopensource.com/iteration/dominictarr/feedopensource/18oBEV9hFdDMEct9Ys2bSAuEY4bi2KSMHi.png

I've tried adding ?blah on the end to make a different url, which does produce a different hash, but I do not know why it's not displaying the image - which works if you open the image in a browser - curl doesn't have the right CA, so you need to use --insecure, but browsers do not complain.

there is no helpful messages in the camo headers, only "not found".

404 on GitHub stars/issues and Shields.io

It seems that, very often but not always, images from Shields.io regarding GitHub "live" scores are giving 404 error on Camo.

• real life example: https://github.com/ZiZuu-store/zzCleanURLs/blob/dev/README.md
• as you can see in the above link non-live images do not rise the problem (the "chat" icon)

certain URLs can crash camo, rather than leading to a standard error response

Don't ask me why, but one of our users tried to get the following URL proxied:

http://www.google.com/url?sa=i&rct=j&q=gods+vs+monsters&source=images&cd=&cad=rja&uact=8&docid=_0iam5px020tzM&tbnid=9ymq63Dc-U7VUM:&ved=0CAUQjRw&url=http%3A%2F%2Fwww.gamepur.com%2Fnews%2F13599-zeus-vs-monsters-math-game-introduce-you-greek-mythological-gods.html&ei=wxk8U4eQBIXjsAT144H4BA&bvm=bv.63934634,d.dmQ&psig=AFQjCNGZm4kOwGHiaXbC9kdyIwNYaWJiWg&ust=1396534065556323

and instead of this causing an error response (since it's sort-of-but-not-really a redirect page), camo actually crashes:

crypto.js:209
  this._binding.update(data, encoding);
                ^
TypeError: Not a string or buffer
    at Hmac.Hash.update (crypto.js:209:17)
    at Server.<anonymous> (/Users/pomax/Documents/Git projects/webmaker-suite/mozillawebmakerproxy.net/server.js:270:14)
    at Server.EventEmitter.emit (events.js:98:17)
    at HTTPParser.parser.onIncoming (http.js:2108:12)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:121:23)
    at Socket.socket.ondata (http.js:1966:22)
    at TCP.onread (net.js:527:27)

Request crashes camo

I didn't catch the request causing it, but maybe the trace is enough.

Apr 17 06:36:23 aeshna.de node[266]: _http_outgoing.js:333
Apr 17 06:36:23 aeshna.de node[266]: throw new Error('"name" and "value" are required for setHeader().');
Apr 17 06:36:23 aeshna.de node[266]: ^
Apr 17 06:36:23 aeshna.de node[266]: Error: "name" and "value" are required for setHeader().
Apr 17 06:36:23 aeshna.de node[266]: at ClientRequest.OutgoingMessage.setHeader (_http_outgoing.js:333:11)
Apr 17 06:36:23 aeshna.de node[266]: at new ClientRequest (_http_client.js:101:14)
Apr 17 06:36:23 aeshna.de node[266]: at Object.exports.request (http.js:49:10)
Apr 17 06:36:23 aeshna.de node[266]: at Object.exports.get (http.js:53:21)
Apr 17 06:36:23 aeshna.de node[266]: at process_url (/usr/lib/camo/server.js:118:25)
Apr 17 06:36:23 aeshna.de node[266]: at Server.<anonymous> (/usr/lib/camo/server.js:301:18)
Apr 17 06:36:23 aeshna.de node[266]: at Server.emit (events.js:110:17)
Apr 17 06:36:23 aeshna.de node[266]: at HTTPParser.parserOnIncoming [as onIncoming] (_http_server.js:491:12)
Apr 17 06:36:23 aeshna.de node[266]: at HTTPParser.parserOnHeadersComplete (_http_common.js:111:23)
Apr 17 06:36:23 aeshna.de node[266]: at Socket.socketOnData (_http_server.js:343:22)

Running 2.2.0.

Preserve http_referer

Compare:

curl -vvv https://makeyourlaws.org/travis-image -e https://github.com/rubysec/bundler-auDRFit/fafa3

Server sees:
x.x.x.x - - [03/Mar/2014:06:25:52 +0000] makeyourlaws.org "GET /travis-image HTTP/1.1" 302 154 "https://github.com/foo/bar/tree/baz/qux" "curl/7.35.0" "-"

Response:
< HTTP/1.1 302 Moved Temporarily
< Location: https://travis-ci.org/foo/bar.png?branch=baz
< Cache-Control: private

With what happens if an image link is put on a Github readme page:

Server sees:
x.x.x.x - - [03/Mar/2014:05:59:26 +0000] makeyourlaws.org "GET /travis-image HTTP/1.1" 404 162 "-" "Camo Asset Proxy 2.0.1" "-"

Because there's no http_referer or even x-forwarded-for, there's no way for me to serve the appropriate Travis CI image.

Please fix by preserving the http referer properly (and respecting cache-control private).

Add support for image/webp mime type

Don't leak secret key to log

This seems like a fundamentally bad practice; attackers could use the log stream as a side-channel to obtain the key:

https://github.com/atmos/camo/blob/master/server.coffee#L218

Invalid upstream response (201)

This image doesn't show up because the server returns a 201:

![201 test](https://img-any-status.herokuapp.com/201)

201 is typical for POST or PUT requests, but I can't find anything that states it would be invalid as response to GET.

I think #77 should have fixed that and can't actually find the code for the error message in the sources, but I thought I'd report here since it seems to be related to camo.

Update changelog

CHANGELOG.md hasn't been updated since 1.1.3.

To make life easy for distributors and end-users, please document the major changes since then, especially any breaking changes between 1.x and 2.x.

.NET port

Not really an issue but wanted to let you know I have created a .NET port of camo.
https://github.com/maartenba/CamoDotNet

Rewriting content-types for URLs

@atmos, how do you feel about rewriting the content-type for URLs that return image data but don't return the correct content-type.

Example URL:
http://www.schwab.com/public/file/Japanese-inflation-adjusted-wages-continue-to-fall/Japanese_inflation_adjusted_wages_continue_to_fall
You'll notice ‰PNG � IHDR in the first few bytes of the HTTP response.

How do you feel about a dependency on the mmmagic library?
https://github.com/mscdex/mmmagic

If camo were to support detection of content-types, I think this feature should be off by default.

Non-Image content-type returned for animated gif

I'm getting the message "Non-Image content-type returned" with this animated gif: http://recordit.co/iEMpbBUfQ0

I added some markdown. In the following line, it renders incorrectly, exactly as it renders on my github wiki page:

to this github wiki page:
https://github.com/eostermueller/performanceGolf/wiki/Install-and-Run

...which generated this url:
https://camo.githubusercontent.com/31065e879f38322ec91c2fb8f6a8479362bfd161/687474703a2f2f7265636f726469742e636f2f69454d70624255665130

Any thoughts on how to make this work?

Thank you,
--Erik

look petros

image not found or error

I have installed camo on nginx + node (v5.4.0) with a custom domain (https://camo.domain.com/).
It seems to work except when i try to load theses images:

https://camo.domain.com/my_key?url=http://www.howtoforge.com/images/linux_screen/2.png
I get the message: Not found

https://camo.domain.com/my_key/http://www.howtoforge.com/images/linux_screen/2.png
I get: this image cannot be display because it contains errors

I have check with several image.
I try to run it on heroku with the same result.

Is there something to do with the config parameters?

possible issue with chunked responses

I ran into an issue during testing, where an upstream server does not provide a content-length, and instead sends a chunked and gzip encoded response. (favicon.ico is an image/x-icon type, and can be gzip compressed).

This causes the browser to hang, and not close the connection (it doesn't think the connection is done -- expects more content).

I have a quasi-fix in a fork branch, but it is tied up with a great deal of unrelated code. If this potential issue is a concern (as it matches the content-type 'image' string test) then let me know and I can try breaking it out.

What I basically did was something to this effect:

# special case content-length. might not be sent by upstream server
# if gzip encoded / chunked response
newHeaders['content-length'] = content_length if content_length?    # this prevents "content-length: undefined" being sent to the client
newHeaders['transfer-encoding'] = srcResp.headers['transfer-encoding'] if srcResp.headers['transfer-encoding']?
newHeaders['content-encoding'] = srcResp.headers['content-encoding'] if srcResp.headers['content-encoding']?

This seems to provide the browser with enough information to close the connection appropriately, and receive and display the content (favicon in this case). This could also be an issue if image/svg+xml images are served gzip/deflate compressed.

Camo not compliant with RFC2046 on case insensitivity

I saw this error in our logs:
Non-Image content-type returned 'image/JPEG'

The code doesn't appear to take case insensitivity of MIME Types into account.

camo/server.coffee

Line 165 in 7af31a0

contentTypePrefix = contentType.split(";")[0]

Nor does the mime-types.json, but I don't recommend adding all permutations of case there.

http://tools.ietf.org/html/rfc2046#section-4.2
4.2. Image Media Type

A media type of "image" indicates that the body contains an image.
The subtype names the specific image format. These names are not
case sensitive. An initial subtype is "jpeg" for the JPEG format
using JFIF encoding [JPEG]

Problem on some images

I'm having few problems on some images. For example this:

https://encrypted-tbn1.google.com/images?q=tbn:ANd9GcR_NbJISu0XkUbARqEYG3k03c4pNsxVWT9t3-U2mitjh3fO4Vyxvg

I use the http://example.org/?url= version:

https://example.com/57725bf7f3846c59ae4bb522278aa32cb2b241aa?url=https%3A%2F%2Fencrypted-tbn1.google.com%2Fimages%3Fq%3Dtbn%3AANd9GcR_NbJISu0XkUbARqEYG3k03c4pNsxVWT9t3-U2mitjh3fO4Vyxvg

But the response is not found... with any other image it works correctly, it's just with these hosted on google and with that url syntax that aren't loaded

I've checked with wget and it returns a 200 response, so it's not using "strange" redirects, and also the content type is image/jpeg

Is it because of the "x-content-type-options: nosniff" they set in the response? Is there a way to bypass it?

why is there a hexdec function in the code?

I noticed the code contains this snippet:

  hexdec = function(str) {
    var buf, i, _i, _ref;
    if (str && str.length > 0 && str.length % 2 === 0 && !str.match(/[^0-9a-f]/)) {
      buf = new Buffer(str.length / 2);
      for (i = _i = 0, _ref = str.length; _i < _ref; i = _i += 2) {
        buf[i / 2] = parseInt(str.slice(i, +(i + 1) + 1 || 9e9), 16);
      }
      return buf.toString();
    }
  };

but what does this actually do? Because it's not a straight up hex to dec function: it turns a string into a string, and it seems to do way more than a simple return parseInt(str, 16).toString(10).

It should be possible to use a unix socket as the port

Hi,

Camo is super and I love it. I am needing to listen on a socket for my deployment at EngineYard, but the parseInt on line 7 of server.coffee prevents that. Removing the parseInt allows me to pass a path to a socket to listen on instead of a port number.

I'm happy to create a patch for this and send a pull request, but I'm not sure if you have a different preferred method to fix it instead.

thanks!

Zupimages does not work

Hi!
The image host http://www.zupimages.net/ does not work.

I think ALL travis-ci badges are broken on github!

when I add a badge like this one to readme.md files like this
[![Build Status](https://travis-ci.org/Tickaroo/tikxml.svg?branch=master)](https://travis-ci.org/Tickaroo/tikxml)

results this url:
https://camo.githubusercontent.com/71f2cc1438ed37e52752b5a1724267b89fbb18ef/68747470733a2f2f7472617669732d63692e6f72672f5469636b61726f6f2f74696b786d6c2e7376673f6272616e63683d6d6173746572

which responds with "Error Fetching Resource" or "Cannot proxy the given URL"

I looked into different repos and they look the same
hope this can be fixed soon before people start fixing their files looking for solutions!

Don't depend on external resources in test

Currently, the test suite attempts to contact a number of sites during testing. Ideally, all these artifacts would be stored locally and DNS resolution mocked out to a local web server also started during test.

Forward Referrer header or other repo info

It would be cool if the referrer header for the original request was sent along. This would be useful for github repo badge images because they could detect the source of the request (i.e. the repo and branch) from this header. Alternatively, maybe just sending that info along in a custom header would be easier.

The idea is that a badge could be put on a readme with a simple url http://badge.com/badge.svg and from the headers figure out which repo requested the badge. This would be great for when people fork repos or otherwise make branches and you want the badge to refer to the correct fork or branch.

Is there a way to detect 404?

Hi guys, and thanks for this great project.
I was wondering if there are some event I can listen to update my backend when a saved URL is not valid anymore.

Any idea for that?

Thanks.

Camofying in the client side

@atmos we were talking about that at GitHub processes the Markdown and camo-fies using html-pipeline. I have a case where I want to do the camo-fying in the client-side instead. First thing that came to mind was just publicly using a shared key that would be visible if somebody checked the javascript source code. Second was adding an option to check for the referral request header for allowed domains. This is definitely not "secure" since those can be spoofed easily. What are your thoughts on this?

Images from flickr are not always displayed

Hi,

I have weird issue with the integration of GIF hosted on Flickr.

On the readme, this gif works well:
https://github.com/matthieu-lapeyre/poppy-ergo-jr/blob/master/README.md

But on the assembly guide the others gif are not displayed:
https://github.com/matthieu-lapeyre/poppy-ergo-jr/blob/master/doc/assembly.md
with plenty of 404 error ...

here I have the same result:

Maybe it's a limit on the image size ?

Add support for image/x-icon mime type

Currently, Camo supports image/vnd.microsoft.icon, but doesn't support image/x-icon which seems to be what all the cool kids are using.

Can I override the proxy, to serve the original image URL?

Is it possible to get the proxy service to leave an image URL alone - either via markdown or response header - so I can serve the image URL directly to the client, unmolested?

I'm trying to show a CI build status badge in my README.md but it's not loading the image. I'm guessing this is because the CI server that is hosting the build status badge image requires authentication, and the camo service isn't able to forward the auth/cookies of the client.

How can I get around this?

Help appreciated.

Allow Customization of Security Headers Set

The massive HSTS age of 365 caught me slightly by surprise.
I've quickly hardcoded it to 0 in case I ever want to repurpose the domain without https, but a general configuration for these sort of things would be nice.

Not following redirect?

Document: https://github.com/GoogleCloudPlatform/kubernetes/blob/3eca6871668a7b4eb6d2a1bc13875a175bc688dd/docs/README.md

The raw markdown is: ![WARNING](http://releases.k8s.io/HEAD/docs/warning.png)

Camo is rendering: https://camo.githubusercontent.com/19023af99ca9a2e65aea15804e54073a29f7ce53/687474703a2f2f72656c65617365732e6b38732e696f2f484541442f646f63732f7761726e696e672e706e67

That camo link is "not found".

$ curl -i http://releases.k8s.io/HEAD/docs/warning.png
HTTP/1.1 301 Moved Permanently
Server: nginx/1.2.1
Date: Wed, 15 Jul 2015 00:04:22 GMT
Content-Type: text/html
Content-Length: 184
Connection: keep-alive
Location: https://github.com/GoogleCloudPlatform/kubernetes/tree/HEAD/docs/warning.png

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.2.1</center>
</body>
</html>

I thought camo would follow redirects?

Versions don't match between package.json and server.coffee

The versions specified in package.json (1.0.2) and server.coffee (0.5.0) do not match.

Error Fetching Resource

https://404forest.com/imgs/blog/2016-3-19-1.png

Hi, when I use images hosted on my server I always can't get it show up and it says: 'Error Fetching Resource'. I read the About anonymized image URLs but it didn't help.

The image's response headers include 'Content-Type: image/png', and removing the image from Camo's cache didn't change the result.

Any thoughts on how to make this work? Thank you.

change log is out of date

Based on commit comments, we now have a release 1.1.1. however, the changelog only goes to 1.0.2.

Adding Travis CI

Would you be interested in having Travis CI testing hooks added to camo? I'd be happy to do this if you're game!

"Socket Timeout" Error after few hours

I'm sure this a not a bug from camo but few hours after starting the camo server and lots of urls proxied through, I start getting "socket timeout" and the images are not proxied anymore.

My guess is the system ran out of avalaible socket which are not "released" or "freed", did anyone had this kind of behaviour and managed to fix it ? Should we raise the file descriptor limit or any kind of linux limit ?

Thanks

Deployment on CentOS 6.6

I want to deploy a small instance of Camo on a phpbb 3.1.5 board (which I will soon update to 3.1.6) that is on a CentOS 6.6 dedi, and I am asking if I installed Camo correctly.
I don't think that it has been installed correctly.

Whenever I start Camo I just get this message
SSL-Proxy running on 8081 with pid:31673 version:2.3.0.

Whenever I connect to http://example.com:8081 HSTS automatically redirects me to https://example.com:8081 and Google Chrome gets an ERR_CONNECTION_CLOSED.
Is this normal behaviour?
I also assume that I need a certificate for this, I can get this easily, but where do I need to install the certificate?

camo_host_exclusions only supports one wildcard (first encountered)

Due to the way that the exclusions environment var is parsed before becoming a RegExp object, only the first wildcard (*) is magically expanded.

This fails:

CAMO_HOST_EXCLUSIONS='(*.example.com|*.example.org)'

There are certainly other ways to write this particular regex that would not fail, but this is an example of one that does.
Perhaps simply passing in a valid regex to start with might be more straightforward?

CAMO_HOST_EXCLUSIONS='(.*\.example\.com|.*\.example\.org)'     

Heroku Deployment Error

During deployment on Heroku, it fails with following:

Error: bash: bundle: command not found.

is there a reason the engine is pinned to node 0.4.x?

0.10.x is out now, so that's 3 versions behind.

original filename not visible in save as-window.

A couple of months a go, a Dutch forum I regularly visit started using Camo to be able to offer all pictures through https.

Unfortunately, because of this, every picture loaded through Camo cannot be saved using the original filename. The save as-window always shows ic.tweakimg.net.jpg.

Is there an option that can be enabled to show the original filename in the save as-window?

Lots of "client request interrupted" recently from Camo

I'm running badge-matrix on Heroku and lately I've been getting a lot of H27 (client request interrupted) from Camo. Is the only reason Camo would close the connection due to CAMO_SOCKET_TIMEOUT? I don't think these requests are taking that long, so I want to make sure it isn't some other issue. (Does GitHub's deployment of Camo use the default timeout?)

You can also see the octopus GIF in Camo's own README is currently broken, but the real URL works fine in a browser. I don't know if the octopus is sad for the same reason my badges are failing, but it seems fishy to me (ba-dum tsss):

For me this is most noticeable on my script-atomic-onload repo, which tests tons of JS script loaders and thus has dozens of badges – often the first 5 or so badge images will be broken due to Camo closing the connection early. This never seemed to happen months ago when I first released this repo, but it's very frequent now.

Reason for 404 with Non-Image content-type returned

I was curious about why camo returns a 404 when a non-image content type is returned.

If servers are misconfigured like this AOL CDN no content type is set so the image appears broken:

http://hss-prod.hss.aol.com/hss/storage/adam/5e9eafe652728052209ba272aea74123/fingerprints-1.jpg

I'm considering turning off this check on my camo, but was wondering if there's a technical or security reason for it?

Camo proxies files when Content-Type is not set in the received header

Camo has a check for mimetypes to prevent it from proxying files that are not images; however, if the server does not return a content type, Camo will skip the check.

As of Firefox 26.0, the lack of a content type causes Firefox to interpret the result as text/html.

Here's a demo (I hosted homakov's Cookiebomb script with a node.js server that does not pass content-type)

Edit:
Firefox and Internet Explorer interprets files served without content-type as HTML, but not Chrome, according to https://twitter.com/homakov/status/429440194333270016 , so this won't work on Chrome.