BW Testing - Image Compression in Proxy Servers about boomerang

[andreas-marschke]

This problem has been addressed recently in this PR: #47
Which was created in reference to: #46

In theory this should fix the problem of compression. However, if this is still an issue in your environment and servers are still hard set to compress your images you may be able to improve your chances to get around it with cache headers and E-Tags which prevent the browser from using a version in their AppCache.

Personally I'd keep static content, as the images are, seperate from your boomerang code and have them configured to use specific headers using the webserver. NodeJSs serve-static does something of that sort providing extra tagging/header adding features prior to sending out content. Being that serve-static and NodeJS isn't necessairly everyones cup of tea and NodeJSs performance for sending static files is less than optimal, I'd suggest doing static serving through something like nginx.
See: http://blog.argteam.com/coding/hardening-node-js-for-production-part-2-using-nginx-to-avoid-node-js-load/

In my testing/development environment I've setup the development server to serve static content inside the NodeJS process. But to avoid the browser barging in on this I set e-tags and
cache-control: private, max-age=0, no-cache, no-store, no-transform
so that it has to fetch the images new everytime I do bandwidth measurements.

The code for the serve-static middleware is here: https://github.com/andreas-marschke/boomerang-express/blob/master/lib/middlewares/index.js

In cases where it is purely cached you may want to use cache-busting. See: http://www.adopsinsider.com/ad-ops-basics/what-is-a-cache-buster-and-how-does-it-work/

Most cases should be fine with an increment or random number at the end as a parameter ie:
http://static.example.org/images/boomerang/bandwidth/image-0.png?n=12345229881921

[vigneshshanmugam]

@andreas-marschke - I don't think the solution on #47 is a rock solid one. It just compressed the image further.

But think about the CDN's and the proxy servers out there which optimises the images heavily based on lot of statistics (Ex -Viewport based either a mobile/desktop, Network based ).

As you said, We can do cache bursting using query params but we cant do that only for specific images since CDN's can be used to cache other images which are required in the page. IMO, This is not a good idea.

@bluesmoon @nicjansma : Just curious to know whether these things will affect the Bandwidth Testing methodologies of boomerang in any way?. If Yes, How can we fix this as a whole?.

[bluesmoon]

@vigneshshanmugam the solution in #47 is the best we can do on the boomerang project side since we do not control web servers. The images are not compressed, instead they have been made incompressible by filling them with random encrypted data. Encrypting data results in increased entropy which makes compression impossible.

This of course only helps if the server does not do other strange things like you've mentioned.

I'm not sure if the server will be able to change image dimensions based on the viewport since boomerang does not send any viewport information back to the server. If you make sure these images are served from a different domain, then you'll also prevent any viewport specifying cookies from being sent across.

The server might still do user agent based transcoding, so for example, change the image to webp for Chrome, but again with the data in the image, it's unlikely that this will result in any change in the image's byte size.

Lastly, since you are using mPulse and not raw boomerang, you should probably send a message to the soasta support team since our recommendations on the boomerang project may not be the same as those for mPulse :)

[vigneshshanmugam]

@bluesmoon Thanks for the detailed Info.

Yup we are using mPulse and I am not doing any such optimisation in the CDN for the images used for BW testing :).

But I wanted to know whether we can solve this problem in general. AFAIK, Most of the CDN servers do image compression in their end to optimise the load time for better experience. Akamai(Uses Adaptive Image Compression) and CloudFare (https://blog.cloudflare.com/introducing-polish-automatic-image-optimizati/) have both lossy and lossless compression which will result in reduced image size.

Reduced image sizes ultimately affects the BW testing. If i understand it correct, this will end up affecting the bandwidth calculation.

[bluesmoon]

I don't think either Akamai or CloudFlare will be able to compress these images. Let me know if you see evidence that they are able to compress them.

[vigneshshanmugam]

Alrighty :) . Will check and get back. Thanks!

[vigneshshanmugam]

Closing it for now. Will reopen if there are new findings.