jwagner / smartcrop.js

Content aware image cropping

smartcrop.js

Smartcrop.js implements an algorithm to find good crops for images. It can be used in the browser, in node or via a CLI.

Image: https://www.flickr.com/photos/endogamia/5682480447/ by Leon F. Cabeiro (N. Feans), licensed under CC-BY-2.0

Note I'm currently working on a more advanced version of smartcrop.js based on machine learning. As part of that I'm looking for a large dataset of manually cropped images. If you know of such a dataset, please let me know.

Demos

• Smartcrop.js Test Suite, contains over 100 images, heavy.
• Smartcrop.js Test Bed, allows you to test smartcrop with your own images and different face detection libraries.
• Automatic Photo transitions, automatically creates Ken Burns transitions for a slide show.

Simple Example

// you pass in an image as well as the width & height of the crop you
// want to optimize.
smartcrop.crop(image, { width: 100, height: 100 }).then(function(result) {
  console.log(result);
});

Output:

// smartcrop will output you its best guess for a crop
// you can now use this data to crop the image.
{topCrop: {x: 300, y: 200, height: 200, width: 200}}

Download/ Installation

npm install smartcrop or just download smartcrop.js from the git repository.

Smarcrop requires support for Promises, use a polyfill for unsupported browsers or set smartcrop.Promise to your favorite promise implementation (I recommend bluebird).

Consider avoiding crops using dont-crop

If you are interested in using smartcrop.js to crop your images you should also consider to avoid cropping them by using dont-crop. Dont-crop gives you matching gradients and colors to pad and complement your images.

Command Line Interface

The smartcrop-cli offers command line interface to smartcrop.js.

Node

You can use smartcrop from nodejs via either smartcrop-gm (which is using image magick via gm) or smartcrop-sharp (which is using libvips via sharp). The smartcrop-cli can be used as an example of using smartcrop from node.

Stability

While smartcrop.js is a small personal project it is currently being used on high traffic production sites. It has a basic set of automated tests and a test coverage of close to 100%. The tests run in all modern browsers thanks to saucelabs. If in any doubt the code is short enough to perform a quick review yourself.

Algorithm Overview

Smartcrop.js works using fairly dumb image processing. In short:

1. Find edges using laplace
2. Find regions with a color like skin
3. Find regions high in saturation
4. Boost regions as specified by options (for example detected faces)
5. Generate a set of candidate crops using a sliding window
6. Rank them using an importance function to focus the detail in the center and avoid it in the edges.
7. Output the candidate crop with the highest rank

Face detection

The smartcrop algorithm itself is designed to be simple, relatively fast, small and generic.

In many cases it does make sense to add face detection to it to ensure faces get the priority they deserve.

There are multiple javascript libraries which can be easily integrated into smartcrop.js.

• ccv js / jquery.facedetection
• tracking.js
• opencv.js
• node-opencv

You can experiment with all of these in the smartcrop.js testbed

On the client side I would recommend using tracking.js because it's small and simple. Opencv.js is compiled from c++ and very heavy (~7.6MB of javascript + 900kb of data). jquery.facedetection has dependency on jquery and from my limited experience seems to perform worse than the others.

On the server side node-opencv can be quicker but comes with some annoying issues as well.

It's also worth noting that all of these libraries are based on the now dated viola-jones object detection framework. It would be interesting to see how more state of the art techniques could be implemented in browser friendly javascript.

Supported Module Formats

• CommonJS
• AMD
• global export / window

Supported Browsers

See caniuse.com/canvas. A polyfill for Promises is recommended if you need to support old browsers.

API

smartcrop.crop(image, options)

Find the best crop for image using options.

image: anything ctx.drawImage() accepts, usually HTMLImageElement, HTMLCanvasElement or HTMLVideoElement.

Keep in mind that origin policies apply to the image source. You may not use cross-domain images without CORS clearance.

options: cropOptions

returns: A promise for a cropResult.

cropOptions

minScale: minimal scale of the crop rect, set to 1.0 to prevent smaller than necessary crops (lowers the risk of chopping things off).

width: width of the crop you want to use.

height: height of the crop you want to use.

boost: optional array of regions whose 'interestingness' you want to boost (for example faces). See boost;

ruleOfThirds: optional boolean if set to false it will turn off the rule of thirds composition weight.

debug (internal): if true, cropResults will contain a debugCanvas and the complete results array.

There are many more (for now undocumented) options available. Check the source and be advised that they might change in the future.

cropResult

Result of the promise returned by smartcrop.crop.

{
  topCrop: crop;
}

crop

An individual crop.

{
  x: 11, // pixels from the left side
  y: 20, // pixels from the top
  width: 1, // pixels
  height: 1 // pixels
}

boost

Describes a region to boost. A usage example of this is to take into account faces in the image. See smartcrop-cli for an example on how to integrate face detection.

{
  x: 11, // pixels from the left side
  y: 20, // pixels from the top
  width: 32, // pixels
  height: 32, // pixels
  weight: 1 // in the range [0, 1]
}

Note that the impact the boost has is proportional to it's weight and area.

Tests

You can run the tests using grunt test. Alternatively you can also just run grunt (the default task) and open http://localhost:8000/test/.

Benchmark

There are benchmarks for both the browser (test/benchmark.html) and node (node test/benchmark-node.js [requires node-canvas]) both powered by benchmark.js.

If you just want some rough numbers: It takes < 20 ms to find a square crop of a 640x427px picture on an i7. In other words, it's fine to run it on one image, it's suboptimal to run it on an entire gallery on page load.

Contributors

• Christian Muehlhaeuser

Ports, Alternatives

• connect-thumbs Middleware for connect.js that supports smartcrop.js by Irakli Nadareishvili
• smartcrop-java by QuadFlask
• smartcrop-android by QuadFlask
• smartcrop.go by Christian Muehlhaeuser
• smartcrop.py by Hideo Hattori
• smartcrop-rails smartcrop wrapped in a ruby gem by Mohammed Sadiq
• smartcrop.net c# .net port by softaware gmbh
• dont-crop a library to avoid cropping by padding images with matching colors or gradients

Version history

2.0.5

Fix TS1046: Top-level declarations in .d.ts files must start with either a 'declare' or 'export' modifier..

2.0.4

Typescript type definitions.

2.0.2

In short: It's a lot faster when calculating bigger crops. The quality of the crops should be comparable but the results are going to be different so this will be a major release.

1.1.1

Removed useless files from npm package.

1.1

Creating github releases. Added options.input which is getting passed along to iop.open.

1.0

Refactoring/cleanup to make it easier to use with node.js (dropping the node-canvas dependency) and enable support for boosts which can be used to do face detection. This is a 1.0 in the semantic meaning (denoting backwards incompatible API changes). It does not denote a finished product.

License

Copyright (c) 2018 Jonas Wagner, licensed under the MIT License (enclosed)

smartcrop.js's Issues

Can add size option in cropOptions?

sometimes we maybe care the file size more but not the width and height, we just want the suitable size to upload or something else, no matter what its width and height are

add support for page detection

It would be great if smartcrop could also auto crop pages that are stored within images.

Add configuration options to disable certain features

Add configuration options (or document ways) to disable saturation and skin detection, among other features that take up time and may not be desirable.

I've noticed that smartcrop can be very time intensive in the client browser if there are many images that need to be processed. It can block the browser when the image gets very large. It would be nice to document ways to reduce the intelligence, and time taken in smartcrop in certain use cases.

The best approach is obviously to create a NodeJS image processor, but some websites don't have that luxury and we could help by not blocking the browser as much.

Release

Thanks for this great job.
I am not a big fan of using * or commit reference into my bower.json.
Could you please create a release?

Error using smartcrop with FileReader()

When I upload an image, I load this uploaded image to a .
But I'm getting this error bellow:

Uncaught TypeError: Failed to execute 'drawImage' on 'CanvasRenderingContext2D': The provided value is not of type '(CSSImageValue or HTMLImageElement or SVGImageElement or HTMLVideoElement or HTMLCanvasElement or ImageBitmap or OffscreenCanvas)'.

Bellow is my code:

$("#avatar").change(function(event) {
var reader = new FileReader();
$(reader).load(function (event) {
image = $("#img-avatar");

        image.attr("src", event.target.result);

        smartcrop.crop(image, {width: 300, height: 300}).then(function(result){
            console.log(result);
        });
    });
    reader.readAsDataURL(event.target.files[0]);
});

Add canvas peer dependency

Could you add canvas as a peer dependency (as described here: https://docs.npmjs.com/files/package.json#peerdependencies ). If you do so, then it will work on tonic (https://tonicdev.com/npm/smartcrop-node) automatically.

Thanks!

Francisco

Multiple Skin Colors

Right now it seems that the skin color defaults to #C69170 and can only take one value. It may be useful to have multiple default values (since not everyone is white) and check all of them in skinDetect.

Trying in test bed

Hi @jwagner ,

Good work. I was trying on the test bed an image and it seems the cat is not detected.

Attaching image for reference. What I was trying was reducing the width and height also, the cat's face was not detected.

1 image transition example

Dear Jonas,

I am very impressed by your code, thanks a lot for that.
Unfortunately I have no knowledge on javascript and I am a newbie in webpage creation. Consequently, I am totally lost with your slideshow example.

Would it be possible to add a simple example (a single image with transition) on your repository?

Regards,
Bruno

Cross origin

A great plugin tho, well done!

I just have a question please, does it work with cross origin images request ?

For example, lets say I load my images from Amazon service, would form any issue ?

Return just focal point

Is it possible just to return a focal point (rather than getting crop info) upon which a library like https://github.com/jonom/jquery-focuspoint can then reposition an image around when used in a re-sizing window?

React

Can somebody help me implement this in react?

how is it possible to record the picture

Hello

Thank you for your great component
how is it possible to record the picture cropped to a folder

Meric in advance for your help

Add a Typescript definitions file

For using this with a Typescript app, it'd be awesome to have a Typescript definitions file (smartcrop.d.ts) for type checking.

Symmetry

Thank you for a useful library!

A possible enhancement: incorporate compositional symmetry into the crop function? The following example from your test page would probably benefit:

I think this would improve the crops for architectural images in particular. If the source image scored highly for symmetry, perhaps optimize crops for symmetry too?

I haven't looked at the source for your project though, nor have I given the implementation much thought, though. Perhaps it's more effort than it's worth.

Thank you for creating this library

I have created a rails gem called smartcrop-rails, a wrapper of smartcrop.js for rails application.

Here are the links:
Github: https://github.com/sadiqmmm/smartcrop-rails
Rubygems: https://rubygems.org/gems/smartcrop-rails

Can you include in your readme / wiki / Docs / any.

Canvas security issue in firefox

When I use firefox, I got the following error:

SecurityError: The operation is insecure.

var input = ctx.getImageData(0, 0, canvas.width, canvas.height),

at smartcrop.js（283)

If I want to do image by image?

Good
If I want to do image by image?

For example:

 <?php do { ?>
   <div class="card bg-dark text-white"> <img class="card-img" src="ad@/fotoslugares/<?php echo $row_fotoslugares['img_name']; ?>">

     <div class="card-img-overlay">
       <h5 class="card-title">Photo</h5>
       <p class="card-text">Photo text</p>
       <p class="card-text">Last updated 3 mins ago</p>
     </div>
   </div>
   <?php } while ($row_fotoslugares = mysql_fetch_assoc($fotoslugares)); ?>

PHP Port

Great Library! Does anyone know already a PHP port or has a desire to develop? GD or Imagick.

License

Please add a license, to let me know if I can use it. Thanks, this project looks promising!

IndexSizeError: Index or size is negative or greater than the allowed amount var input = ctx.getImageData(0, 0, canvas.width, canvas.height),

this in firefox chrome not cause this error, the cropping not working but if i chooesed the imagee twice no problem happen

When decreasing minScale, generating crop size takes astronomically longer

I am generating a square crop (500x500) for a large image (about 3000x3000px). When minScale is at 1 this process is done in a second or two (max) on my machine - when decreasing minScale to 0.7 it takes a good few seconds longer and when decreasing minScale to 0.5 it takes almost a minute (similar results when running on Heroku)!

Is there something I am doing wrong or some way around this?

What format does image have to be in?

The example in the README doesn't show how the image variable gets created. And the documentation says

image: anything ctx.drawImage() accepts, usually HTMLImageElement, HTMLCanvasElement or HTMLVideoElement

Can it be changed to accept a string which is a path to an image file (for NodeJS)?

smartcrop.js as Common.js Module

Hi, is this library useable as common.js module,

as i tried to use it within appcelerator as common.js module but i got error "can't find document".
As document is only available within browser how we can use this library as common.js module?

[Feature] Smart Rotate?

By any chance are there plans to add smart rotate? lol

I've noticed that some users are not rotating their images when uploading them.

Blocked by SecurityError

SecurityError: The operation is insecure.
var input = ctx.getImageData(0, 0, canvas.width, canvas.height),

Content Security Policy

The crop fails when I add

meta http-equiv="Content-Security-Policy" content="default-src 'self'"

for security perposes regarding xss how can I be able to fix this???

I think the cause of this is because your code creates script or img tags somewhere but I dont know where, can you help me with that please

NPM tarball contains non-essential files

While the main smartcrop.js file itself is only 15.7 KB, a local install of smartcrop consumes 31MB of disk space, as can be confirmed by running.

mkdir test
cd test
npm install smartcrop --prod
du -cksh

outputs

31M     total

This is due to several inclusions which probably don't need to be shipped to npm, including:

directory	size
/dist	25 MB
/release	3.9 MB
/.git_	1.9 MB

Any chance you could pull some of these out?

here is a good example of SmartCrop.js: www.wallpaperpcmobile.com

www.wallpaperpcmobile.com uses smartcrop.js to itelligently crop wallpaper, a very good demo site of smartcrop.js
orginal wallpaper:http://www.wallpaperpcmobile.com/wallpaper/bird-on-a-branch-163656
cropped to 480*800 resolution:
http://www.wallpaperpcmobile.com/wallpaper/bird-on-a-branch-163656/download-480x800

document is not defined

Hi !
This is first time I used smartcrop.js. Here is my code:

smartcrop = require('smartcrop'),
smartcrop.crop(dirPath + file_name, {width: 96, height: 96}).then(function(result){
   console.log('\nCrop:')
   console.log(result);
});

But when I run, It throw an error:

Clarify in documentation how input width/height and output width/height correlate

Hello,

From the README:

smartcrop.crop(image, {width: 100, height: 100}).then(function(result){
  console.log(result);
});

Output:

{topCrop: {x: 300, y: 200, height: 200, width: 200}}

In terms of documentation and the above example, I think it would be nice to have a little explanation as to why the output's width/height values (200x200) are larger than the desired input's width/hieght (100x100 values).

Here's an example of desired input widths and heights and smartcrop's output widths and heights:

User-specified input WxH	Smartcrop output WxH	Output scaled to input width
2622x2660	1528x1550	2622x2660
2622x2660	1986x2014	2622x2659
2622x2660	2022x2051	2622x2660
2622x2660	2525x2561	2622x2659
2622x2660	3026x3072	2622x2662
2622x2660	3405x3454	2622x2660

I guess it just took me some exploration time to discover the correlation between the input and output dimensions. Clearly, and correct me if I am wrong, smartcrop is saying:

Based on your input image size, and desired crop area, the resulting relative crop area is (width) and (height) at coordinates (x) and (y). You must crop your image using smartcrop's output width, height, x, y, and then scale up or down to your desired width and height (using tool of choice, like ImageMagick or GraphicsMagick).

With that said, one issue I am running into is that the output numbers are not exact (as you can see above); maybe it should also be mentioned in the docs that if one wants an exact crop, they will have to account for rounding errors and use something like ImageMagick's ^:

....
-crop 3026x3072+100x200 \
-resize 2622x2660^ \
....

Combine with focuspoint

Would be nice if this library, instead of cropping, would simply set the cetral point for use in https://github.com/jonom/jquery-focuspoint

Is there any solution for .net framework?

Hi,

Thank you for your great component, but i need to do it in server side, and server side technology is asp.net MVC, i wonder is there any solution for it ?

Experiment with a hog/svm based face / object detector.

The opencv face detector creates quite a bit of trouble. It might be possible to create or port a better solution based on histograms of oriented gradients and support vector machines.

Disable Rotation on Some Images

Hi,

For some images smartcrop rotates the image to fit it within the size constraints. That sometimes works but, especially with people, this makes the image look worse. Is there a way to disable that?

Test suite images

Just wanted to provide you with some images that are not so well working with your algorithm.

https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Giraffa_camelopardalis_angolensis.jpg/675px-Giraffa_camelopardalis_angolensis.jpg

https://upload.wikimedia.org/wikipedia/commons/thumb/f/f4/Berlin_-_Siegess%C3%A4ule_Gro%C3%9Fer_Tiergarten_2012.jpg/600px-Berlin_-_Siegess%C3%A4ule_Gro%C3%9Fer_Tiergarten_2012.jpg

https://upload.wikimedia.org/wikipedia/commons/thumb/1/15/Cheetah_Feb09_02.jpg/900px-Cheetah_Feb09_02.jpg

https://upload.wikimedia.org/wikipedia/commons/thumb/a/ae/Ursus_arctos_-_Norway.jpg/600px-Ursus_arctos_-_Norway.jpg

Inconsistency while cropping images

I get different results while cropping the same image at differents

The original file :

Resulting files

25x25 :
32x32 :
35x35 :
40x40 :

I have other images with the same behavior, but not necessarely at the same size ( One crops bad at 24, but is fine at 23 and under as well as 25 and above).

This is from Nodejs (v.6.11.1) using the npm smartcrop-sharp (v.1.0.5)

IndexSizeError: Index or size is negative or greater than the allowed amount smartcrop.js:282:0

That happend in Firefox and Safari, in Chrome is working properly for me

Making it smarter

Whereto from here? Are there any efforts to make it smarter and get more accurate? It's already pretty good so far for 80% of the cases, but I would like it to reach a higher level of accuracy.

Mobile camera capture issue

Taking a photo on Samsun s6 portrait mode ( worldstar mode )
Upload it to smartcrop > testbed

The upload shows up sideways, For some reason.

Cropping issues with real-world items

Hi,

My app is related to garments and when trying with the test bed, the cropping was not proper. The cropping happens in different places.

Is it possible to crop only the surrounding empty areas around the main object.

saturationDetect doesn't check saturationThreshold

Looks like a copy/paste error; one of these should be acceptableSaturation:

  if (acceptableLightness && acceptableLightness) {

SmartCrop fails silently in recursive processing. Node + node-canvas.

Hi. I'm trying to implement a batch process to crop large series of images with smartcrop in windows nodejs 0.10.
In the test I'm using 1000x1500px png images (I couldn't use .jpg files because libjpeg-turbo is not working in my W7 machine : ( )

here is the code:

var fs = require('fs'),
    path = require('path'),,
    Canvas = require('canvas'),
    Imagen = new Canvas.Image(),
    SmartCrop = require('smartcrop')
    Crop = 800,
    processFile = function(File, Index) {

        if(FilesList.length > 0) {

            var BaseName = FilesList.shift(),
                SourceFile = FilesDirSource + '\\' + BaseName,
                DestFile = FilesDirDest + '\\' + BaseName;

            Imagen.onerror = function(Error) { console.log(Error); }    
            Imagen.onload = function() {

                SmartCrop.crop(
                    Imagen, 
                    { 
                        width: Crop,
                        height: Crop,
                        canvasFactory: function(w, h) { new Canvas(w, h); } 
                    },
                    function(result){                   

                        var CropData = result.topCrop;
                        console.log(CropData); // print correct data until stop silently

                        // Process & save each file ...
                        // not used in te test to isolate problem

                        processFile();  // recurse next file
                    }
                );
            }   

            fs.readFile(SourceFile, function(Error, Data) {

                if(Error) { console.log(Error); } 
                else { Imagen.src = Data; }
            });

        } else { console.log('Complete'); }
    },
    FilesDirSource = path.join(__dirname, 'path_to_source_files_dir'),
    FilesDirDest = path.join(__dirname, 'path_to_dest_files_dir'),
    FilesList;  

fs.readdir(FilesDirSource, function(Error, Files) {

    if(Error) { console.log(Error); }
    else {

        FilesList = Files;          
        processFile();
    }
});

Seems to SmartCrop create three Canvas instances in each processed image but dont free resources so, after exactly 24 files, recurse process stops silently without throw any error.

Is there a way to clean Canvas resources in node-canvas? or, is it a bug in nodejs, or...

Callback should take error as its first argument

In the signature SmartCrop.crop(image, options, callback), the callback's signature is function(result). But how do errors get discovered and handled. Shouldn't the signature be in the usual JS style like function(error, result)?

Not only would that allow to handle errors but it would also allow smartcrop to be used with any of the promisify libraries (such as pify, thenify, etc) so that it can be used with Babel's ES7 async/await for a non-callback coding style.

Java and Android port

I ported to Java and Android

Getting more done in GitHub with ZenHub

Hola! @sabinmarcu has created a ZenHub account for the jwagner organization. ZenHub is the leading team collaboration and project management solution built for GitHub.

---

How do I use ZenHub?

To get set up with ZenHub, all you have to do is download the browser extension and log in with your GitHub account. Once you do, you’ll get access to ZenHub’s complete feature-set immediately.

What can ZenHub do?

ZenHub adds a series of enhancements directly inside the GitHub UI:

• Real-time, customizable task boards for GitHub issues;
• Burndown charts, estimates, and velocity tracking based on GitHub Milestones;
• Personal to-do lists and task prioritization;
• “+1” button for GitHub issues and comments;
• Drag-and-drop file sharing;
• Time-saving shortcuts like a quick repo switcher.

Add ZenHub to GitHub

Still curious? See more ZenHub features or read user reviews. This issue was written by your friendly ZenHub bot, posted by request from @sabinmarcu.

Investigate adding a center bias

It seems like a good idea to resort to picking a crop near the center when in doubt about what's the best crop.
Investigate adding a center bias to the score function, possibly depending on the spread of results.

Support for images through standard input

Would be nice to be able to pipe an image into the CLI from something else. My use case for that is where a user has uploaded an image through a web interface, and I want to get metadata and crop the photos before uploading them to S3. I could use a tempfile, but it would be nicer to just use the file IO objects as is.

Make gulp/grunt tasks

It would be awesome if we could have this working as a gulp and/or grunt task.

That way we could have it as a step in the build process and send the optimized images to the browser.

For the width and height parameters I'm thinking we could have some configuration in a json file:

{
     "filename.jpg": [{"width": 100, "height": 100}, {"width": 150, "height": 150}]
     ...
}

and for the output we could have something like filename-smartcorp-100-100.jpg so it can be easaly referenced in the templates.

What do you think ? Any other ideas ?

Connect.js / Express.js Middleware

Jonas,

this module is fantastic! I loved it so much that decided to integrate it with connect.js/express.js middleware: https://github.com/inadarei/connect-thumbs in case somebody wants to easily use it on the server-side, in a Node environment. Hope that's helpful.

Cheers!

Irakli