dfdx / piecewiseaffinetransforms.jl

Smooth image transformations for complex shapes

Piecewise Affine Transformations

Package for smooth deformation of complex shapes.

Installation

Pkg.add("PiecewiseAffineTransforms")

Usage Overview

Piecewise affine transformation resembles ordinary affine transformation, but instead of warping single region linearly, it splits down area under the question into a set of triangles and warps each such triangle separately.

Let's say, we have an image of a face and want to warp it to have different expression (destination image is here only for demonstration, we will not use it):

using PiecewiseAffineTransforms

src_img = ...
dst_img = ...

(full version of code is available in examples/ex.jl)

Source image	Destination image

We will also assume that both faces are annotated with corresponding shape landmarks:

src_shape = ... # should be a Nx2 matrix of Float64,
                #  where N is a number of landmarks
dst_shape = ...

First of all, we need to split the shapes into triangles, i.e. triangulate them:

trigs = delaunayindexes(src_shape)  # Tx3 matrix of Int, where T is
                                    #  a number of resuling triangles
# needs ImageView installed (Pkg.add("ImageView"))
triplot(src_img, src_shape, trigs)
triplot(dst_img, dst_shape, trigs)

WARNING: Triangulation is based on VoronoiDelaunay.jl, which currently has a bug resulting in one lost triangle from time to time. To overcome this, just get good sample of triangulation and save it for future use.

Source shape	Destination shape

Warping src_image from src_shape to dst_shape may be as simple as calling this:

@time warped = pa_warp(src_img, src_shape, dst_shape, trigs)
# 1.44 seconds

But if you are going to repeat warping to dst_shape for many source images or just many times, it's worth to prepare warp by creating PAWarpParams object and using it for all future transformation to dst_shape:

@time pa_params = pa_warp_params(dst_shape, trigs, (480, 640))
# 5.92 seconds
@time warped = pa_warp(pa_params, src_img, src_shape)
# 0.075 seconds

But anyway, they both give (almost) the same result:

Original image	Warped image

Acknowledgement

Code for prepared warp was mostly extracted from ICAAM project by Luca Vezzaro.

piecewiseaffinetransforms.jl's Issues

Info about upcoming removal of packages in the General registry

As described in https://discourse.julialang.org/t/ann-plans-for-removing-packages-that-do-not-yet-support-1-0-from-the-general-registry/ we are planning on removing packages that do not support 1.0 from the General registry. This package has been detected to not support 1.0 and is thus slated to be removed. The removal of packages from the registry will happen approximately a month after this issue is open.

To transition to the new Pkg system using Project.toml, see https://github.com/JuliaRegistries/Registrator.jl#transitioning-from-require-to-projecttoml.
To then tag a new version of the package, see https://github.com/JuliaRegistries/Registrator.jl#via-the-github-app.

If you believe this package has erroneously been detected as not supporting 1.0 or have any other questions, don't hesitate to discuss it here or in the thread linked at the top of this post.

ImageView dependency

Is there any chance I could convince you to remove the image view dependency? I intend to use your cool package as a backend for an image augmentation library I am working on (basically to emulate elastic distortions; see image below), and the having ImageView in there makes some stuff a bit tedious (such as travis testing), and it also kinda feels weird.

One proposal I would have is to use https://github.com/JuliaPlots/RecipesBase.jl instead, which is a lightweight package that allows to specify custom plots that can then be used with the Plots.jl package

This is an example of how my recipes for plotting the triangles currently look:

The only current problem with RecipesBase is that it doesn't yet allow to specify custom plot name (such as triplot) but require a custom type. But I have talked to @tbreloff about it and this may be added in the future

What do you think about this in general?

4 Channel Image Support

All image rendering functions appear to generate a similar issue when a four channel image is generated from the Image package.

A sample error from the triplot() function:

ERROR: Cannot infer colorspace of Array, use an AbstractImage type
 in colorspace at /Users/davidchun/.julia/v0.3/Images/src/core.jl:684
 in properties at /Users/davidchun/.julia/v0.3/Images/src/core.jl:625
 in triplot at /Users/davidchun/.julia/v0.3/PiecewiseAffineTransforms/src/view.jl:15
 in include at /Applications/Julia-0.3.11.app/Contents/Resources/julia/lib/julia/sys.dylib
 in include_from_node1 at loading.jl:128
 in process_options at /Applications/Julia-0.3.11.app/Contents/Resources/julia/lib/julia/sys.dylib
 in _start at /Applications/Julia-0.3.11.app/Contents/Resources/julia/lib/julia/sys.dylib
while loading /Users/davidchun/.julia/v0.3/PiecewiseAffineTransforms/examples/ex.jl, in expression starting on line 41

Is this perhaps a deeper issue with determining why my environment is generating a four channel image than to force a three channel image in the package?