spillerrec / overmix

Automatic anime screenshot stitching in high quality

Overmix

Overmix can stitch fractions of smaller images together to create the original full image. It is specifically made for stitching anime screenshots, where a small portion of a scene is shown and the viewpoint slides to show the remaining area.

The idea behind Overmix is to increase the amount of images which is used to stitch it together, and use this to solve MPEG compression, color banding and on-screen text/logo issues. Development is now geared towards understanding the more theoretical parts about Image Reconstruction and how this can be applied to increase quality even further.

Features

• High quality 16-bit (or more) rendering, with dithering for 8-bit output
• Automatic detection of vertical and horizontal offsets, with sub-pixel precision
• Interlacing support
• Multi-threaded to fully exploit your computer's potential
• Support for 10-bit YUV input
• Rendering pipeline supports chroma sub-sampling without upscaling
• Deconvolution for sharpening images
• Logo/credits detection and removal
• Steam minimization (colors kinda broken right now though...)
• Detection of cyclic animations
• Basic Super resolution, GUI kinda lacking right now

Current work

• Separation of foreground and background in slides where foreground and background moves with different speeds
• MPEG2 decoder for minimizing MPEG2 compression artefacts, which should help especially with motion compensation.
• Revamp GUI so it is more easy to add advanced settings for operations
• Command line interface

Future work

• Detection of zooming and rotation
• Figuring out how features such as animation detection and separation of fore/back-ground can be combined

How to contribute

Even if you know nothing about programming, there are several ways to contribute:

• If it currently does not solve all your needs, make a feature request on the issue tracker, or comment on an existing one.
• If you fail to stitch an image properly, create an issue and share a link to the input images. Either I can help find you the right settings, or identify a current limitation of Overmix.
• If you can't figure out how to use some part of the program even after checking the wiki (or even it was just difficult), create an issue/bug report. This mean that either the interface is not intuitive enough, documentation is lacking, or the documentation is not clear enough.

If you do not want to create a Github user, feel free to send me an email at [email protected] about anything.

Avoid using screenshots

Video is not as RGB directly, instead it uses a color transformation to seperate luminance and color information. The color information is then downsampled as it usually isn't noticable. Depending on your video player, it might use lower quality approximations to improve performance/batery life. VLC in particular causes bad results, but mpv can also cause screenshots to be in lower quality than what you see on the screen. By using the built-in video importer you will get the full qualty (in 10bit for those videos as well), while also getting every frame which will further improve quality.

The Dump format

In order to dump Hi10p video frames without them being converted to 8-bit RGB, the dump format was developed. This format supports up to 16-bit YUV images with chroma sub-sampling. Now only needed if you want to store the dumped frames. Several tools related to the format have been developed, most importantly an application to easily extract every unique frame in a video sequence. This and other tools such as Windows extensions can be found in the following repository: https://github.com/spillerrec/dump-tools

Known issues

• Dehumidifier renderer does not "dehumidify" colors. (Forcing RGB mode with the new option might do the trick, but I haven't checked the code.)
• Progress bars are not implemented everywhere, and the "Cancel" option is rarely implemented even though the button appears.

Building

Dependencies

• Qt5
• C++14 (generic lambdas, std::make_unique<>)
• cmake (for compiling)
• ffmpeg
• zlib
• lzma
• libpng
• png++ (header only)
• libjpeg
• libraw
• fftw3
• pugixml
• lcms2 (required for GUI)
• boost
• eigen3
• QCustomPlot 2.x (required for GUI)
• google/benchmark (required for unit-benchmarking)

Optional:

• waifu2x-converter-cpp (with opencv enabled)

Linux only:

• xcb
• qt5x11extras

Building

1. cmake -DCMAKE_BUILD_TYPE=release
2. make

It is recormended to build in a seperate folder, as cmake polutes the all the directories otherwise. You can do it like this:

1. mkdir release
2. cd release
3. cmake ../ -DCMAKE_BUILD_TYPE=release
4. make

overmix's Issues

Create Point and Rectangle classes

QPoint, QPointF, QRect and QRectF is quite limited in which math operations you can do on them. We can't for example multiply a QPoint with another QPoint, i.e.:
QPoint( 32, 54 ) * QPoint( 2, 4 )
This is actually fairly common, as we often want to multiply a size with different x and y scalings.

So proper versions of those classes could simplify a lot of code, if we also add getters in Plane, ImageEx, etc.

I propose we use templates to support both integers and floating point, but also signed vs. unsigned.
Add also common cmath stuff like pow, abs and so on.

Post-processing does not affect dump-output

When saving using the .dump format, the post-processing is ignored, as we save temp_ex which is not affected by the post-processing!

Correct imlpementation of downscaling

The range of values considered when downscaling is incorrect.

Use number of planes in ImageEx to determine valid()

Use planes.size() == 0 for invalid, could simplify it a bit

Support cropping already loaded images

Noticing artifacts caused by black borders after a render is common, so we would like to just crop the loaded images, instead of having to reload them.
Furthermore, we could support cropping images in O(1) time in PlaneBase, without any realloc.

Static difference should output in inverted colors

It is mostly used for masking, but Overmix expect the masks to have black as transparent, not white. For ease of use, invert the colors of the output from static difference.

Save button always enabled, even when no image

Even though there is no image currently, it is still possible to press the Save button and get to the file save dialog, which simply ends up doing nothing. Disable this button when there is no image to save.

Support 16-bit PNG

Support loading and writing PNG in 16-bit color depth (per channel). High color depth is what this project is about, still not having an output format which supports it is silly.

Don't re-render to do post-processing

When changing the post-processing settings, it should apply the changes to the previously rendered image, instead of spending time on re-rendering. Rendering simply takes too long with many input images.

Aligner reset when changing render

Changing the render causes Overmix to realign, wasting a lot of time on that...

Change movement depending on recursive level

In the top levels of the recursion when aligning, both the images bigger and the movement is larger. We therefore need to use an adaptive movement amount depending on the recursion level.

Create GUI for custom render pipeline

We now have a flexible rendering pipeline, but the GUI is made for a simple static pipeline. Create a GUI which allows users to add rendering operations in any order and as many times as they want.

Stop using AContainer::plane()

We need plane() so that AImageAligner can replace it with a processed version of the Image. However we can just as easily wrap it in a ImageEx and use AContainer::image(). This could make the Renders slightly simpler.
We could also start having the AContainer interface tell the information we assume to be static between each image, such as color system.

Add CLI for automation

Add a CLI to make it possible to automate stitching to some degree. Following options should be usable:
Preprocess

• Alpha mask
• De-telecine
• Crop

Merging

• Vertical and Horizontal
• Precision
• All methods, except animation and layers (not ready)

Render and color

• All options

Postprocess

• Scaling
• Deconvolve
• Level
• Binarize

To do this we need to separate the stitching pipeline from the GUI, as it is currently embedded in mainwindow.cpp. At the same time, add the possibility to have separate frames for animation, and reusable alpha masks. Also think about how to specify separate small animations in different parts of the image.

Furthermore, since this is primarily a GUI application, if it is loaded with just images, it should add them in the GUI automatically. Also allow dropping an entire folder, to load all the images therein.

AnimatedAligner needs user interaction

AnimatedAligner is pretty good at finding a good threshold, but there are cases where it isn't possible.

Two cases have been seen several times:

• Sub-pixel mis-alignment causes the errors to be too high, and threshold ends up wrong. Solution is to do sub-pixel aligning, but the user needs to be aware this is the issue.
• The errors between the same image is quite consistent, and it picks up small variances in those, instead of the large variances between the different images.

Showing how the threshold was determined graphically to the user would be quite useful, and we could provide a way to change it manually.

We need to do the following for that:

1. Add a user-interaction step in AImageAligner. This could be used for configuration of other aligners as well.
2. Make a graph widget
3. Make the UI to change the threshold

Performance degradation with alpha-aware aligning

Calculating the difference with alpha is more expensive, obviously. However because the rendering always outputs an alpha plane, a lot of aligning which otherwise could have been done without alpha, now uses the slower method.
For RecurisiveAligner we might want a specialized algorithm for merging two planes, and we could handle it there, but fixing the render would solve the issue for OrderedAligner as well.
Also consider the fact that we ignore small differences. It is mainly useful for AnimatedAligner, so we might want to split it up into two methods. Also consider integer-math versions of the alpha calculations.

Implement dilation

Dilation could possible improve performance of automatic Logo detection, so we need to implement it and try it out.
http://en.wikipedia.org/wiki/Dilation_(morphology)

Update image viewer

The image viewer widget is a stripped down version of the one I made for imgviewer. However it is a quite old version and it has several issues which were fixed for imgviewer a long time ago.
Look into how the image viewer widget can be separated in imgviewer from the rest of the code, so we easily can update it in the future.

Upscale in to_qimage, not in SimpleRender

By not upscaling in SimpleRender, we can export it in Dump format using the non-scaled images

Align images in AnimatedAligner by the movements

Aligning against the averaged image doesn't really work that great, the resulting frames tend to align against the moving objects, not the consistent ones. So we could try to use the fact that they are ordered and slowly moving in an consistent way.

The theory:
We can create a tendency curve over the position of a video frame, using the offsets in a single image, and the frame indexes. If we create such for all images, we can align them by finding the offsets which makes all the tendency curves match each other as best as possible.

Aligner is now desync with ARender, rethink it

AImageAligner is no longer needed for ARender to know how the images are positioned to each others.

First of all, now we no longer need to keep an instance of the aligner anymore, and thus can avoid a raw pointer.
However the "Align&Draw" functionality is now no longer makes sense, as we don't need to align before drawing. Should we just always allow to draw? It is pretty convenient though...

Move edge-aligning to base class

We have specialisations of prepared_plane() in both AverageAligner, AnimatedAligner and Recursive aligner for edge detection, move all this to the super class

Re-evaluate Deteleciner interface

Can we avoid pointers here?

Add way to reload all images

To make it easier to make changes to preprocessing options, add a way to reload all images.

Add an overview over loaded files

We need some way to see all loaded files, and remove, rearrange, and modify masks at will. Being able to see the files directly in Overmix would be nice as well.

Fix rendering when not upscaling chroma, and make it default

Upscaling chroma when rendering is quite slow and uses quite a bit of memory, so we should avoid this and make this the default operation.
This was tried but dropped earlier, because there were minor alignment issues. But thinking about it, it actually makes sense because it actually does sub-pixel rendering. We should let that to the actual sub-pixel or hopefully soon, SR renders.
It also seems broken currently, so fix it...

Alpha not used when aligning

The alpha channel is ignored when calculating the difference between two images. So even if you use an alpha mask, static content such as credits can still interfere alignment. Even worse, stitches which moves a lot in both directions create a lot of transparent areas, but which is treated as solid black by the alignment algorithm.

Both Ordered and Recursive aligner needs to check if there is an alpha channel, and use it available.

Degrade FloatRender to AverageRender when no subpixel alignment

FloatRender should check if there is any subpixel alignment going on, if not it should just call AverageRender.

temp_ex cache not cleared when render is changed

Picking a different render will not invalidate the cache when clicking "Draw"

Add alpha handling to AImageAligner

We want to solve the following two issues:

1. When not upscaling chroma, rendering might need to downscale all alpha masks. Since it is usually the same for all images, we want to downscale it once, not 2*n times.
2. LayeredAligner needs a way to assign alpha masks to planes individually, but can't change it because the images of course are const.

We are going to add this in AImageAligner.
For (1) we need to be able to assign an alpha plane to several images, without making copies. Furthermore, we need to be able to inform ARenders about this information, so it can optimize it away.
For (2) nothing special is needed that (1) doesn't provide, just a nice interface is enough.

Keep in mind that images might already contain an alpha channel. We might want to create a temporary alpha plane which is two planes multiplied with each other. Or we could just ignore it...

Richardson-Lucy deconvolution fails with most PSFs

If the PSF is not Gaussian like and symmetric, deconvolution fails to produce useful results. Investigate why this is the case, as it prevents us for doing any motion-blur removal.

Color incorrectly handled in Dehumidifier

The dehumidifier works by selecting the darkest color, but this does not work for the chroma channels. Either select the darkest pixel based on luminance, or try to see if selecting the color which is farthest away from 0.5, the neutral color, works.

Add edge-aligning to the GUI

We have it in the base class, so might as well add it instead of having to hard-code it.

How to load a png?

I am running Overmix like
Overmix image.png.

I cannot figure out how to load an image.

Make scaling alpha aware

Not keeping alpha into account when scaling can create artifacts, as the otherwise "transparent" colors are included in the interpolation, causing black/white to bleed into the image.

Not redrawing after realign

Does not update image:

1. Click Align&Draw
2. Change parameters and click Align
3. Click Draw
   Image did not update

Use variable diffing when aligning

When doing aligning, we probably don't need high precision diffs when the offsets are large. Decreasing the precision could give large speedups potentially.

Recursive aligner

Instead of the iterative AverageAligner, we could do this recursively with a divide and conquer algorithm.

Overview:

1. Split the remaining images into two halves.
2. Align the two halves separately. In the trivial case, skip this and assume starting position of (0,0)
3. Render the aligned halves
4. Find the alignment between these two halves, and use this to offset the halves
5. Return

The advantage here is that rendering has higher complexity which makes a noticeable slowdown when merging many images. With the recursive algorithm, the renderer will always have n = 2. We can also reduce the memory usage significantly with sub-pixel precision, as we do not need to have all planes upscaled during the whole process.

Optimize alpha counting in AverageRender

Half of the time is spend on counting how many images affects a specific pixel, and how much. When all the images does not containing a alpha channel, we could optimize this.

We could optimize for just the vertical alignment case, as this is 95% of the cases, or we could try to generalize it to both directions. For that we could have a stack for each line which marks when, and how much the amount of images changes.
For now I propose we just optimize it for the vertical case, which just needs a count for each line.

Memoryleak when pressing Draw in the GUI

Memory usage slightly increase each time Draw is pressed. Aligning seems find, thus rendering is likely the same. That makes it likely the leak is in the post-processing pipeline

Fix const-correctness in AImageAligner

Those const removing casts are making me cry...

Update code style

Class names: Big camelcase: ClassName
Method/functions names: lower cammelcase: methodName
Variables: underscores: variable_name
Header files: ClassName.hpp

It is slightly inconsistent currently, which should be corrected.
This should make it easier to see what is classes, functions and variables by just looking at the style conventions.

not upscaling chroma chrashes with alpha (fix architecture)

If using an alpha mask, SimpleRender crashes when not using chroma upscaling.
This is likely because they differ in size, however downscaling all alpha planes will be slow... Usually we don't have unique alpha planes, it is a few used for several images, so we should look into how this can be optimized.

Use vectors in edge-detection inteface

Use Vectors instead of plain Int arrays in the interface, to reduce manual memory management

const issues in MultiPlaneIterator

MultiPlaneIterator does not separate input and output plane. Thus all planes are non-const, but most of the planes we input into it is const. So currently there are a few const_cast in:

• SimpleRender
• PlaneRender

While these planes are not modified, and the cast should thus be safe, it is still possible for us to mess it up later on.

Rethink MultiPlaneIterator, and separate input and output.

Scaling drop-down boxes are not handled

Selecting different scaling algorithms in the drop-down boxes have no effect

Implement move constructors in Plane to reduce pointer usage

Having to return a pointer to a Plane when doing image operations such as scaling similar makes memory management harder and error prone. For the love of pointer-free code, learn how to do it and remove as many pointers as possible.

Better progression info

Right now we only have a single bar going from 0% to 100%. This is tricky to apply to the aligners, since they have several tasks and often use other aligners in the process. Make a multi-level progress bar which can tell what is happening in the individual stages, and show this information to the user.

We also have a recursive aligner, which might be slightly tricky to add. We should also show the progress bar much earlier than QProgress bar...

Investigate GEGL and see if it can be used

Overmix uses its custom image processing code. Could GEGL be an alternative? It would be nice if we had a lot of the basic operations done already, especially OpenCl optimized versions. If some are missing, we could implement them as so they could potentially be added to GEGL.
Try to do some programming with GEGL, check its available operations, and see if it can support the image formats we use. (Chroma sub-sampled, interlacing, etc..)

RenderPipeline does no longer work

Any applied filters result in an empty image, and it will not update image after initial run