phulin / rebook

A collection of tools for cleaning up book scans.

Rebook: various book-scan-processing programs

Cropping

batch.py contains a system for cropping various input formats of collections of images and creating a PDF.

Dewarping

dewarp.py contains implementations of two dewarping algorithms:

• Kim et al. 2015, Document dewarping via text-line based optimization
• Meng et al. 2011, Metric rectification of curved document images

Focal length is currently assumed to be that of the iPhone 7, because that’s what I have been using to test. Change the f value at the top of this file if using a different camera.

The Kim et al. algorithm seems to actually work (and be fast enough to process large numbers of pages in a reasonable amount of time); you can use it directly or via batch.py --dewarp.

Binarization

binarize.py contains a ton of binarization algorithms, which should all have mostly-optimized implementations.

• Niblack binarization
• Sauvola and Pietikäinen 2000, Adaptive document image binarization
• Kamel and Zhao 1993, Extraction of binary character/graphics images from grayscale document images
• Yang and Yan 2000, An adaptive logical method for binarization of degraded document images
• Lu et al. 2010, Document image binarization using background estimation and stroke edges (background estimation portion incomplete)
• Su et al. 2013, Robust document image binarization technique for degraded document images (DIBCO 2013 champion)
• Ntirogiannis et al. 2014, A combined approach for the binarization of handwritten document images

The last algorithm is the best I've found on this set of inputs.

Text Structuring

block.py contains some text-structuring stuff. I intended to use this as a replacement for the current text-line detection system, but I haven't been able to get it to work.

• Koo and Choo 2010, State estimation in a document image and its applciation in text block identification and text line extraction

Super-resolution

upscale.py has some (incomplete) routines for single-image superresolution using text as a prior.

• Lee et al. 2007, Efficient sparse coding algorithms
• Walha et al. 2012, Super-resolution of single text image by sparse representation
• Liu et al. 2014, Blockwise coordinate descent schemes for sparse representation

Building

Requirements: Numpy/Scipy, OpenCV (with python bindings), Cython

I don’t know how to make a proper build system for Cython modules, so just do:

python setup.py build_ext --inplace

I'd love to convert this all to Python 3, but there's something wrong with Cython. PRs are, of course, welcome.

Disclaimer

I have not examined the patent status of any of these algorithms. Use at your own risk.

Do not use this program to violate copyright laws in your country.

License (MIT)

Copyright 2018 Patrick Hulin

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

rebook's Issues

Code for Meng et al. 2011

Hi! I've tried to get results for image dewarp using file mrcdi.py. But I struggle to make it work.

The first error obtained in dewarp.py line 185 in get_AH_lines:
return AH, lines, all_lines
This error was caused by the wrong order of returned values. In mecdi.py line 388:
AH, all_lines, lines = get_AH_lines(im)
So, I easily fixed it.

But the second problem is in mrcdi.py in estimate_directrix line 102:
D_points_arc, _ = arc_length_points(D_points)

arc_length_points() missing 2 required positional arguments: 'ys' and 'n_points'

What am I missing?

image_coords_T is not used in function gcs_to_image in dewarp.py

rebook/rebook/dewarp.py

Lines 232 to 233 in 4bc3920

	image_coords_T = image_coords.T
	image_coords_T += Of

batch.py --dewrap

Document image dewarping using text-lines and line Segments

Hello, thank you for that amazing work you achieved!

Do you have any plan to implement Document image dewarping using text-lines and line Segments by Taeho Kil et al. which is an evolution of Document dewarping via text-line based optimization already implemented?

Text-block alignment constrains implemented?

According to the results, the text-block alignment constrains may not be implemented, is it?

ImportError: inpaint.so: invalid ELF header

I'm trying to run dewarp.py but this error happens. Do you happen to know that is the issue?
By the way here is my python packages, could you please provide something similar so the installation could be done easier?
appdirs==1.4.3 backports.functools-lru-cache==1.6.1 cloudpickle==1.3.0 configparser==4.0.2 contextlib2==0.6.0.post1 cycler==0.10.0 Cython==0.29.15 decorator==4.4.2 distlib==0.3.0 enum34==1.1.10 filelock==3.0.12 importlib-metadata==1.5.0 importlib-resources==1.0.2 kiwisolver==1.1.0 lxml==4.5.0 matplotlib==2.2.5 networkx==2.2 numpy==1.13.0 opencv-python==4.2.0.32 pathlib2==2.3.5 Pillow==6.2.2 pyparsing==2.4.6 python-dateutil==2.8.1 pytz==2019.3 PyWavelets==1.0.3 rawpy==0.13.1 scandir==1.10.0 scikit-image==0.14.5 scipy==1.2.3 six==1.14.0 subprocess32==3.5.4 typing==3.7.4.1 virtualenv==20.0.7 zipp==1.2.0
Thank you very much.

About E_align

Hello, I tested your code and found that the alignment loss function failed to function and failed to realize the up-down perspective deformation

[dewarp.py] RecursionError: maximum recursion depth exceeded while calling a Python object

S1NHdeiMac:rebook s1nh$ python dewarp.py xxx.png 
Traceback (most recent call last):
  File "dewarp.py", line 17, in <module>
    import binarize
  File "/xxxx/rebook/rebook/binarize.py", line 9, in <module>
    import inpaint
  File "inpaint.pyx", line 8, in init rebook.inpaint
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 148, in __enter__
  File "<frozen importlib._bootstrap>", line 166, in _get_module_lock
RecursionError: maximum recursion depth exceeded while calling a Python object

And I set the maximum depth as 100000000 by using sys.setrecursionlimit(100000000)

Segmentation fault: 11

:)

It seems this 0.01 should be 1.01? I guess.

rebook/newton.pyx

Line 78 in 177daa7

u_plus = w * (Rp_x * t * 0.01 - ROf_x - T)

Hi, I just found that in this line, you use u_minus and u_plus, so I guess t should be 0.99 * t and 1.01 * t (rather than 0.01)?

Dewarped image is cropped

Hi,
I tested your dewarp.py and found that the dewarped image is cropped, like below:


I hope the result should be like this:

I want to keep the background, Do you have any advice? TKS

State estimation in a document image and its applciation in text block identification and text line extraction

block.py contains some text-structuring stuff. I intended to use this as a replacement for the current text-line detection system, but I haven't been able to get it to work.

What is the issue(s) you encounter?

Also I noticed that https://github.com/phulin/rebook/blob/master/rebook/block.py#L174 is probably an error. To invert the image, rather use im = 255 - im_inv.

a solution for a bug in binarization.py su2013 algo

Hey,
in the su2013 implementation we have :
def su2013(im, gamma=0.25): W = 5 horiz = cv2.getStructuringElement(cv2.MORPH_RECT, (W, 1)) vert = cv2.getStructuringElement(cv2.MORPH_RECT, (1, W)) I_min = cv2.erode(cv2.erode(im, horiz), vert) I_max = cv2.dilate(cv2.dilate(im, horiz), vert) diff = I_max - I_min C = diff.astype(np.float32) / (I_max + I_min + 1e-16) alpha = (im.std() / 128.0) ** gamma C_a = alpha * C + (1 - alpha) * diff _, C_a_bw = cv2.threshold(C_a, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU) return C_a_bw

the _, C_a_bw = cv2.threshold(C_a, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU) need C_a to be 'uint8' type which is not the case (float64), to resolve it we just need to cast it :
_, C_a_bw = cv2.threshold(C_a.astype('uint), 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)

the algo becomes :
def su2013(im, gamma=0.25): W = 5 horiz = cv2.getStructuringElement(cv2.MORPH_RECT, (W, 1)) vert = cv2.getStructuringElement(cv2.MORPH_RECT, (1, W)) I_min = cv2.erode(cv2.erode(im, horiz), vert) I_max = cv2.dilate(cv2.dilate(im, horiz), vert) diff = I_max - I_min C = diff.astype(np.float32) / (I_max + I_min + 1e-16) alpha = (im.std() / 128.0) ** gamma C_a = alpha * C + (1 - alpha) * diff _, C_a_bw = cv2.threshold(C_a.astype('uint8'), 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU) return C_a_bw

ps : Thanks a lot for this wonderful work 👍 👍

Getting Error when running batch.py --dewarp

Command:
python batch.py --dewarp -f FILE_NAME.jpg

Error:
detected dpi: 200
AH = 25
overall: mean: 6.957979429464889 std: 1.433384350862489
C:\ProgramData\Anaconda3\lib\site-packages\numpy\polynomial_polybase.py:798: RankWarning: The fit may be poorly conditioned
res = cls._fit(xnew, y, deg, w=w, rcond=rcond, full=full)
original lines: 112 merged lines: 112
Bimodal! Splitting page!
split: 32 out of 112 @ 3073
==== PAGE 0 ====
Traceback (most recent call last):
File "batch.py", line 218, in
run(args)
File "batch.py", line 154, in run
_, out_images = process_image(im, dpi=args.dpi)
File "batch.py", line 43, in process_image
dewarped_images = dewarp.kim2014(original_rot90)
File "C:\Users\aimittal\Desktop\Enhanced Copy\rebook\rebook\dewarp.py", line 1044, in kim2014
page_image = page_crop.apply(orig)
File "C:\Users\aimittal\Desktop\Enhanced Copy\rebook\rebook\geometry.py", line 225, in apply
return im[crop.y0:crop.y1, crop.x0:crop.x1]
TypeError: slice indices must be integers or None or have an index method

Could you explain why you compute the initial rotation vector like this?

Hi, I found that you've tried to use the vertical vanishing point to give a good guess of the initial rotation vector.
Although you eventually commented out the theta_0 computed from the vanishing point, I am curious about why you computed the rotation vector like that.
(If possible, could you explain by an illustration?)

rebook/rebook/dewarp.py

Lines 1101 to 1104 in 4bc3920

	xz_ratio = -f / vx # theta_x / theta_z
	norm_theta_sq = (atan2(np.sqrt(vx ** 2 + f ** 2), vy) - pi) ** 2
	theta_z = np.sqrt(norm_theta_sq / (xz_ratio ** 2 + 1))
	theta_x = xz_ratio * theta_z

Did you implement the method in《Robust Document Image Dewarping Method Using Text-Lines and Line Segments》?

Did you implement the method in《Robust Document Image Dewarping Method Using Text-Lines and Line Segments》?

About collate.pyx error when running dewarp.py

Bug in ntirogiannis2014 function, Case of 100% blqck forground

Hey,
In ntigo method, it the contrast estimation is based on the formulat : -50+log10((FG_mean-FG_std)/)BG_mean-BG_std)
in the situation where we have a 100% black foreground, it raise an error ( log(0) ), i checked the article, from my understanding they said to use in the extreme case to put FG_mean-FG_std = 2.5, you can check it for more credit:
here is the proposed solution.

if not (FG_avg + FG_std) == 0:
        C = -50 * np.log10((FG_avg + FG_std) / (BG_avg - BG_std))
        k = -0.2 - 0.1 * C / 10
    else :#This is the extreme case when the FG is 100% black, check the article explaination page before equation 5
        C = -50 * np.log10((2.5) / (BG_avg - BG_std))
        k = -0.2 - 0.1 * C / 10

PS : i ll do a push request as soon as i can, sorry for that