Comments (36)
By the way, you'll probably need a good IPS-monitor to see that fuzz in the lower 2 bits. It doesn't show on my old Medion-monitor.
from archive-pdf-tools.
I ran your image in recode_pdf
:
cp outputbase2-000-raar-effect-onderste-regel-didjvu\ zonder\ tekst.tif img.tif
tesseract img.tif - hocr > img.hocr
recode_pdf --from-imagestack /tmp/img.tif --hocr-file /tmp/img.hocr -o /tmp/out.pdf
Processed 1 pages at 3.73 seconds/page
mrcview /tmp/out.pdf /tmp/mrc.pdf
See file below -- do you see the same background fuzz?
[out.pdf](https://github.com/internetarchive/archive-pdf-tools/files/7597950/out.pdf
mrc.pdf
)
from archive-pdf-tools.
Yes, with Acrobat Reader DC it takes some time to render the foreground, in a fraction of a second I can see lot of fuzzy edges of those letters.
from archive-pdf-tools.
I'm even able to read the fuzz in the background
from archive-pdf-tools.
from archive-pdf-tools.
Right, so you're referring to the background not fully having the text layer removed. There are some tricks you can do, see for example #8 (looks like you found it) - but they also harm the quality in some other cases.
In general the latest releases (the one I shared with you) has gotten much better at making the background just background. Providing it the right DPI for example matters some too, and then the binarisation methods also matter. At archive.org we also deal with a lot of books, if you know that you're dealing only with text, there's a lot more that can be done.
In general, this project is pretty new: it's hardly a year old, and I started it because most of the tools (as you have found) don't do MRC at all, so I decided to write it myself.
from archive-pdf-tools.
BTW: the PDFs I shared about are already ~40x smaller than the tif that you uploaded, which is about as much as you can expect from MRC. If you want more compression, you can downsample the background, like so:
$ recode_pdf --from-imagestack /tmp/img.tif --hocr-file /tmp/img.hocr -o /tmp/out.pdf --bg-downsample 3 -v --dpi 600 --fg-compression-flags '-slope 45000' --mask-compression jbig2
MMX
SSE
SSE2
SSE3
SSSE3
SSE41
POPCNT
SSE42
AVX
F16C
FMA3
AVX2
Creating text only PDF
Starting page generation at 2021-11-24T19:34:25.015312
Finished page generation at 2021-11-24T19:34:25.025512
Creating text pages took 0.0102 seconds
Inserting (and compressing) images
Converting with image mode: 2
MRC time breakdown: {'image_load': 0, 'grey_conversion': 358, 'hocr_mask_gen': 101, 'est_1': 91, 'threshold': 350, 'fast_denoise': 22, 'mask_jbig2': 174, 'fg_partial_blur': 719, 'fg_jp2': 309, 'bg_partial_blur': 770, 'bg_downsample': 270, 'bg_jp2': 82, 'page_image_insertion': 0}
Saving PDF now
Processed 1 pages at 3.63 seconds/page
Compression ratio: 66.214312
which leads to a 64K PDF with the following breakdown:
$ pdfimagesmrc /tmp/out.pdf
backsize: 4.00% 2.43kB
frntsize: 40.16% 24.42kB
masksize: 39.18% 23.82kB
restsize: 16.66% 10.13kB
If you remove the text layer (restsize
), it'd be a bit smaller still.
from archive-pdf-tools.
I've been looking to collect a relatively large sample of different test images. I only have a few here now: https://github.com/internetarchive/archive-pdf-tools/tree/tests/tests/files
If you'd like to contribute some, that could also help in the future when trying to further improve the background generation.
from archive-pdf-tools.
Do you have some pointers to C44 implementations (C or Python)? I'd like to try it on some of the books with images/photos that I have.
from archive-pdf-tools.
from archive-pdf-tools.
from archive-pdf-tools.
Interesting. In the past I had also looked at specifying the region of interest (https://www.researchgate.net/publication/252087916_Selecting_the_don't_care_bits_in_JPEG2000_ROI_coding) with kakadu or openjpeg, but it didn't seem to make a difference.
Do you perhaps have a few instructions on how to use this? I'm working on these kind of improvements in my spare time and I don't have a lot of time to investigate the djvu tooling. The sourceforge website isn't too helpful and I got the same impression when I last checked this out: the project looks a bit dead and it's not clear how they want you (as a user) to use it. Thanks in advance!
from archive-pdf-tools.
Hm -- looks like I have c44
already installed on my system...
from archive-pdf-tools.
It looks like c44
can create a djvu file from a mask file and an input image, and then djvuextract can extract some parts (although I am not sure how to read it yet)
e.g.
c44 /tmp/img.pnm -mask /tmp/out-mask.pbm /tmp/out.djvu
and
$ djvuextract /tmp/out.djvu BG44=/tmp/out-djvu-bg
BG44=/tmp/out-djvu-bg --> "/tmp/out-djvu-bg" (290742 bytes)
from archive-pdf-tools.
from archive-pdf-tools.
For my understanding, do you know how this deals with non-text, like images? I was able to see some output (by converting to postscript and then postscript to pdf) and it seems to perform quantization on the background, which is fine if the background uniform, but not if it contains any kind of drawings or interesting colours?
from archive-pdf-tools.
Here are some examples from (much) older versions of this software: https://archive.org/~merlijn/projects/archive-pdf-tools/index.html#mrc-examples
from archive-pdf-tools.
from archive-pdf-tools.
from archive-pdf-tools.
from archive-pdf-tools.
from archive-pdf-tools.
Manual correction is hard on the scale that I/we usually use this program (many million of pages every day).
You can find a comic I tested on recently here (https://archive.org/details/bruno-de-bever-a01):
The compression ratio is only ~4x because of a specific archive.org policy where we use higher quality compression for the first 10 and last 5 pages of an artifact (and this one only has 33 pages).
from archive-pdf-tools.
If I look at the textballoon https://wizzup.org/bruno.pdf p. 18 "Waar heb je het over, ik heb geen spullen" "Ben je echt alleen"
I see a lot of fuzzy color in the textballoons, where the original has more white in those textballoons, together with some jpeg-artefacts around the text. With the resolution of the color in the textballoons I would expect there to be some blocking at the edge, But not that the color of the hat of the blackbird goes 3 pixels above the black line:
from archive-pdf-tools.
Right, that happens because of an algorithm I wrote that attempts to make the images a bit more smooth to compress, but can cause the colours to bleed through a bit: https://github.com/internetarchive/archive-pdf-tools/blob/master/cython/optimiser.pyx#L65 - I've observed similar things with the PDFs produced by the foxit (luratech) pdf compressor
from archive-pdf-tools.
It might be possible to add some more algorithms, allow tweaking more parameters to be tweaked to create "profiles" for certain input types. I've tried to make it all pretty generic so that it should do a decent job for a wide range of input documents.
from archive-pdf-tools.
(Very open to improvements btw -- I've been toying with this on and off for a year and this is best I could come up with currently)
from archive-pdf-tools.
The first thing I could think of is try to do the same with DjVuSolo 3.1 to see whether there are better options for which patents will be expiring very soon. The American patent however has a strange enddate with no clear relation to the filing date. That should be 20 years max difference. One of the european patents expires in two months.
from archive-pdf-tools.
I've made a DjVu from the with standard settings of DjVuSolo3.1, which has this look in WinDjView:
The resulting djvu of this picture has about the same size as the pdfsam split page of your pdf which has this look in Acrobat DC:
However unfortunately, when I make DjVuToy convert the DjVu to PDF it becomes twice as big.
Documents.zip
Your algorithm seems to preserve more of the drawn lines if you look at the eyes and the nose on the right, but both damage the tree above the character on the right. This was the original:
from archive-pdf-tools.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8404928/#B41-jimaging-07-00153
https://github.com/WangJieying/SDMD-resources
https://github.com/WangJieying/SDMD-resources/blob/main/cartoon.md
So I would propose to diverge from PDF for this purpose, or look whether some vector description language possible within PDF could be used for the vectors.
Unfortunately this SDMD requires CUDA, which I don't have. I could probably rent one at AWS or Azure.
from archive-pdf-tools.
Interesting how the DjVu code does the inverse: removing colour around the beak of the raven and the teeth of the beaver.
from archive-pdf-tools.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8404928/#B41-jimaging-07-00153 https://github.com/WangJieying/SDMD-resources https://github.com/WangJieying/SDMD-resources/blob/main/cartoon.md So I would propose to diverge from PDF for this purpose, or look whether some vector description language possible within PDF could be used for the vectors.
Unfortunately this SDMD requires CUDA, which I don't have. I could probably rent one at AWS or Azure.
Edit: I've tried to bring the contents of these repo's alive, but an important part was missing: the class BSplineCurveGenerate.
I succeeded in getting the missing files, and already saw the compile finish, however I'm now rearranging the dependencies and then will try again to get that bspline-compression done on these pictures on an Azure-cuda server.
from archive-pdf-tools.
It might be worth checking out #33 and also perhaps toying with some of the JPEG2000 compression parameters. I think I currently always use just one layer, but multiple layers with region of interest coding could also help. We could mark the text as 'not interesting' in the background image, and hope that some of the background noise just disappears because of that.
from archive-pdf-tools.
I'm now as far that I can compile the SDMD-code that I had to recompose from some repo's, but still no luck in getting the skeleton. Searching for the goal-to-be-reached: A pdf using fitted bezier curves or something like that to redraw the pictures, I guess using PostScript, I found the complete legacy of Zunzunsite: http://web.archive.org/web/20200315020359/http://zunzun.com/
https://bitbucket.org/zunzuncode/zunzunsite3/src/master/
Unfortunately the owner had to quit due to health reasons:
https://groups.google.com/g/zunzun_dot_com/c/n7Uk9P_CDe4
It's revived! http://www.findcurves.com/
The site offers lots of curve formulas, so I doubt it fits bezier curves right into PostScript in a PDF.
from archive-pdf-tools.
With a little help from @WangJieying I was able to reconstruct and recompose the SDMD-image compression on an Azure server:
https://github.com/rmast/SDMD-resources.
This is the result I get with the default config.txt, of which the compressed output.sir is 211590 bytes:
For convenience, the original:
So, this SDMD algorithm seems not really fit to preserve the black lines of a comic. I wonder if comic compression with splines to create a postscript Bspline-PDF could become a new CUDA-research study subject ;-).
from archive-pdf-tools.
Hmm... interesting, is there another part where the foreground or other components are visible? Like, how is it combined?
from archive-pdf-tools.
from archive-pdf-tools.
Related Issues (20)
- pillow is not working properly HOT 27
- Need some inspiration? HOT 7
- Some scans become inverted HOT 7
- Detect if RGB images in pages are greyscale or even 1bit
- Define scope of tooling and work to improve for that scope
- Create better presets for users with quality-comparable options for openjpeg/grok/pillow and kakadu HOT 1
- Missing test suite? HOT 1
- pdfcomp: new tool, discussion, compression questions HOT 19
- Bug in foreground/background separator choosing massive block instead of character outline. HOT 14
- The choice for inverting, what's the use for perc_larger?
- pdfcomp: problems with inverted text that is often better in hocr. HOT 10
- Wrong resolution of mask image when foreground image is downsampled HOT 1
- First recode_pdf test: 'numpy' has no attribute 'int'. HOT 5
- IndexError: list index out of range (single TIFF file) HOT 5
- HOCR rendering compares unfavorably with tesseract PDF text layer HOT 11
- Installing on MacOS? HOT 29
- Q: accessible tagging/hints? HOT 4
- A certain PDF from Archive.org does not display all of its contents on Mac OS HOT 26
- A user-friendly example for a scanned multipage PDF needed HOT 3
- Recode does not merge hocr into pdf HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from archive-pdf-tools.