Comments (5)
On page 18 /Xi9 appears to be in a circular reference loop that includes itself over and over. I can guard against it but the file is likely malformed at this location. Try Ghostscript to rewrite the PDF.
If it does just happen to be really deep Python has a system call to increase recursion depth which you could use to force the file through. Doubt it would help though.
from ocrmypdf.
Thanks, for my test document, Ghostscript detected the problem and repaired the file.
Changing the python recursion limit does not seem to matter for this issue since the function contains a fixed recursion limit of 10.
from ocrmypdf.
I added a guard that should prevent recursion into previously visited XObjects. Please test and let me know if it works.
It's still probably best to use ghostscript to repair these files.
from ocrmypdf.
The guard does not seem to work. If I understand it correctly, you check against all the included/excluded xrefs before the recursion but the xref are added only after the recursion to the set of included/excluded xrefs. Thus, the function can cycle through the circular references forever before the problematic xrefs are added.
I tried a different approach by passing all the layers of the recursion:
def _find_image_xrefs_container(
pdf: Pdf,
container: Object,
pageno: int,
include_xrefs: MutableSet[Xref],
exclude_xrefs: MutableSet[Xref],
pageno_for_xref: dict[Xref, int],
recursion_layers: tuple[Xref,...] = tuple()
):
"""Find all image XRefs or Form XObject and add to the include/exclude sets."""
if len(recursion_layers) > 10:
log.warning("Recursion depth exceeded in _find_image_xrefs_page")
return
try:
xobjs = container.Resources.XObject
except AttributeError:
return
for _imname, image in dict(xobjs).items():
if image.objgen[1] != 0:
continue # Ignore images in an incremental PDF
xref = Xref(image.objgen[0])
if xref in recursion_layers:
log.warning(f"Skipping {_imname} in page {pageno} since it seems to be recursive.")
continue # Already processed
if Name.Subtype in image and image.Subtype == Name.Form:
# Recurse into Form XObjects
log.debug(f"Recursing into Form XObject {_imname} in page {pageno}")
_find_image_xrefs_container(
pdf,
image,
pageno,
include_xrefs,
exclude_xrefs,
pageno_for_xref,
recursion_layers=tuple([*recursion_layers, xref])
)
continue
if Name.SMask in image:
# Ignore soft masks
smask_xref = Xref(image.SMask.objgen[0])
exclude_xrefs.add(smask_xref)
log.debug(f"xref {smask_xref}: skipping image because it is an SMask")
include_xrefs.add(xref)
log.debug(f"xref {xref}: treating as an optimization candidate")
if xref not in pageno_for_xref:
pageno_for_xref[xref] = pageno
This seems to work with our problematic PDFs.
from ocrmypdf.
@jbarlow83: We found a PDF that could not even be processed with the changes posted above. The new (and so far working) version is this:
def _find_image_xrefs_container(
pdf: Pdf,
container: Object,
pageno: int,
include_xrefs: MutableSet[ocrmypdf.optimize.Xref],
exclude_xrefs: MutableSet[ocrmypdf.optimize.Xref],
pageno_for_xref: dict[ocrmypdf.optimize.Xref, int],
depth: int = 0,
already_processed_xrefs: typing.Optional[MutableSet[ocrmypdf.optimize.Xref]] = None,
strange_xrefs: typing.Optional[MutableSet[ocrmypdf.optimize.Xref]] = None,
):
"""Find all image XRefs or Form XObject and add to the include/exclude sets."""
if depth > 10:
ocrmypdf.optimize.log.warning(
"Recursion depth exceeded in _find_image_xrefs_page"
)
return
try:
xobjs = container.Resources.XObject
except AttributeError:
return
# If no xrefs were passed, an empty set is created.
_already_processed_xrefs: MutableSet[ocrmypdf.optimize.Xref]
if already_processed_xrefs is None:
_already_processed_xrefs = set()
else:
_already_processed_xrefs = already_processed_xrefs
# If no strange xrefs were passed, an empty set is created.
_strange_xrefs: MutableSet[ocrmypdf.optimize.Xref]
if strange_xrefs is None:
_strange_xrefs = set()
else:
_strange_xrefs = strange_xrefs
# This is a mapping of objects' xrefs to the objects' names and objects themself.
to_be_processed_objects = {
ocrmypdf.optimize.Xref(image.objgen[0]): (_imname, image)
for _imname, image in dict(xobjs).items()
}
# Only the xrefs of the to-be-processed objects.
to_be_processed_xrefs: set[ocrmypdf.optimize.Xref] = set(
to_be_processed_objects.keys()
)
# These are the xrefs that have not been processed before.
unprocessed_xrefs = to_be_processed_xrefs - _already_processed_xrefs
# Collecting "strange" xrefs (probably involved in cyclic xrefs?) for later logging.
_strange_xrefs |= to_be_processed_xrefs - unprocessed_xrefs
# This will be passed to further function calls. Thus, we can consider
# these xrefs processed.
to_be_already_processed_xrefs: set[ocrmypdf.optimize.Xref] = (
set(_already_processed_xrefs) | unprocessed_xrefs
)
for xref in unprocessed_xrefs:
_imname, image = to_be_processed_objects[xref]
if image.objgen[1] != 0:
continue # Ignore images in an incremental PDF
if Name.Subtype in image and image.Subtype == Name.Form:
# Recurse into Form XObjects
ocrmypdf.optimize.log.debug(
f"Recursing into Form XObject {_imname} in page {pageno}"
)
_find_image_xrefs_container(
pdf=pdf,
container=image,
pageno=pageno,
include_xrefs=include_xrefs,
exclude_xrefs=exclude_xrefs,
pageno_for_xref=pageno_for_xref,
depth=depth + 1,
already_processed_xrefs=to_be_already_processed_xrefs,
strange_xrefs=_strange_xrefs,
)
continue
if Name.SMask in image:
# Ignore soft masks
smask_xref = ocrmypdf.optimize.Xref(image.SMask.objgen[0])
exclude_xrefs.add(smask_xref)
ocrmypdf.optimize.log.debug(
f"xref {smask_xref}: skipping image because it is an SMask"
)
# Everything that passed the previous filters is relevant for optimization.
include_xrefs.add(xref)
ocrmypdf.optimize.log.debug(
f"xref {xref}: treating as an optimization candidate"
)
if xref not in pageno_for_xref:
pageno_for_xref[xref] = pageno
# Notifying about strange xrefs.
if depth == 0 and _strange_xrefs:
ocrmypdf.optimize.log.warning(
f"{len(_strange_xrefs)} xrefs on page {pageno} were found to be kind of strange."
)
As it turns out, finding suitable variable names is even harder when writing recursive functions. :)
from ocrmypdf.
Related Issues (20)
- [Bug]: crashes with tesseract 5.4.0 HOT 8
- [Bug]: ocrmypdf 16.3.1 fails on a file on Arch that 13.4.0 on Ubuntu handles well HOT 1
- [Feature]: Alternative AI OCR "surya" as opposed to EasyOCR, Just found it today and it dominated the accuracy and speed of Tesseract & EasyOCR HOT 3
- [Bug]: Paperless-ngx Release 2.9.0 Ghostscript rasterizing failed HOT 1
- [Bug]: MetadataProgress does not respect progress_bar=False argument
- [Bug]: No errors and no output for large DPI files HOT 2
- [Bug]: `lots of diacritics - possibly poor OCR` but using standalone tesseract works perfectly HOT 1
- [Bug]: ocrmypdf (16.3.1) and Tesseract 5.4.1 HOT 3
- [Bug]: Existing text is completely replaced with other characters HOT 3
- [Request]: Please make rich logging library an optional dependency HOT 1
- [Feature]: Enable execution on GPU HOT 1
- [Bug]: doesn't always parse Latin with diacritics HOT 3
- Output file images are corrupted HOT 1
- [Bug]: OSError: [Errno 28] No space left on device HOT 4
- [Bug]: problem with tif "DPI is not credible". Estimate dpi HOT 3
- [Bug]: Ghostscript can't create a PDF/A-file (Page object was reserved for an Annotation destination) HOT 3
- [Bug]: KeyError: '/Subtype'
- [Bug]: Ghostscript rasterizing failed HOT 3
- [Bug]: files signed with a-trust are not recognised as digitally signed and hence processed HOT 1
- --sidecar writes text content and messages to file HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ocrmypdf.