jstockwin / py-pdf-parser Goto Github PK
View Code? Open in Web Editor NEWA Python tool to help extracting information from structured PDFs.
Home Page: https://py-pdf-parser.readthedocs.io/en/latest/
License: MIT License
A Python tool to help extracting information from structured PDFs.
Home Page: https://py-pdf-parser.readthedocs.io/en/latest/
License: MIT License
It would be nice if we could draw large box outlines around elements with each section, to make it easy to see where each section is.
This is a little bit tricky, as sometimes the shape will have to be more complicated than a square, for example:
The problem should be something similar to finding the minimum shape containing all the elements in the section. The problem could be slightly more complicated than the above example if the elements are all over the place, but sections always contain continuous groups of elements so it should be okay.
One issue could be that I think at the moment we order based on the centre of an element. Thus if one element is much smaller than another there could be some overlap which shouldn't be there. I think we will have to cope with it not being perfect, but we should try to do pretty well as we don't want it to be misleading.
Note this is the visualise tool so we're not too interested in performance.
Hi @jstockwin ,
I am trying to load 3 similar files which is auto generated from same system, but get 3 different prefix of font.
>>> set(e.font for e in doc.elements)
{'OHKPGR+SourceHanSerifCN-Regular,10.5', 'Helvetica,12.0', 'OHKPGR+SourceHanSerifCN-Regular,26.0', 'OHKPGR+SourceHanSerifCN-Regular,9.0', 'OHKPGR+SourceHanSerifCN-Regular,12.0', 'OHKPGR+SourceHanSerifCN-Regular,8.5', 'OHKPGR+SourceHanSerifCN-Regular,16.0', 'OHKPGR+SourceHanSerifCN-Regular,14.0'}
>>> set(e.font for e in doc2.elements)
{'Helvetica,12.0', 'FZVVCB+SourceHanSerifCN-Regular,8.5', 'FZVVCB+SourceHanSerifCN-Regular,12.0', 'FZVVCB+SourceHanSerifCN-Regular,26.0', 'FZVVCB+SourceHanSerifCN-Regular,14.0', 'FZVVCB+SourceHanSerifCN-Regular,10.5', 'FZVVCB+SourceHanSerifCN-Regular,16.0', 'FZVVCB+SourceHanSerifCN-Regular,9.0'}
>>> set(e.font for e in doc3.elements)
{'PCAGML+SourceHanSerifCN-Regular,16.0', 'PCAGML+SourceHanSerifCN-Regular,14.0', 'PCAGML+SourceHanSerifCN-Regular,26.0', 'PCAGML+SourceHanSerifCN-Regular,12.0', 'PCAGML+SourceHanSerifCN-Regular,8.5', 'PCAGML+SourceHanSerifCN-Regular,9.0', 'Helvetica,12.0', 'PCAGML+SourceHanSerifCN-Regular,10.5'}
What is the prefix:'PCAGML' of font:"PCAGML+SourceHanSerifCN-Regular,16.0"?
And how to use FONT_MAPPING over these diff?
Thanks in advance.
It was pointed out in #100 (comment) that perhaps our documentation is a bit negative, and we could paint the tool as a bit more useful than we do.
@forhonourlx has kindly made a suggestion in #100 (comment), which I'll adapt slightly and add to the documentation.
You can pass show_info=True
to the visualise tool, and this allows you to click on elements and see details etc.
It is unfinished and needs work.
We currently call .height
. However, from pdfminer/pdfminer.six#202 it looks as though LTChar
has a size
attribute. We should use this instead.
That said, we should check the PDFMiner code and how it deals with precision etc.
Note: This is a potential breaking change if the font sizes end up changing slightly.
ignore
property should be renamed to ignored
.ignore()
method, which sets ignore
to True
.ignore_elements
on an ElementList
.For performance, we should keep a set of ignored_indexes
on the document
. A PDFElement
should have a reference to its PDFDocument
so it can check if it's ignored.
Ignored elements will be excluded from all lists, this should be done on __init__
of the ElementList
s, and can be done fast as you can just do - self.document.ignored_indexes
. The ignore_elements
on an ElementList
can also be efficient, rather than calling ignore()
on each of it's contained elements.
Some of our guessed defaults were wrong. See the implementation here.
There is a sort in the table extraction which sorts based on y0
. This must also sort by page number, as currently rows are returned in an incorrect order if the table goes over multiple pages.
Currently tests are running on our private jenkins, which isn't good if we want to open source (we do!), so let's use github actions instead!
It is often the case that if a table goes over a page break then the header is repeated on the new page.
Even though it's not strictly a pdf parsing thing, it might be nice to add a util to handle this case (similar to the fact we handle adding the header to the table even though that's not strictly pdf parsing).
Essentially, if the header row repeats then it should be removed. Essentially I think all rows would be checked against the header row, and if they match both the text and the font, it should be removed from the table. We'd have to keep track of the removed rows sometimes so that the checks pass (we have checks to ensure the correct number of elements were detected).
There should be a parameter to enable this behaviour and it should default to False
.
E.g. by text, by font
It would be good to have some example use cases in the documentation somewhere.
PDFMiner.six has a layout parameter, all_texts
, which, if set to True
, will also perform layout analysis on text within figures.
Doing this in py pdf parser does nothing, since we only look at text boxes. We should also include text from figures when all_texts=True
.
It seems to be a common case that you know the element at which you want to end a section, but you don't want that element to be included (e.g. because this is the start of the next section). We should add an include_last_element=True
) argument to create_section
. Note that this can simply subtract one from the index of the element and then get the new element from the document.
We could also add include_start_element
, but I think this is less useful so maybe we won't bother until it feels like a necessary feature.
We have docstrings, but it would be good to have some proper (mostly auto-generated) documentation
Bug Report
extract_table
re-orders the table rows by the y
axis (top to bottom), which works for most cases.
The issue comes if we have a table with a header which is below any of the other elements of the table, when we have a table in a page split by 2 columns for example:
In the above case, even if element_ordering
is properly set in load
to adjust to the page split, extract_table
would return:
[["C", "D"], ["E", "F"],["HEADER 1", "HEADER 2"], ["A", "B"]]
Should we make extract_table
obey the ordering on which the document elements are defined? Or should we add some sort of rows_sort
and columns_sort
options to the function?
Currently we have
extract_table
which takes an ElementList
and returns a list of list of PDFElement
s.extract_text_from_table
which takes an ElementList
and returns a list of list of strings._extract_text_from_table
which takes a list of list of PDFElement
s and returns a list of list of strings.We have a use case where we want to play with the "table" of PDFElement
s and then extract the text (i.e. we want to call _extract_text_from_table
). Additionally, the extract_text_from_table
name is confusing because it reads as though it expects a table whereas it actually wants an ElementList
.
We should have:
extract_table(as_text=True)
which takes an ElementList
and returns a table (list of list) of either PDFElement
s or strings depending on the as_text
argument. It will also have to take strip=False
after #21 is implemented.get_text_from_table
which takes a table (i.e. list of list of PDFElement
s) and returns a list of list of stringsHi @jstockwin ๏ผ
I am coming with some suggestions from my recent practice, would you please take a look? Thanks in advance.
ElementList.between()
method, with inclusive
param, users are only able to select elements like (e_1, ...e_n)
or [e_1, ..., e_n]
. In first case, I want to select elements like (e_1, ...e_n]
or [e_1, ..., e_n)
(i.e.). In further case, I want to select elements like (e_1-3, ...e_n+4)
, which requires elements outside the 'between' elements, maybe also requires before
, after
slices.The following is some demo of my actual usage without offset and resize, which troubles me.
def locate_contents_between_locs(self, loc_pair_text,
loc_pair_equal_or_contain=['equal','equal'],
loc_pair_idx=[0,0],
loc_pair_offset=[0,0],
elements_scope=None,
output_type=list,
if_print_contents=False,
):
if elements_scope == None:
elements = self.doc.elements
else:
elements = elements_scope
loc_pair = [None, None]
for _ in range(2):
if loc_pair_equal_or_contain[_] =='equal':
loc_pair[_] = elements.filter_by_text_equal(loc_pair_text[_])[loc_pair_idx[_]]
elif loc_pair_equal_or_contain[_] =='contain':
loc_pair[_] = elements.filter_by_text_contains(loc_pair_text[_])[loc_pair_idx[_]]
if loc_pair_offset[_] > 0:#inclusive [0, 1, 2...
loc_pair[_] = elements.after(loc_pair[_], inclusive=True)[loc_pair_offset[_]]
elif loc_pair_offset[_] < 0:#exclusive ...-2, -1]
loc_pair[_] = elements.before(loc_pair[_], inclusive=False)[loc_pair_offset[_]]
contents = elements.between(loc_pair[0], loc_pair[1])
if if_print_contents:
print(f'{contents}:{[c.text() for c in contents]}')
return contents
#73 introduced a bug whereby ALL elements which were cached by font were returned, rather than only the ones for that font.
#69 Added code to show section outlines. Because we wanted to be 100% correct in which elements were within the outline, the construction is quite complicated and can result in strangely shaped outlines. The code isn't too slow, but it's also not super fast.
In the case where a simple rectangle can be drawn around the section (which should be reasonably trivial to check), this should be done.
At the moment, elements are ordered left to right, top to bottom.
This should be configurable, which would allow us to handle:
(1) right to left, top to bottom,
(2) pages which are landscape and so come through as vertical,
(3) PDFs with e.g. two columns of content (slightly more complicated).
We should have an argument which has some presets, but also can be a function so you can pass in your own.
We don't currently have a nice __repr__
for the section class. It would be nice if this e.g. said the name and the number of elements or something similar.
Bug Report
Before submitting an issue, please ensure you have read our CONTRIBUTING.md
, and follow the Code of Conduct.
Thanks for taking the time to report a bug. To help us fix it quickly, please include the following information:
Unable to install via pip as show in the documentation. Running pip3 install py-pdf-parser
I get the error code
ERROR: Command errored out with exit status 1:
command: /home/hank/Development/pdf_parse/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-j1z0oskj/wand/setup.py'"'"'; __file__='"'"'/tmp/pip-install-j1z0oskj/wand/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-j1z0oskj/wand/pip-egg-info
cwd: /tmp/pip-install-j1z0oskj/wand/
Complete output (18 lines):
Traceback (most recent call last):
File "/tmp/pip-install-j1z0oskj/wand/wand/api.py", line 180, in <module>
libraries = load_library()
File "/tmp/pip-install-j1z0oskj/wand/wand/api.py", line 135, in load_library
raise IOError('cannot find library; tried paths: ' + repr(tried_paths))
OSError: cannot find library; tried paths: []
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-j1z0oskj/wand/setup.py", line 8, in <module>
from wand.version import VERSION
File "/tmp/pip-install-j1z0oskj/wand/wand/version.py", line 45, in <module>
from .api import libmagick, library
File "/tmp/pip-install-j1z0oskj/wand/wand/api.py", line 198, in <module>
distname, _, __ = platform.linux_distribution()
AttributeError: module 'platform' has no attribute 'linux_distribution'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
I am trying to install on KDE Neon and using that and the dev version of py-pdf-parser give me the same error. Though it seems to be working on a Windows boot.
sectioning
should provide something similar to .sections
which allows you to filter for sections with a specific (non-unique) name.
PDFMiner has changed the way it gets the font size since this code was written. I think initially the heights had lots of decimal places etc.
We've just seen a PDF where the heights are now e.g. 7.5
. Due to some precision issue (?) this is sometimes rounded up to 8
or down to 7
, for different elements with exactly the same font.
We should switch from using int
for the font to using float
instead. We should also check multiple PDFs to decide if we want to add any rounding, or whether they all seem reasonable.
We allow passing a tolerance
parameter to the horizontally_in_line_with
and vertically_in_line_with
functions. However, if you specify a tolerance which is larger than the width/height of the element, this instantiates an invalid bounding box with either x0 > x1
or y0 > y1
.
We should cap the tolerance at the width/height of the element.
#89 should have had an example added to the documentation. We already have an example that runs through a variety of tables, including ones which go over the page. We should highlight this new option there.
We currently use an image for our own ECR, but should use a public image.
We tried this but it threw up issues with PyQt5 - these need fixing.
Related -> Currently pyqt5 and matplotlib are installed in the dockerfile using apt, but should be added to setup.py
.
There should be an easy way to add a tag to a group of elements
Currently the loaders aren't tested. We should add a (small) real PDF document to the tests and check that it can be loaded using our loaders.
We have some cases (in this case in a table) where elements only just overlap along a certain axis, and we don't want to include these in e.g. "vertically in line with", since they barely touch.
We've decided to add a tolerance
parameter which defaults to 0, but allows you to specify the extent to which elements much overlap along the relevant axis to be considered in line with.
There are some new "high level" functions in PDFMiner, for example extract_pages.
We should be able to use these in our loaders, which should save a few lines of code interacting with PDFMiner.
There is one issue, which is that the high level functions require the file to be on your device, and so whilst we could change load_file(path_to_file: str, ...)
, we will not yet be able to change load(pdf_file: IO, ...)
. I've created an upstream issue for this here, which I'll also submit a fix for. This issue should probably wait until I've closed the upstream issue.
A lot of tables simply have a few gaps, but have at least one row and one column which are complete.
Currently in this case we have to use extract_table
which is much slower than extract_simple_table
(I've not done the maths but extract_simple_table
should be ~n^2
whereas extract_table
is at least n^3
, possibly n^3
.
We should allow gaps in the table for extract_simple_table
, provided there is complete row and column.
Bug Report
Please also check that your bug is not actually caused by pdfminer.six, and is really an issue with this project.
example:
from py_pdf_parser.loaders import load_file
from py_pdf_parser.visualise import visualise
document = load_file("1907-1912_RESULTS.pdf")
visualise(document)
Error:
Traceback (most recent call last):
File "C:\Users\andrewp\AppData\Roaming\Python\Python38\site-packages\wand\api.py", line 180, in <module>
libraries = load_library()
File "C:\Users\andrewp\AppData\Roaming\Python\Python38\site-packages\wand\api.py", line 135, in load_library
raise IOError('cannot find library; tried paths: ' + repr(tried_paths))
OSError: cannot find library; tried paths: [~/CORE_RL_wand_.dll, etc.]
Problem: imagemagic v7+ no longer includes the searched-for dll files.
Resolution:
Uninstall most recent imagemagic (7.0.10-34) and install the latest legacy imagemagic (6.9.11-34)
Solution:
Provide compatibility with v7+ or note that one should install the Legacy version.
Instead of a property element.text
we should have a method element.text(strip=True)
. This means text gets stripped by default, but you can stop this behaviour if you want.
Our filtering currently uses Python's frozensets. This is mainly to be 100% sure we're not editing old element lists etc by applying additional operations. Performance-wise, frozensets are the same as sets.
Calling __getitem__
on an element list means we need to sort the frozenset, which we do by calling sorted. Additionally #113 adds some new functions which requite sorting.
We could use sorted sets (e.g. http://www.grantjenks.com/docs/sortedcontainers/introduction.html#sorted-set). I believe we could just override the functions which allow them to be mutated to be sure we're not doing this. This should give us some performance gain since we won't need to keep explicitly sorting the elements. This should be checked.
We should also check that there is no performance loss when doing set operations on a sorted set vs a set. We also need to e.g. add and remove elements from the set, but not by mutating the set. This might require some copies, which could slow things down?
Pros:
Cons:
Thoughts/ideas on this appreciated.
This is implemented as a dictionary access on the sections dict, which means if you filter for a (unique) section name (i.e. including the _idx
) then you get a KeyError
. Filtering for a non-existent section should probably handle this exception and return an empty section list.
It might be worth adding a get_section
method which raises a SectionNotFoundError
instead of a KeyError
when given a unique section name which doesn't exist.
We always do e.g. self.current_page += 1
, however there may be pages 'missing' from the document if some were blank. This can cause crashes etc.
It's easy to accidentally pass an ElementList
as the start/end element, for example, and then the error thrown is AttributeError: 'ElementList' object has no attribute '_index'
. This is not very clear and should be improved.
Filtering by font occurs frequently, and involves checking the font of each element.
It could be worth caching the filtered elements for each font somewhere on the document. This shouldn't be done at load, only when the function for each font is called.
Some things are mocked a lot, for example most of the output from PDFMiner.
We should run some full examples of extracting things from a simple PDF.
At the moment we're looping through the reference row and then the reference column, and computing the element which is in line with both the reference row element and the reference col element.
This means we're doing to geometry checks to work out each row len(cols)
times, and the checks to work out each col len(rows)
times.
We should do the geometry filtering at the start to create a list of rows and a list of cols. We can then just &
the two to get the element.
This should require much less processing as you only do geometry checks len(rows) + len(cols)
times, rather then len(rows) * len(cols)
times.
I noticed that by setting boxes_flow
outside the documented range, you can actually disable PDFMiner's advanced layout analysis.
We don't need the advanced analysis since we have no hierarchy of text boxes and we order them ourselves, and it's quite a performance gain to leave these out.
I've filed an issue (and fix) to update the documentation and also allow boxes_flow
to be passed as None
to explicitly disable this: pdfminer/pdfminer.six#395
Once that's merged, we should either default or hard-code our boxes_flow
la param to None
. It feels like we should allow it to be overridden, but equally since we ignore the resulting analysis perhaps there's no point and we should hard-code it to None
.
We should publish to PyPI to allow for pip3 install py-pdf-parser
.
Once this is done, we should update the installation instructions on the documentation.
We should add some code coverage checks to the CI.
Initially this will help us find untested areas (note the visualise tool is currently untested as we're not really sure how to go about testing it...).
Moving forwards, it will help catch and PRs which don't add sufficient tests.
The most basic thing to add would be filter_by_regular_expression
(or maybe simply filter_by_regex
). I don't think there's any need to add anything which allows you to provide multiple regular expressions as you can just construct a longer single regex to do this.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.