Expand Basic Metadata extracted for Files, in specific PDF and Clean EXIF History about strawberryfield

[alliomeria]

Ok, still confused about this. @alliomeria may know better.

@DiegoPino I know not better, only that there is/can be considerable variance in EXIF metadata, even for supposedly ISO-standard-compliant PDFs. :)

Question is: Do we de-dup?, do we simply strip from EXIF a list of offenders? I mean i love the idea of indexing in Solr the Colorswatches, but its a lot, like really too much?

@DiegoPino De-duping could be helpful for those history change redundancies, but also don't think it'd hurt much to let be (has a human-readable "this document was changed" vibe). I agree that while cool, the SwatchColorantXX sprawl is too much for Solr. Feel that the need for Solr indexing for this would be an extremely niche (perhaps interesting?) situation/use case.

Speaking of use cases, regarding the PDF info extraction of every page, can certainly see this being important and useful for complex displays as you mention. Also thinking of objects with IIIF annotations down the road, wondering if this could be relevant for potential instances of multipage PDFs with annotations throughout.

[alliomeria]

@DiegoPino (from my humble perspective) The second choice seems the most practical and straightforward. For this choice, how slow is slow (or too slow)?

[DiegoPino]

@alliomeria i agree. It is significantly slower in my local machine (5 seconds for a PDF on identify v/s 400 ms using pdfinfo), but then Docker runs very slow on OSX when doing native File system access as we do in our deployment and Imagemagick is heavier because it is really getting way more info but we are just formatting what we need back.

Pushed the code here now https://github.com/esmero/strawberryfield/tree/ISSUE-101
(need to rebase to i can merge into 1.0.0-RC1)

And the result is this:

 "urn:uuid:ac7fc929-a45e-4abc-9ff4-a6c35ec16c2f": {
            "url": "s3:\/\/7d6\/application-001-ac7fc929-a45e-4abc-9ff4-a6c35ec16c2f.pdf",
            "name": "a3pagepdf.pdf",
            "tags": [],
            "type": "Document",
            "dr:fid": 2357,
            "dr:for": "documents",
            "dr:uuid": "ac7fc929-a45e-4abc-9ff4-a6c35ec16c2f",
            "checksum": "7d644b3e545f8efc2704cafb97c34ffc",
            "flv:exif": {
                "Title": "\/root\/images\/a3pagepdf.pdf",
                "FileSize": "45 MB",
                "MIMEType": "application\/pdf",
                "Producer": "ImageMagick 6.2.4 11\/01\/06 Q16 http:\/\/www.imagemagick.org",
                "PageCount": 3,
                "CreateDate": "2007:01:05 12:08:37",
                "Linearized": "No",
                "ModifyDate": "2007:01:05 12:08:37",
                "PDFVersion": 1.3,
                "EmbeddedImageWidth": 1935,
                "EmbeddedImageFilter": "ASCII85Decode",
                "EmbeddedImageHeight": 3077,
                "EmbeddedImageColorSpace": [
                    "DeviceRGB"
                ]
            },
            "sequence": 2,
            "flv:pronom": {
                "label": "Acrobat PDF 1.3 - Portable Document Format",
                "mimetype": "application\/pdf",
                "pronom_id": "info:pronom\/fmt\/17",
                "detection_type": "signature"
            },
            "dr:mimetype": "application\/pdf",
            "crypHashFunc": "md5",
            "flv:identify": {
                "1": {
                    "width": "1935",
                    "format": "PDF",
                    "height": "3077",
                    "orientation": "Undefined"
                },
                "2": {
                    "width": "2030",
                    "format": "PDF",
                    "height": "1617",
                    "orientation": "Undefined"
                },
                "3": {
                    "width": "2028",
                    "format": "PDF",
                    "height": "1605",
                    "orientation": "Undefined"
                }
            }
        }
    },

now inside flv:identify we get each page characterized with its own size. Means we can now use that data while building the IIIF manifest via the twig template.

[DiegoPino]

Short note: "orientation" here allows me to read back metadata hints on rotation. Since those can be on EXIF and/or TIFF (XMP) tags, orientation can keep track/extract both options transparently.

That can then be used in Cantaloupe to force a certain rotation (RE: the NYSPERSONALHISTORY use case where images are rotated in the phone but cantaloupe fails to acknowledge)

[DiegoPino]

Thanks @alliomeria
So i have 3 choices now
1.- Usign PDFInfo and extracting first page numbers and then PDFInfo again (command shared) Media boxes for every page

pdfinfo 3pagesofdifferentsizes.pdf -f 1 -l 3 -box 
Title:     /root/images2/HJM01_000176/HJM01_000176_001.pdf
Producer:    ImageMagick 6.2.4 11/01/06 Q16 http://www.imagemagick.org
CreationDate:  Fri Jan 5 07:08:37 2007 EST
ModDate:    Fri Jan 5 07:08:37 2007 EST
Tagged:     no
UserProperties: no
Suspects:    no
Form:      none
JavaScript:   no
Pages:     3
Encrypted:   no
Page  1 size: 348 x 554 pts
Page  1 rot: 0
Page  2 size: 365 x 291 pts
Page  2 rot: 0
Page  3 size: 365 x 289 pts
Page  3 rot: 0
Page  1 MediaBox:   0.00   0.00 1935.00 3077.00
Page  1 CropBox:   0.00   0.00  348.00  554.00
Page  1 BleedBox:   0.00   0.00  348.00  554.00
Page  1 TrimBox:   0.00   0.00  348.00  554.00
Page  1 ArtBox:    0.00   0.00  348.00  554.00
Page  2 MediaBox:   0.00   0.00 2030.00 1617.00
Page  2 CropBox:   0.00   0.00  365.00  291.00
Page  2 BleedBox:   0.00   0.00  365.00  291.00
Page  2 TrimBox:   0.00   0.00  365.00  291.00
Page  2 ArtBox:    0.00   0.00  365.00  291.00
Page  3 MediaBox:   0.00   0.00 2028.00 1605.00
Page  3 CropBox:   0.00   0.00  365.00  289.00
Page  3 BleedBox:   0.00   0.00  365.00  289.00
Page  3 TrimBox:   0.00   0.00  365.00  289.00
Page  3 ArtBox:    0.00   0.00  365.00  289.00
File size:   47303172 bytes
Optimized:   no
PDF version:  1.3 (edited) 

Second choice is the somewhat slower identify and gives me some format options so i can split and built back

identify -format 'format:%m|width:%w|height:%h|exif_orientation:%[EXIF:Orientation]@' -quiet 3pagesofdifferentsizes.pdf

format:PDF|width:1935|height:3077|exif_orientation:@format:PDF|width:2030|height:1617|exif_orientation:@format:PDF|width:2028|height:1605|exif_orientation:@ 

And the third one is using actually Drupal directly and its wrapper. Issue is even if the machinery is quite sophisticated its also limiting and easy to break everything when doing normal changes any CMS site builder could do.

It would imply Drupal Imagemagick and Drupal FMD

Any option implies just changes for now in SBF module and in a single file but makes me believe its about time we move to SBF Runners starting from 1.0.1

[DiegoPino]

I don't think touching Technical metadata is an archipelago concern. We might do more harm if we attempt this. EXIF is also a moving target. Closing as won't implement