Corrupt IIIF images at certain zoom levels about iipsrv

[ruven]

Yes, please email them to me

[illtud]

Images sent to your sourceforge address.

[ruven]

I've tried out your images on both Linux and Mac and it works fine in both cases. Are you sure you have used the latest iipsrv source code? Otherwise, perhaps these images is being cached if you're using memcached, for example.

[illtud]

Thanks Ruven . Instructions were for the build to be from a fresh github pull last week, so I assume we're running latest code. We've already seen memcache problems with hosting v1 IIIF & v2 IIIF iipserver binaries on the same server (v1 server serves up v2 IIIF via cache) but I was expecting that any previous versions of the image had long fallen out of the 24 hour cache. I'll disable memcached to check. If it's really that simple then I'll apologise for wasting your time!

[illtud]

No, doesn't seem to be either memcache nor internal cache - restarted lighttpd & memcached, two requests for the 'dodgy leg' - one seems to be generated from scratch, second is from memcache, both produce the same dodgy image. I'll investigate further.

Setting maximum CVT size to 3000
Setting 3D file sequence name pattern to '_pyr_'
Setting max quality layers (for supported file formats) to all layers
Setting up JPEG2000 support via Kakadu SDK
Memcached support enabled. Connected to servers: 'localhost' with timeout 86400


Initialisation Complete.
<----------------------------------->

Full Request is IIIF=/mnt/SCIF2/newspapers/derivatives/apnau/jp2/apnau01300594.jp2/1024,2048,512,512/256,/0/native.jpg
[1/1]: Command / Argument is IIIF : /mnt/SCIF2/newspapers/derivatives/apnau/jp2/apnau01300594.jp2/1024,2048,512,512/256,/0/native.jpg
IIIF handler reached
IIIF :: URL decoded to /mnt/SCIF2/newspapers/derivatives/apnau/jp2/apnau01300594.jp2/1024,2048,512,512/256,/0/native.jpg
FIF handler reached
FIF :: URL decoding/filtering: /mnt/SCIF2/newspapers/derivatives/apnau/jp2/apnau01300594.jp2 => /mnt/SCIF2/newspapers/derivatives/apnau/jp2/apnau01300594.jp2
FIF :: Image cache initialization
FIF :: JPEG2000 image detected
FIF :: Created image
FIF :: Image dimensions are 3590 x 4409
FIF :: Image contains 1 channels with 8 bits per channel
FIF :: Image timestamp: Mon, 15 Oct 2012 11:36:29 GMT
FIF :: Total command time 4855 microseconds
IIIF :: Requested Region: x:1024, y:2048, w:512, h:512
IIIF :: Requested Size: 256x256
IIIF :: Requested Rotation: 0 degrees
IIIF :: Requested Quality: native with format: jpg
IIIF :: image request for /mnt/SCIF2/newspapers/derivatives/apnau/jp2/apnau01300594.jp2 with arguments: region: 1024,2048,512,512; size: 256x256; rotation: 0; mirroring: 0
CVT handler reached
CVT :: Using resolution 4 with size 1795x2204
CVT :: Region: 512,1024,256,256
CVT :: Requested scaled region size is 256x256 at resolution 4. Nearest existing resolution is 256x256
TileManager getRegion :: requesting region directly from image
CVT :: About to JPEG compress strip with height 128
CVT :: Compressed data strip length is 49304
CVT :: About to JPEG compress strip with height 128
CVT :: Compressed data strip length is 47053
CVT :: Total command time 35513 microseconds
IIIF :: Total command time 40888 microseconds
Memcached :: stored 96566 bytes in 1165 microseconds
Total Request Time: 44144 microseconds
image closed and deleted
Server count is 1

Full Request is IIIF=/mnt/SCIF2/newspapers/derivatives/apnau/jp2/apnau01300594.jp2/1024,2048,512,512/256,/0/native.jpg
Memcached hit
Total Request Time: 655 microseconds
image closed and deleted
Server count is 2

[illtud]

code seems to be latest (I don't really know git, but git status -uno says nothing). Here's an ldd of the binary - anything significantly different to yours?

[root@]# ldd iipiiifsrv5.fcgi 
    linux-vdso.so.1 => /usr/local/src/v7_1-01151L/apps/make/libkdu_v71R.so
    libnsl.so.1 => /lib64/libnsl.so.1 
    libmemcached.so.2 => /usr/lib64/libmemcached.so.2
    libpthread.so.0 => /lib64/libpthread.so.0
    libfcgi.so.0 => /usr/lib64/libfcgi.so.0 
    libjpeg.so.62 => /usr/lib64/libjpeg.so.62 
    libtiff.so.3 => /usr/lib64/libtiff.so.3 
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 
    libm.so.6 => /lib64/libm.so.6
    libgomp.so.1 => /usr/lib64/libgomp.so.1
    libgcc_s.so.1 => /lib64/libgcc_s.so.1
    libc.so.6 => /lib64/libc.so.6 
    /lib64/ld-linux-x86-64.so.2 
    libz.so.1 => /lib64/libz.so.1 
    librt.so.1 => /lib64/librt.so.1

[offsets removed as I'm paranoid]

I'll try and build locally and test the images again.

[illtud]

New build on same server, using an earlier version of kakadu, in case that was a problem - /usr/local/src/v7_0-01151L//apps/make/libkdu_v70R.so
Still the same.

I'll try building on another machine altogether.

[burtoncp]

Hi Ruven.

I've taken a look at this with Illtud and have found that corruption in the leg image only occurs if "MAX_LAYERS" => "-1" is set in the lighttpd configuration.

With the second image the corruption only occurs if "JPEG_QUALITY" => "100" is set in the lighttpd configuration. Reducing this to any value lower than 100 will produce an uncorrupted image.

Hope this helps with replicating the problem.

[ruven]

OK, the problem should now be fixed. It was basically because your images are far bigger than the raw data when encoded to JPEG at very high quality levels.

Setting the JPEG quality factor to 100 isn't really a good idea as you don't really get much extra image quality while creating far larger file sizes. An optimal trade-off between size and high quality is a factor of around 90 or so. Beyond that, you are consuming large numbers of bytes for little gain in quality.