Pixel classification performance issues with batch processing about qupath

[petebankhead]

Yeah, I think it was me that introduced that script :) #130 (comment)

There's an option Run → Clear cache (batch processing) built-in now that should do the same thing:

qupath/qupath-gui-fx/src/main/java/qupath/lib/gui/scripting/DefaultScriptEditor.java

Lines 1681 to 1687 in 533ce0c

	if (clearCache.get()) {
	try {
	var store = qupath == null ? null : qupath.getImageRegionStore();
	if (store != null)
	store.clearCache();
	System.gc();
	} catch (Exception e) {

I think it's off by default, since emptying the catch is often not welcome. But I still don't have a very clear idea why it continues to prove useful :/

[yli-hallila]

I had 730 slides with 10 different stains, so I ran them in batches of 73 slides and each of them had the same issue of slowing down significantly after the first few slides. I think I faced an OutOfMemoryError once or twice which led me into finding and using that script, which resolved at least running out of memory (or just "fixed" it that it never happened with my batch size anymore) and might have sped up the process marginally but not to the same speed as running the script without batching.

Below hopefully are the relevant parts of the script and classifiers.

createAnnotationsFromPixelClassifier("ASMA Classifier", 100000.0, 200000.0, "DELETE_EXISTING")

"pixel_classifier_type": "OpenCVPixelClassifier",
     ...
"pixelWidth": {
  "value": 8.048,
  "unit": "µm"
},
"pixelHeight": {
  "value": 8.048,
  "unit": "µm"
}
    ...
"inputWidth": 512,
"inputHeight": 512,

I can try running the analysis sometime and see if I can reproduce the OutOfMemoryError or see any patterns with and without the snippet.

[yli-hallila]

I noticed when running object / pixel classifiers in batches that QuPath would not always clear up the tile cache properly (or that's at least what I thought?) and would occasionally end up running out of memory / being sluggish due to limited available memory.

I added something similar to the snippet below to my scripts and it partially resolved my problems if I recall correctly -- probably worth investigating if nothing else comes up (although my issues were possibly different than the issues brought on image.sc).

Thread.sleep(100)

javafx.application.Platform.runLater {
    getCurrentViewer().getImageRegionStore().cache.clear()
    System.gc()
}

Thread.sleep(100)

[petebankhead]

@yli-hallila did you spot any patterns when it happened?

Like, was it a high-resolution pixel classifier, and did the memory use accumulate over time?
Would it happen after 3 images, 30 images, or 300 images...?

I'm hoping that there is some place where I'm just failing to release memory used by OpenCV / JavaCPP, but I thought I'd tracked down where that was the case.

The script does two things: clears the cache, and calls the garbage collector.

If the thing that helps is clearing the cache, then that suggests to me that it is delaying (rather than solving) the problem - and there is probably still a leak somewhere.

If the thing that helps is running the garbage collector then... I guess I'm confused, because I think that should happen before an out-of-memory error occurs anyway.

I quite like hunting for leaks (as long as they are in code I can control), but here I'm not sure where exactly I should start.

[petebankhead]

Excellent, thanks. That looks like a fairly low resolution... do you remember if you used a large number of features in the classifier, or just stuck with the defaults?

And do you happen to know how much RAM you had...?

[yli-hallila]

I tried reproducing the error a few times with that classifier with 4 GB of RAM and I wasn't able to reproduce it. I bumped up the resolution to 1,01 µm/px and two OutOfMemoryError came up with only one stacktrace (this happened without any GC or cache clearing; didn't try with it enabled either). This was running on 0.3.2, I can give it a try on newer versions if any relevant parts have changed in the code.

I've attached the whole classifier below (it's just a simple thersholder to filter out the background). I think the original analysis was done with 6 GB or 8 GB of RAM but I also had other scripts with qupath.imagej.detect.cells.WatershedCellDetection running and I can't say for 100% certainty whether it was the Pixel Classifier, Cell Counting or both which caused OutOfMemoryErrors.

ERROR: OutOfMemoryError: Java heap space

ERROR: qupath.opencv.tools.OpenCVTools.matToBufferedImage(OpenCVTools.java:765)
    qupath.opencv.ml.pixel.OpenCVPixelClassifier.applyClassification(OpenCVPixelClassifier.java:115)
    qupath.lib.classifiers.pixel.PixelClassificationImageServer.readTile(PixelClassificationImageServer.java:299)
    qupath.lib.images.servers.AbstractTileableImageServer.lambda$getTile$0(AbstractTileableImageServer.java:213)
    qupath.lib.images.servers.AbstractTileableImageServer$$Lambda$1691/0x0000000800796740.call(Unknown Source)
    java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    qupath.lib.images.servers.AbstractTileableImageServer.getTile(AbstractTileableImageServer.java:217)
    qupath.lib.images.servers.AbstractTileableImageServer.readBufferedImage(AbstractTileableImageServer.java:287)
    qupath.lib.images.servers.AbstractTileableImageServer.readBufferedImage(AbstractTileableImageServer.java:60)
    qupath.lib.analysis.images.ContourTracing.traceGeometriesImpl(ContourTracing.java:1157)
    qupath.lib.analysis.images.ContourTracing.traceGeometries(ContourTracing.java:1143)
    qupath.lib.analysis.images.ContourTracing.lambda$traceGeometriesImpl$9(ContourTracing.java:1022)
    qupath.lib.analysis.images.ContourTracing$$Lambda$2595/0x00000008009b7a18.apply(Unknown Source)
    qupath.lib.analysis.images.ContourTracing.lambda$invokeAll$8(ContourTracing.java:1000)
    qupath.lib.analysis.images.ContourTracing$$Lambda$2596/0x00000008009b7c58.call(Unknown Source)
    java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    java.base/java.lang.Thread.run(Unknown Source)

{
  "pixel_classifier_type": "OpenCVPixelClassifier",
  "metadata": {
    "inputPadding": 0,
    "inputResolution": {
      "pixelWidth": {
        "value": 1.006,
        "unit": "µm"
      },
      "pixelHeight": {
        "value": 1.006,
        "unit": "µm"
      },
      "zSpacing": {
        "value": 1.0,
        "unit": "z-slice"
      },
      "timeUnit": "SECONDS",
      "timepoints": []
    },
    "inputWidth": 512,
    "inputHeight": 512,
    "inputNumChannels": 3,
    "outputType": "CLASSIFICATION",
    "outputChannels": [],
    "classificationLabels": {
      "0": {
        "name": "Area",
        "colorRGB": -6895466
      },
      "1": {
        "colorRGB": -12566464
      }
    }
  },
  "op": {
    "type": "data.op.channels",
    "colorTransforms": [
      {
        "channelName": "Red"
      }
    ],
    "op": {
      "type": "op.core.sequential",
      "ops": [
        {
          "type": "op.gaussian",
          "sigmaX": 4.0,
          "sigmaY": 4.0
        },
        {
          "type": "op.constant",
          "thresholds": [
            239.0
          ]
        }
      ]
    }
  }
}

[petebankhead]

Ah, it might be different in v0.4... but might not.

In any case, this PR should substantially improve memory use and performance for at least some applications of the pixel classifier: #1332

There may still be issues, but I couldn't find any other memory leaks - so I'll close this issue for now. I'll explore again if anything emerges that can be reproduced with the latest code.

Thanks!