Hi, I'm using a FLIR Blackfly BFLY-PGE-50A2C-CS GigE camera with are

I will have a closer look once I find time. <p dir="aut

memory leak using ADSpinnaker and SoftwareTrigger,about areadetector/adspinnaker

Comments (25)

MarkRivers commented on May 28, 2024

Because this camera cannot set the frame rate independently of the exposure time, I have built a work around ...

Are you sure about that? Can you post screen shots of the main ADSpinnaker screen and the camera-specific screens?

the memory consumption of the ioc process starts to continously increase with a rate of around 7 mByte/hour at 1s Scan interval

That works out to about 2 KB per image, which is quite small.

What happens if you use a normal internal trigger and acquire at 10 frames/s. Does the memory usage also increase at 2 KB per image?

from adspinnaker.

MarkRivers commented on May 28, 2024

I just ran ADSpinnaker on Ubuntu 18 with an Oryx 51S5M camera.

What command did you run to get the memory usage you plotted above? With "top" I see this:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
30904 epics     20   0   55316   5024   3544 R   1.0  0.0   0:03.90 top
29077 epics     20   0 7776280 726956  42916 S   0.7  1.5   0:48.55 spinnakerApp

So I am seeing the spinnakerApp using about 7.8 GB of virtual memory, much more than the 95 MB you are seeing. I clicked repeatedly on the TriggerSoftware button and watched the Virtual Memory in "top". It moved up and down a bit, but I did not see any upward trend.

from adspinnaker.

rettigl commented on May 28, 2024

Hi Mark,
thanks for your questions.

Because this camera cannot set the frame rate independently of the exposure time, I have built a work around ...

Are you sure about that? Can you post screen shots of the main ADSpinnaker screen and the camera-specific screens?

The options to do that are there, but the camera does not want to do that.

This is the main screen of the ADSpinnaker for my camera. If I set ExposureAuto to Off, and Frame Rate Enable to On, and then set a FrameRate, I get an error in the Epics shell:
epics> 2021/05/19 08:55:05.510 Param[GC_FRAMERATE] GenICamFeature::write: node AcquisitionFrameRate is not writable
I also cannot change the frame rate in the independently in the FLIR software, so I think it is a limitation of the camera.

That works out to about 2 KB per image, which is quite small.

I think it is not related to the actual size of the image, because it does not change if I change the binning or region size, as far as I can tell.

What happens if you use a normal internal trigger and acquire at 10 frames/s. Does the memory usage also increase at 2 KB per image?

Do you just mean free running? That's what I did in the example above. Actually, in all the example above the Trigger was actually off, and the camera was free running at 130 Hz (with a small region size as you can see in the screen shot). If I do not process the SoftwareTrigger PV, the memory stays ~constant.

What command did you run to get the memory usage you plotted above? With "top" I see this:

I used a profiling bash snipped based on ps, which records the "resident set size", i.e. the non-swapped memory. This corresponds to the RES column in top :
logpid() { while sleep 60; do ps -p $1 -o pcpu= -o rss= ; done; }

The virtual memory actually does not increase, it appears.

So I am seeing the spinnakerApp using about 7.8 GB of virtual memory, much more than the 95 MB you are seeing. I clicked repeatedly on the TriggerSoftware button and watched the Virtual Memory in "top". It moved up and down a bit, but I did not see any upward trend.

I am using a fairly small image size (because of a small region size), maybe that makes a difference. The relevant parts of the ioc startup file are these:

# Really large queue so we can stream to disk at full camera speed
epicsEnvSet("QSIZE",  "2000")
# The maximim image width; used for row profiles in the NDPluginStats plugin
epicsEnvSet("XSIZE",  "2592")
# The maximim image height; used for column profiles in the NDPluginStats plugin
epicsEnvSet("YSIZE",  "1944")
# The maximum number of time series points in the NDPluginStats plugin
epicsEnvSet("NCHANS", "2048")
# The maximum number of frames buffered in the NDPluginCircularBuff plugin
epicsEnvSet("CBUFFS", "500")
# Define NELEMENTS to be enough for a 2592x1944x3 (color) image
epicsEnvSet("NELEMENTS", "15116544")
# ADSpinnakerConfig(const char *portName, const char *cameraId, int numSPBuffers,
#                   size_t maxMemory, int priority, int stackSize)
ADSpinnakerConfig("$(PORT)", $(CAMERA_ID),40)
# Main database.  This just loads and modifies ADBase.template
dbLoadRecords("$(ADSPINNAKER)/db/spinnaker.template", "P=$(PREFIX),R=cam1:,PORT=$(PORT)")
# Load the autogenerated file of GenICam features
dbLoadRecords("$(GENICAM_DB_FILE)", "P=$(PREFIX),R=cam1:,PORT=$(PORT)")
# Create a standard arrays plugin
NDStdArraysConfigure("Image1", 5, 0, "$(PORT)", 0, 0)
# Use this line for 8-bit or 16-bit data
dbLoadRecords("$(ADCORE)/db/NDStdArrays.template", "P=$(PREFIX),R=image1:,PORT=Image1,ADDR=0,TIMEOUT=1,NDARRAY_PORT=$(PORT),TYPE=Int16,FTVL=SHORT,NELEMENTS=$(NELEMENTS)")

With the trigger scanning off, I see also with top and initial small increase in memory consumption, and then a flattening at ~95 mbytes (5 sec interval):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
10795 epics     20   0 2754276  90824  38248 S  73.3  2.4   0:16.13 st.cmd
10795 epics     20   0 2754276  91124  38248 S  66.2  2.4   0:19.44 st.cmd
10795 epics     20   0 2754276  91812  38248 S  62.9  2.4   0:22.59 st.cmd
10795 epics     20   0 2754276  92524  38248 S  67.0  2.4   0:25.94 st.cmd
10795 epics     20   0 2746080  92756  38248 S  67.1  2.4   0:29.30 st.cmd
10795 epics     20   0 2754276  92736  38248 S  67.0  2.4   0:32.65 st.cmd
10795 epics     20   0 2746080  92760  38248 S  66.6  2.4   0:35.98 st.cmd
10795 epics     20   0 2754276  92776  38248 S  63.9  2.4   0:39.18 st.cmd
10795 epics     20   0 2754276  92756  38248 S  64.6  2.4   0:42.41 st.cmd
10795 epics     20   0 2746080  92764  38248 S  66.7  2.4   0:45.75 st.cmd
10795 epics     20   0 2754276  93064  38248 S  65.6  2.4   0:49.03 st.cmd
10795 epics     20   0 2746080  93040  38248 S  63.8  2.4   0:52.22 st.cmd
10795 epics     20   0 2754276  93016  38248 S  63.5  2.4   0:55.40 st.cmd
10795 epics     20   0 2754564  93636  38248 S  66.6  2.4   0:58.73 st.cmd
10795 epics     20   0 2746368  93604  38248 S  65.1  2.4   1:01.99 st.cmd
10795 epics     20   0 2754564  93892  38248 S  65.8  2.4   1:05.28 st.cmd
10795 epics     20   0 2746368  93860  38248 S  65.3  2.4   1:08.55 st.cmd
10795 epics     20   0 2754564  94096  38248 S  66.2  2.5   1:11.86 st.cmd
10795 epics     20   0 2754564  94248  38248 S  66.1  2.5   1:15.17 st.cmd
10795 epics     20   0 2746368  94216  38248 S  64.4  2.5   1:18.39 st.cmd
10795 epics     20   0 2754564  94524  38248 S  63.9  2.5   1:21.59 st.cmd
10795 epics     20   0 2746368  94492  38248 S  65.6  2.5   1:24.87 st.cmd
10795 epics     20   0 2754564  94468  38248 S  66.1  2.5   1:28.18 st.cmd
10795 epics     20   0 2754564  94444  38248 S  65.8  2.5   1:31.47 st.cmd
10795 epics     20   0 2746368  94412  38248 S  65.8  2.5   1:34.76 st.cmd
10795 epics     20   0 2754564  94388  38248 S  67.3  2.5   1:38.13 st.cmd
10795 epics     20   0 2746368  94732  38248 S  66.4  2.5   1:41.45 st.cmd
10795 epics     20   0 2754564  94972  38248 S  64.3  2.5   1:44.67 st.cmd
10795 epics     20   0 2754564  94948  38248 S  66.4  2.5   1:47.99 st.cmd
10795 epics     20   0 2746368  94916  38248 S  64.4  2.5   1:51.21 st.cmd
10795 epics     20   0 2754564  94984  38248 S  63.8  2.5   1:54.40 st.cmd
10795 epics     20   0 2746368  94952  38248 S  66.7  2.5   1:57.74 st.cmd
10795 epics     20   0 2754564  95248  38248 S  67.2  2.5   2:01.10 st.cmd
10795 epics     20   0 2754564  95224  38248 S  66.9  2.5   2:04.45 st.cmd
10795 epics     20   0 2746368  95192  38248 S  66.4  2.5   2:07.77 st.cmd
10795 epics     20   0 2754564  95488  38248 S  66.6  2.5   2:11.10 st.cmd
10795 epics     20   0 2746368  95456  38248 S  67.3  2.5   2:14.47 st.cmd
10795 epics     20   0 2754564  95432  38248 S  65.1  2.5   2:17.73 st.cmd
10795 epics     20   0 2746368  95604  38248 S  65.4  2.5   2:21.00 st.cmd
10795 epics     20   0 2754564  95580  38248 S  66.7  2.5   2:24.34 st.cmd
10795 epics     20   0 2754564  95556  38248 S  66.6  2.5   2:27.67 st.cmd
10795 epics     20   0 2746368  95528  38248 S  66.0  2.5   2:30.97 st.cmd
10795 epics     20   0 2754564  95504  38248 S  67.7  2.5   2:34.36 st.cmd
10795 epics     20   0 2746368  95472  38248 S  65.2  2.5   2:37.62 st.cmd
10795 epics     20   0 2754564  95448  38248 S  65.4  2.5   2:40.89 st.cmd
10795 epics     20   0 2754564  95424  38248 S  66.2  2.5   2:44.20 st.cmd
10795 epics     20   0 2746368  95392  38248 S  65.1  2.5   2:47.46 st.cmd
10795 epics     20   0 2754564  95628  38248 S  63.8  2.5   2:50.65 st.cmd
10795 epics     20   0 2746368  95596  38248 S  64.9  2.5   2:53.90 st.cmd
10795 epics     20   0 2754564  95572  38248 S  65.8  2.5   2:57.19 st.cmd
10795 epics     20   0 2754564  95548  38248 S  67.4  2.5   3:00.56 st.cmd
10795 epics     20   0 2746368  95516  38248 S  64.3  2.5   3:03.78 st.cmd
10795 epics     20   0 2754564  95492  38248 S  65.6  2.5   3:07.06 st.cmd
10795 epics     20   0 2746368  95460  38248 S  64.9  2.5   3:10.31 st.cmd
10795 epics     20   0 2754564  95436  38248 S  65.8  2.5   3:13.60 st.cmd
10795 epics     20   0 2754564  95412  38248 S  64.3  2.5   3:16.82 st.cmd
10795 epics     20   0 2754564  95380  38248 S  64.0  2.5   3:20.02 st.cmd
10795 epics     20   0 2754564  95620  38248 S  63.4  2.5   3:23.19 st.cmd
10795 epics     20   0 2746368  95588  38248 S  62.9  2.5   3:26.34 st.cmd
10795 epics     20   0 2754564  95564  38248 S  64.2  2.5   3:29.55 st.cmd
10795 epics     20   0 2746368  95540  38248 S  64.9  2.5   3:32.80 st.cmd
10795 epics     20   0 2754564  95516  38248 S  67.0  2.5   3:36.15 st.cmd
10795 epics     20   0 2754564  95492  38248 S  66.1  2.5   3:39.46 st.cmd
10795 epics     20   0 2746368  95460  38248 S  65.8  2.5   3:42.75 st.cmd
10795 epics     20   0 2754564  95436  38248 S  65.4  2.5   3:46.02 st.cmd
10795 epics     20   0 2754564  95616  38248 S  66.6  2.5   3:49.35 st.cmd
10795 epics     20   0 2746368  95584  38248 S  66.1  2.5   3:52.66 st.cmd
10795 epics     20   0 2754564  95560  38248 S  66.4  2.5   3:55.98 st.cmd
10795 epics     20   0 2746368  95528  38248 S  65.7  2.5   3:59.27 st.cmd
10795 epics     20   0 2754564  95504  38248 S  64.2  2.5   4:02.48 st.cmd
10795 epics     20   0 2754564  95480  38248 S  65.2  2.5   4:05.74 st.cmd
10795 epics     20   0 2746368  95624  38248 S  67.5  2.5   4:09.12 st.cmd
10795 epics     20   0 2754564  95600  38248 S  66.8  2.5   4:12.46 st.cmd
10795 epics     20   0 2746368  95568  38248 S  64.7  2.5   4:15.70 st.cmd

Now I start manually clicking the button 500 times:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
10795 epics     20   0 2746368  95424  38248 S  66.7  2.5   5:56.66 st.cmd
10795 epics     20   0 2754564  95400  38248 S  64.1  2.5   5:59.87 st.cmd
10795 epics     20   0 2754564  95636  38248 S  66.6  2.5   6:03.20 st.cmd
10795 epics     20   0 2746368  95620  38248 S  67.8  2.5   6:06.59 st.cmd
10795 epics     20   0 2754564  95604  38248 S  67.9  2.5   6:09.99 st.cmd
10795 epics     20   0 2746368  95600  38248 S  68.0  2.5   6:13.39 st.cmd
10795 epics     20   0 2754564  95704  38248 S  68.1  2.5   6:16.80 st.cmd
10795 epics     20   0 2754564  95680  38248 S  68.2  2.5   6:20.21 st.cmd
10795 epics     20   0 2754564  95648  38248 S  69.2  2.5   6:23.67 st.cmd
10795 epics     20   0 2754564  95624  38248 S  68.5  2.5   6:27.10 st.cmd
10795 epics     20   0 2746368  95592  38248 S  67.8  2.5   6:30.49 st.cmd
10795 epics     20   0 2754564  95568  38248 S  67.9  2.5   6:33.89 st.cmd
10795 epics     20   0 2746368  95800  38248 S  68.2  2.5   6:37.30 st.cmd
10795 epics     20   0 2754564  95776  38248 S  67.5  2.5   6:40.68 st.cmd
10795 epics     20   0 2754564  95756  38248 S  68.2  2.5   6:44.09 st.cmd
10795 epics     20   0 2746368  95728  38248 S  68.2  2.5   6:47.50 st.cmd
10795 epics     20   0 2754564  95704  38248 S  68.3  2.5   6:50.92 st.cmd
10795 epics     20   0 2746368  95672  38248 S  67.6  2.5   6:54.30 st.cmd
10795 epics     20   0 2754564  95660  38248 S  67.9  2.5   6:57.70 st.cmd
10795 epics     20   0 2754564  95664  38248 S  68.2  2.5   7:01.11 st.cmd
10795 epics     20   0 2746368  95884  38248 S  67.4  2.5   7:04.48 st.cmd
10795 epics     20   0 2754564  95876  38248 S  68.3  2.5   7:07.90 st.cmd
10795 epics     20   0 2746368  95856  38248 S  67.8  2.5   7:11.29 st.cmd
10795 epics     20   0 2754564  95844  38248 S  68.1  2.5   7:14.70 st.cmd
10795 epics     20   0 2754564  96096  38248 S  69.0  2.5   7:18.15 st.cmd
10795 epics     20   0 2746368  96080  38248 S  68.2  2.5   7:21.56 st.cmd
10795 epics     20   0 2754564  96072  38248 S  67.9  2.5   7:24.96 st.cmd
10795 epics     20   0 2746368  96220  38248 S  68.2  2.5   7:28.37 st.cmd
10795 epics     20   0 2754564  96208  38248 S  67.9  2.5   7:31.77 st.cmd
10795 epics     20   0 2754564  96208  38248 S  67.6  2.5   7:35.15 st.cmd
10795 epics     20   0 2754564  96260  38248 S  68.3  2.5   7:38.57 st.cmd

That's an increase of about 800 kbyte in RES, no change in VIRT.

I repeat the same with scan at 0.1 s:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
10795 epics     20   0 2754564  96488  38248 S  68.4  2.5   9:01.11 st.cmd
10795 epics     20   0 2746368  96456  38248 S  66.5  2.5   9:04.44 st.cmd
10795 epics     20   0 2754564  96440  38248 S  70.2  2.5   9:07.95 st.cmd
10795 epics     20   0 2746368  96520  38248 S  70.4  2.5   9:11.47 st.cmd
10795 epics     20   0 2754564  96768  38248 S  69.7  2.5   9:14.96 st.cmd
10795 epics     20   0 2754564  96824  38248 S  70.0  2.5   9:18.46 st.cmd
10795 epics     20   0 2754564  96792  38248 S  69.7  2.5   9:21.95 st.cmd
10795 epics     20   0 2754564  96752  38248 S  66.2  2.5   9:25.26 st.cmd
10795 epics     20   0 2746368  96724  38248 S  68.0  2.5   9:28.66 st.cmd
10795 epics     20   0 2754564  96700  38248 S  69.7  2.5   9:32.15 st.cmd
10795 epics     20   0 2746368  96668  38248 S  70.0  2.5   9:35.65 st.cmd
10795 epics     20   0 2754564  96644  38248 S  69.7  2.5   9:39.14 st.cmd
10795 epics     20   0 2754564  96848  38248 S  70.2  2.5   9:42.65 st.cmd
10795 epics     20   0 2746368  96812  38248 S  69.2  2.5   9:46.11 st.cmd
10795 epics     20   0 2754564  96828  38248 S  70.1  2.5   9:49.62 st.cmd
10795 epics     20   0 2746368  96796  38248 S  69.6  2.5   9:53.10 st.cmd
10795 epics     20   0 2754564  96820  38248 S  70.5  2.5   9:56.63 st.cmd
10795 epics     20   0 2754564  96832  38248 S  69.6  2.5  10:00.11 st.cmd
10795 epics     20   0 2746368  96808  38248 S  70.5  2.5  10:03.64 st.cmd
10795 epics     20   0 2754564  97148  38248 S  69.6  2.5  10:07.12 st.cmd
10795 epics     20   0 2746368  97612  38248 S  69.7  2.5  10:10.61 st.cmd
10795 epics     20   0 2754564  97668  38248 S  69.6  2.5  10:14.09 st.cmd
10795 epics     20   0 2754564  97644  38248 S  69.6  2.5  10:17.57 st.cmd
10795 epics     20   0 2746368  97700  38248 S  69.9  2.5  10:21.07 st.cmd
10795 epics     20   0 2754564  97676  38248 S  70.0  2.5  10:24.57 st.cmd
10795 epics     20   0 2746368  97644  38248 S  69.9  2.5  10:28.07 st.cmd
10795 epics     20   0 2754564  97620  38248 S  70.2  2.5  10:31.58 st.cmd
10795 epics     20   0 2754564  97596  38248 S  69.2  2.5  10:35.04 st.cmd
10795 epics     20   0 2746368  97564  38248 S  69.7  2.5  10:38.53 st.cmd
10795 epics     20   0 2754564  97800  38248 S  70.0  2.6  10:42.03 st.cmd
10795 epics     20   0 2746368  97768  38248 S  69.3  2.6  10:45.50 st.cmd
10795 epics     20   0 2754564  97780  38248 S  69.0  2.6  10:48.95 st.cmd
10795 epics     20   0 2754564  97772  38248 S  70.6  2.6  10:52.48 st.cmd
10795 epics     20   0 2754564  97776  38248 S  70.3  2.6  10:56.00 st.cmd
10795 epics     20   0 2754564  97768  38248 S  69.6  2.6  10:59.48 st.cmd
10795 epics     20   0 2746368  97748  38248 S  70.3  2.6  11:03.00 st.cmd
10795 epics     20   0 2754564  97740  38248 S  69.8  2.6  11:06.49 st.cmd

RES continuously grows, VIRT is not affected, as far as I can tell.

from adspinnaker.

MarkRivers commented on May 28, 2024

epics> 2021/05/19 08:55:05.510 Param[GC_FRAMERATE] GenICamFeature::write: node AcquisitionFrameRate is not writable
I also cannot change the frame rate in the independently in the FLIR software, so I think it is a limitation of the camera.

I think that your problem is that some FLIR cameras have a GenICam feature called AcquisitionFrameRateAuto. I suspect that feature is set to On on your camera. In that case you cannot control the frame rate. Look at the second medm screen shot on this page for an example with the BFLY-PGE-20E4C camera: https://areadetector.github.io/master/ADGenICam/ADGenICam.html. Note that AcquisitionFrameRateAuto is set to Off on that screen. That is what you need to do.

RES continuously grows, VIRT is not affected, as far as I can tell.

Then that is not a memory leak. Memory leaks cause VIRT to grow, because that is the total memory that has been allocated but not freed. RES is resident memory. The OS controls that to optimize performance, and so it can grow as new memory locations are written to.

from adspinnaker.

rettigl commented on May 28, 2024

I think that your problem is that some FLIR cameras have a GenICam feature called AcquisitionFrameRateAuto. I suspect that feature is set to On on your camera. In that case you cannot control the frame rate. Look at the second medm screen shot on this page for an example with the BFLY-PGE-20E4C camera: https://areadetector.github.io/master/ADGenICam/ADGenICam.html. Note that AcquisitionFrameRateAuto is set to Off on that screen. That is what you need to do.
Yes, you are right that there is this AquisitionFramRateAuto setting.

This indeed allows some control, but still not what I need. Here is what I observe:

If I set AquisitionFramRateAuto to off and AquisitionFramRateEnable to on, and also Exposure auto to off, I can control expsoure time and Frame rate independently, but, with the following limitations: Maximal exposure time is capped at 66 ms, and minimal frame rate at 15.148 Hz:

These settings do not produce errors in the ioc shell, the numbers are just capped.
If I set AquisitionFramRateAuto to continous, and AquisitionFramRateEnable to off, I can increase the exposure time above 66ms:

As I need this range of expsoures, this is the mode I have been running before, and used the trigger to control the frame rate to around 1 Hz, independent of acquisition time.

Then that is not a memory leak. Memory leaks cause VIRT to grow, because that is the total memory that has been allocated but not freed. RES is resident memory. The OS controls that to optimize performance, and so it can grow as new memory locations are written to.

RES is the actual physical memory used by the program, no? So whatever it is, it's causing the IOC to use more and more physical memory, and the computer with 3-4 of these iocs running for several days went out of memory and started to show huge load and became ineccessible, that's how I became aware of the problem. So definitely something I need to solve if I want to keep using this approach. And so far, I don't see a different one with my camera.

from adspinnaker.

MarkRivers commented on May 28, 2024

If I set AquisitionFramRateAuto to off and AquisitionFramRateEnable to on, and also Exposure auto to off, I can control expsoure time and Frame rate independently, but, with the following limitations: Maximal exposure time is capped at 66 ms, and minimal frame rate at 15.148 Hz:

Are you sure that you are setting the parameters in the correct order? For example if FrameRate=20 then you cannot set the exposure time above 0.05. But if you first decrease the frame rate to 2 then you should be able to increase the exposure time to 0.5. I just tested this on a Blackfly camera with ADSpinnaker and it worked fine.

You had not previously reported that RES kept increasing forever and was causing a problem. That does seem incorrect. I will test and see if I can reproduce the problem.

from adspinnaker.

MarkRivers commented on May 28, 2024

I ran 2 tests with my BlackFly camera, 1600x1200 pixels, Mono8 mode.

TriggerMode=Off, ImageMode=Multiple, NumImages=1000. RES increased by 20 during the run (20 KB), which is the noise in the values when nothing is happening.
TriggerMode=On, TriggerSource=Software, ImageMode=Multiple, NumImages=1000, script wrote 1 to TriggerSoftware 1000 times. RES increased by 1776 (1.8 MB).

So I can reproduce the problem, RES does not increase when using the internal frame rate, but it does increase when using the software trigger.

There is really no code in ADSpinnaker that does anything when the TriggerSoftware record is processed. It just writes to the TriggerSoftware GenICam feature, and then everything is done in the Spinnaker library. I don't see any place there can be a memory leak in the areaDetector code.

I suggest running the FLIR viewer and seeing whether RES also increases when you use SoftwareTrigger with that.

from adspinnaker.

rettigl commented on May 28, 2024

Hi Mark,
thanks again for taking this on. Without finding time yet to do more tests myself, I had a glance at the release notes of the Spinnaker SDK. In version 2.3.0.77, they mention a fixed issue:

Memory usage keeps increasing when executing Single image acquisition in SpinView
And version 2.1.0.95 mentions:
memory leak with DeviceEvent

The ADSpinnaker currently used 2.0.0.147, no?
so this could be related to the problem. I will have a closer look once I find time.

from adspinnaker.

MarkRivers commented on May 28, 2024

I will have a closer look once I find time.

I would suggest also trying to see if you can make FrameRate and AcquireTime work with normal triggered acquisition. I am able to control those over the full range on a BlackFly as long as I change them in the correct order.

from adspinnaker.

MarkRivers commented on May 28, 2024

I agree that updating the Spinnaker SDK is the first thing to try.

from adspinnaker.

rettigl commented on May 28, 2024

Are you sure that you are setting the parameters in the correct order? For example if FrameRate=20 then you cannot set the exposure time above 0.05. But if you first decrease the frame rate to 2 then you should be able to increase the exposure time to 0.5. I just tested this on a Blackfly camera with ADSpinnaker and it worked fine.

I tried again what you suggested. AquisitionFramRateAuto=0, AquisitionFramRateEnable=1
Now, when I try to set the frame rate to e.g. 10 Hz, it will set it to 15.148 Hz. See my screenshot above. No matter what I do.
And I cannot increase the Exposure time above 0.066 s.
What I can reproduce is that if I set frame rate e.g. to 40 Hz, then the highest exposure time is 0.02 s. But I think the limitation in Frame rate comes from the camera model.

from adspinnaker.

rettigl commented on May 28, 2024

Just changing the libraries to the new version in the ioc did not do the trick, apparently they changed something in the SDK. It builds, but produces the following error when running:

ADSpinnakerConfig("SP1", 16119088,40)
c++ error: Spinnaker: System instance cannot be acquired. Could not load producer. Make sure that the environment variable FLIR_GENTL64_CTI exists, and points to the location of the file FLIR_GenTL.cti [-1012]

Just setting this variable to the file copied from the respective package also does not help yet...

from adspinnaker.

MarkRivers commented on May 28, 2024

I have created a new spinnaker2.4 branch in this repository. It contains the files from the latest version of the Spinnaker SDK, 2.4.0.143. It builds and runs fine for me on Ubuntu 18.

I tested first acquiring 1000 frames in TriggerMode=Off, 0.1 second acquisition.

When	VIRT	RES
Before	7708904	317868
After	7708904	317868

There was no increase in VIRT or RES during this acquisition.

I then repeated the testing using TriggerMode=On, TriggerSource=Software, and an IDL script that wrote 1 to the TriggerSoftware PV 1000 times.

When	VIRT	RES
Before	7708904	318048
After	7708904	320508

RES has increased by 2460 KB in 1000 images, or about 2.5 KB per image. This is the same as with the previous version of Spinnaker, 2.0.0.147. So the new version of Spinnaker does not seem to have fixed the problem.

from adspinnaker.

MarkRivers commented on May 28, 2024

Run on Windows, SDK 2.4.0.144, Trigger=Off.

Before

Name	PID	Status	User name	CPU	Memory (private working set)	Commit size	Description
spinnakerApp.exe	11632	Running	rivers	00 	264,676 K	303,500 K	SpinnakerApp

After

Name	PID	Status	User name	CPU	Memory (private working set)	Commit size	Description
spinnakerApp.exe	11632	Running	rivers	00 	264,636 K	303,444 K	SpinnakerApp

So there is not increase in private working set of Commit size when TriggerMode=Off.

I then repeated the testing using TriggerMode=On, TriggerSource=Software, and an IDL script that wrote 1 to the TriggerSoftware PV 1000 times.

Before

Name	PID	Status	User name	CPU	Memory (private working set)	Commit size	Description
spinnakerApp.exe	11632	Running	rivers	01 	264,636 K	303,436 K	SpinnakerApp

After

Name	PID	Status	User name	CPU	Memory (private working set)	Commit size	Description
spinnakerApp.exe	11632	Running	rivers	00 	266,348 K	305,184 K	SpinnakerApp

So both the private working set and the Commit size increased by about 1700 KB for 1000 images, or about 1.7 KB per image.

It seems that this problem occurs on both Linux and Windows.

from adspinnaker.

rettigl commented on May 28, 2024

I've tested with a small program derived from the BufferHandling.cpp delivered with the SDK/also in ADSpinnaker.
I configured it to just trigger and read one camera 1000x, and here I saw no memory increase during the read loop, just some small increase before.

I also could not reproduce the problem with the spinView QT application. So maybe the problem is not inside the SDK after all?
Here is the code I used:
TriggerDebug.zip

from adspinnaker.

rettigl commented on May 28, 2024

I tested the behavior with ADAravis, and observed the same thing. So it appears likely to me this is something rooted in the ADGenICam code, rather than in the code of the SDK.

from adspinnaker.

MarkRivers commented on May 28, 2024

I tested the behavior with ADAravis, and observed the same thing.

That's what I was going to test next, so thanks for doing it,

I have done another test which I believe proves that the issue has nothing to do with the TriggerSoftware PV, or even with image acquisition at all. I wrote a script which continuously writes 0.05 to the AcquireTime PV at about 100Hz, when TriggerMode=Off and acquisition is stopped. I see a growth in RES that continues as long as the script is running. The ReadStatus.SCAN PV was Passive when doing this.

I can also reproduce the problem by simply setting ReadStatus.SCAN to 0.1 second. I see RES increase by about 256 KB every few 10s of seconds.

This looks like a memory leak in the code that reads GenICam feature values.

from adspinnaker.

MarkRivers commented on May 28, 2024

I believe I have fixed the problem. The problem occurred with GenICam enum features. The main problem was that epicsStrDup was being called when it did not need to be, and the duplicated string was never freed. I also fixed an issue where unneeded enum callbacks were being done, and I made the code internally consistent regarding when an enum feature is set to have a single "N.A." state, i.e. when it is not implemented, or is not available, or is not readable.

I have pushed the fix to the master branch.

from adspinnaker.

MarkRivers commented on May 28, 2024

I have merged the spinnaker2.4 branch into master, so it is now using SDK version 2.4.0.143 (Linux) and 2.4.0.144 (Windows).

from adspinnaker.

rettigl commented on May 28, 2024

I believe I have fixed the problem. The problem occurred with GenICam enum features. The main problem was that epicsStrDup was being called when it did not need to be, and the duplicated string was never freed. I also fixed an issue where unneeded enum callbacks were being done, and I made the code internally consistent regarding when an enum feature is set to have a single "N.A." state, i.e. when it is not implemented, or is not available, or is not readable.

I have pushed the fix to the master branch.

Great, thanks for looking into this. I will test this and let you know.

I have merged the spinnaker2.4 branch into master, so it is now using SDK version 2.4.0.143 (Linux) and 2.4.0.144 (Windows).

I compiled the new 2.4 version of the ADSpinnaker on linux, but still could not get it to run with my camera. I get a segfault:

# ADSpinnakerConfig(const char *portName, const char *cameraId, int numSPBuffers,
#                   size_t maxMemory, int priority, int stackSize)
ADSpinnakerConfig("SP1", 16119088,1)
Segmentation fault (core dumped)

(independent on the stackSize)

from adspinnaker.

MarkRivers commented on May 28, 2024

ADSpinnakerConfig("SP1", 16119088,1)

In that command you are setting numSPBuffers=1. Try removing the last argument completely, which will then use the default of 100 buffers.

If it still crashes then run the IOC under gdb

gdb ../../bin/linux-x86_64/spinnakerApp
run st.cmd
(wait to get the segfault)
backtrace

from adspinnaker.

MarkRivers commented on May 28, 2024

@rettigl be sure to do the following when you rebuild the new release of ADSpinnaker:

make clean uninstall
make -sj

from adspinnaker.

MarkRivers commented on May 28, 2024

I forgot to mention that the fix for the memory leak is not in ADSpinnaker, it is in ADGenICam. You also need to get the master branch of ADGenICam and build that before you build ADSpinnaker.

from adspinnaker.

rettigl commented on May 28, 2024

@rettigl be sure to do the following when you rebuild the new release of ADSpinnaker:
make clean uninstall
make -sj

Thanks, that apparently did the trick. I did a "make clean" before, but not uninstall. Did not know this was necessary...
Now it runs.

In that command you are setting numSPBuffers=1. Try removing the last argument completely, which will then use the default of 100 buffers.

With my camera, I experienced that the default of 100 Framebuffers does not work. Depending on settings, only values of 20-40 or so work:

ADSpinnakerConfig("SP1", 16119088)
2021/05/26 15:58:12.649 ADSpinnaker::connectCamera exception Spinnaker: GenICam::OutOfRangeException= Value = 100 must be equal or smaller than Max = 26. : OutOfRangeException thrown in node 'StreamBufferCountManual' while calling 'StreamBufferCountManual.SetValue()' (file 'IntegerT.h', line 79) [-2002]
2021/05/26 15:58:12.649 ADSpinnaker:ADSpinnaker:  camera connection failed (3)

I forgot to mention that the fix for the memory leak is not in ADSpinnaker, it is in ADGenICam. You also need to get the master branch of ADGenICam and build that before you build ADSpinnaker.

Thanks, this I surely did before, however also without uninstall, but seemed to have worked.

The memory consumtion looks good now, this is in the triggered mode I used before:

I suppose we can close this then.

from adspinnaker.

MarkRivers commented on May 28, 2024

I did a "make clean" before, but not uninstall. Did not know this was necessary...

"uninstall" is not normally necessary. But in this case the vendor spinnaker library in spinnakerSupport/os/linux-x86_64/ uses the normal Linux soft links. When gnumake installs the files the soft links are lost. gnumake was probably confused about the file dates, and so it did not install the new spinnaker library when it should have. "uninstall" deletes the ADSpinnaker/lib directory and that fixes it.

from adspinnaker.

memory leak using ADSpinnaker and SoftwareTrigger about adspinnaker HOT 25 CLOSED

Comments (25)

Related Issues (10)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent