Data corruption while reading Kmall data with version 0 #MWC datagrams about mb-system

[dwcaress]

Aaron, Can you make some sample files available? It looks like the Jason data package I got through Bill Chadwick does not include the Jason EM2040 data. Certainly there is a bug - the intent is fix the data to be consistent with the current version and to only write out records in the current version. Thanks, Dave

…

On Nov 5, 2020, at 4:52 PM, Aaron Marburg ***@***.***> wrote: I'm working with *.kmall files collected with ROV Jason this summer. mbinfo segfaults on some, but not all of the files. With some investigation, I found the data in the EMdgmMWCrxBeamData_def struct within the MWC packet was being corrupted for beams >= 1 within the packet. This leads to struct justification problems, incorrect calculation of the size of data following the struct, and from there occasional reads beyond the end of the buffer, causing the segfault. I traced the problem to this line in the mbr_kemkmall_rd_mwc function: https://github.com/dwcaress/MB-System/blob/01db042a51646480a33a83c197e1471255960b23/src/mbio/mbr_kemkmall.c#L2403 ThedetectedRangeInSamplesHighResolution field was added to the struct in Version 1 of the datagram. The code correctly handles the version 0 case (not reading the extra field) for the 0'th beam, then manually overwrites the flag in the datagram header and mis-reads all subsequent beams. The question is whether this is a simple bug or if overwriting the version number has some greater purpose elsewhere in the system. I am working with master branch built from source, on Ubuntu 18.04. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

---------------------------------------------------- David W. Caress Principal Engineer Seafloor Mapping Lab Monterey Bay Aquarium Research Institute 7700 Sandholdt Road Moss Landing, CA 95039 [email protected] http://www.mbari.org/~caress/ Phone: [phone]

[amarburg]

Hi Dave,

I've posted two test snippets to Val's Python module which contain version 0 MWC packets -- the {Hi,Low}ResPhase files here: https://github.com/valschmidt/kmall/tree/master/data

They don't segfault, but:

> mbinfo -VVVVV -I data/0004_20200917_014959_HiResPhase_subset.kmall 2>&1 | grep -n -A6  "MWC receiver beam data"

45421:dbg5       #MWC receiver beam data  0/256:
45422-dbg5       tiltAngleReTx_deg:                     30.000000
45423-dbg5       startRangeSampleNum:                   0
45424-dbg5       detectedRangeInSamples:                0
45425-dbg5       beamTxSectorNum:                       0
45426-dbg5       numSampleData:                         3596
45427-dbg5       detectedRangeInSamplesHighResolution:  0.000000
--
45828:dbg5       #MWC receiver beam data  1/256:
45829-dbg5       tiltAngleReTx_deg:                     29.764708
45830-dbg5       startRangeSampleNum:                   0
45831-dbg5       detectedRangeInSamples:                0
45832-dbg5       beamTxSectorNum:                       0
45833-dbg5       numSampleData:                         3596
45834-dbg5       detectedRangeInSamplesHighResolution:  -0.000000
--
46235:dbg5       #MWC receiver beam data  2/256:
46236-dbg5       tiltAngleReTx_deg:                     0.000000
46237-dbg5       startRangeSampleNum:                   0
46238-dbg5       detectedRangeInSamples:                3596
46239-dbg5       beamTxSectorNum:                       32896
46240-dbg5       numSampleData:                         32896
46241-dbg5       detectedRangeInSamplesHighResolution:  -0.000000
--
49572:dbg5       #MWC receiver beam data  3/256:
49573-dbg5       tiltAngleReTx_deg:                     -7100336854532096.000000
49574-dbg5       startRangeSampleNum:                   56805
49575-dbg5       detectedRangeInSamples:                57056
49576-dbg5       beamTxSectorNum:                       57816
49577-dbg5       numSampleData:                         55771
49578-dbg5       detectedRangeInSamplesHighResolution:  -1831.182129
--
55197:dbg5       #MWC receiver beam data  4/256:
55198-dbg5       tiltAngleReTx_deg:                     8520064000.000000
55199-dbg5       startRangeSampleNum:                   25367
55200-dbg5       detectedRangeInSamples:                59671
55201-dbg5       beamTxSectorNum:                       12551
55202-dbg5       numSampleData:                         29961
55203-dbg5       detectedRangeInSamplesHighResolution:  -0.000000

...

The tiltAngleReTx_deg field is clearly going off the rails. and the other fields are random.

If the suspect line of code is commented out:

> mbinfo -VVVVV -I data/0004_20200917_014959_HiResPhase_subset.kmall 2>&1 | grep -n -A6  "MWC receiver beam data"

45421:dbg5       #MWC receiver beam data  0/256:
45422-dbg5       tiltAngleReTx_deg:                     30.000000
45423-dbg5       startRangeSampleNum:                   0
45424-dbg5       detectedRangeInSamples:                0
45425-dbg5       beamTxSectorNum:                       0
45426-dbg5       numSampleData:                         3596
45427-dbg5       detectedRangeInSamplesHighResolution:  0.000000
--
45828:dbg5       #MWC receiver beam data  1/256:
45829-dbg5       tiltAngleReTx_deg:                     29.764708
45830-dbg5       startRangeSampleNum:                   0
45831-dbg5       detectedRangeInSamples:                0
45832-dbg5       beamTxSectorNum:                       0
45833-dbg5       numSampleData:                         3596
45834-dbg5       detectedRangeInSamplesHighResolution:  0.000000
--
46235:dbg5       #MWC receiver beam data  2/256:
46236-dbg5       tiltAngleReTx_deg:                     29.529415
46237-dbg5       startRangeSampleNum:                   0
46238-dbg5       detectedRangeInSamples:                0
46239-dbg5       beamTxSectorNum:                       0
46240-dbg5       numSampleData:                         3596
46241-dbg5       detectedRangeInSamplesHighResolution:  0.000000
--
46642:dbg5       #MWC receiver beam data  3/256:
46643-dbg5       tiltAngleReTx_deg:                     29.294121
46644-dbg5       startRangeSampleNum:                   0
46645-dbg5       detectedRangeInSamples:                0
46646-dbg5       beamTxSectorNum:                       0
46647-dbg5       numSampleData:                         3596
46648-dbg5       detectedRangeInSamplesHighResolution:  0.000000
--
47049:dbg5       #MWC receiver beam data  4/256:
47050-dbg5       tiltAngleReTx_deg:                     29.058828
47051-dbg5       startRangeSampleNum:                   0
47052-dbg5       detectedRangeInSamples:                0
47053-dbg5       beamTxSectorNum:                       0
47054-dbg5       numSampleData:                         3596
47055-dbg5       detectedRangeInSamplesHighResolution:  0.000000

...

[dwcaress]

Aaron, Thanks for the thorough debugging - you are correct that I was resetting the datagram version number prematurely, and thus messing up the reading of subsequent beam data. I just committed a few updates to the master branch, including what I hope is a fix for the problem you identified. I have not made a new release yet. Cheers, Dave

…

On Nov 5, 2020, at 8:20 PM, Aaron Marburg ***@***.***> wrote: Hi Dave, I've posted two snippets to Val's Python module which contain version 0 MWC packets -- the {Hi,Low}ResPhase files here: https://github.com/valschmidt/kmall/tree/master/data They don't segfault, but: > mbinfo -VVVVV -I data/0004_20200917_014959_HiResPhase_subset.kmall 2>&1 | grep -n -A6 "MWC receiver beam data" 45421:dbg5 #MWC receiver beam data 0/256: 45422-dbg5 tiltAngleReTx_deg: 30.000000 45423-dbg5 startRangeSampleNum: 0 45424-dbg5 detectedRangeInSamples: 0 45425-dbg5 beamTxSectorNum: 0 45426-dbg5 numSampleData: 3596 45427-dbg5 detectedRangeInSamplesHighResolution: 0.000000 -- 45828:dbg5 #MWC receiver beam data 1/256: 45829-dbg5 tiltAngleReTx_deg: 29.764708 45830-dbg5 startRangeSampleNum: 0 45831-dbg5 detectedRangeInSamples: 0 45832-dbg5 beamTxSectorNum: 0 45833-dbg5 numSampleData: 3596 45834-dbg5 detectedRangeInSamplesHighResolution: -0.000000 -- 46235:dbg5 #MWC receiver beam data 2/256: 46236-dbg5 tiltAngleReTx_deg: 0.000000 46237-dbg5 startRangeSampleNum: 0 46238-dbg5 detectedRangeInSamples: 3596 46239-dbg5 beamTxSectorNum: 32896 46240-dbg5 numSampleData: 32896 46241-dbg5 detectedRangeInSamplesHighResolution: -0.000000 -- 49572:dbg5 #MWC receiver beam data 3/256: 49573-dbg5 tiltAngleReTx_deg: -7100336854532096.000000 49574-dbg5 startRangeSampleNum: 56805 49575-dbg5 detectedRangeInSamples: 57056 49576-dbg5 beamTxSectorNum: 57816 49577-dbg5 numSampleData: 55771 49578-dbg5 detectedRangeInSamplesHighResolution: -1831.182129 -- 55197:dbg5 #MWC receiver beam data 4/256: 55198-dbg5 tiltAngleReTx_deg: 8520064000.000000 55199-dbg5 startRangeSampleNum: 25367 55200-dbg5 detectedRangeInSamples: 59671 55201-dbg5 beamTxSectorNum: 12551 55202-dbg5 numSampleData: 29961 55203-dbg5 detectedRangeInSamplesHighResolution: -0.000000 The tiltAngleReTx_deg field is clearly going off the rails. and the other fields are random. If the suspect line of code is commented out: 45421:dbg5 #MWC receiver beam data 0/256: 45422-dbg5 tiltAngleReTx_deg: 30.000000 45423-dbg5 startRangeSampleNum: 0 45424-dbg5 detectedRangeInSamples: 0 45425-dbg5 beamTxSectorNum: 0 45426-dbg5 numSampleData: 3596 45427-dbg5 detectedRangeInSamplesHighResolution: 0.000000 -- 45828:dbg5 #MWC receiver beam data 1/256: 45829-dbg5 tiltAngleReTx_deg: 29.764708 45830-dbg5 startRangeSampleNum: 0 45831-dbg5 detectedRangeInSamples: 0 45832-dbg5 beamTxSectorNum: 0 45833-dbg5 numSampleData: 3596 45834-dbg5 detectedRangeInSamplesHighResolution: 0.000000 -- 46235:dbg5 #MWC receiver beam data 2/256: 46236-dbg5 tiltAngleReTx_deg: 29.529415 46237-dbg5 startRangeSampleNum: 0 46238-dbg5 detectedRangeInSamples: 0 46239-dbg5 beamTxSectorNum: 0 46240-dbg5 numSampleData: 3596 46241-dbg5 detectedRangeInSamplesHighResolution: 0.000000 -- 46642:dbg5 #MWC receiver beam data 3/256: 46643-dbg5 tiltAngleReTx_deg: 29.294121 46644-dbg5 startRangeSampleNum: 0 46645-dbg5 detectedRangeInSamples: 0 46646-dbg5 beamTxSectorNum: 0 46647-dbg5 numSampleData: 3596 46648-dbg5 detectedRangeInSamplesHighResolution: 0.000000 -- 47049:dbg5 #MWC receiver beam data 4/256: 47050-dbg5 tiltAngleReTx_deg: 29.058828 47051-dbg5 startRangeSampleNum: 0 47052-dbg5 detectedRangeInSamples: 0 47053-dbg5 beamTxSectorNum: 0 47054-dbg5 numSampleData: 3596 47055-dbg5 detectedRangeInSamplesHighResolution: 0.000000 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

---------------------------------------------------- David W. Caress Principal Engineer Seafloor Mapping Lab Monterey Bay Aquarium Research Institute 7700 Sandholdt Road Moss Landing, CA 95039 [email protected] http://www.mbari.org/~caress/ Phone: [phone]

[amarburg]

Dave, cheers. Just confirming that HEAD of master is now working with my data.