oceanmapping / community Goto Github PK

From the EM / SIS Troubleshooting Notes:

(EX2101, SIS v5.6.0.441) In acoustically soft sediments, lower-frequency systems like the EM304 and EM124 may suffer from a bottom detection artifact near nadir known as ‘Erik’s Horns’ or ‘railroad tracks’ because of the appearance in the grid.

Summarized from EM122 SIS 4 machine troubleshooting by scientists aboard R/V Marcus G. Langseth (July 2023).

Context:
EM122 operating with SIS 4 on Windows machine. All indicators are green and EM122 is mapping as normal.

Symptoms:

1. SIS 4 machine is not receiving surface sound speed or sending center beam depth datagram (DPT) to other systems over serial connection.
2. SV is coming through to other systems / loggers.
3. COM ports not showing up in 'External Sensors' setup screen ('normal' screenshot below).

Troubleshooting:

1. Restarted EM122 TRU and SIS machine.
2. Device Manager did not show the expected COM ports on the SIS machine.

Solution:

1. Turned off the SIS machine and reseated the serial card and all serial connections.
2. Restarted SIS machine and the COM ports showed up normally.

SSV Workaround:
While the surface sound speed sensor was not available, the EM122 was configured to use the 'Profile' value (Runtime Parameters --> Sound Speed --> Sound Speed at Transducer).

Julianna Diehl alerted the MAC on 2021/10/26 to feedback from a PI about intermittent missing starboard coverage on an EM122 in the UNOLS fleet, appearing to occur only for the second swath in dual-swath mode.

This behavior was intermittent, as shown by the 'rejected' flags (yellow) on two swath coverage test lines:

Context and symptoms: see attached PDF.

EM122 missing second swath RX data.pdf

Solution: replacing RX and BSP boards after discussion with Kongsberg technical support.

Science party says the sonar is broken.

Starting this issue thread to document some of the issues, or maybe more correctly limitations, we've found with GSF (Generic Sensor Format) files. As background, we've used GSF as our preferred processed point data export and have used it as an internal/requestable archive format as well as submitted it to public archive (though I don't think it is made available on NCEI) for work with OET and others. We were and are using this format because it is an open format, is supported by many mapping software programs, supports processing flags so that all of the processing work we do isn't lost, and can, theoretically at least, be re-processed or have additional processing done to it. But we have hit some limitations and I'm sure other have as well. Documenting here for comment and tracking.

ISSUE: When setting a survey to use a UTM projection (so that I could bring in a background GeoTiff, see issue #35, the Helmsman tool will not open.

Symptoms:
Warning light in SIS 4 (EM122) that network attitude velocity was either missing or being received at lower than the expected rate (100 Hz expected).

Attitude velocity started missing for both navigation inputs (Seapath and POS MV) at the same time. No change or improvement.

WARNING: FM transmission is disabled if attitude velocity is not received at the expected rate. This limits the system to CW-only transmission, even in deep modes intended to use CW/FM Mix or full FM swaths. Check the 'TX Pulse Form' field under the Numerical Display to ensure CW/FM Mix or FM is displayed when FM sectors are expected.

Troubleshooting:
Check all communication settings against last known good configuration.

Solution:
Restart EM122 TRU. Both attitude velocity feeds returned to full rate.

Ongoing topic from RVTEC email 2023-05-16 (Paul Duncan, SOI), posted with permission:

Symptoms:
POS MV errors traced down to interference from Inmarsat C transmission close to the GNSS antennas.

Suggestions from industry and RVTEC community:

1. Replace with Trimble GA-830 or Zephyr 3 Rugged antennas with better rejection of Iridium frequencies.
2. Install choke rings and General Dynamics L1/L2 filters
3. Move Inmarsat C antennas higher so POS MV antennas are out of main transmissions

SOI will attempt replacing these with GA-830 antennas at next opportunity, then report back and take further action if necessary.

On extensive multi-day surveys, it seems that SIS (both SIS4 and SIS5) almost always gets to a point where it will stop updating the real-time coverage grid. Our solution so far has been to restart SIS, and when it starts failing daily, start a new SIS project. Are there any other solutions or workarounds for this issue?

ISSUE: Unable to upload a background GeoTiff into SIS 5, even when creating a grid in World Mercator to match the default survey settings.

SOLUTION: I was able to get the background grid in by creating a new survey and selecting the most applicable UTM zone as the projection (note, that there are only a few zones to choose from), and bringing in a GeoTiff in the same UTM projection. I had to restart SIS after uploading the GeoTiff for it to work.

Symptom:
Realtime heave is greyed out on the POS MV display (POSView).

Troubleshooting:
Tried restarting, new antennas, new cables, communication settings, etc.

Response from Applanix technical support to earlier / identical issue:

"The Heave will be greyed out by one of two scenarios.

First, please try opening the Heave plot under View. After three minutes, the True Heave and the Real time should be scrolling across. Once this window is activated, then the Heave status indicator should change from grey to either red or green.

If this proves to be unsuccessful, please go to the Logging->Ethernet realtime and make sure that UDP Broadcast is selected and not TCP as TCP doesn't allow for the Heave status to be updated/sent back to the POS PCS."

Credit: from an RVTEC discussion between Finn Sougioultzoglou (UDEL) and Val Schmidt (UNH CCOM)

Adapted from email conversation between Shannon Hoy (NOAA) and Jonathan Beaudoin (HydroOctave Consulting) in Feb 2024.

Context:
Survey files are not incremented during turns, leaving turn data in the files and complicating the 'standard' survey line processing.

Problem:
The impacts of swaths in the turns are especially clear and problematic in backscatter mosaics (e.g., in FMGT).

Various solutions:

1. Mosaic editing:
Remove turn data from mosaics in the backscatter processing software.

Pro: This does not require earlier processing or export to GSF; the raw data format can be used.

Con: Edits made to the mosaics are not carried through to other products (e.g., other mosaics at different resolutions). Also, the swath removal step in backscatter software may be less intuitive / more time-consuming than swath or navigation rejection in other packages (see below).

2. Swath editing:
Remove turn data in bathymetry processing software (e.g., Qimera), export to GSF, and import into backscatter processing software (e.g., FMGT).

Pro: This provides the opportunity to address other bathymetric data quality issues (e.g., outlier removal) and carries the edit all the way through bathy and backscatter processing.

Con: This requires swath editing and export to GSF from another software package.

3. Navigation editing:
Remove turn data by flagging / rejecting the corresponding heading data in the time series editor (e.g., in Qimera). This causes a navigation lookup failure during bathy processing that will reject the turning swaths and carry this through to GSF export.

Pro: may be much faster than swath editing; GSF carries these edits through all mosaics in the backscatter processing step.

Con: This requires navigation editing and export to GSF from another software package.

SIS 5.11.1 Build date 2022-11-11

After replacing the SIS machine, the telnet reported:
Reading backscatter calibration file failed on: Calib124.txt : (4000) Mode: 6 Swath: 2 N sectors: 6 Header parse error:
Followed by:
** No RX backscatter correction file for head EM 124 . Using default.

We found a copy of the Calib124.txt file and uploaded it to the PU via FTP, but there was still an issue. In the meantime, we were not collecting ANY backscatter -- waterfall display was showing nothing, water column display was showing a purple wedge (aka nothing).

With help from Kongsberg (thanks Jørgen) we discovered that if there is a Calib124.txt file present on the PU and it has issues, the system will NOT use the default backscatter correction file, even though it continues to report that it is via the telnet. In order to get it to use the default, you have to FTP into the PU and rename the Calib124.txt file so it can't find it (broken_Calib124.txt or something). Then it will use the default and you will get backscatter.

Easy enough fix but we lost several hours of data while we got it sorted out. Put this here in the hopes someone else with the issue will be able to fix it more quickly.

Context: EM2040 testing with a portable processing unit (PPU) in an autonomous vessel.

Symptoms:

1. After a few days of routine operations and all BISTs passing, the RX Unit BIST failed during pre-survey checks.
2. Within the RX Unit test data (saved to text), the 5 V power supply was reading <1 V. All other components of the RX Unit test passed.
3. Multiple restarts of the PPU and power cycles for the whole vessel did not fix the issue. All other BISTs passed, except RX Channels.

GitHub shows only 'open' Issues when the page is opened. Closed (resolved) Issues are not shown immediately.

To see ALL Issues, clear "is:issue is:open" from the search bar and hit Enter.

Alternatively, click Open or Closed to see only those topics.

There is no way to change this default filter, so please leave this 'Issue' open so it will always show up.

Information provided by Mary Huey (SIO).

Context / symptoms:

1. The ship's networks experienced excessive traffic of BPDU packets originating from the CPU3 port on the PU (EM124 on Sally Ride).

Some users have reported Parameter Setup (and other) windows not starting after updating SIS to 5.9.3.

EM712 went from collecting data swimmingly to a depth read out of -0.3m or -0.2m during turn. Unable to collected data after.

Troubleshooting: Power cycle of computer and PU did not remedy the problem. Powered cycle the PU twice.. once for 5 minutes and once for 3 hours. This is the error print out on telnet session:

CBMF 1 : Timed out waiting for samples OBJ_TIMEOUT 4004 remaining: : 4004 was expected

In July 2023, we had a different issue with the CBMF that was resolved by editing the Spanning Tree in the switch.; this issue was related to network connections on system. Note - this problem occurred on both our EMs onboard (EM304 and EM712) and was not unqiue to one system.

Simultaneously, we've been experiencing a bug (Togrims words from KM Support) where on initialization of sonar, the K-Controller will automatically terminate SIS and sent the system into an automatic start-up sequence. The only way out of this loop and to collect data is to restart the HWS and turn off system PU for at least ten minutes.

I am emailing KM support with this issue, but wanted to report on this wiki in case anyone has experienced anything similar. Screen shot with telnet attached.
OXR20230824_0901_0355_EM712.txt

UPDATE 2 SEPT 2023

The error --> "CBMF 1 : Timed out waiting for samples OBJ_TIMEOUT 4004 remaining: : 4004 was expected" corresponds to losing connection with the RX Unit. We confirmed we lost connection by checking the link lights in the ethernet connection on the physical unit in our Sonar Room (see attached picture). Further testing showed that we have 230V going into the power supply but nothing coming out of the power supply - the board is dead! We are working with Kongsberg to getting a spare delivered (luckily one was already purchased as part of a recent spare order).

Note, this error is not linked to the separate CBMF issues as noted above.

Recently upgraded EM304 and EM712 from SIS 5.11.1 to SIS 5.12.1.

When going through my sanity checklist to ensure full functionality for data acquisition and processing, I noticed that there was no data in the water column display. This occurs on both EM712 and EM304. I tried the following troubleshooting steps:

1. Restart software
2. geographical restart (separate icon on desktop)
3. SIS User Interface Restart found in the SIS Software
4. Exiting out of the Water Column Display and then bringing back into view
5. Rebooting PU.

Because this is happening to both sonars, I am thinking there is a bug related to the software. I've reached out to Kongsberg for insight..

A couple of bugs were noted with the update of SIS to 5.10.1 (also observed with 5.9.3):

1. In Parameter Setup, though the Logging Interval is set to 3600, the files split at 30 rather than 60 minutes. No restart has been able to effectuate the change.
2. In Parameter Setup, though the Auto Start Echosounder on Start-Up is set to 1, the EM 304 will not auto start.
3. When starting a New Survey, the vessel name will not append to the file name due to a bug.

There is an ongoing issue with significant consistent nadir mistracking going up and down slopes (especially on seamounts). We reached out to Kongsberg but have been unable to alleviate this issue. We will troubleshoot further during our upcoming shakedown.

Symptom:
Gridding .gsf files from a deepwater transit resulted in gaps between files (50 m grid in ~5000 m water).

Software: Qimera v2.4.1
Files: .gsf exported from light Qimera processing underway

Context: EM2040 testing with a portable processing unit (PPU) in an autonomous vessel.

Symptoms:

1. The portable processing unit (PPU) would power up but not boot.
2. The screen showed 'Power On' but no other information, and SIS / K-Controller would not connect.
3. Waiting 10-15 minutes and multiple restarts (via relay, not the PPU power button) did not resolve the issue.

Information provided by Mary Huey (SIO).

Context / symptoms:

1. Valeport Thru-Hull SVS data originally provided to SIS in 'Valeport' probe output format.

2. This feed was no longer accepted by SIS after upgrading to version 5.7.0 and later.

3. Note: the Thru-Hull SVS models may use different software than other Valeport sensors (https://www.valeport.co.uk/products/through-hull-svs/)

Information provided by Mary Huey (SIO).

Context / symptoms:

1. The EM124 TXU has two ethernet ports.
2. The TXU had been periodically losing connection to the PU (as seen in "Module not found" errors on the Start -> K-Controller -> EMSWUpdater).
3. The connection could be restored by power cycling, but only temporarily.
4. This reached a point where it was no longer detected by the PU until the ethernet cable was moved from one port to the other.

GitHub allows any users to post Issues but allows tagging and notification options only for users who have been added as collaborators on the oceanmapping/community repository.

Simply contact the admins (https://github.com/oceanmapping/community/wiki/Contributing) to be added as a collaborator. Then sit back, relax, and enjoy Issue tagging and notifications among many of the wiki's all-inclusive benefits.

(Thanks to Mike Smith and Lindsay Gee for pointing out this nuance of the Issues section.)

When attempting to sign up to the Kongsberg software emailing list, I could not get the email form box to show.
I am using Chrome.

Did not have this issue when using an alternative browser (edge).

Distances in Qimera are grid distances in the project coordinates, and not true distances. This may not be a problem when in a small survey in a UTM projection, BUT it's a significant problem if you are in World Mercator projection at 52N ... or anywhere not on the equator.

This applies to all distances shown in the profile tool (2D, 3D, surface), the measurement tool, and swath extinction plot (swath angles incorrect). The NOAA export csv of line statistics is also based on the projection; average line heading is grid heading, line distances are grid distances and not sure if calculation of average line speed in knots is correct.

There are issues raised with QPS support for the error in the swath extinction output and to better support geographic coordinates and will post any updates here.

[Adapted from troubleshooting notes from EX2101 EM304MKII SAT, running SIS v5.6.0441)

Symptoms:

1. A selected .asvp file does not update correctly in the Sound Velocity Management window.
2. A warning light flashes and an error message is displayed; however, these might be missed by the user.

Documenting a just-resolved issue here.

We noticed a really dramatic wobble in our EM712 data. EM304 data be collected simultaenously did not have any noticeable issues. The wobble varied but at it's worse, it was easy to spot real time in the SIS display, and was very evident in the Qimera surface during processing. The extremeness varied over the course of the survey. We also noticed a really large number in the PU - POS difference in the Numerical Display (red box in SIS display image).

.

[Adapted from EX2101 notes / SIS v5.6.0.441]

Symptoms:

1. SIS stops gridding altogether if the grid becomes too large or a portion of it becomes corrupted (e.g., network crash or power loss)

2. Multiple SIS / SIS machine / PU restarts and ‘Restart HDDS’ do not improve / recover the grid

Context:
NOAA Ship Okeanos Explorer with EM304 MKII and POS MV/Seapath navigation inputs.

Symptoms:
The EM304 intermittently and unpredictably stops pinging / tracking seafloor, leaving large data gaps of variable duration. Logging continues as normal (no file increment).

Troubleshooting:
Extensive troubleshooting was conducted by the EX team on board in communication with Kongsberg support in Norway.

Multiple software and firmware updates were applied with no improvement.

The PU ethernet switch firmware was updated to v1.40 to address an issue with excessive BPDU packets (#15)

EM dropouts were found to correlate with brief VSAT satellite internet outages, suggesting a broader ship network interruption impacting the EM network.

Solution:
The relationship between VSAT interruptions and EM network dropouts was traced to the on-board network’s spanning tree protocol (STP).

When major changes occur, such as loss of VSAT, the switches block traffic temporarily to protect the network from possible looping paths or other related problems.

The solution was to set up filters and block STP updates to/from the Kongsberg switch as well as disable STP on the port within the ship’s switches connected to the EM304 PU "CPU3" connector.

To prevent further dropouts, the on-board team recommends STP should be disabled on the Kongsberg CPU3 connector, if not globally within their switch.

Follow-up:
These network changes have reliably solved the dropout issue.
Kongsberg does not utilize STP and will block this feature on all future deliveries.

Credit:
Okeanos Explorer, Global Foundation for Ocean Exploration (GFOE), and Kongsberg technical personnel