Converting encoding of waypoints names about gpsbabel

[tsteven4]

Hurray. glad to have help. I'll push something up that is a start in a bit.

[tsteven4]

please pull the latest commits to the branch, rebuild and try
./gpsbabel -vs -w -i garmin,codec=windows-1251 -f usb: -o gpx -F test1.gpx

This should also support encoding on write (using -o garmin,codec=windows-1251)

🤞

[tsteven4]

You can get the list of codecs through the back door by:
gpsbabel -i unicsv,codec=xxx -f /dev/null

[tsteven4]

What version of gpsbabel?

What platform? windows, linux, macos?

What is the command line that you are using?

We do have this comment in garmin.cc that implies this might have been broken years ago. It is difficult to maintain this module without a battery of devices to test against, and such testing is manual. Robert will tremble, but it does look possible to try to use the correct codec, but we would require help with testing.

  /*
   * This used to mean something when we used cet, but these days this
   * format either uses implicit QString conversions (utf8),
   * or we have hard coded QString::fromLatin1, CSTRc, or CSTR.  These
   * are likely bugs.
   * However, this is still used for garmin_fs_garmin_after_read,
   * garmin_fs_garmin_before_write.
   */

[svintuss]

I use GPSBabel 1.8.0 GUI on macOS. Didn't find any mention of text encoding while transfer, so I've decided to ask.

I have a couple of old devices (GPSmap 60Cx and GPS V) where I can test latin1 <> cp1251 conversion.

[tsteven4]

Can you build the code yourself?

[svintuss]

Yes, I can.

[tsteven4]

You can try this branch.

I think it will fix the garmin reader. It doesn't touch the garmin writer, so that will still be broken. Add option -vs to get verbose status which should print a message "reciever charset detected as ..." to stdout when you try to use the garmin format to read from your device. To start read from the device and write to gpx, let's see if that looks better.

[svintuss]

Thanks. Hope I'm not in over my head with this.

[svintuss]

Thanks again. After building the garmincodec branch I have the charset detected as windows-1252. That should be technically correct, but I thought of manually specifying encoding. E. g. "source encoding" and "destination encoding" options like-se windows-1251 and -de UTF8.

% ./gpsbabel -vs -w -i garmin -f usb: -o gpx -F test1.gpx
reciever charset detected as windows-1252.
100/46/46
100/46/46

[tsteven4]

There currently isn't a way to override the encoding. Based on the type of device we assign a character set. Does the device have support for more than one character set?

how does the gpx look? it should be correctly encoded in utf-8.

[svintuss]

No, these Garmins are from pre-unicode era, they only support windows-1252. The GPX produced is correctly encoded in utf8 but extended latin characters are treated as extended latin and not cyrillic.

I thought of a workaround when I open the .gpx file > save it with Latin1 encoding > open with windows-1251 encoding > save again with utf8.

[tsteven4]

Hmmm. so the device only supports 1252 but your using it with 1251?

[svintuss]

Yes, that's how old devices were localised. Extended latin font characters were replaced with Cyrillic counterparts and MapSource was forced to use cp1251 instead of cp1252 with a registry hack.

[tsteven4]

ok. writer fixes about ready, but you also need an option to use with garmin to force the charset.

[svintuss]

That worked, cool!

  <wpt lat="57.142150495" lon="33.117158515">
    <name>5 КАЛИНИНА</name>
    <cmt>5 КАЛИНИНА</cmt>
    <desc>5 КАЛИНИНА</desc>
    <sym>Lodging</sym>
  </wpt>

[tsteven4]

Now can you see if we write correctly encoded data to the device?

[svintuss]

Sending data works too.

[robertlipe]

Shouting from the peanut gallery... The Garmin spec said they're ASCII. We "know" certain newer models (your 2006 model is "new" in Garmin protocol perspective) were capable of more so we enabled those at runtime. Some units in some markets, and we have poor data on the curve of those, have yet additional (or different) code pages. I do have a US 60cs and 60csx and MacOS in the lab. I'll admit most of my personal use (as a geocacher) was writing waypoints from GPX, not reading. I can spot test things tonight if needed.

…

On Mon, May 15, 2023, 1:15 PM svintuss ***@***.***> wrote: Sending data works too. — Reply to this email directly, view it on GitHub <#1115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD3YSA2QBXOHK2ECPI53XGJXF3ANCNFSM6AAAAAAYCLHQTI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[svintuss]

I'm a bit confused: sending .gpx works fine with this branch. But I can't send tracks from .kml, only waypoints. This happens either with or without codec=windows-1251 option. Main branch ver. 1.8.0 sends .kml tracks correctly (apart from cyrillic).

[tsteven4]

The garmin reader and writers don't know where the data comes from or goes to. The readers all convert the recovered 8 bit data, usually assuming it is UTF-8, to UTF-16, and the writers all convert this UTF-16 data to something, usually UTF-8. The garmin codec option sets that conversion to/from UTF-16 to use whatever codec you asked for. Note you do need to specificy the codec option with -i garmin,codec= AND -o garmin,codec=, or you will get the default.

There were multiple changes to the garmin module that independently effect waypts, routes and tracks.

What exactly do you mean by "I can't send tracks"? Is there an error message? crash? Is the data not send to the device? Or is the wrong data sent do the device?

[tsteven4]

I do see something I missed. It might cause the waypoints in routes to be mis-encoded when writing to the device, just like 1.8.0.

[svintuss]

Sorry for not being specific enough. No .kml tracks are sent to device with garmincodec branch.
If I checkout the main branch and recompile gpsbabel .kml tracks are sent to device just fine.

No errors or log data is written to output with either branch.

[tsteven4]

Please add the "-D 2" option and look for differences. That should enable further debug output, but not having a device I am not sure what you will see.

Perhaps we are sending an illegal character to the device.

[svintuss]

BTW what is the list of possible codecs?
And is it possible to add a translit codec which simply converts Cyrillic letters to its Latin analogues (this is one example of transliteration tool: https://translit.cc)?

[tsteven4]

The minimal set of supported codecs are listed in https://doc.qt.io/qt-5/qtextcodec.html#details
However, many more are available based on your system and the availability of ICU. We used to output the list of available codecs but that seems to have become disconnected.

[tsteven4]

On MY system (ubuntu jammy, with ICU):
Available Codecs:
MIBenum Name Aliases
3 US-ASCII iso-ir-6, ANSI_X3.4-1968, ANSI_X3.4-1986, ISO_646.irv:1991, ISO646-US, us, IBM367, cp367, csASCII
4 ISO-8859-1 latin1, CP819, IBM819, iso-ir-100, csISOLatin1
5 ISO-8859-2 ibm-912_P100-1995, ibm-912, ISO_8859-2:1987, latin2, csISOLatin2, iso-ir-101, l2, 8859_2, cp912, 912, windows-28592
6 ISO-8859-3 ibm-913_P100-2000, ibm-913, ISO_8859-3:1988, latin3, csISOLatin3, iso-ir-109, l3, 8859_3, cp913, 913, windows-28593
7 ISO-8859-4 ibm-914_P100-1995, ibm-914, latin4, csISOLatin4, iso-ir-110, ISO_8859-4:1988, l4, 8859_4, cp914, 914, windows-28594
8 ISO-8859-5 ibm-915_P100-1995, ibm-915, cyrillic, csISOLatinCyrillic, iso-ir-144, ISO_8859-5:1988, 8859_5, cp915, 915, windows-28595
9 ISO-8859-6 ibm-1089_P100-1995, ibm-1089, arabic, csISOLatinArabic, iso-ir-127, ISO_8859-6:1987, ECMA-114, ASMO-708, 8859_6, cp1089, 1089, windows-28596, ISO-8859-6-I, ISO-8859-6-E, x-ISO-8859-6S
10 ISO-8859-7 ibm-9005_X110-2007, ibm-9005, 8859_7, greek, greek8, ELOT_928, ECMA-118, csISOLatinGreek, iso-ir-126, ISO_8859-7:1987, windows-28597, sun_eu_greek
11 ISO-8859-8 ibm-5012_P100-1999, ibm-5012, hebrew, csISOLatinHebrew, iso-ir-138, ISO_8859-8:1988, ISO-8859-8-I, ISO-8859-8-E, 8859_8, windows-28598, hebrew8
12 ISO-8859-9 ibm-920_P100-1995, ibm-920, latin5, csISOLatin5, iso-ir-148, ISO_8859-9:1989, l5, 8859_9, cp920, 920, windows-28599, ECMA-128, turkish8, turkish
13 ISO-8859-10 iso-8859_10-1998, iso-ir-157, l6, ISO_8859-10:1992, csISOLatin6, latin6
16 ISO-2022-JP-1 ISO_2022,locale=ja,version=1, JIS_Encoding, csJISEncoding, ibm-5054, JIS, x-windows-50221
17 Shift_JIS ibm-943_P15A-2003, ibm-943, MS_Kanji, csShiftJIS, windows-31j, csWindows31J, x-sjis, x-ms-cp932, cp932, windows-932, cp943c, IBM-943C, ms932, pck, sjis, ibm-943_VSUB_VPUA, x-MS932_0213, x-JISAutoDetect
18 EUC-JP euc-jp-2007, Extended_UNIX_Code_Packed_Format_for_Japanese, csEUCPkdFmtJapanese, X-EUC-JP, eucjis, ujis
30 US-ASCII iso-ir-6, ANSI_X3.4-1968, ANSI_X3.4-1986, ISO_646.irv:1991, ISO646-US, us, IBM367, cp367, csASCII
36 windows-949 windows-949-2000, KS_C_5601-1987, KS_C_5601-1989, KSC_5601, csKSC56011987, korean, iso-ir-149, ms949, x-KSC5601
37 ISO-2022-KR ISO_2022,locale=ko,version=0, csISO2022KR
38 windows-949 windows-949-2000, KS_C_5601-1987, KS_C_5601-1989, KSC_5601, csKSC56011987, korean, iso-ir-149, ms949, x-KSC5601
39 ISO-2022-JP ISO_2022,locale=ja,version=0, csISO2022JP, x-windows-iso2022jp, x-windows-50220
40 ISO-2022-JP-2 ISO_2022,locale=ja,version=2, csISO2022JP2
57 GBK windows-936-2000, CP936, MS936, windows-936
81 ISO-8859-6 ibm-1089_P100-1995, ibm-1089, arabic, csISOLatinArabic, iso-ir-127, ISO_8859-6:1987, ECMA-114, ASMO-708, 8859_6, cp1089, 1089, windows-28596, ISO-8859-6-I, ISO-8859-6-E, x-ISO-8859-6S
82 ISO-8859-6 ibm-1089_P100-1995, ibm-1089, arabic, csISOLatinArabic, iso-ir-127, ISO_8859-6:1987, ECMA-114, ASMO-708, 8859_6, cp1089, 1089, windows-28596, ISO-8859-6-I, ISO-8859-6-E, x-ISO-8859-6S
84 ISO-8859-8 ibm-5012_P100-1999, ibm-5012, hebrew, csISOLatinHebrew, iso-ir-138, ISO_8859-8:1988, ISO-8859-8-I, ISO-8859-8-E, 8859_8, windows-28598, hebrew8
85 ISO-8859-8 ibm-5012_P100-1999, ibm-5012, hebrew, csISOLatinHebrew, iso-ir-138, ISO_8859-8:1988, ISO-8859-8-I, ISO-8859-8-E, 8859_8, windows-28598, hebrew8
104 ISO-2022-CN ISO_2022,locale=zh,version=0, csISO2022CN, x-ISO-2022-CN-GB
105 ISO-2022-CN-EXT ISO_2022,locale=zh,version=1
106 UTF-8
109 ISO-8859-13 ibm-921_P100-1995, ibm-921, 8859_13, windows-28603, cp921, 921, x-IBM921
110 ISO-8859-14 iso-8859_14-1998, iso-ir-199, ISO_8859-14:1998, latin8, iso-celtic, l8
111 ISO-8859-15 ibm-923_P100-1998, ibm-923, Latin-9, l9, 8859_15, latin0, csisolatin0, csisolatin9, iso8859_15_fdis, cp923, 923, windows-28605
113 GBK windows-936-2000, CP936, MS936, windows-936
114 GB18030 gb18030, ibm-1392, windows-54936
1000 UTF-16
1001 UTF-32
1011 SCSU ibm-1212, ibm-1213
1012 UTF-7 windows-65000, unicode-1-1-utf-7, unicode-2-0-utf-7
1013 UTF-16BE
1014 UTF-16LE
1015 UTF-16
1016 CESU-8 ibm-9400
1017 UTF-32
1018 UTF-32BE
1019 UTF-32LE
1020 BOCU-1 csBOCU-1, ibm-1214, ibm-1215
2004 hp-roman8 ibm-1051_P100-1995, ibm-1051, roman8, r8, csHPRoman8, x-roman8
2005 Adobe-Standard-Encoding ibm-1276_P100-1995, ibm-1276, csAdobeStandardEncoding
2009 IBM850 ibm-850_P100-1995, ibm-850, cp850, 850, csPC850Multilingual, windows-850
2013 IBM862 ibm-862_P100-1995, ibm-862, cp862, 862, csPC862LatinHebrew, DOS-862, windows-862
2016 IBM-Thai ibm-838_P100-1995, ibm-838, IBM838, csIBMThai, cp838, 838, ibm-9030
2024 Shift_JIS ibm-943_P15A-2003, ibm-943, MS_Kanji, csShiftJIS, windows-31j, csWindows31J, x-sjis, x-ms-cp932, cp932, windows-932, cp943c, IBM-943C, ms932, pck, sjis, ibm-943_VSUB_VPUA, x-MS932_0213, x-JISAutoDetect
2025 GBK windows-936-2000, CP936, MS936, windows-936
2026 Big5 windows-950-2000, csBig5, windows-950, x-windows-950, x-big5, ms950
2027 macintosh macos-0_2-10.2, mac, csMacintosh, windows-10000, macroman, x-macroman
2028 IBM037 ibm-37_P100-1995, ibm-37, ibm-037, ebcdic-cp-us, ebcdic-cp-ca, ebcdic-cp-wt, ebcdic-cp-nl, csIBM037, cp037, 037, cpibm37, cp37
2030 IBM273 ibm-273_P100-1995, ibm-273, CP273, csIBM273, ebcdic-de, 273
2033 IBM277 ibm-277_P100-1995, ibm-277, cp277, EBCDIC-CP-DK, EBCDIC-CP-NO, csIBM277, ebcdic-dk, 277
2034 IBM278 ibm-278_P100-1995, ibm-278, cp278, ebcdic-cp-fi, ebcdic-cp-se, csIBM278, ebcdic-sv, 278
2035 IBM280 ibm-280_P100-1995, ibm-280, CP280, ebcdic-cp-it, csIBM280, 280
2037 IBM284 ibm-284_P100-1995, ibm-284, CP284, ebcdic-cp-es, csIBM284, cpibm284, 284
2038 IBM285 ibm-285_P100-1995, ibm-285, CP285, ebcdic-cp-gb, csIBM285, cpibm285, ebcdic-gb, 285
2039 IBM290 ibm-290_P100-1995, ibm-290, cp290, EBCDIC-JP-kana, csIBM290
2040 IBM297 ibm-297_P100-1995, ibm-297, cp297, ebcdic-cp-fr, csIBM297, cpibm297, 297
2041 IBM420 ibm-420_X120-1999, ibm-420, cp420, ebcdic-cp-ar1, csIBM420, 420
2043 IBM424 ibm-424_P100-1995, ibm-424, cp424, ebcdic-cp-he, csIBM424, 424
2011 IBM437 ibm-437_P100-1995, ibm-437, cp437, 437, csPC8CodePage437, windows-437
2044 IBM500 ibm-500_P100-1995, ibm-500, CP500, ebcdic-cp-be, csIBM500, ebcdic-cp-ch, 500
2045 cp851 ibm-851_P100-1995, ibm-851, IBM851, 851, csPC851
2010 IBM852 ibm-852_P100-1995, ibm-852, cp852, 852, csPCp852, windows-852
2046 IBM855 ibm-855_P100-1995, ibm-855, cp855, 855, csIBM855, csPCp855, windows-855
2047 IBM857 ibm-857_P100-1995, ibm-857, cp857, 857, csIBM857, windows-857
2048 IBM860 ibm-860_P100-1995, ibm-860, cp860, 860, csIBM860
2049 IBM861 ibm-861_P100-1995, ibm-861, cp861, 861, cp-is, csIBM861, windows-861
2050 IBM863 ibm-863_P100-1995, ibm-863, cp863, 863, csIBM863
2051 IBM864 ibm-864_X110-1999, ibm-864, cp864, csIBM864
2052 IBM865 ibm-865_P100-1995, ibm-865, cp865, 865, csIBM865
2053 IBM868 ibm-868_P100-1995, ibm-868, CP868, 868, csIBM868, cp-ar
2054 IBM869 ibm-869_P100-1995, ibm-869, cp869, 869, cp-gr, csIBM869, windows-869
2055 IBM870 ibm-870_P100-1995, ibm-870, CP870, ebcdic-cp-roece, ebcdic-cp-yu, csIBM870
2056 IBM871 ibm-871_P100-1995, ibm-871, ebcdic-cp-is, csIBM871, CP871, ebcdic-is, 871
2062 IBM918 ibm-918_P100-1995, ibm-918, CP918, ebcdic-cp-ar2, csIBM918
2063 IBM1026 ibm-1026_P100-1995, ibm-1026, CP1026, csIBM1026, 1026
2084 KOI8-R ibm-878_P100-1996, ibm-878, koi8, csKOI8R, windows-20866, cp878
2085 HZ-GB-2312 HZ
2086 IBM866 ibm-866_P100-1995, ibm-866, cp866, 866, csIBM866, windows-866
2087 IBM775 ibm-775_P100-1996, ibm-775, cp775, csPC775Baltic, windows-775, 775
2088 KOI8-U ibm-1168_P100-2002, ibm-1168, windows-21866
2089 IBM00858 ibm-858_P100-1997, ibm-858, CCSID00858, CP00858, PC-Multilingual-850+euro, cp858, windows-858
2091 IBM01140 ibm-1140_P100-1997, ibm-1140, CCSID01140, CP01140, cp1140, ebcdic-us-37+euro
2092 IBM01141 ibm-1141_P100-1997, ibm-1141, CCSID01141, CP01141, cp1141, ebcdic-de-273+euro
2093 IBM01142 ibm-1142_P100-1997, ibm-1142, CCSID01142, CP01142, cp1142, ebcdic-dk-277+euro, ebcdic-no-277+euro
2094 IBM01143 ibm-1143_P100-1997, ibm-1143, CCSID01143, CP01143, cp1143, ebcdic-fi-278+euro, ebcdic-se-278+euro
2095 IBM01144 ibm-1144_P100-1997, ibm-1144, CCSID01144, CP01144, cp1144, ebcdic-it-280+euro
2096 IBM01145 ibm-1145_P100-1997, ibm-1145, CCSID01145, CP01145, cp1145, ebcdic-es-284+euro
2097 IBM01146 ibm-1146_P100-1997, ibm-1146, CCSID01146, CP01146, cp1146, ebcdic-gb-285+euro
2098 IBM01147 ibm-1147_P100-1997, ibm-1147, CCSID01147, CP01147, cp1147, ebcdic-fr-297+euro
2099 IBM01148 ibm-1148_P100-1997, ibm-1148, CCSID01148, CP01148, cp1148, ebcdic-international-500+euro
2100 IBM01149 ibm-1149_P100-1997, ibm-1149, CCSID01149, CP01149, cp1149, ebcdic-is-871+euro
2101 Big5-HKSCS ibm-1375_P100-2008, ibm-1375, big5hk, HKSCS-BIG5
2102 IBM1047 ibm-1047_P100-1995, ibm-1047, cp1047, 1047
2250 windows-1250 ibm-5346_P100-1998, ibm-5346, cp1250
2251 windows-1251 ibm-5347_P100-1998, ibm-5347, cp1251, ANSI1251
2252 windows-1252 ibm-5348_P100-1997, ibm-5348, cp1252
2253 windows-1253 ibm-5349_P100-1998, ibm-5349, cp1253
2254 windows-1254 ibm-5350_P100-1998, ibm-5350, cp1254
2255 windows-1255 ibm-9447_P100-2002, ibm-9447, cp1255
2256 windows-1256 ibm-9448_X100-2005, ibm-9448, cp1256, x-windows-1256S
2257 windows-1257 ibm-9449_P100-2002, ibm-9449, cp1257
2258 windows-1258 ibm-5354_P100-1998, ibm-5354, cp1258
2259 TIS-620 windows-874-2000, windows-874, MS874, x-windows-874
2107 TSCII

[svintuss]

Please add the "-D 2" option and look for differences. That should enable further debug output, but not having a device I am not sure what you will see.

Perhaps we are sending an illegal character to the device.

I've found out the culprit. garmincodec branch works fine, it's just gpsbabel won't send both waypoints and tracks from .kml at the same time. E. g.
% ./gpsbabel -vs -wt -i kml -f /Users/alex/Desktop/Северка.kml -o garmin,codec=windows-1251 -F usb:
didn't transfer tracks whereas
% ./gpsbabel -vs -t -i kml -f /Users/alex/Desktop/Северка.kml -o garmin,codec=windows-1251 -F usb:
worked OK.

[tsteven4]

OK. I still missed encoding of waypoint names in routes when writing to the device.

[robertlipe]

-wt is a different flag than the two flags you want: -w -t

…

On Mon, May 15, 2023, 3:44 PM svintuss ***@***.***> wrote: Please add the "-D 2" option and look for differences. That should enable further debug output, but not having a device I am not sure what you will see. Perhaps we are sending an illegal character to the device. I've found out the culprit. garmincodec branch works fine, it's just gpsbabel won't send both waypoints and tracks from .kml at the same time. E. g. % ./gpsbabel -vs -wt -i kml -f /Users/alex/Desktop/Северка.kml -o garmin,codec=windows-1251 -F usb: didn't transfer tracks whereas % ./gpsbabel -vs -t -i kml -f /Users/alex/Desktop/Северка.kml -o garmin,codec=windows-1251 -F usb: worked OK. — Reply to this email directly, view it on GitHub <#1115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD334CDPVWAQP6EQM4CDXGKISZANCNFSM6AAAAAAYCLHQTI> . You are receiving this because you commented.Message ID: ***@***.***>

[tsteven4]

It looks like this existing code, which so far is unchanged with #1117,

gpsbabel/garmin.cc

Lines 1012 to 1021 in 488aa41

	char* d = rte->ident;
	for (auto idx : wpt->shortname) {
	int c = idx.toLatin1();
	if (receiver_must_upper && isalpha(c)) {
	c = toupper(c);
	}
	if (strchr(valid_waypt_chars, c)) {
	*d++ = c;
	}
	}

can write past the end of rte->ident if the shortname has sufficient valid characters.

[tsteven4]

It looks like this existing code, which so far is unchanged with #1117,

gpsbabel/garmin.cc

Lines 1012 to 1021 in 488aa41

	char* d = rte->ident;
	for (auto idx : wpt->shortname) {
	int c = idx.toLatin1();
	if (receiver_must_upper && isalpha(c)) {
	c = toupper(c);
	}
	if (strchr(valid_waypt_chars, c)) {
	*d++ = c;
	}
	}

can write past the end of rte->ident if the shortname has sufficient valid characters.

I assert this is fixed in 9bea843

Please pull the latest changes, rebuild and test writing a track. please build with "-DCMAKE_BUILD_TYPE=Debug". This will enable assertions, and we want to make sure none fail.

For some historical reason the name of waypoints that make up a route/track is restricted to a very limited set of characters, "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 ", and maybe "+-", and maybe the lower case equivalents. Other characters are thrown out.

[GPSBabelDeveloper]

Honestly, that reason is cowardice. The original Garmin spec said uppercase characters and digits only. Some devices allowed you to do more than that and some devices would crash at upload time or, worse, in the field if you sent characters outside that space. At the time the 60C's hit, Garmin refused to publish the specs and USB was my day job so I reverse engineered the protocol, happily finding it was mostly their serial protocol smuggled inside a (poorly implemented) USB protocol. After they'd shipped a couple of years, Garmin eventually published a spec for their windows USB driver which revealed some secrets and then, later, an actual USB protocol spec and we then went back and filled in our missing understandings. Geocachers were a HUGE percentage of our users at the time and they only really cared about waypoints. I could push a release to that crowd (especially via GSAK) and get meaningful testing on dozens of models within a few days. We discovered which boundaries we could push and which we couldn't. The route code and track code (especially writing tracks) were relatively unused. It's entirely probable that the units that take CP1252 for waypoitns will take them for routes and tracks, too. If he's in a good position to test this, go for it. Thanx for poking at the history.

…

On Mon, May 15, 2023 at 7:25 PM tsteven4 ***@***.***> wrote: It looks like this existing code, which so far is unchanged with #1117 <#1117>, https://github.com/GPSBabel/gpsbabel/blob/488aa41264c3c94c8c6e1d3de41ebd0c54d0c0d9/garmin.cc#L1012-L1021 can write past the end of rte->ident if the shortname has sufficient valid characters. I assert this is fixed in 9bea843 <9bea843> Please pull the latest changes, rebuild and test writing a track. please build with "-DCMAKE_BUILD_TYPE=Debug". This will enable assertions, and we want to make sure none fail. For some historical reason the name of waypoints that make up a route/track is restricted to a very limited set of characters, "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 ", and maybe "+-", and maybe the lower case equivalents. Other characters are thrown out. — Reply to this email directly, view it on GitHub <#1115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC3VADZRECKQMLBIOAUZNZLXGLCN7ANCNFSM6AAAAAAYCLHQTI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[svintuss]

Please pull the latest changes, rebuild and test writing a track. please build with "-DCMAKE_BUILD_TYPE=Debug". This will enable assertions, and we want to make sure none fail.

I have successfully built the latest garmincodec commit with -DCMAKE_BUILD_TYPE=Debug. Track sending works fine, their names are properly converted to cp1251.

TBH I don't quite understand what exactly should I test with track/route waypoints. Maybe I just don't have tracks with custom point names. Am I getting it right that track waypoint name resides in <ele> .gpx tag?

      <trkpt lat="69.444419732" lon="30.763811963">
        <ele>164.042</ele>
      </trkpt>

[robertlipe]

Track *points* dont' have names, but the overall tracks do. From memory, it's like: <trk> <name>my ride</name <trkseg> <trkpt > once for each point</trkpt> </trkseg> </trk> If "my ride" has non-ASCII characters (i.e. "my řȋɗê" or whatever is actually representable in 1252) does it now successfully roundtrip to/from the receiver? Some receivers have a UI to allow you to toggle visibility of specific routes/tracks to reduce screen clutter (and draw times) and let you focus on a specific target. Historically, tracks were records of where you HAD BEEN and as such were records FROM the GPS TO the host. It was only the later Garmins that allowed you to send a track from the computer to the GPS at all. I can't remember which side of that line the 60/76 family fell upon; it's possible that you may be unable to send tracks to it at all, in which case you can really only test this by fiddling with a saved track on the device, renaming it, and verifying that it shows up in the GPX correctly. THanx for your help. RJL

…

On Tue, May 16, 2023 at 2:53 AM svintuss ***@***.***> wrote: Please pull the latest changes, rebuild and test writing a track. please build with "-DCMAKE_BUILD_TYPE=Debug". This will enable assertions, and we want to make sure none fail. I have successfully built the latest garmincodec commit with -DCMAKE_BUILD_TYPE=Debug. Track sending works fine, their names are properly converted to cp1251. TBH I don't quite understand what exactly should I test with track/route waypoints. Maybe I just don't have tracks with custom point names. Am I getting it right that track waypoint name resides in <ele> .gpx tag? <trkpt lat="69.444419732" lon="30.763811963"> <ele>164.042</ele> </trkpt> — Reply to this email directly, view it on GitHub <#1115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD36PFPFUK5HORGH7ARTXGMXALANCNFSM6AAAAAAYCLHQTI> . You are receiving this because you commented.Message ID: ***@***.***>

[svintuss]

If "my ride" has non-ASCII characters (i.e. "my řȋɗê" or whatever is actually representable in 1252) does it now successfully roundtrip to/from the receiver?

Thank you for explanation.
My GPSmap 60Cx handles non-ASCII track and route names correctly. And now I can send them to and from device.

[tsteven4]

There were multiple other places we could have overwritten data structures that also have been fixed. Each of these 4 memcpy calls used the size of the source, and in some cases, although unlikely, it could have been larger than the destination. All of these were fixed previously so @svintuss has already ran this code.

gpsbabel/garmin.cc

Lines 879 to 900 in 488aa41

	memcpy(tx_waylist[i]->ident, ident, strlen(ident));
	if (global_opts.synthesize_shortnames) {
	xfree(ident);
	}
	tx_waylist[i]->ident[sizeof(tx_waylist[i]->ident)-1] = 0;
	// If we were explicitly given a comment from GPX, use that.
	// This logic really is horrible and needs to be untangled.
	if (!wpt->description.isEmpty() &&
	global_opts.smart_names && !wpt->gc_data->diff) {
	memcpy(tx_waylist[i]->cmnt, CSTRc(wpt->description), strlen(CSTRc(wpt->description)));
	} else {
	if (global_opts.smart_names &&
	wpt->gc_data->diff && wpt->gc_data->terr) {
	snprintf(obuf, sizeof(obuf), "%s%u/%u %s",
	get_gc_info(wpt),
	wpt->gc_data->diff, wpt->gc_data->terr,
	CSTRc(src));
	memcpy(tx_waylist[i]->cmnt, obuf, strlen(obuf));
	} else {
	memcpy(tx_waylist[i]->cmnt, CSTRc(src), strlen(CSTRc(src)));

[tsteven4]

The waypoints that compose our tracks and routes can have names. Reading a Garmin eTrexLegend manual, their route are composed of waypoints that can have names. The suspect cleaning of all but A-Z, 0-9, space and perhaps + and - is related to the names of waypoints that make up a route or a track. Perhaps the waypoint names of the waypoints that make up a track get dropped somewhere in garmin land, but it does seem like the waypoint names of waypoints that make up a route should exist in garmin land. But I think we butcher them with the suspect cleaning mentioned above.

@svintuss, can you read this gpx file and write the route to your device, and then look at the names of the points that make up the route? The points that make up the route are name RPT001, rpt005 and RPT011. I think your device is recognized with id=292, which won't force uppercase. Therefore I think on your device you will see name RPT001, 005, and RPT011. I think if you rename these in the gpx file to Cyrillic characters they will all be filtered out and the names on your device will be empty (unless garmin makes something up).

testrte.gpx.txt

[svintuss]

The waypoints that compose our tracks and routes can have names. Reading a Garmin eTrexLegend manual, their route are composed of waypoints that can have names. The suspect cleaning of all but A-Z, 0-9, space and perhaps + and - is related to the names of waypoints that make up a route or a track. Perhaps the waypoint names of the waypoints that make up a track get dropped somewhere in garmin land, but it does seem like the waypoint names of waypoints that make up a route should exist in garmin land. But I think we butcher them with the suspect cleaning mentioned above.

@svintuss, can you read this gpx file and write the route to your device, and then look at the names of the points that make up the route? The points that make up the route are name RPT001, rpt005 and RPT011. I think your device is recognized with id=292, which won't force uppercase. Therefore I think on your device you will see name RPT001, 005, and RPT011. I think if you rename these in the gpx file to Cyrillic characters they will all be filtered out and the names on your device will be empty (unless garmin makes something up).

testrte.gpx.txt

Hi.
I've uploaded testre.gpx to my GPSmap 60cx with gpsbabel. Route points were indeed converted to uppercase and any Cyrillic characters I've manually added (like <name>Тчк 3</name>) were truncated to ASCII and numerals.
However I tested that same .gpx with Cyrrillic points against MapSource on a Win7 machine and it uploaded both upper- and lowercase as well as Cyrillic symbols in route waypoint names without issue.

So I think we should be more flexible with point names limitations: GPSmap 60Cx can display cp1251 and cp1252 in route and track waypoints.

[robertlipe]

This confirms our suspicion in #1117 (comment) that the copy is being naughty. But if I look at the current state of this change at https://github.com/GPSBabel/gpsbabel/pull/1117/files# and see that naughty loop at 1012 on the left, the code on right right (the newest version) is about exactly what I'd described to fix it. I don't know if the version you got was beforr or after this? If your line 1012 of garmin.cc looks like that loop with theisalpha in it on the left, that's the old version. If so, can you please update, rebuilt, and retest to confirm.

…

On Wed, May 17, 2023 at 4:00 AM svintuss ***@***.***> wrote: The waypoints that compose our tracks and routes can have names. Reading a Garmin eTrexLegend manual, their route are composed of waypoints that can have names. The suspect cleaning of all but A-Z, 0-9, space and perhaps + and - is related to the names of waypoints that make up a route or a track. Perhaps the waypoint names of the waypoints that make up a track get dropped somewhere in garmin land, but it does seem like the waypoint names of waypoints that make up a route should exist in garmin land. But I think we butcher them with the suspect cleaning mentioned above. @svintuss <https://github.com/svintuss>, can you read this gpx file and write the route to your device, and then look at the names of the points that make up the route? The points that make up the route are name RPT001, rpt005 and RPT011. I think your device is recognized with id=292, which won't force uppercase. Therefore I think on your device you will see name RPT001, 005, and RPT011. I think if you rename these in the gpx file to Cyrillic characters they will all be filtered out and the names on your device will be empty (unless garmin makes something up). testrte.gpx.txt <https://github.com/GPSBabel/gpsbabel/files/11491591/testrte.gpx.txt> Hi. I've uploaded testre.gpx to my GPSmap 60cx with gpsbabel. Route points were indeed converted to uppercase and any Cyrillic characters I've manually added (like <name>Тчк 3</name>) were truncated to ASCII and numerals. However I tested that same .gpx with Cyrrillic points against MapSource on a Win7 machine and it uploaded both upper- and lowercase as well as Cyrillic symbols in route waypoint names without issue. So I think we should be more flexible with point names limitations: GPSmap 60Cx can display cp1251 and cp1252 in route and track waypoints. — Reply to this email directly, view it on GitHub <#1115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD37AK5SCZHEC33CLNJTXGSHSHANCNFSM6AAAAAAYCLHQTI> . You are receiving this because you commented.Message ID: ***@***.***>

[tsteven4]

@robertlipe

1. It doesn't fix it. See below.
2. There are bigger problems as reported #1115 (comment)

This test, cut from the latest garmin.cc, outputs "ABC123", the lower case letters abc are dropped.

#define MILITANT_VALID_WAYPT_CHARS "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"

/* Technically, even this is a little loose as spaces arent allowed */
static const char* valid_waypt_chars = MILITANT_VALID_WAYPT_CHARS " ";
static QRegularExpression invalid_char_re;

int main()
{
  assert(!QString(valid_waypt_chars).contains('\\'));
  assert(!QString(valid_waypt_chars).contains('^'));
  assert(!QString(valid_waypt_chars).contains('-'));
  assert(!QString(valid_waypt_chars).contains('['));
  assert(!QString(valid_waypt_chars).contains(']'));
  invalid_char_re = QRegularExpression(QStringLiteral("[^%1]").arg(valid_waypt_chars));
  assert(invalid_char_re.isValid());

  QString cleanname = "abcABC123";
  bool receiver_must_upper = false;

  if (receiver_must_upper) {
    cleanname = cleanname.toUpper();
  }
  cleanname = cleanname.remove(invalid_char_re);
  qDebug() << cleanname;
}

[tsteven4]

@robertlipe
Of course this will fix the lower case cleaning issue. But we also use valid_waypt_chars with shortname, so it is a bit more involved.

invalid_char_re = QRegularExpression("[^a-zA-Z0-9 +-]");

"abc +-ABC123*&@!" -> "abc +-ABC123"

But why are we cleaning the names of waypoints in routes or tracks this way, while we clean the names of isolated waypoints a different way (mkshort)? I am cautious of changes here as they will affect all the different garmin units, but it is a little hard to believe the requirements on the names of isolated waypoints are different than the requirements on the names of waypoints in routes.

[tsteven4]

@robertlipe
The difference in name cleaning came in here de992ba

Previously we used mkshort on isolated waypoint names, and nothing on route/track waypoint names.

[tsteven4]

but it is a little hard to believe the requirements on the names of isolated waypoints are different than the requirements on the names of waypoints in routes.

In looking at various versions of the Garmin device interface spec I see they did have different requirements for "User Waypoint Idendifiters" (upper-case letters, numbers) and "Route Waypoint Identifiers" (any ASCII character). They also say the host should be able to receive any ASCII character, but only send the restricted set to the GPS. And the say some products may allow additional characters, but no attempt is made to document these product specific additions. ☠

[tsteven4]

Maybe we follow this lead with rtept names, only cleaning them if receiver_must_upper. This would work for @svintuss, the question is do we break anybody other devices that set receiver_must_upper to 0.

gpsbabel/garmin.cc

Lines 330 to 338 in 4aa594a

	/*
	* Until Garmins documents how to determine valid character space
	* for the new models, we just release this safety check manually.
	*/
	if (receiver_must_upper) {
	setshort_goodchars(mkshort_handle, valid_waypt_chars);
	} else {
	setshort_badchars(mkshort_handle, "");
	}

[tsteven4]

Which would gets us to the following, sidestepping the issue of cleaning a-z.

  QString cleanname = wpt->shortname;
  if (receiver_must_upper) {
    cleanname = cleanname.toUpper().remove(invalid_char_re);
  }
  write_char_string(rte->ident,
                    str_from_unicode(cleanname).constData(),
                    sizeof(rte->ident));

[svintuss]

I've made a git pull and recompiled garmincodec branch. However seems like I've already built the latest commit last time:

% cmake --build .
ninja: no work to do.

Nevertheless I reuploaded testrte.gpx with the following data to GPSmap 60cx:

<name>Road Дорога</name>
<desc>Generated from track Route</desc>
<rtept lat="36.339560000" lon="-117.422570000">
  <ele>479.3</ele>
  <name>Point 1</name>
</rtept>
<rtept lat="36.345010000" lon="-117.354270000">
  <ele>624.3</ele>
  <name>ТОЧКА 2</name>
</rtept>
<rtept lat="36.344640000" lon="-117.349030000">
  <ele>663.7</ele>
  <name>_~-$#``;""|\/ ABC321</name>
</rtept>

Gpsbabel got me Road Дорога route name and P 1, 2 and ABC321 route points.
Mapsource got me Road Дорога route name and Point 1, ТОЧКА 2 and _~-$#``;""|\/ ABC321 route point names on the unit.

May I suggest a manual switch to enable/disable point names truncation?
And maybe a transliteration function to upload Cyrillic letters on old units with original Latin1 firmware?

[tsteven4]

@svintuss How did you do this to your GPSmap 60cx: "Extended latin font characters were replaced with Cyrillic counterparts"

We are chewing over how to proceed with the rtept names.

I don't think we know how to upload alternate character sets to the device.

[svintuss]

I've installed a custom firmware like the ones hosted here:
As I understand it back in the day it was the only way to implement Cyrillic support. Here is a link to flawia project, which hosts a font editor tool:

Basically there are units with unmodified firmware where users tend to upload maps and use waypoints in Latin1 with transliteration (i. e. Я -> JA, Ч -> CH) and the ones with alternative/modified firmware which accept cp1251 maps and points.

[tsteven4]

Thanks. The hosted here part and the links didn't come through, but I get the idea. ICU has such a transform, but we don't always have ICU available and I think it is unlikely we would invest the development effort in this day of unicode dominance.

However, I am hopefully that we can stop trashing the rtept names when writing to the device. We know how to do this, but we want to be careful because many devices are supported and they may not all tolerate unexpected characters. One choice would be to throw it onto the user with an option, but we already are making this decision for users on some devices, like yours, when sending other fields besides the rtept names.

[svintuss]

I'll fix a GPS V in a couple of days and could check whether it supports non-alfanumeric and lower-case letters in route points.
As for GPSmap 60cx, it definetely can handle non-truncated rtepts.

Thanks. The hosted here part and the links didn't come through, but I get the idea. ICU has such a transform, but we don't always have ICU available and I think it is unlikely we would invest the development effort in this day of unicode dominance.

It leaves us with a unix-way approach: pipe the gpx output through an external converter which will replace Cyrillic with translit.

[robertlipe]

So all this is about/for a custom, modified, unofficial firmware version? I don't love the nested levels of "unsupported" in all this. TSteven4, I chased the context of the submit you referenced up to this email thread with backing information: ***@***.***/ To the sidebar question: no, I think if Qt doesn't find it important enough to include libicu that we won't take it on ourselves. The affected hardware has been off the market for years. The hardware itself doesn't even really support CP1251. So while it's reasonable for us to try to help with issues that are within the blast radius of "supported", I don't see taking on thousands of lines of code to support obsolete hardware with modified firmware. That's an unreasonable carrying cost for us to build and support.

…

On Thu, May 18, 2023 at 8:18 AM svintuss ***@***.***> wrote: I'll fix a GPS V in a couple of days and could check whether it supports non-alfanumeric and lower-case letters in route points. As for GPSmap 60cx, it definetely can handle non-truncated rtepts. Thanks. The hosted here part and the links didn't come through, but I get the idea. ICU has such a transform <https://unicode-org.github.io/icu/userguide/transforms/general/#scriptlanguage>, but we don't always have ICU available and I think it is unlikely we would invest the development effort in this day of unicode dominance. It leaves us with a unix-way approach: pipe the gpx output through an external converter which will replace Cyrillic with translit. — Reply to this email directly, view it on GitHub <#1115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD35LOF6HHQLIZTUTK63XGYOTRANCNFSM6AAAAAAYCLHQTI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[svintuss]

So all this is about/for a custom, modified, unofficial firmware version? I don't love the nested levels of "unsupported" in all this.

First of all I'm very grateful to you guys for implementing encoding conversion. It really makes sending and receiving POIs much simpler without the need of post-processing gpx'es.

Secondly I do agree that the devices I mention are indeed obsolete, but they are still useful and functional especially for those of us who don't want to produce more e-waste.
As of levels of "unsupported" I can't completely agree. Back in the day there were officially translated Garmin units (e. g. eTrex H series) but they were poorly translated and generally buggy and the market went for hacking Latin1 units. So although my device has unofficial firmware installed there is nothing unsupported in cp1251 units in general. After all MapSource sends cp1251 maps, POIs, routes and tracks without issue.

[tsteven4]

We needed to do this work anyway to get cp1252 fixed which we use for some devices by default.

In any event, the rtept names should work on your 60cx now if you build from the master branch or use our pre-release installer https://github.com/GPSBabel/gpsbabel/releases/tag/Continuous-macOS

Thanks for the debugging. It definitely wouldn't have happened without your help.

[robertlipe]

Thank you both. To be clear, I'm happy to help resolve this level of issue and help keep older units somewhat working. I just don't think we should take on internalizing ICU for the project, especially for obsolete units used in a specific market with modified aftermarket firmware. I'm a little intrigued whether the firmware in question "just" modifies the bitmap or if there are any code changes in the image. Is there a source where I can learn more about that?

…

On Fri, May 19, 2023, 3:22 PM tsteven4 ***@***.***> wrote: We needed to do this work anyway to get cp1252 fixed which we use for some devices by default. In any event, the rtept names should work on your 60cx now if you build from the master branch or use our pre-release installer https://github.com/GPSBabel/gpsbabel/releases/tag/Continuous-macOS Thanks for the debugging. It definitely wouldn't have happened without your help. — Reply to this email directly, view it on GitHub <#1115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD32ILKFSU7FOMRK6WQDXG7JB7ANCNFSM6AAAAAAYCLHQTI> . You are receiving this because you were mentioned.Message ID: ***@***.***>