sillsdev / libpalaso Goto Github PK

When SLDR is initialized, it currently does a synchronous request to download the langtags.json file and this request can be slow on machines with slow internet connections.

It would be better if the download of the langtags.json file could be separated out from the initialization code and allow applications to decide when to make the call.

Also, the SLDR code currently uses the IF-MODIFIED-SINCE header and for a while this was broken on the server. It is working now, but WSTech would like us to switch to using the ETag method instead since this is now more common. They gave this as a reference: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag

For the applications that use this dialog (WeSay, Bloom) people should always be choosing just "en", "fr", etc. The on exception to this is Chinese Mandarin which ends up being zh-CN.

("packaging" below means creating the Debian packages)

• Create and use nuget packages in the following projects

  • libpalaso
  • chorus
  • flexbridge
  • liblcm
  • LfMerge
  • lfmerge-autosrtests
  • FieldWorks (the current work is in https://gerrit.lsdev.sil.org/c/FieldWorks/+/7869) (#1001)

• Verify that using the designer for winforms projects is still possible on Windows

  • test using the designer with VS 2019 16.8 in the projects
  • remove separate Designer projects that were added as a hack
  • test building on Linux
  • libpalaso
  • chorus (sillsdev/chorus#224 currently fails on AppVeyor, maybe because it doesn't yet have the current VS version available) (#1002)
  • flexbridge (#1003)
  • liblcm (#1004)
  • FieldWorks (#1005)

• update flexbridge installer/packaging - adjust build scripts (sillsdev/flexbridge#289) (#1006)

  • Windows installer
  • Linux packaging

• update/test LfMerge packaging (#1007)

• update/test FieldWorks packaging (#1008)

• document developing/debugging a project with dependencies that are now nuget packages (#1009)

• update and make available documentation of what changed and how things work now with the nuget packages (#1010)

• update documentation with release process

• make the feature/nuget branch the main development branch

  • merge master into feature/nuget again on the different projects
  • announce that the merge of the feature/nuget branch will be made the main development branch
  • merge feature/nuget into master on the different projects. Doing this requires adjusting the path to the CHANGELOG.md file in the .csproj as well as adjusting GitVersion.yml (removing settings for feature/nuget branch).
  • create a release of the nuget packages on each project, and use these released nuget packages in the other projects. Note: also create releases for other projects like L10NSharp that still have a pre-release version number (#1018).
    • libpalaso (#1011)
    • chorus (#1012)
    • flexbridge (#1013)
    • liblcm (#1014)
    • LfMerge (#1015)
    • lfmerge-autosrtests (#1016)
    • SIL.FwBuildTasks (#1017)
    • FieldWorks

Reported by Jonathan Coombs:

I did go ahead and create a sample project that illustrates the problem (attached): you can use the first "Set..." button as many times as you like, but the second button (which relies on a persistent _wsModel) will only work the first time you click it. It's simply the difference between this (which works)...

        IWritingSystemRepository store = AppWritingSys.WritingSystems;
        WritingSystemSetupModel wsModel = new WritingSystemSetupModel(store);
        wsModel.SetCurrentIndexFromRfc46464(textBoxWs.Text);
        using (var d = new WritingSystemSetupDialog(wsModel))
        {
            d.ShowDialog(wsModel.CurrentRFC4646);
            this.textBoxWs.Text = wsModel.CurrentRFC4646;

and this (which crashes on second run, once you click a WS):

        _store = _store ?? AppWritingSys.WritingSystems;
        _wsModel = _wsModel ?? new WritingSystemSetupModel(_store);
        using (var d = new WritingSystemSetupDialog(_wsModel))
        {
            d.ShowDialog(_wsModel.CurrentRFC4646);
            this.textBoxWs.Text = _wsModel.CurrentRFC4646;

On the Palaso end of things, these two BetterLabels in WritingSystemSetupView are ending up getting disposed too early:

    private Palaso.UI.WindowsForms.Widgets.BetterLabel _rfc4646;
    private Palaso.UI.WindowsForms.Widgets.BetterLabel _languageName;

It seems that either the WS dialog is damaging the passed-in _wsModel, or gets it partly unwired on closing the dialog the first time. (I'm suspicious of BindToModel(), and of LocalizationHelper.OnControlDisposed() but don't really know.) My workaround in Solid has been to make sure that I use the first approach above.

Also, while debugging I noticed some excessive event ping-ponging between methods in BetterLabel, which could be reduced by swapping these lines in BetterLabel_SizeChanged() to:
_previousWidth = Width;
DetermineHeight();

Jonathan also sent around a zip of a project that reproduces the problem. The core is here:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;

using Palaso.UI.WindowsForms.WritingSystems;
using Palaso.WritingSystems;

namespace TestPalaso
{
    public partial class FormTestWs : Form
    {
        public FormTestWs()
        {
            InitializeComponent();
        }

        IWritingSystemRepository _store;
        WritingSystemSetupModel _wsModel;


        //Works: Totally non-persistent, other than putting a string value into a textbox.
        private void buttonOpenWsDialog_Click(object sender, EventArgs e)
        {
            IWritingSystemRepository store = AppWritingSys.WritingSystems;
            WritingSystemSetupModel wsModel = new WritingSystemSetupModel(store);
            wsModel.SetCurrentIndexFromRfc46464(textBoxWs.Text);
            using (var d = new WritingSystemSetupDialog(wsModel))
            {
                d.ShowDialog(wsModel.CurrentRFC4646);
                this.textBoxWs.Text = wsModel.CurrentRFC4646;
            }

        }

        // Crashes on second launch, once you click on a WS:
        private void buttonOpenWsDialogPersistent_Click(object sender, EventArgs e)
        {
            _store = _store ?? AppWritingSys.WritingSystems;
            _wsModel = _wsModel ?? new WritingSystemSetupModel(_store);
            using (var d = new WritingSystemSetupDialog(_wsModel))
            {
                d.ShowDialog(_wsModel.CurrentRFC4646);
                string s = _wsModel.CurrentRFC4646;
                this.textBoxWs.Text = s;
                _wsModel.SetCurrentIndexFromRfc46464(s); //prob redundant
            }
        }


    }
}

The licenses are too small. They should be replaced by large ones and then scaled down as needed.

• test using the designer with VS 2019 16.8 in the projects
• remove separate Designer projects that were added as a hack
• test building on Linux

In this section of TestTask.cs, the Verbose flag causes anything on stderr to be logged as an error. This is causing build failures like this one where GDK warnings that were logged to stderr are being treated (incorrectly) as errors and causing a build failure when there's no actual problem with the build.

A lot of Linux software treats stderr not as a place to print error messages, but as a place to print any notification that is meant for the user to see even though he might be redirecting stdout. E.g., progress messages, and so on. We should change the logic in TestTask.cs to not consider stderr output as an error.

Also, the Verbose flag is documented as "If true print the output of NUnit immediately, otherwise print it ater [sic] NUnit finished". Yet it's the Verbose flag that's being used to treat stderr output as an error, which means it changes the semantics of how stderr is treated. This made it rather hard to track down this issue, as it's not at all obvious from that documentation that Verbose is going to change the behavior of the TestTask class.

• merge master into feature/nuget again
• announce that the merge of the feature/nuget branch will be made the main development branch
• create a git tag on master (pre-nuget ?) before doing the merge
• merge feature/nuget into master. Doing this requires adjusting the path to the CHANGELOG.md file in the .csproj as well as adjusting GitVersion.yml (removing settings for feature/nuget branch).
• create a release of the nuget packages, and use these released nuget packages in the other projects.

This change would allow applications to more easily and broadly use vernacular versification.

The methods char.IsDigit and char.GetNumericValue (used in VerseRef.TryGetVerseNum) only handle base-10, non-surrogate numerals (Complete list here). A broader alternative to these methods would have to be found or made.

The value calculation in TrySetVerseUnicode also currently assumes a decimal base:
vNum = (short)(vNum * 10 + (romanOnly ? ch - '0' : char.GetNumericValue(ch)));
This would have to be changed.

See discussion on #999 for additional context and (ignored) test cases.

Search on "a". Scroll up / down repeatedly. Eventually the scrollbar thumb starts losing the mouse capture. The longer you scroll, the worse it seems to get.

Crossref: https://jira.sil.org/browse/BL-482

See sillsdev/flexbridge#289

• Windows installer
• Linux packaging #1028

The upload and validation of the symbol packages for SIL.Windows.Forms.dll and SIL.Windows.Forms.Keyboarding.dll fails. The reason is that both projects use ILRepack to merge with some non-strongnamed assemblies. Using ILRepack removes the necessary debug sections from the PE file.
A solution would be to create/update nuget packages for the assemblies that are not strong-named and remove ILRepack. For the Interop dlls we might want to include them in the SIL.Windows* nuget package.

Nuget packages that have to be strong-named:

• DialogAdapters
• MarkdownDeep.Patched
• Sign the assemblies included in lib

This leads to a crash:

Msg: Could not load file or assembly 'Keyman10Interop, Version=10.0.0.0, Culture=neutral, PublicKeyToken=e33b67d3bb5581e4' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
Class: System.IO.FileLoadException
Source: SIL.Windows.Forms.Keyboarding
Assembly: SIL.Windows.Forms.Keyboarding, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cab3c8c5232dfcf2
Stack:    at SIL.Windows.Forms.Keyboarding.Windows.KeymanKeyboardAdaptor.GetInstalledKeymanVersion()
  at SIL.Windows.Forms.Keyboarding.KeyboardController.SetDefaultKeyboardAdaptors()
  at SIL.Windows.Forms.Keyboarding.KeyboardController.Initialize(IKeyboardRetrievingAdaptor[] adaptors)
  at SIL.Transcelerator.TxlPlugin.<>c__DisplayClass9_0.<Run>b__0() in C:\Projects\Transcelerator\Transcelerator\TxlPlugin.cs:line 151
  at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
  at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
  at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
  at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
  at System.Threading.ThreadHelper.ThreadStart()
Thread: Transcelerator
Thread UI culture: en-US
Exception: System.IO.FileLoadException
--Error Reporting Properties--
Version: 1.3.0.0 (apparent build date: 9/25/2019)
CommandLine: "C:\Windows\Microsoft.NET\Framework\v4.0.30319\AddInProcess32.exe" /guid:9733f9a6-c557-4203-b92b-d21dcaec29b6 /pid:21204
CurrentDirectory: C:\Program Files (x86)\Paratext 9
MachineName: SWD-BOGLE-LT
OSVersion: Windows 10
DotNetVersion: 4.0.30319.42000
WorkingSet: 23691264
UserDomainName: WAXHAW
UserName: Bogle
Culture: en-US
Plugin Name: Transcelerator
Host Application: Paratext 9.0.63.5

There's no actual need to initialize m_fileDialog (etc.) until we actually get ready to show the dialog for the first time. By accessing abstract properties in the constructor, it prevents Subclasses from having arguments passed in to their constructors that might be needed to implement the abstract members. In this case, there are times when subclasses might need to have a more meaningful Title based on something that is not static.

The alltags.json file is now langtags.json in the upstream WS Tech sldr repo
The update url for alltags.json

The cached file in SIL.WritingSystems/Resources needs to be changed and in LanguageRegistryResources
The code in SIL.WritingSystems/Sldr.cs needs to be updated to update the new file
plus any comments that refer to alltags

When the HTML code that gets displayed in the SILAboutBox or ShowReleaseNotesDialog contains links, the user can click on the link and the new page loads in the dialog. This happens on both WinForms and Gecko browsers, and is independent of the AllowNavigation setting. Clicking on a link should open the link in an external web browser (e.g. call Process.Start with the URL).

From the MSDN documentation it sounds like setting AllowNavigation to false should do what we want, but it doesn't seem to work.

LoadIanaSubtags() in StandardTags.cs is using "\r\n" to split tags. This fails on Linux which uses "\n" for a linefeed only. There are other instances of "\r\n" being used in the Palaso Library for linefeeds and these may also cause problems on Linux. Error can be recreated using test console app below.

using System;
using Palaso.IO;
using Palaso.Reporting;
using Palaso.UI.WindowsForms.WritingSystems;

namespace Palaso_LookupIsoCodeModel_test
{
class MainClass
{
public static void Main (string[] args)
{
LookupIsoCodeModel _lookupIsoCode = new LookupIsoCodeModel();
}
}
}

If we can figure out an appropriate way in ReflectionHelper to detect that we're running under Visual Studio Test Platform, it should be possible to change the logic to return the executing assembly. Then maybe in FileLocationUtilities.GetFileDistributedWithApplication, we could note that we're running from inside a package location and search within the package contents to find the desired "distfile."
This issue came up because required distfiles were added to GlyssenEngine (for use in Vessel), but when running unit tests using microsoft.testplatform.testhost, it works very differently from nunit, and it cannot locate those files.

The recent changes to the keyboard switching code (#953) causes a crash on Ubuntu 20.04 because it's using a different version of libgnome-desktop-3 (libgnome-desktop-3.so.19 instead of libgnome-desktop-3.so.17).

This only happens on the feature/nuget branch.

• merge master into feature/nuget again
• announce that the merge of the feature/nuget branch will be made the main development branch
• create a git tag on master (pre-nuget ?) before doing the merge
• merge feature/nuget into master. Doing this requires adjusting the path to the CHANGELOG.md file in the .csproj as well as adjusting GitVersion.yml (removing settings for feature/nuget branch).
• create a release of the nuget packages, and use these released nuget packages in the other projects.

WritingSystemFromWindowsLocaleProvider:GetLanguageAndKeyboardCombinations() can crash here:

yield return Tuple.Create(cleaner.GetCompleteTag(), language.LayoutName);

with

Object reference not set to an instance of an object. Source: System.Windows.Forms Assembly: System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 Stack: at System.Windows.Forms.InputLanguage.get_LayoutName() at Palaso.UI.WindowsForms.WritingSystems.WritingSystemFromWindowsLocaleProvider.<GetLanguageAndKeyboardCombinations>d__4.MoveNext() in

We cannot reproduce this, but a user is getting it. Presumably he has something corrupt in his languages installation on Windows. At the very least, we could improve the error message to list the name of the language and suggest that it may be corrupt on his machine.

• merge master into feature/nuget again
• announce that the merge of the feature/nuget branch will be made the main development branch
• create a git tag on master (pre-nuget ?) before doing the merge
• merge feature/nuget into master. Doing this requires adjusting the path to the CHANGELOG.md file in the .csproj as well as adjusting GitVersion.yml (removing settings for feature/nuget branch).
• create a release of the nuget packages, and use these released nuget packages in the other projects.

Keyman keyboards don't show up as available keyboards in Ubuntu 20.04.

See LT-20189 and LT-20196

447 of the ethnologue entries in our LanguageIndex.txt are marked as pejorative.

• test designer
• remove separate Designer projects that were added as a hack - does not apply
• test building on Linux - waiting on sillsdev/liblcm#164

If you add multiple attachments to an IEmailMessage and send that email, all except the first attachment are silently and unexpectedly ignored by most implemented email providers although at least some of underlying email systems accept multiple attachments.

I checked this file;
https://github.com/sillsdev/libpalaso/blob/master/SIL.WritingSystems/Resources/alltags.txt
Why does some of the tags have a star in front of them and others have not?
Is there a documentation somewhere on how this file is structured?

AcquireToken tries to get the token, then tries repeatedly over a number of seconds while saying "waiting". However, when the other app closes, no matter how long you wait, it never acquires the token, always gives up.

The reason is cleare: AcquireTokenQuietly(), which is calls repeatedly, starts with

if (s_fileLock != null)
      return false;

• merge master into feature/nuget again
• announce that the merge of the feature/nuget branch will be made the main development branch
• create a git tag on master (pre-nuget ?) before doing the merge
• merge feature/nuget into master. Doing this requires adjusting the path to the CHANGELOG.md file in the .csproj as well as adjusting GitVersion.yml (removing settings for feature/nuget branch).
• create a release of the nuget packages, and use these released nuget packages in the other projects.

The RichTextBox control uses the default size limit of MaxValue as its MaxLength. We need to check to see whether the text to append will fit, as described here.

While debugging a failing unit test in the LfMerge project, I got the following exception:

SIL.PlatformUtilities.NativeException : An error with the number, 4, ocurred.
  at SIL.Threading.GlobalMutex+LinuxGlobalMutexAdapter.Wait () [0x0002d] in /var/lib/TeamCity/agent/work/60b6a3b495b7759c/SIL.Core/Threading/GlobalMutex.cs:208 
  at SIL.Threading.GlobalMutex+LinuxGlobalMutexAdapter.Init (Boolean initiallyOwned) [0x0007a] in /var/lib/TeamCity/agent/work/60b6a3b495b7759c/SIL.Core/Threading/GlobalMutex.cs:198 
  at SIL.Threading.GlobalMutex.InitializeAndLock (System.Boolean& createdNew) [0x00006] in /var/lib/TeamCity/agent/work/60b6a3b495b7759c/SIL.Core/Threading/GlobalMutex.cs:98 
  at SIL.FieldWorks.FDO.Infrastructure.Impl.SharedXMLBackendProvider.StartupInternal (Int32 currentModelVersion) [0x00000] in <filename unknown>:0 
  at SIL.FieldWorks.FDO.Infrastructure.Impl.FDOBackendProvider.StartupInternalWithDataMigrationIfNeeded (IThreadedProgress progressDlg) [0x00000] in <filename unknown>:0 
  at SIL.FieldWorks.FDO.Infrastructure.Impl.FDOBackendProvider.StartupExtantLanguageProject (IProjectIdentifier projectId, Boolean fBootstrapSystem, IThreadedProgress progressDlg) [0x00000] in <filename unknown>:0

The error number in SIL.PlatformUtilities.NativeException comes from Marshal.GetLastWin32Error(), which on Linux returns the latest value of errno, the Unix C library's all-purpose error number. http://www.virtsync.com/c-error-codes-include-errno lists errno code 4 as EINTR, "Interrupted system call". According to this SO question, this blog post, and this libc manual entry, the right thing to do when EINTR is received is usually to restart the interrupted system call. (If EINTR was received because the user hit Ctrl-C or ran kill (your process ID), then your code should already be handling that signal elsewhere and shutting down the program.)

In this case, that's certainly the right approach. The LinuxGlobalMutexAdapter needs to check for EINTR and handle it by retrying the appropriate system call, up to small number of times (say, 5). I don't currently have time to work on a patch for this, but I will have some time next month if nobody gets to this issue before then.

At least in test app, haven't tried in FW.

It looks like GeckoFxWebBrowserAdapter.AllowNavigation is implemented incorrectly. If AllowNavigation == false then no page can be loaded at all. If AllowNavigation == true the page loads, but it also allows to navigate on that page.

MSDN states that WebBrowser.AllowNavigation indicates whether the control can navigate to another page after its initial page has been loaded. GeckoFxWebBrowserAdapter.AllowNavigation should behave identical.

The behaviour can be observed by opening the SILAboutBox with geckofx from the PalasoUIWinForms test app. The browser will only show an empty browser control on the right side.

Please connect this repo at https://reviewable.io/repositories so that all PRs get linked to reviews. Thanks!

• test using the designer with VS 2019 16.8 in the projects
• remove separate Designer projects that were added as a hack
• test building on Linux

• merge master into feature/nuget again
• announce that the merge of the feature/nuget branch will be made the main development branch
• merge feature/nuget into master. Doing this requires adjusting the path to the CHANGELOG.md file in the .csproj as well as adjusting GitVersion.yml (removing settings for feature/nuget branch).

• test using Designer with VS 2019 16.8
• remove separate Designer projects that were added as a hack
• test building on Linux

The current work is in https://gerrit.lsdev.sil.org/c/FieldWorks/+/7869

Build succeeds (at least on my machine on Linux, haven't checked on Windows), but some test failures.

• Get Windows build.bat working with VS 2019

• Update NUnitReport to look in the right folder. Issue #1038

• Fix COM errors in tests

• Fix 32-bit Windows build that is still failing with ICU errors on some Tom's and Eberhard's machine

• Re-Introduce LCMTemplate file copying (GOLDEtic, POS, ect.)

• Diagnose and repair issues around icu data files

• Update the wiki page(s) about dependencies (at least his one: https://github.com/sillsdev/FwDocumentation/wiki/Dependencies-on-Other-Repos#building-dependencies-locally)

Currently xkb keyboards are not supported on Cinnamon if ibus is not installed.

Currently any ICU collation rule that contains a comment (# this is a comment) is considered invalid by IcuRulesCollator.ValidateSortRules().

The parser should follow the syntax that ICU uses (http://userguide.icu-project.org/collation/customization, http://www.unicode.org/reports/tr35/tr35-collation.html#Rules).

https://www.w3.org/International/questions/qa-choosing-language-tags#regionsubtag mentions that deprecated subtags should be avoided when creating language tags. Yet they appear in the Region list of the "Script and Variant" dialog (e.g. German Democratic Republic). The SIL.WritingSystems/Resources/ianaSubtagRegistry.txt file contains the Deprecated information.

Examples: Papua New Guinea: etr, gaj, oro, snc, tbg. Nigeria: kcg, mzk, jen

Comparing those languages with the problem with those without it, the pattern I see is that, on Ethnologue.com, those languages where the script information is displayed have grown this "-Latn" part of their tag:

• merge master into feature/nuget again
• announce that the merge of the feature/nuget branch will be made the main development branch
• merge feature/nuget into master. Doing this requires adjusting the path to the CHANGELOG.md file in the .csproj as well as adjusting GitVersion.yml (removing settings for feature/nuget branch).
• create a release of the nuget packages, and use these released nuget packages in the other projects.

• merge master into feature/nuget again
• announce that the merge of the feature/nuget branch will be made the main development branch
• create a git tag on master (pre-nuget ?) before doing the merge
• merge feature/nuget into master. Doing this requires adjusting the path to the CHANGELOG.md file in the .csproj as well as adjusting GitVersion.yml (removing settings for feature/nuget branch).
• create a release of the nuget packages, and use these released nuget packages in the other projects.