aiongg / khiin Goto Github PK

The keyboard viewmodel should handle communication with the EngineManager and provide UI state for the on screen keyboard.

For the eventual Mac & iOS apps, the engine will need to be buildable for these targets. I'm not sure what the latest status on this topic is, since I haven't had an Apple machine for a while. It should be possible to either use the C++ directly, or wrap it in Objective-C. See this question on SO.

See also the UIKit docs for a custom keyboard and InputMethodKit.

Feature breakdown:

• Building khiin engine for iOS & Mac
• Loading and calling khiin engine methods from iOS / Mac code
• Passing protobuf data back and forth between engine & client app
• Keyboard UI (keys & candidate display)
• Additional UI (settings, etc.)

There is at least one or potentially a few different crashes that I believe are the result of null pointers. (Edit: or not, see below.) I have not been able to locate the source of them yet. Debugging is a hassle because you need to attach to an app like Notepad, but the IME DLL will then be loaded into whatever application you are using for debugging (e.g. Visual Studio) and subsequently cause another crash.

I've tried to use iterators as much as possible to limit the possibility of going beyond array bounds, but apparently I missed some that have still not been resolved. I believe the problem areas are around BufferMgr.cpp#L561 or potentially in BufferElement.cpp.

The khiin::engine::Engine is created as a std::unique_ptr. It should be released and delivered back to Kotlin as a regular Long to keep the Engine instance available throughout the app lifecycle. Engine methods should pass this pointer back through JNI, together with the protobuf command to send to the engine. Engine instance must be deleted when the app/service shuts down.

This was working prior at 54ce721 prior to the big DB refactor. Right now a simple input of lisi in automatic mode, which should produce 汝是 as the top candidate, does not show either. It looks like it has something to do with the tones not being typed out. I did change some things in the search process / CandidateFinder to do with the TaiToken::input_size property, which might affect this.

It could also be that because the key_sequences are now generated ahead of time in Python (see aiong/khiin-data), we are tagging the input_size property incorrectly. Maybe it should be the size of the original input, before any tones were stripped. (e.g. "li" should have an input size of 3 when it represents "li2".) I'm not sure if this has anything to do with it or not, but it might.

Needs more research.

Currently for many key combinations, we collect them but do not handle them. We need to decide which combinations we will handle and for what purposes, and otherwise allow them to be passed back to Windows. For example, Shift + Arrow keys normally allows the user to select text in the input field. We probably want to keep this behavior, so we should not collect Shift + Arrow.

These checks and pass/reject should happen somewhere in KeyEventSink.

Basic information: https://developer.android.com/reference/android/inputmethodservice/InputMethodService

Getting compose lifecycle in the service: https://stackoverflow.com/questions/65755763/inputmethodservice-with-jetpack-compose-composeview-causes-composed-into-the

Communication between the Windows client app and the engine is currently synchronous. This is particularly problematic for longer database operations like initialization, which causes a very short hiccup that blocks the UI thread (it used to be half a second and is now <100ms). The client/engine communication should happen asynchronously to prevent any UI blocking.

This is the method in the Windows client application that calls the engine library and needs to be revised: https://github.com/aiongg/khiin/blob/master/windows/tip/EngineController.cpp#L187-L200

Manual input mode is stubbed but not yet implemented.

Manual input mode should be fairly straightforward: we do not show any candidate window by default as the user is typing. Tone keys and spaces are consumed as their usual functions (vs. Hanji modes where space begins candidate selection). Manual input mode should work with tone number keys or our Tai Telex schema:

2 > s
3 > f
5 > l
7,8 > j (depending on the final consonant being h)

User should still have the option to bring up the candidate window by pressing tab to select a Hanji, but it is otherwise not shown.

• UI should be themeable. User can select colors and whether the keys are flat or colored in.
• Follow standard Google keyboard design patterns for best user experience.
• Keys should popup briefly on tap, and more options should pop up on long press (select by dragging).

The code to use a custom font, either installed or a plain font file from a directory, is already in place here: https://github.com/aiongg/khiin/blob/master/windows/tip/RenderFactory.cpp

We are still missing a UI option in the Settings application for the user to select a font (and saving it in the preferences, etc.)

Currently, the database contains entries in Lômájī, and we generate the raw input sequences (e.g. Lo5ma2ji7) here. This logic should be moved out of the engine and into khiin-data so that it is available statically within the engine and does not need to be generated dynamically. The original idea was to allow users to select their own keys for certain items like tones, and regenerate these input sequence maps on demand. However, this generation is too slow and making it happen in a background thread was more difficult than expected. Moving it into khiin-data (in Python) would be much easier at this point.