xtuer / ayase

🥥 Control everything by keyboard.

Ayase

简体中文 | For Simplified Chinese version, please click here

Ayase is a tool which can help you to control everything by keyboard. This is specifically designed for lazy people like me and blind people. It is known to all that in the most cases, we have to control the computer by both mouse and keyboard. However, the frequent switch for our hand between mice and keyboards would severely threaten the health of our wrists. Meanwhile, for blind people, though in most cases, it is convenient enough to use the screen readers to control elements on screen, most softwares cannot control both precisely and swiftly. You have to wait for the element or switch many times to find the element you want to control. With Ayase, these problems are all solved.

About Naming (Ayase)

I love Mitsukasa Ayase forever!

https://zh.moegirl.org.cn/三司绫濑

Features

• Support high DPI, different DPI across multiple monitors, cross monitor display
• Support Chrome-based Apps (e.g. Github Desktop)
• Support Electron Apps (e.g. Visual Studio Code)
• Support WinForms, WPF, UWP (e.g. Netease Music UWP)
• Support the self-developed UI frameworks (e.g. Wechat & QQ of Tencent Inc.)
• Scanning of UI tree with high speed
• Support pinyin for Chinese (e.g. 网易云 → wyy / wangyiyun)
• Ignore upper and lower case

For the discussion on accessibility development and speed optimization, please refer to my following articles. Finally, the implementation scheme I adopted is to use C++ to invoke Win32API / uiautomation.h and compile into dll (dynamic link library). Then, I used P/Invoke in the C# / WPF project to use the library. In most cases, the performance was improved for about 80%.

• .NET C# AutomationElement 获取桌面UI元素
• .NET C# 手撕线程池 | 县城撕裂者
• .NET C# AutomationElement 加速探索
• .NET C# UIAutomationElement 获取桌面元素
• .NET C# 通过Win32API获取窗口及控件的信息
• .NET C# 获取桌面UI元素总结
• .NET C# MSAA 获取桌面UI元素
• .NET C# C++ 混编问题

Note that I do not have additional time and effort to translate those articles into English. Using translators can also understand the articles well. If you still have any questions or advice, you can contact me directly. You can find ways to contact me in my GitHub homepage. This is the first time I write WPF / C# this formally, so if you have any advice, please don't hesitate to raise issues or pull requests.

Project Structure

.
├── Ayase                                   # Ayase Project
│   ├── AccessibilityBridge                 # P/Invoke bridge for .dll
│   │   ├── GNativeIUIAutomationManager.cs
│   │   └── GNativeUIElement.cs
│   ├── App.xaml
│   ├── App.xaml.cs                         # Entrance
│   ├── AssemblyInfo.cs
│   ├── AutomationLib                       # Most of them are no longer used. Serve as alternatives
│   │   ├── GAutomationManager.cs
│   │   ├── GIUIAutomationManager.cs
│   │   ├── GMSAAManager.cs
│   │   ├── GRawUIManager.cs
│   │   ├── GUIElement.cs
│   │   └── Wrapper                         # Wrapper for Win32API used in this section
│   │       ├── ObjectIdentifiers.cs
│   │       ├── RECT.cs
│   │       ├── ReferenceIdentifiers.cs
│   │       └── Win32API.cs
│   ├── Ayase.csproj
│   ├── Ayase.csproj.user
│   ├── HotkeyLib                           # Global hotkey wrapping
│   │   ├── GHotKey.cs
│   │   ├── GKeybdEvent.cs
│   │   └── VirtualKeys.cs
│   ├── MouseLib                            # Mouse control wrapping
│   │   └── GMouse.cs
│   ├── Properties
│   │   └── launchSettings.json
│   ├── ScreenLib                           # Screen library wrapping
│   │   └── PrimaryScreen.cs
│   ├── ThreadLib                           # Private thread pool
│   │   └── GThreadPool.cs
│   └── UI                                  # UI
│       ├── FormMaskWindow.xaml
│       ├── FormMaskWindow.xaml.cs
│       ├── MaskWindow.cs
│       ├── NotationLabel.cs
│       ├── ReuseWindow.cs
│       ├── ScreenMaskWindow.xaml
│       ├── ScreenMaskWindow.xaml.cs
│       ├── SearchWindow.xaml
│       ├── SearchWindow.xaml.cs
│       ├── SettingsWindow.xaml
│       ├── SettingsWindow.xaml.cs
│       └── WindowManager.cs                # Windows scheduling
├── Ayase.Accessibility                     # dll project
│   ├── Ayase.Accessibility.vcxproj
│   ├── Ayase.Accessibility.vcxproj.filters
│   ├── Ayase.Accessibility.vcxproj.user
│   ├── GConstant.h                         # Some constants
│   ├── GIUIAutomationManager.cpp           # Main implementation
│   ├── GIUIAutomationManager.h
│   ├── GUIElement.cpp                      # Wrapping for data structure
│   ├── GUIElement.h
│   ├── dllmain.cpp
│   ├── framework.h
│   ├── pch.cpp
│   └── pch.h
└── Ayase.sln

The TOC for generated project:

.
├── ...
├── Ayase.Accessibility.dll
└── <.net target>
    ├── ...
    └── Ayase.exe

Dependencies

• Microsoft.Toolkit.Uwp.Notifications
• Notifications.Wpf
• ToolGood.Words

Screenshots

Use Ayase to control Netease Music UWP:

Use Ayase to surf on the Internet:

Usage

• Start UI Scanning: Alt + CapLocks (We would adjust the status of CapLocks automatically)
• Exit UI Scanning: Esc
• Next Element: Tab
• Previous Element: Shift + Tab
• Click the Selected Element: Enter
• Move the mouse to the Selected Element: Shift + Enter

TODO

• Add voice module
• Add settings module
• Add support for control by AWSD
• Add pure mouse mode