takeyamayuki / nonmouse

a webcam-based virtual gesture mouse that is easy to use with hands on the desk.

a webcam-based virtual gesture mouse that is easy to use with hands on the desk

---

Features

• No need for anything other than a webcam and host PC.
• Gesture-based HCI has been around for some time, but shooting from above makes the mouse easier to use.
• NonMouse can be invoked by the global hotkey even when this application is inactive.
• Works well with typing.
• Just download from the latest release(windows, mac only).

Installation

📁 Executable file

Download the zip file that matches your environment from the latest release.

OR

🐍 PyPI

Run the following script.

$ pip install nonmouse

(If you have trouble installing mediapipe, please visit the official website.)

Usage

1. Install a camera

The following three ways of placing the device are assumed.

• Normal: Place a webcam normally and point it at yourself (or use your laptop's built-in camera)

• Above: Place it above your hand and point it towards your hand.

• Behind: Place it behind you and point it at the display.

2. Run

• Run the executable as described in the GitHub wiki.

  OR

• Run the following script from the continuation of the installation.

  For windows and linux(global hotkey function does not work in linux.)

  $ nonmouse

  For MacOS, you need execute permission.

  $ sudo nonmouse

3. Settings

When you run the program, You will see a screen similar to the following. On this screen, you can set the camera and sensitivity.

• Camera
  Select a camera device. If multiple cameras are connected, try them in order, starting with the smallest number.

• How to place
  Select the location where you placed the camera. Place the camera in one of the following positions: Normal, Above, Behind in [📷 Install a Camera].

• Sensitivity
  Set the sensitivity. If set too high, the mouse cursor will shake slightly.

When you are done with the settings, click continue. The camera image will then be displayed, and you can use NonMouse with the settings you selected.

4. Hand Movements

stop cursor	left click	right click	scroll

The following hand movements are enabled only when you hold down Alt(Windows), Command(MacOS). You can define your own global hotkeys by rewriting here. You can use this function even if the window is not active.This feature is only available on windows and mac.

• cursor
  • Mouse cursor: tip of index finger → A blue circle will appear at the tip of your index finger.
  • Stop mouse cursor: Attach the tip of your index finger to the tip of your middle finger. → The blue circle disappears.
• left click
  • Left click: Attach the fingertips of your thumb to the second joint of your index finger → A yellow circle will appear on the tip of your index finger.
  • Left click release: Release the thumb fingertip and the second joint of the index finger. → The yellow circle disappears.
  • Double click: Left click twice within 0.5 seconds.
• other
  • Right click: Hold the click state for 1.5 second without moving the cursor. → A red circle will appear at the tip of your index finger.
  • Scroll: Scroll with the index finger with the index finger folded → a black circle will appear.

Note

• Use it with a bright light at hand.
• Keep your hand as straight as possible to the camera.

5. Quit

Press Ctrl+C, when a terminal window is active.
Press close button(Valid only on windows, linux) or Esc key, when an application window is active.

Build

Note
The built binary files can be downloaded from latest realease.

In app-mac.spec and app-win.spec, change pathex to fit your environment.
Run the following scripts for each OS.

• windows

  Copy and paste the location obtained by pip show mediapipe into datas, referring to what is written originally.
  Run the following script.

  $ pip show mediapipe
  ...
  Location: c:\users\namik\appdata\local\programs\python\python37\lib\site_packages
  ...
  #Copy and paste into the datas in win.spec
  $ pyinstaller config/win.spec
  ... ````

• mac

  Create a venv environment and perform pip install, because the directory specified in datas is for an assumed venv environment.

  $ git clone https://github.com/takeyamayuki/NonMouse.git
  $ cd NonMouse
  $ python3 -m venv venv
  $ . venv/bin/activate
  (venv)$ pip install -r requirements.txt
  (venv)$ pyinstaller config/mac.spec

nonmouse's Issues

need refactor

• Low code quality
• No environment-dependent resolution -> PyPI support
• Need appropriate build environments (pyinstaller usage)
• Need CI with github actions

ZeroDivisionErrorでクラッシュする

cfps = int(cap.get(cv2.CAP_PROP_FPS))
でcfpsが1になって
if cfps < 30: cap.set(cv2.CAP_PROP_FRAME_WIDTH, cap_width) cap.set(cv2.CAP_PROP_FRAME_HEIGHT, cap_height) cfps = int(cap.get(cv2.CAP_PROP_FPS))
を通してもcfpsが1になり、
ran = int(cfps/10)
でranが0になってしまい、ZeroDivisionErrorでクラッシュします。

環境は以下の通りです
PC: MacBook Pro 2020 Intel
OS: macOS Monterey
カメラ: Mac内蔵Webカメラ
Pythonバージョン: 3.9.7

tasks2

• タッチディスプレイモード追加
• アプリケーション化

カメラや手が傾くと、自分の指先移動量に対するマウス移動量が異なってしまう

たとえば、カメラを斜めに取り付けると、カメラに映る指は縦方向に圧縮されたものになり、当然移動量もその分小さくなる。
現在の自分の環境だと手に向けてまっすぐ取り付けられているので問題はないが、初めて使う人にとってはこのアプリケーションを使う上での大きな壁となる。

「解決策」
まず、基準となる指の関節座標を用意しておく。
その基準となる座標群と現在の座標群を比較して手の傾き具合を求める。
それによって、dx,dy変える

USB cameras not working?

Sorry for not posting a lot of debug info here, but I am trying to test this and only can get it working for my mounted webcam. Anytime I try to use a usb web cam the device does not appear to work properly for the program.

I will try to pull down the project soon and see if I can get some more info as to what or why they are not working.

tasks

• スクロールの機能変更(手を回転)
• 手首基準座標軸追加
• app-official.pyにargs追加
• GUI化
• スクロールの機能変更(親指以外全部くっつける)

PyObjCの依存関係エラー

requirements.txtの通りにインストールすると、pyobjcによるエラーが発生する。そのときは、以下のように、pyobjcのダウングレードをすること
moses-palmer/pynput#420

pip install cannot find mediapipe==0.810.1

When I run pip install nonmouse as mentioned in the readme, I get the following error: Perhaps the mediapipe version can be updated?

ERROR: Could not find a version that satisfies the requirement mediapipe==0.8.10.1 (from nonmouse) (from versions: 0.9.0.1, 0.9.1.0, 0.9.2.1, 0.9.3.0, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.5, 0.10.7, 0.10.8)
ERROR: No matching distribution found for mediapipe==0.8.10.1

Thank you for this package.

How to set screen size?

Hello the mouse only works on left two thirds of the screen. I gues it has something to do with screen size, my lapto has 1920x1080. Can I reconfigure it somewhere?

[bug report] Sometimes the orientation of the hand does not match the orientation of the image

NonMouse/app.py

Lines 115 to 155 in 4603acc

	# 手の向きでモードを変える
	nowLandX = hand_landmarks.landmark[4].x - \
	hand_landmarks.landmark[20].x
	nowLandY = hand_landmarks.landmark[12].y - \
	hand_landmarks.landmark[0].y
	if m == 0:
	preLandX = nowLandX
	preLandY = nowLandY
	m = 1
	if nowLandX > 0:
	mode = 2 # 左右反転
	if nowLandY > 0:
	mode = 1 # 上下反転
	if nowLandY < 0 and nowLandX < 0:
	mode = 0
	if mode == 0:
	if nowLandX > 0 and np.sign(nowLandX) != np.sign(preLandX):
	mode = 2 # 左右反転
	if nowLandY > 0 and np.sign(nowLandY) != np.sign(preLandY):
	mode = 1 # 上下反転
	if nowLandY < 0 and nowLandX < 0 and \
	(np.sign(nowLandY) != np.sign(preLandY) or np.sign(nowLandX) != np.sign(preLandX)):
	mode = 0
	elif mode == 1:
	if nowLandX > 0 and np.sign(nowLandX) != np.sign(preLandX):
	mode = 2 # 左右反転
	if nowLandY < 0 and np.sign(nowLandY) != np.sign(preLandY):
	mode = 1 # 上下反転
	if nowLandY > 0 and nowLandX < 0 and \
	(np.sign(nowLandY) != np.sign(preLandY) or np.sign(nowLandX) != np.sign(preLandX)):
	mode = 0
	elif mode == 2:
	if nowLandX < 0 and np.sign(nowLandX) != np.sign(preLandX):
	mode = 2 # 左右反転
	if nowLandY > 0 and np.sign(nowLandY) != np.sign(preLandY):
	mode = 1 # 上下反転
	if nowLandY < 0 and nowLandX > 0 and \
	(np.sign(nowLandY) != np.sign(preLandY) or np.sign(nowLandX) != np.sign(preLandX)):
	mode = 0
	preLandX = nowLandX
	preLandY = nowLandY

error: PyObjC requires macOS to build

Hello. When installing requirements on Linux I get this error:

Collecting pyobjc-core==7.3
  Downloading pyobjc-core-7.3.tar.gz (684 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 684.2/684.2 KB 5.4 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [2 lines of output]
      running egg_info
      error: PyObjC requires macOS to build
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

MacOS MontereyでカメラFPSが5Hz程度になる

NonMouse/app.py

Lines 117 to 121 in 1e7f7bc

	cfps = int(cap.get(cv2.CAP_PROP_FPS))
	if cfps < 30:
	cap.set(cv2.CAP_PROP_FRAME_WIDTH, cap_width)
	cap.set(cv2.CAP_PROP_FRAME_HEIGHT, cap_height)
	cfps = int(cap.get(cv2.CAP_PROP_FPS))

cfps = int(cap.get(cv2.CAP_PROP_FPS))で取得するカメラのFPSが5Hzになってしまい、カーソルが小刻みに揺れてしまう。
これは恐らく、cv2の問題だと思われる。

I cannot run newest version on win 11

App wont start, any ideas why?