hofstadter-io / self-driving-desktop

Desktop Automation Framework. Drive your mouse and keyboard with an easy to use language.

self-driving-desktop

Desktop Automation Framework. Drive your keyboard and mouse with text files.

pip3 install self-driving-desktop

Running

Playing a playlist:

sdd playlist.txt

Recording a playlist:

sdd --record recording.txt

Playlists

# Import other playlist files
import "test/main.txt";

# Create Coordinates
coords {
  "center": {
    "1080p": [960, 540],
    "720p":  [640, 360]
  }
};

# Create a playlist
playlist "openChrome" {
  # Run programs in the shell
  shell "google-chrome";
  sleep 2.0;

  # Name the new window
  active "hofChrome";
  sleep 0.5;

  # Use hotkeys to arrange
  hotkeys "winleft" "right";
  sleep 1;
};

playlist "closeChrome" {
  # Focus a named window
  focus "hofChrome";
  hotkeys "alt" "f4";
  sleep 1;
};

playlist "readTheDocs" {
  # Go to a webpage
  focus "hofChrome";
  sleep 0.2;

  # Type the URL
  write "https://docs.hofstadter.io\n" 0.05;

  # Goto an imported coordinate
  coord "getting-started" 0.5;
};

# Move the mouse in a square
playlist "repeatTest" {
  mm 100 100 1;
  mm 1000 100 1;
  mm 1000 500 1;
  mm 100 500 1;
};

# Our main playlist
playlist "main" {
  # Goto a named coordinate, also with offset
  coord "center" 1;
  coord "center" 250 -250 1;

  # Operate the browser
  play "openChrome";
  play "readTheDocs";
  play "closeChrome";

  # Play a playlist multiple times
  play "repeatTest" 4;

};

# Set screen size
screen "1080p";

# Set the global delay between steps
delay 0.025;

# Finally, play our main playlist
play "main";

Grammar

Top-level:

• file has steps and playlists
• steps are the only thing run
• play runs a playlist

# relative imports from file
import "relative/path.txt";

# named coordinates
coords {
  # coord name
  "center": {
    # screen identifier
    "1080p": [960, 540],
    "720p":  [640, 360]
  }
};

# define playlists
playlist "my-playlist" {
  steps...;
};

playlist "main" {
  steps...;
  # run playlists from playlists
  play "my-playlist"
}

# set the screen identifier
screen "1080p";

# run a playlist
play "my-playlist" "main";

Steps:

• play "name" "nameB" ... [N];: run one or more playlists, optionally repeat N times.
• delay x.y;: set delay between steps to x.y seconds
• sleep x.y;: sleep for x.y seconds
• screen "screen";: set the screen resolution identifier
• shell "quoted strings"+;: exec a command from the program

windows:

• active "someName";: name the active window
• focus "someName";: focus a named window

mouse:

• mouse x y s;: move the mouse to x,y in s seconds
• coord "name" s;: move the mouse to a named coordinate in s seconds
• coord "name" x y s;: move to a named coordinate with offset in s seconds
• click;: click the left mouse button
• btnclick [left,middle,right];
• btndown [left,middle,right];
• btnup [left,middle,right];
• drag [left,middle,right] x y s;: drag the mouse to x,y in s seconds
• scroll n;: scroll n lines, negative is up
• hscroll n;: horizontal scroll n "clicks", negative is left

keyboard:

• keypress "key";
• keydown "key";
• keyup "key";
• hotkeys "quoted" "keys" ...;: press some keys together
• write "quoted string\n";: type a string, "\n" is enter

clipboard:

• copy;, just ctrl-c
• paste;, just ctrl-v
• save_clipboard "name"; save the clipboard contents to "name"
• load_clipboard "name"; load the clipboard contents from "name"
• copy_clipboard "name"; copy && save the clipboard contents to "name"
• paste_clipboard "name"; load the clipboard contents from "name" && paste

all keys are from pyautogui

Grammer Definition

Recording

You can record your mouse and keyboard to a playlist file by:

sdd record.txt --record

Note, not all keys are working yet.

A keymap to fix some is here.

Development Setup

virtualenv --python python3 penv
source penv/bin/activate
pip install -r requirements.txt
export PYTHONPATH=.
python self_driving_desktop/__main__.py ...

Install from local repository:

git clone https://github.com/hofstadter-io/self-driving-desktop
pip3 install ./self-driving-desktop/

self-driving-desktop's Issues

Can't install with pip

Python noob here...
if I try: pip install self-driving-desktop
I get this output with errors:

  Using cached https://files.pythonhosted.org/packages/3c/09/457c88f2df7ae33534bf811d30a7e3bc75aed976c666d379671c448e54ec/self-driving-desktop-0.0.1.tar.gz
Building wheels for collected packages: self-driving-desktop
  Running setup.py bdist_wheel for self-driving-desktop ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-b02n7A/self-driving-desktop/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpKOsM0xpip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_scripts
  creating build
  creating build/scripts-2.7
  error: file '/tmp/pip-build-b02n7A/self-driving-desktop/sdd' does not exist
  
  ----------------------------------------
  Failed building wheel for self-driving-desktop
  Running setup.py clean for self-driving-desktop
Failed to build self-driving-desktop
Installing collected packages: self-driving-desktop
  Running setup.py install for self-driving-desktop ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-b02n7A/self-driving-desktop/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-wYTCD7-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    running install
    running build
    running build_scripts
    creating build
    creating build/scripts-2.7
    error: file '/tmp/pip-build-b02n7A/self-driving-desktop/sdd' does not exist
    
    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-b02n7A/self-driving-desktop/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-wYTCD7-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-b02n7A/self-driving-desktop/```

data files & usage

would like to be able to read a data file during playback, and fill in fields based on access to that data

pip install seems to be broken

make pip installable

• make necessary changes
• get into PyPy or where ever
• profit

repeat

repeat N { ... }

named clipboards

• copy and paste
• want to be able to save to a named clipboard system
• want to be able to collect data items from different applications and/or place them into other windows / applications

sdd playlist.txt error

python 2 support

tbd... are all deps py2 compat?

What would one use this for?

It's cool, but I can't think of example use cases.

named coordinates

named coordinates, support for different screen resolutions

OS support

detect and switch commands based on the OS, tbd...

Include a license

Can we have demo in GIF

imports

implemented in 0.0.6

expand functionality

there is a bunch of pyautogui that would be nice to add