dictation-toolbox / aenea

Client-server library for using voice macros from Dragon NaturallySpeaking and Dragonfly on remote/non-windows hosts.

dictation-toolbox

Meta-repository for tracking and providing information about the organization itself.

Summary

dictation-toolbox is a loose organization intended to organize projects for users of voice recognition software, with a focus on customizability, flexibility, and programming/system administration/etc. We hope to help fill in the gap between commercially available solutions for common workflows and other workflows which require customization.

We also intend to provide a contingency plan for projects to be "adopted" by new maintainers if their current maintainers leave without transferring control of the project. Each repository in dictation-toolbox has one or more owners responsible for it, and admins of dictation-toolbox agree not to use their admin privileges on repositories where they are not an owner, except in the case of all owners dropping out of all contact for four months.

We are open to including other projects under these terms to better serve the needs of the dictation community. Please open an issue in this repository if you are interested. Historically, we have made all owners of all repositories in dictation-toolbox admins of the organization.

Projects

The listed owners of each project, rather than the admins of dictation-toolbox, retain full control and responsibility for all aspects of their project.

aenea

Owners: @calmofthestorm

Client-server library for using voice macros from Dragon NaturallySpeaking and Dragonfly on remote/non-windows hosts using NatLink and Dragonfly.

aenea-grammars

Owners: @calmofthestorm

A collection of grammars written for use with Aenea, with an eye toward programming and use on Linux.

dragonfly-scripts

Owners: @nirvdrum

This repository contains a collection of Dragonfly Python-scripts, that can be used with Dragon NaturallySpeaking Professional.

aenea-resources

Owners: @calmofthestorm

Miscellaneous resources (cheatsheets, etc) for working with aenea and aenea-grammars.

aenea's Issues

Relative mouse movement is broken when one of the coordinates is "-0"

For example, Mouse("<-0, -0>") breaks relative mouse movement. (The problem is that the -- is not passed to xdotool so it has the usual issues with negative mouse movement.

Socket error connecting to aenea server

First of all, thanks for this awesome project, I highly appreciate the work and the throrough documentation!

I basically completed all the steps of the short tutorial and the host-only adapter has been set up and can be pinged from the Guest as well as the Host (aenea.json and config.py have been adapted with the same IP accordingly). However, when I try to start DNS with, I get the following error:

Socket error connecting to aenea server. To avoid slowing dictation, we won't try again for 5 seconds.

It makes tracking down the point of failure quite hard since everything else seems to work fine.

Does anyone have an idea?

Thanks a lot in advance!

Make it clear that server plugins are optional

After I pulled from master and started up the server, I was greeted with the message "Cannot import yapsy; server plugin functionality not available.". I assumed this was a new requirement and something I needed to have running, rather than something that was optional. A softer worded message, that indicates the functionality is optional, would probably be better.

stopgap doesn't load on Windows

I'm not really sure what the stopgap grammar is. It looks like maybe it's not even meant to run on Windows. But in any case, it fails when loading because the "h" modifier doesn't work with the built-in Key implementation. Knowing a bit more what this should be doing will make fixing it a lot easier, so any help is appreciated.

how to get active window title etc.

I tried a function action like

def diagnostics_command():
    window = Window.get_foreground()
    print (window.executable, window.title, window.handle)

        "diagnostics": Function(diagnostics_command),

but now when I say diagnostics, it prints the window title of the client (my Windows VM) instead of that of my X11 app. Can I somehow access the window title of the X11 app?

Server plugins don't work out-of-the-box

If yapsy is installed, server plugins are automatically enabled. However, the default configuration file doesn't set the plugin directory and it's assumed that that configuration item exists. The net result is things just break upon server start up until that configuration entry is explicitly added.

Basic integration test for server_x11.py

I'd like to see a simple integration test for the server. This could be as simple as firing up xev, setting it full screen, running the script, and verifying that the correct events were produced.

Local keys very slow if remote server not up

If the configured remote server is not up and the proxy server is enabled, local keys are severely impacted. The out-of-the-box config, for example, will point at an IP that almost certainly will fail for everyone. I'm seeing that commands like "disable proxy server" and dictating into notepad (both local actions) takes ~12s to process. While I'd certainly expect issues sending commands to the aenea server, I'd expect local keys to work unimpacted.

json-rpc ProtocolError

Hey,​
I'm trying to setup aenea on a Windows 10 machine to use in order to take advantage of the added functionality of aenea and the grammars contributed by the community.

Currently I'm stuck at the setup aenea itself. If you have a spare moment and you could look at the trace I would be grateful.

I launch windows server.
aenea.exe -a 127.0.0.1

`C:\NatLink\NatLink\MacroSystem\aenea.json`  
{
    "host": "127.0.0.1",
    "port": 8240,
    "platform": "proxy",
    "use_multiple_actions": false,
    "screen_resolution": [6400, 1440],
    "project_root": "C:\\NatLink\\NatLink\\MacroSystem",
    "restrict_proxy_to_aenea_client": false
}

The _hello_world_aenea.py test fails upon attempting to send the RPC request. Trace:

 Vocola not active
Aenea hello world grammar: Loaded.

--- WARNING: Speech Model BestMatch V is used for this User Profile
The performance of many NatLink grammars is not good with this model.
Please choose another User Profile with for example Speech Model BestMatch IV.
See http://unimacro.antenna.nl/installation/speechmodel.html
----
natlinkmain started from C:\NatLink\NatLink\MacroSystem\core:
  NatLink version: 4.1lima
  DNS version: 12
  Python version: 27
  Windows Version: 8

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\dragonfly\engines\engine_natlink.py", line 248, in results_callback
    r.process_recognition(root)
  File "C:\Python27\lib\site-packages\dragonfly\grammar\rule_mapping.py", line 181, in process_recognition
    self._process_recognition(item_value, extras)
  File "C:\Python27\lib\site-packages\dragonfly\grammar\rule_mapping.py", line 201, in _process_recognition
    value.execute(extras)
  File "C:\Python27\lib\site-packages\dragonfly\actions\action_base.py", line 127, in execute
    if self._execute(data) == False:
  File "C:\NatLink\NatLink\MacroSystem\aenea\wrappers.py", line 201, in _execute
    self._data = ensure_execution_context(data)
  File "C:\NatLink\NatLink\MacroSystem\aenea\wrappers.py", line 111, in ensure_execution_context
    data['_server_info'] = aenea.proxy_contexts._server_info()
  File "C:\NatLink\NatLink\MacroSystem\aenea\proxy_contexts.py", line 82, in _server_info
    _refresh_server()
  File "C:\NatLink\NatLink\MacroSystem\aenea\proxy_contexts.py", line 64, in _refresh_server
    _last_server_info = aenea.communications.server.server_info()
  File "C:\NatLink\NatLink\MacroSystem\aenea\communications.py", line 104, in call
    return self._execute_batch([(meth, a, kw)])
  File "C:\NatLink\NatLink\MacroSystem\aenea\communications.py", line 87, in _execute_batch
    batch[0][0])(*batch[0][1], **batch[0][2])
  File "C:\Python27\lib\site-packages\jsonrpclib\jsonrpc.py", line 276, in __call__
    return self.__send(self.__name, kwargs)
  File "C:\Python27\lib\site-packages\jsonrpclib\jsonrpc.py", line 225, in _request
    check_for_errors(response)
  File "C:\Python27\lib\site-packages\jsonrpclib\jsonrpc.py", line 529, in check_for_errors
    raise ProtocolError((code, message))
jsonrpclib.jsonrpc.ProtocolError: (-32601, u'Method not found: server_info')

I have the python libs in the global-sites packages.

dragonfly==0.6.5
jsonrpclib==0.1.3
pygame==1.9.1
pyparsing==2.0.3
pywin32==218
PyXML==0.8.4

I noticed that if I print the address from Proxy class's _execute_batch it points to u'localhost'. no matter what I pout in the aenea.json .

If I set the server to 192.168.56.1 it gives the error pasted above.

If the server is set to localhost I get the error bellow.

Socket error connecting to aenea server. To avoid slowing dictation, we won't try again for 5 seconds.

Any help much appreciated.

OSX test-client.py hangs

I'm trying to setup aenea server on OSX El Capitan. When I try to run server_osx.py followed by test-client.py I get aAB printed to my terminal after which the script seems to hang all while seemingly keeping the Shift key pressed all the time.

Is this a known issue, how good is osx support for aenea. It is not really clear from the README files.

PS: Great project, I think it this impacts/helps a lot of people in a meaningful way. I hope it to include me as well in the near future.

recent foloders choose does not work

when recent folders dialog appear, I say choose one, it make no sense. I use chinese Gui, and I have add my language to the explorer select funtion, I see the function gotoRecentFolder. but I do not see the caller.

bug: aenea raises WindowsError when Userdirectory is not set in natlink

pull request #131 introduced a bug: userDirectory can be ''

Error loading _translated_grammars from C:\NatLink\NatLink\MacroSystem\_translated_grammars.py
Traceback (most recent call last):
  File "C:\NatLink\NatLink\MacroSystem\core\natlinkmain.py", line 309, in loadFile
    imp.load_module(modName,fndFile,fndName,fndDesc)
  File "C:\NatLink\NatLink\MacroSystem\_translated_grammars.py", line 2, in <module>
    import dragonfly_grammars
  File "c:\users\nihlaeth\git\dragonfly-grammars\dragonfly_grammars\__init__.py", line 5, in <module>
    import dragonfly_grammars.aenea_
  File "c:\users\nihlaeth\git\dragonfly-grammars\dragonfly_grammars\aenea_.py", line 26, in <module>
    import aenea
  File "c:\users\nihlaeth\git\aenea\client\aenea\__init__.py", line 18, in <module>
    import aenea.communications
  File "c:\users\nihlaeth\git\aenea\client\aenea\communications.py", line 30, in <module>
    _server_config.write()
  File "c:\users\nihlaeth\git\aenea\client\aenea\configuration.py", line 56, in write
    os.makedirs(os.path.split(self._path)[0])
  File "C:\Python27-32bit\Lib\os.py", line 157, in makedirs
    mkdir(name, mode)
WindowsError: [Error 3] The system cannot find the path specified: ''

document issue with keyboard layout

Grammar configuration needs to be written

Grammars may have configuration files (NAME.py) placed in grammars_config. These files need to be copied to the VM whenever configuration is reloaded. Additionally, the existing grammars which would benefit from external configuration need to be rewritten to use this format.

Reduce duplication in verbal_emacs and multiedit grammars

Both the verbal emacs and the multiedit grammars define a lot of common formatting functions. It'd be nice if the duplication is removed and maybe the common elements extracted to a common grammar not tied to either.

Where is numbers defined?

On trying the 'zip 00' sample command, I get the following error:

...
  File "C:\NatLink\NatLink\MacroSystem\_stopgap.py", line 30, in _process_recognition
    x = numbers.get(x, x)
NameError: global name 'numbers' is not defined

I don't see where numbers is defined, even when grepping the python files.

But I guess this is evidence that at least it's working!

aenea client configuration no longer imports under linux

Seems to be due to some missing mocks from the alias stuff.

Aenea Dictation Client keystroke errors with OS X server

Thank you for creating and maintaining this excellent set of tools. I'm excited to get back to programming after many months of RSI.

I have everything working with a Windows 10 client and OS X 10.11.3 server except that I'm having trouble with the Aenea Dictation Client.

With the current code, when I start capturing dictation with the Aenea Dictation Client I get the error:

ProtocolError: (-32603, u'Server error:   File "server_osx.py", line 414, in key_press | RuntimeError: Don\'t know how to handle keystroke Control_R')

I assume the purpose of pressing control is to test that the client is working. If I comment out line 74 of aenea_client.py then dictation is captured and transferred to the OS X server until a key outside of LITERAL_KEYS is pressed, say through a command from a grammar. Then I get errors such as:

ProtocolError: (-32603, u'Server error:   File "server_osx.py", line 414, in key_press | RuntimeError: Don\'t know how to handle keystroke Num_Lock')

or

jsonrpclib.jsonrpc.ProtocolError: (-32603, u'Server error:   File "server_osx.py", line 414, in key_press | RuntimeError: Don\'t know how to handle keystroke ctrl')

And dictation no longer transfers from the client to server until I restart Dragon on Windows and restart the Aenea Dictation Client.

I assume this is a problem with key translation. Is there a way that I can fix this so that it works with the OS X server?

Here's a full traceback:

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Python27\lib\lib-tk\Tkinter.py", line 1486, in __call__
    return self.func(*args)
  File "aenea_client.py", line 217, in start_capture
    self.proxy_buffer.start_capture()
  File "aenea_client.py", line 74, in start_capture
    aenea.ProxyKey('Control_R').execute()
  File "C:\Python27\lib\site-packages\dragonfly\actions\action_base.py", line 127, in execute
    if self._execute(data) == False:
  File "C:\Python27\lib\site-packages\dragonfly\actions\action_base.py", line 179, in _execute
    self._execute_events(self._events)
  File "C:\NatLink\NatLink\MacroSystem\aenea\proxy_actions.py", line 130, in _execute_events
    aenea.communications.server.execute_batch(commands)
  File "C:\NatLink\NatLink\MacroSystem\aenea\communications.py", line 100, in execute_batch
    self._execute_batch(batch, aenea.config.USE_MULTIPLE_ACTIONS)
  File "C:\NatLink\NatLink\MacroSystem\aenea\communications.py", line 87, in _execute_batch
    batch[0][0])(*batch[0][1], **batch[0][2])
  File "C:\Python27\lib\site-packages\jsonrpclib\jsonrpc.py", line 290, in __call__
    return self.__send(self.__name, kwargs)
  File "C:\Python27\lib\site-packages\jsonrpclib\jsonrpc.py", line 238, in _request
    check_for_errors(response)
  File "C:\Python27\lib\site-packages\jsonrpclib\jsonrpc.py", line 567, in check_for_errors
    raise ProtocolError((code, message))
ProtocolError: (-32603, u'Server error:   File "server_osx.py", line 414, in key_press | RuntimeError: Don\'t know how to handle keystroke Control_R')

Key("control:down") not working

Aenea is super slick, thanks again for releasing this!

I see that Key("c:down").execute() works but repeats 'c'. I have been unable to get control:down to work, although perhaps I'm not trying the right keyword (this is the word for Dragonfly).

Let me know your thoughts? Thanks!

Better cross-platform support

We should add some additional classes to the core to make working across platforms easier. You can currently do it with ProxyPlatformContext, and that may be necessary for more complex cases, but simple things like different keystrokes should be much easier.

• ProxyPlatformAction -- performs a different action based on whether currently active is local or a server, and if a server based on what platform (and possibly WM?) it reports.
• ProxyPlatformKey/Mouse -- sugar for ProxyPlatformAction with Key objects (IE, enter different keystrokes based on platform).
• Some kind of custom RPC context that makes it easier to work with plugins maybe? Like ProxyRPCContext("get_vim_mode") would call the get_vim_mode RPC (added by a plugn presumably) and match based on the value. Maybe we don't need this and should let plugin authors do it themselves, I welcome opinions here.

Anything else in the same vein?

aenea_client batching sometimes gets text "stuck"

While it generally works perfectly, the batching code I wrote for @poppe1219's awesome dictation capture client has the property that occasionally text gets caught in the buffer.

I'll investigate this more when I have a chance, but I just wanted to make a note that if you are using it and something you say doesn't show up, saying something else usually should flush the buffer and get both the old and new speech.

OSX : Aenea server setup : test-client.py does not work for 192.168.x.x but works for loopback (127.0.0.1)

Hi All,

First of all, thanks for setting up this project. I am really looking forward to use this to get rid of my RSI.
I have followed all the steps in server setup. When I try to test it out, I could connect server with loopback address in config.py but it does not work when using 192.168.x.x IP. I tried with both 'Host only adapter' settings and 'NAT', but no luck. The server starts, but client says 'Operation timed out'.

Please help.

Here are logs.
Terminal 1 - Server

:: python server_osx.py
started on host = 192.168.1.104 port = 8240

Terminal 2 - Client

::python test-client.py
HOST=
192.168.1.104
PORT=
8240
Traceback (most recent call last):
  File "test-client.py", line 129, in <module>
    main()
  File "test-client.py", line 123, in main
    all_tests(communication.server)
  File "test-client.py", line 107, in all_tests
    test_key_press(distobj)
  File "test-client.py", line 44, in test_key_press
    distobj.key_press(key='a')
  File "/Library/Python/2.7/site-packages/jsonrpclib/jsonrpc.py", line 290, in __call__
    return self.__send(self.__name, kwargs)
  File "/Library/Python/2.7/site-packages/jsonrpclib/jsonrpc.py", line 237, in _request
    response = self._run_request(request)
  File "/Library/Python/2.7/site-packages/jsonrpclib/jsonrpc.py", line 255, in _run_request
    verbose=self.__verbose
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1292, in single_request
    self.send_content(h, request_body)
  File "/Library/Python/2.7/site-packages/jsonrpclib/jsonrpc.py", line 126, in send_content
    connection.endheaders()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 571, in create_connection
    raise err
socket.error: [Errno 60] Operation timed out

Help getting started

Having trouble getting this started. Followed instructions, and installed the packages on an xp virtual box. I am trying to connect to an Ubuntu 12.04 system. I think I have the natlink side of things working.

The server seems to start properly when I run it from util: python server_x11.py

However, when i start util/aenea.py in the windows vm it exits immediately with a 0 error code.
The experimental aenea_client.py brings up a window, but it doesn't seem to send the data properly to the server.

On the ubuntu server:

enea$ python server_x11.py
eddyks-MacBook-Pro.local - - [28/Jun/2014 22:37:28] "POST / HTTP/1.1" 200 -
type: unrecognized option '--file'
Usage: type [--window windowid] [--delay milliseconds] <things to type>
--window <windowid>    - specify a window to send keys to
--delay <milliseconds> - delay between keystrokes
--clearmodifiers       - reset active modifiers (alt, etc) while typing
--args N  - how many arguments to expect in the exec command. This is
            useful for ending an exec and continuing with more xdotool
            commands
--terminator TERM - similar to --args, specifies a terminator that
                    marks the end of 'exec' arguments. This is useful
                    for continuing with more xdotool commands.
-h, --help             - show this help output
eddyks-MacBook-Pro.local - - [28/Jun/2014 22:37:49] "POST / HTTP/1.1" 200 -

When I'm normal or dictation mode, with the capture window focused, the Dragon dictation dialog pops up. Then I need to tell it to transfer or manually click it myself. Is this right?

Thanks for any help getting this going. It's made more challenging because I'm not familiar with python yet, and I'm in the middle of a worsening rsi episode.

test server x11 does not work

The test server seems to be based on an outdated version of aenea, and does not work.

Error connecting to server

Hello, I have everything set up until testing with _hello_world_aenea.py.
I have a winxp vm with the host-only adapter running dragon.

I get an error in the natlink window:

Socket error connecting to aenea server. To avoid slowing dictation, we won't try again for 5 seconds.

Which is the result of:

[Errno 10065] A socket operation was attempted to an unreachable host

I have tried many things but have not been able to get the guest to connect to the server.
Any help - exactly what configuration to use for host/port in config.py and aenea.json, would be appreciated.
Thank you

Port server integration test to xvfb

It would be nice if we could run the server tests in xvfb for isolation from user desktop, better control of window geometry so we can test absolute mouse movements more reliably, and (wishlist) integration with CI like Travis.

I took a quick look at this today, but ran into a number of problems -- getactivewindow fails with unsupported, and key_press events were not arriving (possibly the window was not activated).

ReloadGrammarsRule attempts to open and write to directories.

ReloadGrammarsRule in the aenea grammar should only append " " to files when executed. Method previously tried to open directories as files for writing which caused errors on some systems. (win xp)

Network protocol should support authentication

Given we allow full remote control of the keyboard and mouse, it would be possible to be rather evil if an attacker could connect to the server. Investigate TLS, Keyczar, and srp. Priority should be for simplicity so people don't just disable it, and ideally not reliant on certificate authorities.

Long action chains take a significant amount of time to execute

After playing with aenea for some time I started digging into trying to tighten the latency between "say something" and "do something".

Occasionally I noticed a delay AFTER recognition when a long chain of commands was to be executed so I came up with this test

from aenea import Text

def time_actions(count=20, data=None):
    s = time.time()
    for _ in range(0, count):
        Text('abc').execute(data=data)
    return time.time() - s

# >>> time_actions(count=20)
#0.43 seconds

That means that aenea takes almost half a second to perform 20 actions client to server. This is on the high end of what we might see with a long-ish dictation but not too uncommon. Latency is high enough as it is. We can definitely shave down ~0.4 seconds.

I noticed I wasn't passing the optional data parameter to action.execute. So I tried again.

# >>> time_actions(count=20, data=ensure_execution_context({}))
#0.20 seconds

Explicitly passing execution context prevents aenea from having to query it between nearly ever action. This saves the server from having to shell out repeatedly to get context.

I did one last test to see just how much making multiple rpc calls hurts on the longer chains

class BatchTextActionExecutor(object):
    def __init__(self):
        self.batch = []
        self.original_write_text = None

    def __enter__(self):
        batch = self.batch

        def write_text_patch(*args, **kwargs):
            batch.append(('write_text', args, kwargs))

        self.original_write_text = aenea.communications.server.write_text
        aenea.communications.server.write_text = write_text_patch

    def __exit__(self, exc_type, exc_val, exc_tb):
        aenea.communications.server.multiple_actions(self.batch)
        aenea.communications.server.write_text = self.original_write_text

def time_batch_actions(count=20, data=None):
    s = time.time()
    with BatchTextActionExecutor():
        for _ in range(0, count):
            Text('abc').execute(data=data)
    return time.time() - s

# time_batch_actions(count=20, data=ensure_execution_context())
#0.10 seconds 

The above context manager is a proof of concept. It patches aenea's proxy to waits for multiple actions to be executed. Upon leaving the context manager all commands are flushed to the aenea server in a single RPC. By doing so the post recognition latency is reduced even further. This could be improved further by not shelling out 20 times to xdotool on the server.

I'd like to know if anyone else thinks batch action execution is something worth investing more time in. I feel that recognition is slow enough as it is and that any reduction in latency is a win.

TL;DR Post recognition latency between nearly 0.5 seconds on longish action chains. We can optimize to ~0.1 seconds. Is it worth the effort?

gist containing some of the test hacks:
https://gist.github.com/mzizzi/fd2924a8d68a33ef0ec2

Plugin-based chaining grammars

Grammars such as multiedit and verbal_emacs work by building up a bunch of commands, each of which the user would think of as a single command, and then creating a repetition rule. This allows the user to string together commands, rather than having to wait for each to be recognized before speaking the next (or at the very least without pausing between commands). To Dragon, this looks like one large command each time the user speaks.

This is great for usability, and it is wonderful that Dragon can support such complicated commands. I would argue that it is the capability of doing this alone that makes editing text possible in this way. Unfortunately, this tends to lead to conflating the natural concept of a command from a users perspective with the logic necessary to support the chaining. This leads to issues such as the duplication of code between verbal_emacs and multiedit. As the project grows, and more people want to use different languages, different programs, etc., I feel like this architecture will become more and more unwieldy.

I propose the following as a long-term design for how to solve this. I basically see three distinct problems that need to be composed in various combinations to make any particular grammar along the lines of verbal_emacs, multiedit, shell commands, etc:

• The actual logic of chaining (take a list of commands, put them together in a repetition rule, delegate the actual command processing as appropriate). You can also think of this as the interface to dragonfly/natlink.
• Various "plug-ins" that each constitute a family of commands. Examples of this include the variable name formatting functions in multiedit and verbal_emacs, shell functionality such as pipes and redirects, editing commands in VIM, and the Python-specific and Eclipse-specific commands in multiedit.
• The tie together that determines which plug-ins should be enabled in which contexts, possibly also with some glue logic (such as managing the mode you are in in VIM). This could possibly also integrate with the work in https://github.com/poppe1219/dragonfly-scripts/tree/master/dynamics to allow dynamic loading of plug-ins as appropriate.

This is how I would build this in the absence of constraints. Unfortunately I need to look into what constraints Natlink/Dragonfly/Dragon impose (in particular I suspect some of the issues with nested grammars being thrown out for being too complex that I ran into with verbal_emacs will likely reappear here; Dragon seems to object more to grammars that are too deep (even when unambiguous) then to grammars which recognize a large number of commands, which is what I would expect based on how most parsing tends to go) and I suspect that is likely to dominate the design, unfortunately.

This is also complicated by #17.

I welcome feedback, both on my design in ideal land and on the realities anyone has run into with similar projects. @poppe1219 @nirvdrum

Changing commands in vocabulary.json results in KeyError

I think I've fixed this here but it's really hacky:
sweetmandm@d9f3be3

Basically, it looks like in most cases a vocabulary should specify {spoken: executed}. However in the case of _vocabulary.py it's expecting {executed: spoken}. If you change vocabulary.json to try to use a different spoken command, the original key is referenced later and breaks on KeyError.

This is what I get when I change vocabulary.json to include "enable vocab <vocabulary>": "enable vocabulary <vocabulary>"

Error loading _vocabulary from C:\NatLink\NatLink\MacroSystem\_vocabulary.py
Traceback (most recent call last):
  File "C:\NatLink\NatLink\MacroSystem\core\natlinkmain.py", line 340, in loadFile
    imp.load_module(modName,fndFile,fndName,fndDesc)
  File "C:\NatLink\NatLink\MacroSystem\_vocabulary.py", line 62, in <module>
    class EnableRule(dragonfly.CompoundRule):
  File "C:\NatLink\NatLink\MacroSystem\_vocabulary.py", line 63, in EnableRule
    spec = command_table['enable vocabulary <vocabulary>']
KeyError: 'enable vocabulary <vocabulary>'

Grammars don't work if mic turned off then on

Natlink reloads modules when the microphone is turned off and then back on. Unfortunately, it only reloads grammars -- those that begin with _. This leads to two issues:

• Modules that grammars may import will not be reloaded.
• Due to reliance on isinstance and similar, after the reload, you run into x-is-not-an-x type of errors. This occurs whether or not you attempt to manually reload modules that grammars rely on.

For now, the workaround is to exit and start again Dragon whatever you make a change to grammar. I agree this is not ideal, but at this time the only way I know to fix this would be to make every grammar a standalone file. This would result in an unacceptable degree of duplication of code, and make configuration a nightmare.

Key translations requires a rethink

Right now, the way we handle key translations is less than ideal. The fundamental issue is simple: Dragonfly, Windows, X11, and OS X all use different names for the various keys on the keyboard. We want to make things work with each other as much as possible while avoiding mapping things that aren't equivalent.

Currently, the X11 server maps most Dragonfly keysyms into X11 keysyms. #64 adds a translation layer in the client for < and >. Additionally, Aenea will accept X11 keysyms directly and pass them along to the server.

So the questions I see is:

• Do we need two levels of translation (client and server) or should it entirely be the responsibility of one?
• Should Aenea accept the union of all keys across all platform, just keys for the currently active one (strict and lax do some of this already), or only Dragonfly keys?
  • If only Dragonfly keys, how do we handle platform specific keys (meta, hyper, super, command, etc)?
• Should the client prefer to send Dragonfly symbols, or attempt to translate to the active platform keysyms?

In general, my instinct is it's better for Aenea to pass along anything it can than for it to police what a grammar and the server are allowed to do, insofar as it mediates communication between them. But I do want the same grammar to be able to work across platforms where possible.

My favored solution would probably be to accept union of all platforms on the client, translate all that can be to Dragonfly syms and pass along the rest as are, then the servers translate Dragonfly syms to local platform ones, and executes the rest natively. This means Aenea will let you send a Linux keysym to a Windows server, and the latter will experience an error, but I think this is an acceptable price to pay.

I welcome your thoughts, especially if you have any experience with internationalization and localization -- I've only worked with various layouts of US keyboards, and ideally this would work well for those on other hardware (or at the very least not actively get in their way).

Expose natlink onChangeCallback in aenea server

This is a feature request.

natlink provides callbacks for various state changes. We could proxy these events off to the server and do useful things with them like this: https://github.com/mzizzi/aenea-indicator

This feature would slightly get away from the pure "remote input" spirit of Aenea so I'm not sure how others would feel about exposing it.

Thoughts?

Mangled text over the wire

I need to track the problem down, but I wanted to file the issue here so it doesn't get forgotten. But the basic issue is sometimes certain text forms come across mangled.

E.g., I have this mapped rule:

"cap deploy to ": Text("SKIP_ASSETS=true RUBBER_ENV=") + Function(lib.format.lowercase_text) + Text(" cap deploy")

And when I say "cap deploy to kevin" with > 0.5 probability, I'll get something like the following typed:

SKIP_ASSETS=true RUBBER_ENV=ke cap deployvin

Install requirements with pip

Would it be an idea to have an requirements.txt and then you could install with pip?

observations on setup

Having spent days setting up (Virtualbox with Windows 7 Ultimate 64-bit, Natlink 4.1papa, Python 2.7, OSX 10.8.5), I'd like to relay some of my problems and solutions for possible inclusion in the readme.

1. Natlink required installing Microsoft Visual C++ 2010 Service Pack 1 Redistributable Package (ONLY 32 BIT VERSION WILL WORK), which isn't in the natlink documentation.
2. I wasted some time trying to test the server when the VM was not running, which causes an error. Start the VM or set the ip in the server config to your ip on your LAN.
3. Wrappers.py was unable to import everything from dragonfly and so was loading the mock instead, causing an error in the hellow_world_aenea script. The fix was to comment out FormatState (line 48) and Word (87). I don't know yet if this will cause problems later on. My version of dragonfly is 0.6.6b1. Do people think my solution is OK?
4. The readme states you need to install pyobjc for mac, but this is already included with the native python install on 10.8.5. I actually tried to add it to my brew python as well but couldn't get that to work. Just make sure you execute the server with the Apple-installed python.
5. I think the applescript commands applicable to my OSX (10.8.5) must be different as the server-test does not perform all actions correctly. I have not been able to look into this yet.

Grammer does not get loaded

I changed my project_root to a virtualbox share folder, but the grammers does not get loaded. They do if they are in the MacroSystem folder.

I searched the issues and did not find anything about it.

How do you organize your grammars? :)

What to do after _hello_world_aenea.py?

Really awesome project! I got the whole setup running, and I am able to run the _hello_world_aenea command and see it show up on my OSX system, but I am then confused about what to do next?

I can't seem to dictate anything and I don't really know how to write my own grammers with dragonfly.

I have also copied in _aenea.py.

What are the easiest way to get started? :)

Grammars not working in some apps

I have a vim grammar 'quit': Key('escape') that works when using MacVim, but when using IntelliJ with IDEAVim, it does not:

• The word quit usually doesn't get recognised when IntelliJ is the current window, but when MacVim the current window, quit is able to be recognised more accurately.
• When it is recognised, the text quit is entered instead of pressing the escape key.

configuration refactoring

We were talking about it in the ssl pull request, and well, I did a thing... It's early stages right now, and the documentation looks atrocious, but I thought it might come in handy here. If you want any features added to make it more suitable for aenea, I'm all ears.

Allow matching by command args

I'm not sure if the new refactoring handles this, but I ran into a case where it would be nice to match against an application's command-line arguments. In this case, I wrote a grammar for Terminator. Terminator is a Python app and is apparently launched via /usr/bin/python /usr/bin/terminator. This means the app context needs to match "python", which is rather broad. Being able to match on the full procline would allow the grammar to be scoped properly.

"force natlink to reload all grammars" occasionally causes errors

Specifically, sometimes modules used in grammars are set to None after running this command but this does not always happen. This causes obvious problems when grammars attempt to refer to those modules...

Presumeably, this command sometimes unloads modules it shouldn't (modules imported from grammars are unloaded and then reloaded in the implementation). It's also possible that it might reload things in the wrong order which could also cause problems (this seems like the most likely cause).

Project root aenea.json not optional

I was trying to use aenea with all the default config, but this unfortunately meant there was no way for me to point at my Linux X11 server. So, I copied aenea.json.example into the Natlink directory. I made the host change and made project_root point at my aenea installation. In particular, I wanted the settings I had in aenea's grammar_config directory to be picked up.

In any event, my project root didn't have an aenea.json file in there and so things blow up. If a config file is found, the logic goes to the project root and starts recursively trying to load additional configuration. However, it doesn't seem to handle the case when a file doesn't exist.

https://github.com/dictation-toolbox/aenea/blob/master/client/aenea/config.py#L37

is the current line that's problematic.

If the configuration isn't optional, then handling the missing file and printing a useful help message would be preferable. If it should be optional, then it should just silently fail. I can pull together a PR for either case, but please advise on the correct behavior.

Add convenience classes for cross-platform and proxy keystrokes.

Currently I have to say AeneaAction(Key('local'), Key('proxy')). It would be nice to be able to say AeneaKey('local', 'remote'). Similar situation for different remote OS, and possibly combine the two.

Windows VM config

This isnt really an issue with aenea but rather an discussion about configs for windows vms for those using aenea.

I first tried to use aenea with a windows xp vm with 2 cores, 2G ram, and dragon using bestmatch IV. Dictation was accurate but terribly slow due to XP not supporting multi core systems well. I then tried updated windows kernel and removed support omfor multiple cores and got a significant performace boost. Unfortunately Dragon's bestmatch IV algorithm is terribly inaccurate in single core mode.

So I'm currently left configuring a windows 7 ultimate vm to attempt to use aenea again.

Im curious to hear what configs work best for others.

_hello_world_aenea.py prints successful when not actually successful

Probably due to proxy config defaults in the absence of aenea.json.

Bad path for server_state.json

I've been updating the dragonfly-scripts repo to work with the aenea. It took a while to figure everything out, but I think I got there. But I'm currently running into an issue with a bad path for the server_state.json file. At start-up, I get:

Error writing config file .\.\server_state.json: [Errno 13]
Permission denied: '.\\.\\server_state.json'.

Text is mangled when sent to a windows VM via xdotool

This may be related to #28 but I wanted to make this a separate issue since I did not experience any problems prior to commit 775af7c.

Basically some characters appear to get dropped so that only a subset go through. It's not always a problem but seems to happen intermittently.

aenea_client should clear window sometimes

It would be nice if text that appeared in the client window cleared itself periodically (this could be either time-based or after N flushes, where I prefer the latter).