donna-ai / donna

Donna = Virtual Assistant + Internet of Things, "Donna, the best Virtual Secretary platform" (Work In Progress)

Donna

Donna, the best Virtual Secretary platform

---

About

Donna is a platform for developing an artificial intelligent, virtual secretary. You can create your own instance of Donna and extend her with a wide variety of plugins. Plugins range from Senses to Cortices to Controls.

• Senses are responsible for receiving stimulus as inputs into Donna.
  • Audio from microphones, Visuals from cameras, textual messages from SMS (texting), and more.
• Cortices are modular pieces of her brain that allow Donna to process Thoughts.
  • Text to Intent (Wit.ai), Speech Recognition, Facial Recognition, and more.
• Controls are ways Donna can interact and output information or responses to the world.
  • Show content on Chromecast on the TV, send SMS (text) messages, move robot limbs, and more.

For more technical information see the documentation.

Features

• Extensible with plugins for Senses, Cortices, and Controls.
• Distributed over many devices: Mac, Linux, Windows, iOS, Android, Raspberry Pi.

Documentation

See docs/ directory for list of documentation pages. A good place to start is archtecture.md.

Goals and Roadmap

Unfiltered description by Glavin Wiechert:

My initial goal was to create a solid foundation for connecting modular plugins that handle either inputs (Sense plugins), outputs (Control plugins), or processes data ("Cortex" plugins). These plugins would then be switched together like high-level neural pathways passing relevant information to their neighbouring respecting plugins. While the human brain works on synapses connecting neurons, Donna would attempt to accomplish similar goals but at a higher level. Donna does not have a volume of brain that grew/learned organically to specialize in speech recognition: she has a module specifically designed for said purpose. Oversimplification: Think of her as LEGO blocks for brain functions. Data would flow through Donna and be interpreted with the context and could optionally output to controls (text to speech and many other mediums). After seeing Node-RED I now want to utilize Node-RED for that initial foundation and leverage its Web UI for connecting flows and use its existing packages. I then want to build Node-RED packages that are also used by Donna, for the purpose of: natural language processing (exists: use Witai), object recognition (computer vision), optical character recognition, speech to text (exists: Witai, AT&T, Google, PocketSphinx, Sphinx4, etc etc). I have tried a lot of current open source virtual assistants, and none have really impressed me enough to stick with them. I've seen many become abandoned and others be rewritten many times. I want IoT to be the foundation for the next virtual assistant: Donna. Input can come from limitless sources and data is passed thru Node-RED modules. Donna could express information in a wide variety of ways: auditory with text to speech, visually maybe on a chrome cast or custom web app, or push to your hand held phone's companion app because you're on the go. This is of course a very stretchy goal, but my main desire is to design a system for developing a virtual assistant that is extensible and connected, unlike many others. And then see what developers want to teach her.

I want to learn more and create an open source solution that I can grow upon and utilizes the power of IoT and the many existing technologies/APIs available today.

Donna Paulsen Talks About Donna.ai

Wit.ai handles the Natural Language Processing and Donna takes care of interpretting the users' intent and interacting with the world.

Get excited!

donna's Issues

Speaker Recognition

See http://stackoverflow.com/a/20780781/2578205

• http://mistral.univ-avignon.fr/index_en.html
• http://marf.sourceforge.net/

IntentEntity can query/select specific Output plugins to use for handling OutputEntity

Take for example:

“Turn on lights in bedroom” → intent “toggle_lights” → extract from context {room: “bedroom”, state: “on” } → send request to Donna server that has access to the Bedroom lights → turn on bedroom lights

Many Output plugins could handle the toggle_lights Intent, however the IntentEntity specified the bedroom and so only a subset (1) output device could be used.

• By default, the Output plugin associated with that Input/Sense plugin is used
  • For instance, iOS Input plugin and iOS Output plugin on the same device
• Send Output to all / multiple
• Send Output to only one

Migrate docs from Google Drive for Donna into docs/ directory

• Architecture: https://docs.google.com/document/d/1ZPISzJEnbK3d7o9YS_jmvuUUHh2tKESrMbIg89uK-yM/edit#

Tasks/Queries Donna should be able to handle

• Show the photos I took today on the TV
  • Query Photos with today's date
  • Push Photos to TV (DLNA, Chromecast, etc)
  • Provide UI for selecting which photo to display on mobile/desktop device
  • Allow voice control (next, previous, etc)
• Share files with so-and-so
  • Looking so-and-so in Contacts
  • See if they have Donna
  • Provide their Donna with access to those files
  • Notify their Donna to provide a message to so-and-so
• Remind me next time I see so-and-so to give them that money I owe them
• Call me Star Lord from now on
  • Add an alias for the User to be "Star Lord" and make it the default alias
• Play some music everyone will like
  • Query for other Donnas on the local network (the houseguests are over and will be connected to the Wi-Fi, of course!)
  • Host (my) Donna asks the other Donnas for permission to access their music library
    • This prompts their owners to say "Yes" to allowing permissions
  • All of the Donnas share their music library and perform analysis based on artists, songs, and genres.
  • Host Donna will use the data to create a playlist aggregating all of user's combined songs
  • Host Donna will command the other Donnas when to play their respective songs and send to the speakers (Chromecast, DLNA on TV, or Desktop / Mobile Donna that are connected to speakers, etc)
• Wake me up tomorrow 30 minutes before sunrise
  • Set alarm for sunrise time of tomorrow - 30 minutes
• Wake me up every day 30 minutes before sunrise
  • See http://stackoverflow.com/a/4141239/2578205 for how Cron internally works
  • Set alarm for sunrise time of tomorrow - 30 minutes
  • When that alarm happens, recalculate for the next day and repeat
• If I am on my computer at 6pm, remind me to go start making dinner
  • Schedule event for 6pm
  • At 6pm, process event:
    • Check computer activity
    • If computer activity is recent then display reminder

Donna builder

Make installing a customized Donna easier and more user-friendly by providing a web interface for inputting customization.

• Select plugins to use
• Use Wit.ai plugin for Intention extraction
  • Load corpus of expressions and intents for plugins

Learning System with Genetic Algorithms and Neural Networks

Walking:

• http://rednuht.org/genetic_walkers/
• https://vimeo.com/79098420
• http://www.cs.ubc.ca/~van/papers/2013-TOG-MuscleBasedBipeds/2013-TOG-MuscleBasedBipeds.pdf

Other:

• A genetic algorithm learns how to fight! - https://www.youtube.com/watch?v=u2t77mQmJiY
• Tetris AI - Genetic Programming Vs Tetris Game - https://www.youtube.com/watch?v=_nklY5lFZAY
• Computer program that learns to play classic NES games - https://www.youtube.com/watch?v=xOCurBYI_gY

Donna-ai CLI

• NPM package named donna-ai
• setup CLI / bin
  • Usage similar to https://github.com/blandinw/cherry
  • donna-ai config.yaml

Setup Unit Testing

Use Mocha: http://mochajs.org/

Medium severity vulnerability affecting this repo

Hi there,

We noticed that your repo has a medium severity vulnerability:

• Remote Memory Exposure
• Affected package: request <2.68.0 >2.2.5

Here is the test report for this repo.
If you’d like to fix this vulnerability, Snyk lets you generate a pull request that recommends the best upgrade path - there’s a link to fix this vulnerability on the test report.

Stay secure :-)
Snyk Community

Use Senses on demand

Normally the senses and running in the background and emit to Donna. However it may be beneficial to have Donna be able to request usage of her senses. For instance, Donna could start listening (request Wit.ai speech recognition plugin activation) or read a file (a file system Sense plugin that would not emit to Donna but Donna could still utilize).

Documentation Generation

See http://stackoverflow.com/questions/6096649/documenting-node-js-projects

Memory system

• Memory system
  • Short-term
  • Long-term

Plan Architecture for Donna

See https://docs.google.com/document/d/1ZPISzJEnbK3d7o9YS_jmvuUUHh2tKESrMbIg89uK-yM/edit#

• Configuration system (moved to #8 )
  • Configuration file
    • Loaded on start
    • List of plugins to load
      • Check if Node package has already been installed / can be required: https://github.com/substack/node-resolve
      • Load plugins
        • See http://stackoverflow.com/a/15957574/2578205 for how to automatically / programmatically install plugins
• Memory system (moved to #10)
• Plugin system
  • Supported types:
    • Sense plugin
      • Types:
        • auditory
        • visual
        • haptic
        • olfactory
        • gustatory
    • Cortex plugin
      • Input router
      • Intent router
      • Output router
    • Output plugin
      • Unit test for example output plugin
• Input Router
  • InputEntity
• Intent Router
  • IntentEntity
• Output router
  • OutputEntity

Thought pathways

Similar to Neural pathways except on a larger, more modular scale.

While neural pathways connect neurons throughout distant areas of the brain, Donna's thought pathways would represent traversals through thought handlers.

A thought handler is a function that transforms a thought entity, given it has an expected input and output data type that the handler supports processing. While the brain has clusters of neurons that connect to each other and eventually learn how to process information passing through them (neural network), a thought handler is like a pre-computed neural network that is specialized for processing certain type of data and can output another specific type of data. For instance, there are areas of the brain responsible for understanding speech. In Donna, those areas would be represented as individual thought handlers, such as Speech-to-Text handler that supports input type audio or speech and output type of text. Donna could have another thought handler for text to Intent, such as wit.ai.

A thought entity can be thought as a unit describing a chunk of related information that can be processed by Donna.

Sensory input ( Input entities ) will be received by Donna and then be converted to Thought entities that can be passed through thought pathways in the same way information in the human body's nervous system passes through the neurons in the brain. Graph traversal algorithms could be implemented to improve efficiency / precision of traversal from input entity through thought handlers. Neural networks could also be used to further optimize which thought handlers are used more often in different cases.

Move Wit.ai plugin into separate dependency repo

https://github.com/Donna-ai/donna-wit-ai

Register plugin accept an Object, not a Function

Instead of having the registerPlugin method accept a Function, have it accept an Object that is formatted as follows:

{
  "name": "donna-plugin-example",
  "description": "Interact with the foobar API; assign foos, list bars, etc.",
  "version": "0.0.1",
  "plugins": {
     "senses": [ ], // array of sense emitters
     "cortices": [ ], // array of cortices / processing units
     "controls": [ ] // array of controllers
  }
}

Config system

• Configuration file
  • Loaded on start
  • List of plugins to load
    • Check if Node package has already been installed / can be required: https://github.com/substack/node-resolve
    • Load plugins
      • See http://stackoverflow.com/a/15957574/2578205 for how to automatically / programmatically
• Move Config persistence into Memory system
  • Currently the Config is separated from the Memory System. It'd be great to mix these such that configuration would also be distributed, and cached in short-term and persisted in long-term, and not require a distinct API for getting Config information versus other information.

maryTTS

I would be nice if Donna would support maryTTS. (http://mary.dfki.de/).
This is a java server for text to speech. Since it listens on a deciated port, it should be easy to stitch replys from a config file with node-red and maryTTS. I've looked into espeak but the voices are very bad. Apple's "say" commands are very good but then that's only osx