waterrmalann / friday

🤖 Modular Desktop Virtual Assistant in Python

Hey 👋, I'm Alan :)

I'm a self-taught developer with nearly 7 years of experience, excited about crafting immersive web experiences and scalable apps, dabbling in various creative projects, among other silly things. Beyond web, I'm also into game development and 3D modeling. Tools of the trade are Python, JavaScript/TypeScript (MERN/MEVN/Svelte), and C# (.NET, Unity, Godot).

🚀 Me

• ⌨️ I'm working on a skill building platform for creatives.
• 📚 Learning React Native.
• ✒️ I sometimes write on my blog.
• 🎬 Occasionally, I also put stuff on my YouTube.
• 💬 Ask me about Python, JS/TS, and C#. I'd love to help!
• 📫 You can reach me on Discord (@waterrmalann), Telegram, or via E-Mail

🛠️ Stack

• JavaScript x TypeScript
  • React (Next.js), Vue, Svelte (+ SASS / TailwindCSS)
  • MongoDB (Mongoose), SQL (SQLite, MySQL)
  • Express (Node.js), Elysia (Bun.js)
  • Ionic Framework, ElectronJS
• Python
  • Flask, FastAPI
  • Discord.py
• C#
  • Unity, Godot
  • WinForms, WPF
• Microservices
  • Docker (+ Compose)
  • Kubernetes (+ Helm)
  • RabbitMQ, Kafka, NATS
  • Redis

⏳ Stats

Check out some of my repositories. Show some ❤️ and maybe leave a ⭐ or two 😅.

---

friday's Issues

Feature Availability Status

Feature Availability Status

An active dictionary (or named tuple) of the assistant that keeps track of which features are available. This information can be used by skills, handlers, and response engines to take advantage of additional functionality or to adapt to restricted environments.

Features to keep track of (for now):

• Internet Availability (Boolean)
• Operating System (String)
• Assistant Uptime (Integer/Float)
• System UI: Dark Mode / Light Mode (Boolean)
• On Battery (Boolean)
• Bluetooth Availability (Boolean)

Skills that rely on an API for example will be able to check for Internet Availability before informing the user that it can only work when online or switching to an offline version of the skill dynamically.

[Skill] Jokes

Skill Proposal | Jokes

A skill that gives the assistant the ability to tell jokes (including anti-jokes like dad jokes and puns) on request.

Implementation

Jokes could preferably be stored locally as part of the skill, or alternatively fetched using an API. Perhaps both of these can be implemented in a way that the skill initially attempts to fetch a joke via some API but gets one from a locally stored database if it fails.

Types of Jokes

• Plain Delivery Jokes
• Dad Jokes / Question-Type Delayed Delivery Jokes
• Puns
• Comebacks
• Yo Mama Jokes
• ..? (more could be added)

Input Patterns

(Tell/Tell me/Give me) a (joke/dad-joke/comeback/pun)

Commands like:

• Tell me a joke.
• Tell a dad-joke.
• Give me a comeback.

Examples

Sorry I only got bad jokes to show examples.

Plain Delivery:

User: Tell me a joke.
Assistant: I heard there were a bunch of break-ins over at the car park. That is wrong on so many levels.

Delayed Delivery:

User: Tell me a joke.
Assistant: Why do we tell actors to “break a leg?”
User: Why?
Assistant: Because every play has a cast.

Context Manager

Context Manager

A module for storing and retrieving context information that plugs into the assistant and runs before any response engine does. It can then pipe user input into a response engine while also providing context information that was stored in the manager.

Why is a context manager necessary?
Context managers help skills/response engines keep track of data across multiple messages and if implemented properly can be used to chain messages realistically.

Example Scenario 1:

User: Who is Alan Turing?

Assistant: Alan Mathison Turing was an English mathematician, computer scientist, logician, cryptanalyst, philosopher, and theoretical biologist.

User: What was he known for?

Assistant: Turing was highly influential in the development of theoretical computer science, providing a formalisation of the concepts of algorithm and computation with the Turing machine, which can be considered a model of a general-purpose computer. He is widely considered to be the father of theoretical computer science and artificial intelligence.

As seen here, context for the follow-up question was automatically understood by the assistant even though the two inputs triggered two separate events. This kind of context-aware chaining of inputs is only possible with a context manager storing data between the request and response exchanges.

Example Scenario 2:

User: What's the value of Pi?

Assistant: Approximately 3.14159. Want me to get more precise?

User: Subtract 1 from it.

Assistant: Alright. That makes it 2.14159.

Here, two inputs that triggered two separate events from two different unrelated skills are sharing information between one another. This kind of communication is also possible with a context manager, though is a bit more sophisticated and perhaps something for the future.

Implementation

Since the assistant should theoretically be able to handle multiple contexts simultaneously. It's better to store them as a collection of Context objects, each of which are special classes designed to hold contextual information.

Context Class:

• engine: Which engine's context is being stored.
• timestamp: The unix timestamp for when the context was stored.
• content: A dictionary holding the reference content (structure is defined by the skills/engine)
• exchange: A tuple containing both the Message and Response objects that triggered the context management.
• active: Whether the context object is active, context-aware processing will only happen if this is true.

These classes may then be stored in the Working Memory of the assistant as a list under the key name "ContextList".

When an input is given to the assistant, the context manager checks for if any stored context is active and if it is, the data is rephrased and transferred to the engine specified in the engine attribute alongside the Context data. The rest of the processing is handled by the engine.

Commands System

Command System

A non-essential (for users) yet important feature (for devs) of the assistant is the ability to run tests and debug the assistant in real time. The proposed method is via slash commands that are given in the regular input field just like normal requests except with the assistant parsing and processing them directly instead of sending it off to a response engine.

An example command: /ping
Returns the input latency of the assistant (ie: the time it takes for a command to process and return a result)

Implementation

I imagine a good way to define and attach commands to the assistant is something like this. It's simple while also modular.

# Define the command, where 'args' will be the arguments passed from the input.
def cmd_version(*args):
    print("Friday v3.0.0")

# Attach the command function to the assistant instance.
assistant.register_command(cmd_version, aliases=['ver', 'version'])

Something I hadn't thought of is error handling bad arguments/input. This could possibly be done through special command function decorators or inside the command handler itself. Certain commands should also be restricted based on the user's access level (which is also something that's not implemented yet).

Additional Notes:

• Commands should only run when the developer_mode of the [developer] table in the config.toml file is set to true. It is disabled by default for safety reasons.
• The prefix used for such commands should be defined and read from prefix of the [developer] table in the config.toml file. A forward slash / is preferred by default.

Memory Component

Memory

An important component that the assistant is lacking is the ability to remember data in the short term by storing things in the RAM as well as the long term by storing things as files. This issue describes the proposed memory types.

Working Memory (Short-Term Storage)

A temporary memory used to store information like contexts and previous exchanges or clipboard data for quick access by the assistant. It is a primarily key value based storage. Persistence is possible by storing the whole class in binary serialized form using pickle.

Persistent Working Memory

A variant of working memory that's easily serializable as JSON/YAML that can be saved as a file locally and retrieved later by the assistant, then deleted after use. Example use cases include todo lists and reminder skills.

Persistent Memory (Long-Term Storage)

A semi-permanent memory used to store information like facts and things that the assistant has learnt from the user. Data is serialized and stored in a relational SQL database with additional JSON/YAML support. Data can also be stored in pickled binary form or as plaintext or CSV file.

Persistent Configuration Memory

A variant of persistent memory exclusively for storing user preferences on how the assistant should behave, which features are modified, and in what ways.

Data Memory (Heavy Long-Term Storage)

A truly permanent means of storage for handling large amounts or size of data of varying types such as images, videos, zip, etc... Files downloaded through the assistant are usually kept in this memory in compressed form. Usually recovery images of the assistant are stored as binary files here as well.

Implementation

[no implementation details yet]

Event Threads

Event Loops/Threads

The assistant needs separate event threads that repeatedly calls some functions every pre-determined amount of seconds so that much more can be done than what is already possible. Implementation details described below.

1. Primary Event Loop

A parallel-run, precise-to-second (by default) timer thread for the execution of functions or checks at every interval. Response engines like the Skill Based Response Engine can take advantage of this loop to for example, run code for passive skills in the background.

This loop calls the assistant.mainloop.tick() function every second (by default, can be adjusted in config.toml) which executes all the functions in mainloop.funcs[] consecutively. The time delay between each tick could also be called the 'tickrate' as it is kind of the equivalent of the heartbeat of the assistant.

Ticker Class

A component class that plugs into the assistant just like interpreter or the memory components.

assistant.mainloop = EventLoop(self.config['developer']['main_loop_delay'])
assistant.mainloop.start()

Once activated, functions can be added to the main loop (from a response engine for example).

assistant.mainloop.add_function(function, args, kwargs)

Where function is the function to run at each interval with specified arguments and keyword-arguments if any.

2. Scheduler Event Loop

Another parallel-run, precise-to-second (by default) timer thread for the execution of single-use functions at specified times. For example, if a task needs be scheduled to run at 5:00 A.M just once, it should be registered on the scheduler.

This loop calls the assistant.scheduler.tick() function every second (by default, can be adjusted in config.toml) which runs a check to see if the time has passed for any scheduled events in the queue (scheduler.queue[]), and if it did, run them and dispose them from the queue.

Scheduler Class

A component class that plugs into the assistant like the ticker class.

assistant.scheduler = Scheduler(self.config['developer']['sched_loop_delay'])
assistant.scheduler.start()

Once activated, events can be added to the scheduler queue (from a response handler, for example)

assistant.scheduler.queue_event(function, time)

Where function is the function to run at scheduled time which should be a unix timestamp.

3. Update Event

The main thread used by the assistant also fires an assistant.on_update() method everytime the assistant receives an input or gives an output. This event method should not be relied on for most use cases as it would only execute when the user is actively interacting with the assistant. It's better for things like auto-save or to maybe do some calculations, etc..

There are currently no plans to allow interpreters, handlers, or engines to add functions to the on_update() event dynamically. In order to add things to this event, you would need to inherit the Assistant class and override the on_update() method but be sure to call super().on_update() first as well.

Input Interpreter

Input Interpreter

Current: Input from the user gets sent to the assistant, which handles it to the active response engine and gets a response.

Proposal: Input from the user gets sent to the assistant, which handles it to an interpreter. The interpreter is tasked with identifying the type of input (whether it was a debug command or a regular request). If it turns out to be a debug command, the input is handled to the command handler which will take care of it and give back an output. If it turns out to be a request instead, the input is handled to the request handler. An additional handler called a context handler (Issue #2) runs ahead of these two and redirects inputs to either the command handler or directly to a response engine based on contextual information it has stored that matches with the input.

All ways input can flow

• Input → [Input Interpreter] → [Context Handler] → [Command Handler] → Output
  • Input is identified to contain contextual information and is recognized as a debug command and handled to the command handler which produces an output.
• Input → [Input Interpreter] → [Context Handler] → [Response Engine] → Output
  • Input is identified to contain contextual information and is recognized as a request and handled directly to the response engine which produces an output.
• Input → [Input Interpreter] → [Command Handler] → Output
  • Input is recognized as a debug command and handled by the command handler which produces an output.
• Input → [Input Interpreter] → [Request Handler] → [Response Engine] → Output
  • Input is recognized as a request and handled by the request handler, which gives it to the proper response engine, producing an output.
• Input → [Input Interpreter] → Output/Exit
  • A special case for the input interpreter that realizes that the user input requests to terminate the assistant, fulfilling the request.

Structure

• Input Interpreter
  • Command Handler
  • Request Handler
    • Skill Based Engine
    • Query Engine
    • Chat Engine
  • Context Handler

Implementation

• The interpreter would be a component that plugs into the assistant.
• It must be modular and extendable so that more handlers can be added in the future.
• It needs to accept user input and identify which handler needs the input, and pipe it to that handler.
• It needs to process the text before it is sent to other handlers.
• It also needs to process and pipe back the result/response from the handlers.

[Skill] 8-ball

Skill Proposal | Magic 8-Ball

An assistant skill that gives a fortune reply when a question is asked and 8-ball is mentioned.

Implementation

The responses can be stored locally either as plaintext or JSON. It would however be preferable to use JSON as then the responses could be categorized into types which the assistant can use to understand if the 8ball response was positive, neutral or negative. This information can be stored in the contextual memory in case the user says something related to the response obtained from the 8ball.

The skill may trigger whenever "8ball", "8-ball", "magic ball", or "eight-ball" is mentioned alongside some other text content which may be easily identified as a question (if a question mark is present), otherwise the skill could still trigger if there are no other higher priority skills that were caught by the input.

Priority: 0 (Extremely low, other skills can take priority)

Possible Responses

Affirmative	Non-Committal	Negative
It is certain	Reply hazy, try again.	Don't count on it.
It is decidedly so.	Ask again later.	My reply is no.
Without a doubt.	Better not tell you now.	My sources say no.
Yes, definitely.	Cannot predict now.	Outlook not so good.
You may rely on it.	Concentrate and ask again.	Very doubtful.
As I see it, yes.
Most likely.
Outlook good.
Yes
Signs point to yes.

Examples

User: Ask 8ball if today will be a good day.
Assistant: 8-ball says: "As I see it, yes."

User: 8-ball, will I ever be successful?
Asssitant: Concentrate and ask again.

User: 8ball, does this work?
Asssitant: 8-ball says yes.

The responses may be varied as seen above. The assistant may either reply directly or act as a translator/mediator for the magic 8ball.