basedhardware / omi

AI wearables

Omi (prev. Friend)

Meet Omi, the world’s leading open-source AI wearables that revolutionize how you capture and manage conversations. Simply connect Omi to your mobile device and enjoy automatic, high-quality transcriptions of meetings, chats, and voice memos wherever you are.

       

Homepage | Documentation | Buy Assembled Device

Documentation:

• Introduction
• App setup
• Buying Guide
• Build the device
• Install firmware
• 3rd Party Integrations.

Contributions

• Check out our contributions guide.
• Check out the current tasks.
• Earn from contributing! Check the paid bounties 🤑.
• Join the Discord.
• Build your own Plugins/Integrations.

Made by the Community, with -❤️-:

Licensing

Friend is available under MIT License

omi's Issues

Structure the memory asking JSON output `{"title", "summary"}`, in that way we can have better parsed data.

When the user sends a question in the chat, extract from the AppStandalone the `function_calling` that determines if the message requires context, if that's the case, retrieve the top 10 most similar vectors ~~ For an initial version we can read all memories from sqlite or SharedPreferences, and compute the formula between the query and each vector.

Question: Is there a way to record selectively?

Like, have a button, and only when it is pressed will the device record/ transmit?

Thanks!

Push an app to app store

App stops running when it’s in background or locked

Currently the App isn’t Actively running when it’s minimized or the if the user locks the device in itself.

In case an API key fails, either Deepgram WebSocket connection fails, or GPT requests, let the user know the error message, either has no more credits, api key is invalid, etc.

Automatically recognize Friend Device

Currently user needs to scan and select the device from the App

Cloud update for firmware ($500)

Firmware on Friend's wearable should be able to be updated via cloud

reference

Ideal scenario:
All friend's firmware should check our github folder and auto-update once it has internet connection>

"ok" scenario: Friend's firmware should be able to update via the mobile app "appwithwearable"

User-story: As a user, I want to be able to open Friend's app, connect to my wearable, then click "update". It should update the firmware from github

Summarization

We need to add summarization of conversations to an AppWithWearable

The summarization is implemented in AppStandalone => we just need to move it to AppWithWearable

Summarization should be saved by chunks of conversations.

Chunks are cut by a 30-seconds pause

For example, if there is a long conversation of 1h, the summarization chunk will be 1h.

if there is a 30-seconds, it is considered a new conversation. For example
2 people talked for 5 minutes (chunk), then one of them left to a restroom for 2 minutes. After coming back, they continuted to talk again for 10 minutes. Those will be 2 different chunks of 5 and 10 mintes

See good examples how it was done in AppStandalone

Friend App UI not centered on fold interior display.

Found a minor UI bug where the Friend App UI doesn't seem to be centered on the interior screen of the Samsung Fold 5. I've attached screenshots to display the issue, which persists in all orientations.

Use from the AppStandalone `_ragContext` function as a baseline for creating the query embedding.

Device disconnected, display dialog to user asking to reconnect, or take the user back to `find_devices` page.

test

curious about the react version..

it says failed in the readme, does it build or is it mostly just an earlier attempt?

iOS] memories and chat page on the bottom do not have the blurred colors pattern, but plain primary color

Add a Firmware Setup for Arduino-based Prototyping and Testing

Hi everyone!

I'm a big fan of the Friends project, but as a beginner, I've found the current firmware setup to be quite challenging. Therefore, I've taken the initiative to simplify things by rewriting the code into a straightforward .ino file.

Setup:
This code utilizes the mic.h (please download the ZIP file from the existing project as the current release is non-functional.) and ArduinoBLE libraries.

To set up the Seeed nRF52 mbed-enabled Boards library (more details here):

Go to File > Preferences, and add the following URL to "Additional Boards Manager URLs": https://files.seeedstudio.com/arduino/package_seeeduino_boards_index.json
Navigate to Tools > Board > Boards Manager..., type "seeed nrf52" in the search box, select the latest version of Seeed nRF52 mbed-enabled, and install it.

If you encounter a Permission Denied issue, you can find a fix here.

Please note that it's not fully functional yet. I attempted to replicate the original code, but when running the Python script to generate audio recordings, it doesn't work. I suspect there are some issues with the sampling rate, samples, buf_size, etc., but I'm struggling to grasp how to fix this on my own.

To Do:

• Rectify the data transmission process.
• Incorporate blinking LEDs.
• Determine how to monitor battery level.

The code :

#include <ArduinoBLE.h>
#include <mic.h>

#define SAMPLES 400
mic_config_t mic_config = {
  .channel_cnt = 1,
  .sampling_rate = 16000,
  .buf_size = 1600,
  .debug_pin = LED_BUILTIN
};
NRF52840_ADC_Class Mic(&mic_config);

int16_t recording_buf[SAMPLES];
volatile static bool record_ready = false;

#define SERVICE_UUID "19B10000-E8F2-537E-4F6C-D104768A1214"
#define CHARACTERISTIC_UUID_AUDIO "19B10001-E8F2-537E-4F6C-D104768A1214"

BLEService audioService(SERVICE_UUID);
BLECharacteristic audioDataCharacteristic(CHARACTERISTIC_UUID_AUDIO, BLERead | BLENotify, sizeof(recording_buf), true);

void setup() {
  Serial.begin(115200);
  while (!Serial);

  Mic.set_callback(audio_rec_callback);

  if (!Mic.begin()) {
    Serial.println("Mic initialization failed");
    while (1);
  }

  if (!BLE.begin()) {
    Serial.println("Failed to start BLE!");
    while (1);
  }

  BLE.setLocalName("Friend");
  BLE.setDeviceName("Friend");
  BLE.setAdvertisedService(audioService);

  audioService.addCharacteristic(audioDataCharacteristic);
  BLE.addService(audioService);

  BLE.advertise();
  Serial.println("BLE Peripheral is now advertising");
}

void loop() {
  BLEDevice central = BLE.central();

  if (central) {
    Serial.println("Connected to central device");

    while (central.connected()) {
      if (record_ready) {
        audioDataCharacteristic.writeValue((uint8_t*)recording_buf, sizeof(recording_buf));
        record_ready = false;
      }
    }
    Serial.println("Disconnected from central device");
  }
}


static void audio_rec_callback(uint16_t *buf, uint32_t buf_len) {
  // Convert uint16_t buffer to int16_t buffer
  for (int i = 0; i < SAMPLES; i++) {
    recording_buf[i] = (int16_t)buf[i];
  }
  record_ready = true;
}

Improve structured memory results performance by sending n previous memories as part of the structuring but as context, not as part of the structure, so that if there's some reference to a person, and then you use a pronoun, the LLM understands what you are referring to.

Add Chat Interface

Adding chat interface will allow users to chat with their memories

Fix scroll and add saving of conversations

Problem 1
now: when I restart the app, all conversations are deleted.
Should be: everything is saved, and once the app is restarted, the user is redirected to the list of conversations/summaries

Problem 2:
Scroll is broken

See video for reference: https://drive.google.com/file/d/15rxa_ZyM89LamahmRwCKH47h4gxdc09F/view?usp=sharing

[Request] Merge AppWithWearable and AppStandalone

Ideally there should be a single app, you can record with the app, and 'if' you have the wearable connect it.

I scenario example would be, if you have the wearable, but your battery in it died, the current split app approach means you cannot record audio at all.

Received empty audio data array (V1 Firmware Build)

Hello everyone,

I've successfully installed the V1 Release of the Firmware on the chip, and the device is recognized with the LED blinking as expected.

However, upon testing it using either the Python script on my laptop or the online firmware tester tool (Chrome), I encountered the following errors:

Python Script Logs:

Recording audio...
Recording stopped
0
Warning: Received empty audio data array.

Online Firmware Tester Tool Logs:

Starting Bluetooth...
Bluetooth started: started
Opening device...
Device opened: Friend (zkSRZ5uwGXfndd3Ka6qSIQ==)
Service: 19b10000-e8f2-537e-4f6c-d104768a1214
- Characteristic: 19b10001-e8f2-537e-4f6c-d104768a1214
  - canRead: true
  - canWrite: false
  - canNotify: true
- Characteristic: 19b10002-e8f2-537e-4f6c-d104768a1214
  - canRead: true
  - canWrite: false
  - canNotify: false
Target protocol found: pcm-8
Waiting for data...
Total received: 0 bytes, speed: 0 bytes/sec, lost: 0 packets or 0 packets
Total frames: 0
Decoding...
Done!

Additional Information:

• MacBook Air Early 2015 running Monterey 12.6
• I've attempted to reinstall the software multiple times without success.

I would greatly appreciate any assistance in resolving these issues. Thank you in advance!

Train Own Voice for "Speaker 0" Naming

As a user, I want to be able to train Deepgram's transcription functions to identify my own voice so that anything I speak is labeled with my name and summaries are able to identify what I said compared to other people.

Hope to support Chinese

Hope to support Chinese

Test/Implement [speaker diarization](https://developers.deepgram.com/docs/diarization) to recognize multiple speakers in transcription, use that for better context when creating the structured memory.

Speaker Diarization ($1000)

THIS IS A PAID TASK. WHOEVER COMPLETES IT, GETS $500 IN CASH

As a user, I want the app to identify when I was speaking, vs someone else and see it in transcriptions/memories

For that, during the app's onboarding, i want to provide a few seconds sample of my voice (like Siri), and then

And then the app should do this:

It shouldn't require to reconnect every time that you open the app, it should load `ConnectDeviceWidget` and in there listen/reconnect to the device.

Settings bottom sheet, improve way of handling `***` blurring of api keys, as if you save it while is blurred with *, it sets the key to that value, and you have to set them again

Add animations for processing stages

The app needs to show when there is audio is coming from the device above certain noise level.

similarly whe. The app has sent transcript to the LlM and is waiting for response, it should show a different animation.

Migrate MemoryRecord from SharedPreferences to sqlite

Improve connected device page UI, including transcription text, and when memory creates after 30 seconds, let the user know

• improve the appearance of the transcribed text
• add some sort of UI event to notify the user that a new memory was created

App does not receive audio with New Device Firmware

The latest App works with old firmware, but with the new firmware no audio is received

Unable to find Friend device from AppWithWearable on Android w/o manually granting location permision

The app can't find the Friend device unless going into App Permissions and manually granting location permissions.

Device: Pixel 8 Pro
OS: Android 14

The user should be prompted when launching the app for all necessary permissions.

Note: It did prompt me for nearby devices and notifications

Settings Panel

Create settings panel for Api keys and other custom properties such as prompts

About Firebase

I've noticed that there are Firebase related configurations in FlutterFlow. Are there any tutorials on how to configure it? Or can I just create a Firebase project with the default settings? Additionally, I hope it can support Supabase because it allows for self-hosting. Thanks. :)

Change device discovery based on the service GUiD

App should discover the devices based on the service guid, not on name.

Use that as context, and ask to the LLM. Retrieve the prompt from the AppStandalone.

AppWithWearable: when using the app cannot find the device even the device is showing up

when i run the app and am looking for the friend wearable device. The device is showing on Bluetooth search but app cannot detect any device

When a memory is created, compute the vector embedding and store it locally.

Permissions on app are altered

clicking on We need access to send you notifications opens → Bluetooth Dialog
clicking on We need access to your microphone opens → Notifications Dialog

Better `AppWithWerable` folders structure.

Recording Logic is broken

Device doesn’t record continuously.

Gauge of the wire

I have been trying to use recommended 26mm gauge wire to solder on the seed studio, and having hard time, does anyone suggest better size, that's easy.

What was your experience when you soldered the wires!?

Whisper Out returns Gibberish

At times when voice isn’t recognized properly or language spoken isn’t English, whisper throws gibberish output as transcription. Apply English filter to Whisper output to sanitize such responses.

How to this?
Only permit English letters, numbers etc.. (Regex should do)

Design a good 3d trangle-hexagon case that would fit the 250mAH battery

Are there videos or demos somewhere?

Hi,

Very hyped by this project but I'm surprised to not find any pictures or videos online about it. Are there any?

Might be due to this: I'll be blunt but I do think it's better to hear this sooner than later: In my opinion the name "Friend", although very nice, seems terrible in SEO-wise. I really had trouble refinding this repo after having heard about it :/ I terribly want this kind of projects to thrive and find its community but this name really seems like trying in hardcore mode. I mean well :)

Thanks!

Settings Deepgram + openAI key are forced to be set

Implement [similarity search](https://www.pinecone.io/learn/vector-similarity/) locally

Device disconnects when app is closed

So I got the latest APK installed, added my deepgram and openai apis. It connected, the red light turned blue, started working correctly and I saw the text being generated. However, about a minute after leaving the app, the blue light turns red. When I open up the app again, there's no memory of any of the previously recorded text, but a device does reconnect and then starts working again. Is there some sort of permission on my pixel 8 that I haven't enabled? (Allow background usage is already enabled)

Define flutter code style rules.

Stored device address on first pair and connect only to that device

This is part of general improvement for app/device connectivity.

On first pair, app should store the MAC address of the device and subsequently connect only to that device, u mess the user explicitly chooses to pair with another device (whether to replace existing pair or support multiple devices)