neurofusioninc / fusion Goto Github PK

Basically this - https://p5.eegedu.com

• The dino game (#19) is the first experiment
• User can create and share them for others to run on Fusion
• Including support for lab.js experiments
• And three.js too!

We already have a flow to:

• authenticate a user account and fetch recently played songs for a user

• process and return the features of songs a user is listening to

We should integrate an OAuth flow to the spotify api that allows us to fetch listening history for a user periodically and update:

• on the settings page, user connects account
• client sends token to the backend server to store in keystore
• adds a controller endpoint to periodically fetch from spotify on behalf of user
• call the upload api to store data for user

https://github.com/dgendill/spotify-play-history has a good javascript example for connecting to spoify api and fetching periodically.

Data should be stored in remote blob (if user consents) - /y/m/d/-listening_history.json

This'll eventually make creating the desktop version of fusion.

The goal here is to explore how much brain activity can be stored locally. For context, indexed db can go up to 20GB (or 1% of total device storage -- need to validate this part)

• https://dexie.org/
• https://developer.chrome.com/docs/apps/offline_storage/#unlimited

ToDesktop for electron generation from the web app - https://app.todesktop.com/

https://docs.expo.dev/versions/latest/sdk/sqlite/#installation

At the moment, I'm using AsyncStorage - which is pretty much like LocalStorage on the web. Works now but will change in next release

https://github.com/expo/examples/blob/master/with-sqlite/App.js

• Allow custom options be listed as a response type
• Ensure notifications are handled
  • create categories based
  • accept button identifier type

There's some documentation on expo on this, just need to follow it

https://docs.expo.dev/build/setup/

this one is big! redefines how we onboard a fusion user - basically making you anon by default

[will update]

We need to guide the user on more expressive in terms of what user should expect (e.g after creating prompt, tell them what happens next, why are prompts important?)

Figma link

• Overview screens ( what the app is )
• Our commitment to user privacy (see rewind flow)
• Ask what is top of mind for user & save it in database
• suggest prompts for them to add based on entry.

• call /api/storage/search (https://github.com/oreHGA/fusion/blob/master/neurofusion/server/index.js#L163)
• display the list per experiment
• add download button that calls (https://github.com/oreHGA/fusion/blob/master/neurofusion/server/index.js#L173)

• Ensure the saved events are done with the prompt/notificationID
• Response view display
  • Yes/No response
• Allow user to click notification to respond
  • Page for responding to prompt
• Make the edit/create prompt view scrollable
• Account tab
  • Allow user send feedback from app!

In the /datasets route, allow the user delete a particular dataset

just like we do with magicflow, we need to setup period data fetch for

• sleep - https://docs.tryvital.io/api-reference/sleep/get-summary
• activity - https://docs.tryvital.io/api-reference/activity/get-summary
• workout

Will involve adding more methods to the vital controller & processors

• add Privacy policy page link to notion
• add the dynamic page titles & basic SEO (og:title etc)
• screenshot of the app in the center similar to neurosity
• add "start neuro recording" to the landing page
• change logos under the integration section (add apple health & google fit)
• Include pioneer questions & responses in FAQs

• Add this as an experiment/exercise - https://github.com/Rajesh42/chromeDino
• Collect device data from neurosity
• Generate json of events - {startTimestamp e.g 1674750807, eventDetail e.g 'eyes closed')
• Upload data to storage

start with simple flow

• ios app that works
• accepts configuration for:
  • question
  • cadence
  • input
• when entry is complete make api call to move to data store

• add fusion to allowed urls that can fetch from local activitywatch
  • enable CORS
• make fetch calls from browser using https://github.com/ActivityWatch/aw-client-js

• Show more information about selected frequency band
• Filter out noise
• Need to be descriptive about what the change means
• Allow to view neurosity focus metrics against context switches

Here's a quick example of using it in javascript - https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/keyvault/keyvault-secrets/samples/v4/javascript/helloWorld.js

It's actually quite straight forward

• stores token in keyvault -> writes keyvaultTokenId to database

• generate a neurofusionEncryption key for user and store in keyvault too

doesn't need to be perfect. just communicate direction and what exist right now. (all pieces are there)

This needs some more detailed thinking but the gist here is that:

• An "explorer/researcher" on fusion defines a quest ( a set of experiments & workflow that a user should do) with duration

  • create db table & web route
  • define what variables to collect
    - [ ] descriptions
    - [ ] what prompts
    - [ ] what experiments

• Explorer can share for other users to subscribe to

  • subscription means
    • functionality to see active quests (study/trials) and join on web & mobile
    • you'd get notified to do the recordings
    • if self-prompting is involved, we allow you easily configure & set them up on the mobile app
    • you get instant analysis/ summary of your results
    • you chose to share your data (anonymously with the quest creator)

• Researcher should be able to track adherence & see raw data

from the granular event entries, we're looking to:

• display the things that a user will likely care about

• stuff around personality (how's a person like? - introverted etc)

Just did a 30min recording and upload failed because for some reason I had been logged out from fusion backend.

Currently the token expiry is set for 7days I believe... need to add a refresh flow

• Connect fusion account
• Start eeg recording ( need to figure out what the design flow for this is )
• Validate that neurosity token is stored in the backend

Related #43

we need to be able to get the probability of something happening given another event

intro article - https://towardsdatascience.com/conditional-probability-with-a-python-example-fd6f5937cd2

stack overflow ref - https://stackoverflow.com/questions/33468976/pandas-conditional-probability-of-a-given-specific-b

pandas documentation - https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.crosstab.html

extra - https://corporatefinanceinstitute.com/resources/knowledge/other/conditional-probability/

After a single dataset exists, we need to apply some correlation to the data to:

• know what events are tightly correlated
• trend analysis : how to a sequence of these events affect an outcome

Approaches

• (more articles from imessage to add here)
• https://www.questionpro.com/blog/pearson-correlation-coefficient/

We also display the signal quality values for each in the circle

This will grow over time but the core objective here is around sending relevant messages to:

• self sample a question of interest. e.g 'energy levels' - the user gets to set this

Rough concept:

• (try with just activity information)
• top events, when they occur
• current thinking is to parse this as context to a LLM
• fusion initiates a conversation with the user
• user engages with fusion

Include interface in web & mobile

Show relevant information from ActivityWatch & Magicflow

I have some scripts from a while ago that auto predicted the "project of interest" I was working on and general app usage summaries.

I think there's value in looking at this data over time.

A useful view is something similar to this

Script for fetching data is already available.

• accept refreshToken from frontend and store in keystore frontend changes - inprogress + backend changes
• periodically call the magicflow fetch script for user link
• upload it the blob storage using the storageController link

In prep for launch this weekend, we need to make sure all our endpoints are 🔐

Now, when you login to fusion, we make a request to /api/userlogin which then (creates a user if not & returns details) See implementation

• We will update this to just return a JWT token
• #33

One thing to keep in mind though is that our analytics runner will still need to be able to access the api ( not sure what the best approach here is since it needs to access "regardless" of the user

• Encrypt data during upload
• Decrypt when fetched

after having conditional probability is done, we need to present this.

• what questions are we answering
• chart display
• electron app with results

Current implementation of magicflow daily fetch should have it happen nightly, but something's off...

I think it has to do with the token not being in the UserProvider table ... will investigate

Current integration hardcodes Oura in the controller for generating the token.

We should update this

• accept user public key / (generate one for them}
• display terms of service
• create "public key" column in UserMetadata

We have this on notion but it's no where near standard. https://www.notion.so/Fusion-Data-Use-Privacy-Policy-e6f36d7cf99e4c1180bcafdece020e73?pvs=4

• Privacy policy - https://usefusion.app/privacy
• Terms of use

Repro: https://usefusion.app

Try to add email to substack :

Continuous data pipeline to pre-process from github, spotify, web history from google chrome, twitter

• read raw brainwaves & signalquality values
• split into epochs - 3-5seconds
• average resting state for a single recording
• then compare two datasets together. (this will be the basis for defining baseline vs. new dataset)
• see how data has changed over time - "morning", "afternoons", "evenings"

Dataset

• Resting state EEG data Link

• upload dataset to hugging face

Not sure why this is the case, I've added a Notifications.dismissNotificationsAsync(identifier) call.

• This works for Yes/No type prompts,
• Number & Input prompts get logged but don't dismiss automatically unless the user cancels it themselves.

This is annoying but good enough to ship for now (spent the last 30mins trying to fix this but no luck ._.)

Next steps:

• Try using handleLastNotificationResponse instead of using the addNotificationResponseListener
• File a bug with expo dev team

• #75
• Flow for doing recordings
• Start / stop recordings without timer (one click to start a recording)
• "Always on" recording
  • Requested by @ErikBjare in https://discord.com/channels/1067959240217608222/1107889131750707221/1126112660627410994
• Trigger recordings from the server

Vital gives us access to different providers (oura, fitbit etc) Supported providers - Vital (tryvital.io)

• Make request to:
  • create user Create User - Vital (tryvital.io)
    • store the vital user id in key store
  • backend to generate token Generate a link token - Vital (tryvital.io)
• [Frontend] obtain consent from user & create account with tryVital
• Use generated token to display “initiate onboarding button”
  • code sample Launching link - Vital (tryvital.io)
• On Auth complete, make another request to backend
• #37

From the existing data sources, we will take the following columns:

• activity_watch_summarized = summary_day, total_time_surfing, what_app_did_user_spend_x_percent_of_time_on (one column per percent) , events_usage (tuple: event_classification (encoded), sum_total_time)

• google_calendar = summary_day, total_time_in_meetings, average_meeting_duration, mode_meeting_duration, count_of_meetings_initiated, count_of_meetings_invited, avg_number_of_people_per_meeting

• oura = (drop some rows) - remove combination identifiers

• tweets = summary_day, count_of_tweets, count_of_total_positive, total_neutral, total_negative, sum_likes_positive, sum_likes_negative, sum_likes_neutral, sum_retweets_positive, sum_retweets_negative, sum_retweets_neutral

these datasets will then be joined by summary_day for a single day's entry

Things to keep in mind during combination:

activity watch

• cluster events into groups (max 5 & then make them columns)
• encode all events, consider magnitude = 0: 70, 1: 30

google calendar

• considering how many meetings were created impromptu
• meetings initiated vs invited ( giving / taking time )
• how start_time vs end_time fit into daily activities

twitter

• original tweets vs retweets
• how are people interacting with tweets
• people interacting with