Feedback about kbd-audio

[dhenson02]

k im officially never getting on public discord servers again

[weddy2019]

Hey man love the work you done with this. I was wondering can it be used with a microphone plugged into the laptop as opposed to the built in microphone in the laptop?

Cheers

[weddy2019]

In addition what setup did you use and how much machine learning did you do to get your results?

[ggerganov]

The capturing tools do support an option to select the microphone to use (i.e. the capture device).
You simply need to provide the command line argument -cN where you replace N with the capture device ID.

For example, on my machine, when I run the record tool I get this info printed:

$ ./record test.kbd
Usage: ./record output.kbd [-cN]
    -cN - select capture device N

Recording 5 frames per key press
Found 3 capture devices:
    - Capture device #0: 'Built-in Microphone'
    - Capture device #1: 'Soundflower (2ch)'
    - Capture device #2: 'Soundflower (64ch)'
Attempt to open capture device 0 : 'Built-in Microphone' ...
Opened capture device succesfully!
    Frequency:  24000
    Format:     33056 (4 bytes)
    Channels:   1
    Samples:    512

You can see the 3 available capture devices listed in the standard output. By default, the tools select the capture device with ID = 0. For example, if I wanted to use capture device 1 instead, then I would do:

$ ./record test.kbd -c1
Usage: ./record output.kbd [-cN]
    -cN - select capture device N

Recording 5 frames per key press
Found 3 capture devices:
    - Capture device #0: 'Built-in Microphone'
    - Capture device #1: 'Soundflower (2ch)'
    - Capture device #2: 'Soundflower (64ch)'
Attempt to open capture device 1 : 'Soundflower (2ch)' ...
Opened capture device succesfully!
    Frequency:  24000
    Format:     33056 (4 bytes)
    Channels:   1
    Samples:    512

Regarding the machine learning question:
On my iMac, using the build-in mic, I can record just 2-3 keytaps of each key and be able to recognise the key audio with very high accuracy (as demonstrated in the video). The result really depends on the setup you have - mic, keyboard, surroundings, etc. More details about the approach are in the blog post.

[weddy2019]

Thank you for the quick response. I tried using the built in microphone but I seem to get a very low recall rate on my laptop. I don’t know if you have any suggestions? Also one other thing was the use of command line ./keytap-gui input0.kbd, when trying to open the command line states there isn’t a input0.kbd file? I replace it with output.kbd but the recall rate inst of that on the video. Any suggestions on how to help would be brilliant. On 2 Apr 2019, at 22:42, Georgi Gerganov <[email protected]<mailto:[email protected]>> wrote: The capturing tools do support an option to select the microphone to use (i.e. the capture device). You simply need to provide the command line argument -cN where you replace N with the capture device ID. For example, on my machine, when I run the record tool I get this info printed: $ ./record test.kbd Usage: ./record output.kbd [-cN] -cN - select capture device N Recording 5 frames per key press Found 3 capture devices: - Capture device #0: 'Built-in Microphone' - Capture device #1: 'Soundflower (2ch)' - Capture device #2: 'Soundflower (64ch)' Attempt to open capture device 0 : 'Built-in Microphone' ... Opened capture device succesfully! Frequency: 24000 Format: 33056 (4 bytes) Channels: 1 Samples: 512 You can see the 3 available capture devices listed in the standard output. By default, the tools select the capture device with ID = 0. For example, if I wanted to use capture device 1 instead, then I would do: $ ./record test.kbd -c1 Usage: ./record output.kbd [-cN] -cN - select capture device N Recording 5 frames per key press Found 3 capture devices: - Capture device #0: 'Built-in Microphone' - Capture device #1: 'Soundflower (2ch)' - Capture device #2: 'Soundflower (64ch)' Attempt to open capture device 1 : 'Soundflower (2ch)' ... Opened capture device succesfully! Frequency: 24000 Format: 33056 (4 bytes) Channels: 1 Samples: 512 Regarding the machine learning question: On my iMac, using the build-in mic, I can record just 2-3 keytaps of each key and be able to recognise the key audio with very high accuracy (as demonstrated in the video<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D2OjzI9m7W10&data=02%7C01%7Cs4910341%40bournemouth.ac.uk%7C6aeccd5b908a44d1d63508d6b7b40ee3%7Cede29655d09742e4bbb5f38d427fbfb8%7C0%7C0%7C636898381311686403&sdata=5WO1tFBSb7%2BhfTNoHISmhSo%2FJP2Rk4ME6qonWoVFfuE%3D&reserved=0>. The result really depends on the setup you have - mic, keyboard, surroundings, etc. More details about the approach are in the blog post<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fggerganov.github.io%2Fjekyll%2Fupdate%2F2018%2F11%2F30%2Fkeytap-description-and-thoughts.html&data=02%7C01%7Cs4910341%40bournemouth.ac.uk%7C6aeccd5b908a44d1d63508d6b7b40ee3%7Cede29655d09742e4bbb5f38d427fbfb8%7C0%7C0%7C636898381311696412&sdata=yB0nuqTyUzT1PQNJ7rldzinPIJoBTs3It5mq8OTO0vk%3D&reserved=0>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fggerganov%2Fkbd-audio%2Fissues%2F3%23issuecomment-479216304&data=02%7C01%7Cs4910341%40bournemouth.ac.uk%7C6aeccd5b908a44d1d63508d6b7b40ee3%7Cede29655d09742e4bbb5f38d427fbfb8%7C0%7C0%7C636898381311706421&sdata=RPEwHxSDIUMbq13xH%2BweoQbBtTQ7HMAV3df87MY8Uuk%3D&reserved=0>, or mute the thread<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAuuk4EuKRc71lChUsQe5r-sWn7AqZO0kks5vc86vgaJpZM4Y6BqR&data=02%7C01%7Cs4910341%40bournemouth.ac.uk%7C6aeccd5b908a44d1d63508d6b7b40ee3%7Cede29655d09742e4bbb5f38d427fbfb8%7C0%7C0%7C636898381311706421&sdata=0hJ97GxMGIAYrEq2j4lby9BdIXLImXf%2FR5%2BskFl9or0%3D&reserved=0>.

[weddy2019]

Also how do you open the application in google? On 2 Apr 2019, at 22:42, Georgi Gerganov <[email protected]<mailto:[email protected]>> wrote: The capturing tools do support an option to select the microphone to use (i.e. the capture device). You simply need to provide the command line argument -cN where you replace N with the capture device ID. For example, on my machine, when I run the record tool I get this info printed: $ ./record test.kbd Usage: ./record output.kbd [-cN] -cN - select capture device N Recording 5 frames per key press Found 3 capture devices: - Capture device #0: 'Built-in Microphone' - Capture device #1: 'Soundflower (2ch)' - Capture device #2: 'Soundflower (64ch)' Attempt to open capture device 0 : 'Built-in Microphone' ... Opened capture device succesfully! Frequency: 24000 Format: 33056 (4 bytes) Channels: 1 Samples: 512 You can see the 3 available capture devices listed in the standard output. By default, the tools select the capture device with ID = 0. For example, if I wanted to use capture device 1 instead, then I would do: $ ./record test.kbd -c1 Usage: ./record output.kbd [-cN] -cN - select capture device N Recording 5 frames per key press Found 3 capture devices: - Capture device #0: 'Built-in Microphone' - Capture device #1: 'Soundflower (2ch)' - Capture device #2: 'Soundflower (64ch)' Attempt to open capture device 1 : 'Soundflower (2ch)' ... Opened capture device succesfully! Frequency: 24000 Format: 33056 (4 bytes) Channels: 1 Samples: 512 Regarding the machine learning question: On my iMac, using the build-in mic, I can record just 2-3 keytaps of each key and be able to recognise the key audio with very high accuracy (as demonstrated in the video<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D2OjzI9m7W10&data=02%7C01%7Cs4910341%40bournemouth.ac.uk%7C6aeccd5b908a44d1d63508d6b7b40ee3%7Cede29655d09742e4bbb5f38d427fbfb8%7C0%7C0%7C636898381311686403&sdata=5WO1tFBSb7%2BhfTNoHISmhSo%2FJP2Rk4ME6qonWoVFfuE%3D&reserved=0>. The result really depends on the setup you have - mic, keyboard, surroundings, etc. More details about the approach are in the blog post<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fggerganov.github.io%2Fjekyll%2Fupdate%2F2018%2F11%2F30%2Fkeytap-description-and-thoughts.html&data=02%7C01%7Cs4910341%40bournemouth.ac.uk%7C6aeccd5b908a44d1d63508d6b7b40ee3%7Cede29655d09742e4bbb5f38d427fbfb8%7C0%7C0%7C636898381311696412&sdata=yB0nuqTyUzT1PQNJ7rldzinPIJoBTs3It5mq8OTO0vk%3D&reserved=0>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fggerganov%2Fkbd-audio%2Fissues%2F3%23issuecomment-479216304&data=02%7C01%7Cs4910341%40bournemouth.ac.uk%7C6aeccd5b908a44d1d63508d6b7b40ee3%7Cede29655d09742e4bbb5f38d427fbfb8%7C0%7C0%7C636898381311706421&sdata=RPEwHxSDIUMbq13xH%2BweoQbBtTQ7HMAV3df87MY8Uuk%3D&reserved=0>, or mute the thread<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAuuk4EuKRc71lChUsQe5r-sWn7AqZO0kks5vc86vgaJpZM4Y6BqR&data=02%7C01%7Cs4910341%40bournemouth.ac.uk%7C6aeccd5b908a44d1d63508d6b7b40ee3%7Cede29655d09742e4bbb5f38d427fbfb8%7C0%7C0%7C636898381311706421&sdata=0hJ97GxMGIAYrEq2j4lby9BdIXLImXf%2FR5%2BskFl9or0%3D&reserved=0>.

[ggerganov]

Laptop keyboards usually have very bad recall rate - the main reason is that the sound of the keys is very low. I myself have had a nice recall rate only when plugging a mechanical keyboard to my iMac - the clicks are much louder and distinguishable.

The web-version of the tool is at the following link:
https://ggerganov.github.io/jekyll/update/2018/11/24/keytap.html

[Rob-pw]

Not relevant to the software itself: but on ideas of how to circumvent this - go somewhere where there is no microphone and load your passwords onto a HID (like a Teensy 3.0), then have it type the passwords; but for every day typing it wouldn't help as the approximate keystrokes could be reconstructed - this project highlights the importance of securing every device in proximity (a secure PC with a compromised phone microphone nearby, for instance). I'd known this was possible for a number of years, but the accuracy with such a small training dataset is totally unnerving - thanks!