Overlapping speech detection about pyannote-audio

[hbredin]

In latest version of pyannote.audio (v1.0), pyannote.audio.labeling.tasks.SpeechActivityDetection has a overlap boolean parameter that can be used to train SAD with 3 classes (non-speech vs. one speaker vs. 2+ speakers) instead of 2 (non-speech vs. speech).

[claudiv]

Hi there! Thank you for opensourcing this project 👍 .

I want to detect overlapping speech regions using pyAnnote.Audio version 1.0.1.
I trained a model with speech activity detection and enabled overlap detection (as you proposed in the previous comment). Then I applied the model to AMI corpus.
Now it's unclear to me how/where I can see the detected overlapping speech regions.
When I load my precomputed SAD-scores, I would normally binarize these SAD-scores to obtain speech regions, but where can I see regions with amount of detected speakers > 1 ?

Best wishes

[hbredin]

With overlap=True, SAD scores are now 3-dimensional:

• 1st dimension is for non-speech
• 2nd dimension is for exactly one speaker
• 3rd dimension is for 2 or more speakers

Therefore, you should binarize the 3rd dimension, ie use dimension=2 in

pyannote-audio/pyannote/audio/signal.py

Line 161 in b678920

def apply(self, predictions, dimension=0):

FYI, I never really used this overlap detection so I'd be curious to receive some feedback on how well it performs...

[chitwansaharia]

Hi,

I am also working on the problem of overlap detection for AMI. For training, I followed your tutorial, just adding overlap: True parameter to config.yml, and I got a training loss curve like this (comparison b/w overlap and non-overlap)

However, I am not sure how to evaluate this model using pyannote-metrics after binarizing the 3rd dimension.

[hbredin]

You can take inspiration for pyannote-overlap-detection validation code or use the script directly

$ pyannote-overlap-detection --help

It will compute recall at fixed precision.

[chitwansaharia]

Thanks a lot!

[hbredin]

Closing this stalled issue. Please re-open if needed.

See https://arxiv.org/abs/1910.11646 for more up-to-date overlapped speech detection results (obtained with pyannote.audio).