Behavior of `autotune` when `n_clusters == 1` about spectralcluster

[wq2012]

Ah I see.

Spectral clustering (or to be more specific, the eigen-gap criterion) doesn't work well at distinguishing single speaker vs multi speakers. We usually always set min_clusters=2 for spectral clustering.

To handle single speaker utterances, what we do is:

• We use an RNN-T speaker turn detection model to detect speaker turns.
• If there is at least one highly-confident speaker turn, then we trigger spectral clustering, with `min_clusters=2.
• Otherwise, we assume there is only one single speaker.

However since the RNN-T speaker turn model is not open sourced, I guess we will have to do something else to handle single speaker cases.

A few ideas I can think of:

• Use some other implementations of speaker turn/change detection.
• Pre-processing: always run another faster clustering (e.g. hierarchical) first to decide whether there are at least two speakers.
• Pre-processing: compute max-distance between any pair of embeddings (or perhaps only neighboring pairs?) and threshold that max-distance to decide whether there are at least two speakers.
• Post-processing (the one you mentioned): if n_clusters=2, compare centroids with simple threshold.

[wq2012]

I can update this library to implement some of these ideas. Stay tuned!

[hbredin]

I see from the GitHub logs that you have been working on this. Thanks!

What about fitting the values of the affinity matrix with

• 1-gaussian GMM
• 2-gaussians GMM
  and use the BIC criterion to decide which is best.

If 2-gaussians GMM is best, it would mean that there are actually two modes in the distribution of affinity value (one for intra cluster affinity, one for inter cluster affinity)... and therefore strictly more than one cluster. Otherwise there is probably just one cluster.

This approach (if succesfull) would have the extra benefit of not relying on any additional threshold.

[wq2012]

That's a good idea! I will give it a try.

[wq2012]

I've changed the default value of single_cluster_condition to AffinityGmmBic.

[wq2012]

Just realized one thing.

The diagonals of the affinity matrix will be 1. So if we use GMM BIC, it's very likely it's always going to be GMM-2 wins, as the diagonals would be significantly larger than anything else.

We should remove/crop diagonal before we fit GMM to the affinity values.

[hbredin]

Good point. Same goes for values close to the diagonal in case embeddings are extracted using an overlapping sliding window.

[wq2012]

OK, I added a parameter single_cluster_affinity_diagonal_offset, which defaults to 1

[wq2012]

Please use version 0.2.11.