andabi / voice-vector

Deep neural networks for getting text-independent speaker embedding written in TensorFlow

Text-independent voice vectors

Subtitle: which of the Hollywood stars is most similar to my voice?

• Authors: Dabi Ahn([email protected]), Noah Jung, Jujin Lee and Kyubyong Park
• Demo: Check out who your voice is like!

Prologue

Everyone has their own voice. The same voice will not exist from different people, but some people have similar voices while others do not. This project aims to find individual voice vectors using VoxCeleb dataset, which contains 1,251 Hollywood stars' 145,379 utterances. The voice vectors are text-independent, meaning that any pair of utterances from same speaker has similar voice vectors. Also the closer the vector distance is, the more voices are similar.

Architectures

The architecture is based on a classification model. The utterance inputted is classified as one of the Hollywood stars. The objective function is simply a cross entropy between speaker labels from ground truth and predictions. Eventually, the last layer's activation becomes the speaker's embedding.

The model architecture is structured as follows.

1. memory cell
   • CBHG module from Tacotron captures hidden features from sequential data.
2. embedding
   • memory cell's last output is projected by the size of embedding vector.
3. softmax
   • embedding is logits for each classes.

Training

• VoxCeleb dataset used.
  • 1,251 Hollywood stars' 145,379 utterances
  • gender dist.: 690 males and 561 females
  • age dist.: 136, 351, 318, 210, and 236 for 20s, 30s, 40s, 50s, and over 60s respectively.
• text-independent
  • at each step, the speaker is arbitrarily selected.
  • for each speaker, the utterance inputted is randomly selected and cropped so that it does not matter to text.
• loss and train accuracy
  • 
    

Embedding

• Common Voice dataset used for inference.
  • hundreds of thousands of English utterances from numerous voice contributors in the world.
• evaluation accuracy
  • 
    
• embedding visualization using t-SNE
  • voices are well clustered by gender without any supervision in training.

    • 

  • but we could not find any tendency toward age.

    • 

How to run?

Requirements

• python 2.7
• tensorflow >= 1.1
• numpy >= 1.11.1
• librosa == 0.5.1
• tensorpack == 0.8.0

Settings

• configurations are set in two YAML files.
• hparams/default.yaml includes default settings for signal processing, model, training, evaluation and embedding.
• hparams/hparams.yaml is for customizing the default settings in each experiment case.

Runnable python files

• train.py for training.
  • run python train.py some_case_name
  • remote mode: utilizing more cores of remote server to load data and enqueue more quickly.
    • run python train.py some_case_name -remote -port=1234 in local server.
    • run python remote_dataflow.py some_case_name -dest_url=tcp://local-server-host:1234 -num_thread=12 in remote server.
• eval.py for evaluation.
  • run python eval.py some_case_name
• embedding.py for inference and getting embedding vectors.
  • run python embedding.py some_case_name

Visualizations

• Tensorboard
  • Scalars tab: loss, train accuracy, and eval accuracy.
  • Audio tab: sample audios of input speakers(wav) and predicted speakers(wav_pred)
  • Text tab: prediction texts with the following form: 'input-speaker-name (meta) -> predicted-speaker-name (meta)'
    • ex. sample-022653 (('female', 'fifties', 'england')) -> Max_Schneider (('M', '26', 'USA'))
• t-SNE output file
  • outputs/embedding-[some_case_name].png

Future works

• One-shot learning with triplet loss.

References

• Nagrani, A., Chung, J. S., & Zisserman, A. (2017, June 27). VoxCeleb: a large-scale speaker identification dataset. arXiv.org.
• Zhang, C., & Koishida, K. (2017). End-to-End Text-Independent Speaker Verification with Triplet Loss on Short Utterances (pp. 1487–1491). Presented at the Interspeech 2017, ISCA: ISCA. http://doi.org/10.21437/Interspeech.2017-1608
• Li, C., Ma, X., Jiang, B., Li, X., Zhang, X., Liu, X., et al. (2017, May 6). Deep Speaker: an End-to-End Neural Speaker Embedding System. arXiv.org.

voice-vector's Issues

Can't get a dependency to work

In train.py and eval.py, the line from data_load import DataLoader, AudioMeta seems to cause the following error: ModuleNotFoundError: No module named 'tensorpack.dataflow.prefetch'.

I've verified that tensorpack is installed but can't seem to get passed this. I've actually tried this on multiple computers and get the same error. Has anyone encountered this?

Dataset files confusion

Which datapath should be put in path .
There are 3 variables speaker_id ,data_path, meta_path please specify what path should I keep in following
blanks.
I have voxceleb dataset and metadata .

I think take the last timestep output should use code like 'out = out[:, -1, :]'

Hi, @andabi
in master branch model.py line 54 your code is:
out = out[..., -1] # (n, h)

is this want to take the last timestep output?

Pre Trained Model

Thank you for this code @andabi. Is it possible to share your model?

Using your model on another project

Hello,

I have a speech recognition project (written in tensorflow), and I would like to know how could I be able to use your trained model inside my project. My initial idea is to use the layer before the last one of your model as a feature vector (that will work as a speaker identification).

Unfortunately, I am not quite familiar with tensorpack, and I am having trouble to do a simple task as: (1) load your trained model in tensorflow; (2) remove the last layer; and (3) use it as a feature extractor. Is it possible to do that? or your model needs the tensorpack to work?

how to get embeddings from a speaker

I wanted extract Embeddings from a Audio File...Because I wanted to do Voice Similarity.
I'm Confusing with ReadMe File......to get Embeddings,Any Simple Commands...

I found These code on README....

• train.py for training.
  • run python train.py some_case_name

In Above case,what is some_case_name

And Also Some Help for Extracting Embeddings with embeddings.py....

What is the name of the paper to which this project belongs?

Hello, this is a wonderful project, but I can't seem to find a paper for this project on the Internet. So, what is the title of this paper? looking forward to your reply

embedding with pretrained model

Hi!

How to only to embed with pretrained model? Without downloading vox dataset

how preprocessing voxceleb data and acc==0?

how process voxceleb data for run trian.py?

why I iter 200 using 2GPU, but acc=0?

No such file or directory: 'datasets/vcc2016_training'

'vcc2016_training' corresponds to which dataset : VoxCeleb or Common Voice?