Spatio-Temporal Graph Convolutional Networks for Nonverbal Language in Entrepreneurial Pitching Sessions
Nonverbal language plays a role when entrepreneurs pitch their business ideas to potential investors. Some authors have proposed that several crucial clues from nonverbal communication can affect on accessing early stage investments and even long-term firm survival. In our paper and this repository, we propose a deep learning strategy grounded on Spatial-temporal Graph Convolutional Networks (ST-GCN) that is able to automatically infer a set of nonverbal language characteristics of a speaker from a monocular video.
We make use of HumanNoVeLa, our dataset with 3D human pose data from entrepreneurial pitching sessions recorded on video that can be used to train our network to estimate human nonverbal features. It contains data from 218 pitching sessions in the form of human skeletal information of speakers and ground truth of 6 nonverbal characteristics.
We propose a ST-GCN regression model that learns patterns in body poses that correlate with those personal characteristics. The obtained results exhibit good performance and outperform existing learning techniques with hand-crafted features.
This code has been branched and modified from MMSkeleton, an open source toolbox for skeleton-based human understanding and part of the open-mmlab in the charge of Multimedia Laboratory, CUHK.
a. Clone HumanNoVeLa dataset.
git clone https://bitbucket.org/fserracant/humannovela.git
b. Clone this repository.
git clone https://bitbucket.org/fserracant/stgcnnonverbal.git
c. Get into STGCNNonVerbal repository.
cd stgcnnonverbal
d. Follow installation steps in Installation and activate open-mmlab environment.
a. Create an experiment from the HumanNoVeLa dataset. An experiment is a folder with a data set is prepared for training and testing purposes. prepareExperiment.py
script will perpare the HumanNoVeLa dataset for our network and partition it into training and testing sets. It uses template yaml files for training and test that can be found at the templates directory. Run python prepareExperiment.py -h
for full list of optional arguments.
cd experiments
python prepareExperiment.py --dataset ../../humannovela/entrepreneurs/dataset.pkl --exppath <your_experiment_name>
b. (Optional) Change train.yaml
or test.yaml
to your needs.
c. (Optional) Change checkpoint
file in argparse_cfg
and processor_cfg
in test.yaml
file to ../checkpoints/st_gcn.HumanNoVeLa.pth
in order to use our trained weights.
a. Run training on your new experiment.
python ../mmskl.py <your_experiment_name>/train.yaml
b. Run testing on your experiment.
python ../mmskl.py <your_experiment_name>/test.yaml
c. Find your checkpoints, logs and results in your experiment directory.
For any question, feel free to contact
Joan Francesc Serracant : [email protected]
Coloma Ballester : [email protected]