aniket1998 / disentangled-sequential-autoencoder

Variational Autoencoder for Unsupervised and Disentangled Representation Learning of content and motion features in sequential data (Mandt et al.).

Disentangled Sequential Autoencoder

PyTorch implementation of Disentangled Sequential Autoencoder (Mandt et al.), a Variational Autoencoder Architecture for learning latent representations of high dimensional sequential data by approximately disentangling the time invariant and the time variable features.

Results

We test our network on the Liberated Pixel Cup dataset consisting of sprites of video game characters of varying hairstyle, clothing, skin color and pose. We constrain ourselves to three particular types of poses, walking, slashing and spellcasting. The network learns disentangled vector representations for the static (elements like skin color and hair color) and dynamic aspects (motion) in the vectors f, and z1, z2, z3, .. z8 (one for each frame), respectively

Style Transfer

We perform style transfer by learning the f and z encodings of two characters that differ in both appearance and pose, and swap their z encodings. This causes the characters to interchange their pattern of motion while preserving appearance ,allowing manipulations like "blue dark elf walking" swapped with "lightskinned human spellcasting" gives "blue dark elf spellcasting" and "lightskinned human walking" respectively

Sprite 1	Sprite 2	Sprite 1's Body With Sprite 2's Pose	Sprite 2's Body With Sprite 1's Pose