pearl_v1's Introduction

Pretrained Encoders are All You Need

Mina Khan, P Srivatsa, Advait Rane, Shriram Chenniappa, Rishabh Anand, Sherjil Ozair, Pattie Maes

This repo provides code for the benchmark and techniques from the paper Pretrained Encoders are All You Need

📦 Install -- Install relevant dependencies and the project
🏃 Usage -- Learn how to use PEARL in colab
🔧 Change Configurations -- Change configuration to run different experiments
💾 Save Embeddings -- Generate clip embeddings for selected games

Install

$ git clone https://github.com/PAL-ML/PEARL_v1.git pearl
$ cd pearl
$ pip install -r requirements.txt

Complete installation to run on colab can be found in any of the notebooks in notebooks/experiments.

Usage

In your Google Drive, add a shortcut to our processed clip embeddings drive folder
Open any of the jupyter notebooks in notebooks/experiments in Google Colab and update the section Initialization & constants. Make sure the paths point to where the clip embeddings are saved as in step 1 and where probe and encoder checkpoints should be saved in your drive.
Run the notebook

To run without using our saved clip embeddings, change training_input in Initialization & constants from embeddings to images. Note that making this change would require you to make several changes to the notebooks we provide, including the encoder used (to CLIPEncoder).

Change configurations

Change game

To run a different game using the same parameters, change the env_name in Initialization & constants. Refer to game_names.txt for complete list of supported games.

Change parameters

To change parameters, refer to relevant section in Initialization & constants.

Change training methods for encoder/probe

To change the training methods for encoders, refer to template notebooks in notebooks/experiments.

Change probe used

Change probe_type in Initialization & constants to match any of the available probes in src/benchmark/probe.py

Save embeddings

To generate and save the CLIP embeddings we used in our experiments, refer to the notebooks in notebooks/save_embeddings. These would save the embeddings to Google Drive. Before running these notebooks , make sure to add a link to your drive folder as the parameter drive_link. The link to a folder on drive can be obtained by right-clicking on a folder and choosing the Get Link option.