Coder Social home page Coder Social logo

esbenkc / othello_world Goto Github PK

View Code? Open in Web Editor NEW

This project forked from likenneth/othello_world

0.0 0.0 0.0 10 MB

Emergent world representations: Exploring a sequence model trained on a synthetic task

License: MIT License

Shell 0.04% Python 11.57% HTML 7.31% Jupyter Notebook 81.07%

othello_world's Introduction

Update 02/13/2023 ๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ

Neel Nanda just released a TransformerLens version of Othello-GPT (Colab, Repo Notebook), boosting the mechanistic interpretability research of it.

Othello World

This repository provides the code for training, probing and intervening the Othello-GPT in Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task, to be present at ICLR 2023.
The implementation is based on minGPT, thanks to Andrej Karpathy.

Abstract

Language models show a surprising range of capabilities, but the source of their apparent competence is unclear. Do these networks just memorize a collection of surface statistics, or do they rely on internal representations of the process that generates the sequences they see? We investigate this question by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network and create "latent saliency maps" that can help explain predictions in human terms.

Table of Contents

  1. Installation
  2. Training Othello-GPT
  3. Probing Othello-GPT
  4. Intervening Othello-GPT
  5. Attribution via Intervention Plots
  6. How to Cite

Installation

Some plotting functions require Latex on your machine: check this FAQ for how to install.
Then use these commands to set up:

conda env create -f environment.yml
conda activate othello
python -m ipykernel install --user --name othello --display-name "othello"
mkdir -p ckpts/battery_othello

Training Othello-GPT

Download the championship dataset and the synthetic dataset and save them in data subfolder.
Then see train_gpt_othello.ipynb for the training and validation. Alternatively, checkpoints can be downloaded from here to skip this step.
The default experiment setting requires $8$ GPU's and takes up to roughly $12$ Gigabytes memory on each. Once you set up the code, we can use jupyter nbconvert --execute --to notebook --allow-errors --ExecutePreprocessor.timeout=-1 train_gpt_othello.ipynb --inplace --output ckpts/checkpoint.ipynb to run it in background.

Probing Othello-GPT

Then we will use train_probe_othello.py to train probes.
For example, if we want to train a nonlinear probe with hidden size $64$ on internal representations extracted from layer $6$ of the Othello-GPT trained on the championship dataset, we can use the command python train_probe_othello.py --layer 6 --twolayer --mid_dim 64 --championship.
Checkpoints will be saved to ckpts/battery_othello or can be alternatively downloaded from here. What produces the these checkpoints are produce_probes.sh.

Intervening Othello-GPT

See intervening_probe_interact_column.ipynb for the intervention experiment, where we can customize (1) which model to intervene on, (2) the pre-intervention board state (3) which square(s) to intervene.

Attribution via Intervention Plots

See plot_attribution_via_intervention_othello.ipynb for the attribution via intervention experiment, where we can also customize (1) which model to intervene on, (2) the pre-intervention board state (3) which square(s) to attribute.

How to Cite

@inproceedings{
li2023emergent,
title={Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task},
author={Kenneth Li and Aspen K Hopkins and David Bau and Fernanda Vi{\'e}gas and Hanspeter Pfister and Martin Wattenberg},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=DeG07_TcZvT}
}

othello_world's People

Contributors

likenneth avatar neelnanda-io avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.