facebookarchive / babi-tasks

Task generation for testing text understanding and reasoning

bAbI tasks

This repository contains code to generate the bAbI tasks as described in the paper Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. Please cite the paper if you use this code in your work (bibtex entry here).

depth

2

Installation

This project requires Torch to be installed. The easiest way to install Torch is by following the installation instructions at torch/distro. To use the library, install it with LuaRocks by running the following command from the root directory.

luarocks make babitasks-scm-1.rockspec

Usage

To generate a task, run the command

babi-tasks <task-id>

where <task-id> is either a class name (like PathFinding) or the task number (e.g. 19). To quickly generate 1000 examples of each task, you can use

for i in `seq 1 20`; do babi-tasks $i 1000 > task_$i.txt; done

Tasks

The tasks in babi/tasks correspond to those from the original dataset as follows:

#	Task	Class name
1	Basic factoid QA with single supporting fact	WhereIsActor
2	Factoid QA with two supporting facts	WhereIsObject
3	Factoid QA with three supporting facts	WhereWasObject
4	Two argument relations: subject vs. object	IsDir
5	Three argument relations	WhoWhatGave
6	Yes/No questions	IsActorThere
7	Counting	Counting
8	Lists/Sets	Listing
9	Simple Negation	Negation
10	Indefinite Knowledge	Indefinite
11	Basic coreference	BasicCoreference
12	Conjunction	Conjunction
13	Compound coreference	CompoundCoreference
14	Time manipulation	Time
15	Basic deduction	Deduction
16	Basic induction	Induction
17	Positional reasoning	PositionalReasoning
18	Reasoning about size	Size
19	Path finding	PathFinding
20	Reasoning about agent's motivation	Motivations

Note: This code is a rewrite of the original code that was used to generate the publicly available dataset at fb.ai/babi. As such, it is not possible to produce exactly the same dataset. However, we have verified that numbers obtained are very similar.

Task flags

Some tasks accept configuration flags that will change their output.

In both the PathFinding and Size the number of inference steps required to answer the question can be changed. You can also control the number of "decoys" (locations that are not part of the path).

babi-tasks PathFinding --path-length 3 --decoys 1
babi-tasks Size --steps 3

Currently the path length plus the number of decoys has to be 5 or less. Similarly, the number of size comparisons cannot be more than 5.

For tasks involving people moving around, the use of coreferences and conjunctions can be controlled with the flags --coreference and --conjunction. These flags take a number between 0 and 1 as an argument, determining the fraction of the time coreferences and conjunctions are used respectively.

babi-tasks WhereIsActor --coreference 1.0

Tasks can also be rendered in a more symbolic manner. Use the flag --symbolic true to enable this.:

1 H teleport N
2 H teleport F
3 eval H is_in  F       2

Code Overview

Tasks are generated through simulation: We have a world containing entities, and actions that can add new entities to the world, or modify entities' states. Simulations then just take the form of sampling actions that are valid.

We often want to ask questions that require some sort of logical inference. Some types of inference can be re-used in multiple tasks, for example the deduction that a person and the object they are holding are in the same place is used several times. For this reason, some of the reasoning has been factored out: We keep track of what the reader of a story knows about the world, and each time a new line is read, we update this knowledge.

What follows is a brief overview of the classes and concepts used, which should help guide the understanding of the code.

World

A world is a collection of entities. Worlds can be loaded from text files such as those found in babi/tasks/worlds using the world:load(filename) command.

Entity

All concepts and objects in the simulations are entities. They are effectively Lua tables that describe the entity's properties.

Actions

Actions modify the state of the world. Each action is performed by an entity, even actions like setting the location or size of another entity (these are usually performed by the entity "god").

An action's is_valid method will test whether an action can be performed e.g. John cannot move to the kitchen if he is already there. The perform method assumes that the action is valid, and modifies the world accordingly i.e. it will change the location of John.

Lastly, actions can update the reader's knowledge of the world. For example, if we know that John is in the kitchen, the action "John grabs the milk" informs the reader that the milk is in the kitchen as well.

Knowledge

The Knowledge class keeps track of what a reader currently knows about the world. When actions are performed, the Action.update_knowledge method can update this knowledge accordingly. For example, when Knowledge contains the information that John is in the kitchen, the action of dropping the milk will result in the knowledge being updated to say that the milk is in the kitchen, and that it isn't being held by anyone.

The Knowledge class takes into account some basic logical rules. For example, some properties are "exclusive" in the sense that only one value can be true (John cannot be in the kitchen and the garden at the same time, but he can be not in the kitchen and not in the garden at the same time). Reversely, this means that if John is in the garden, the reader knows that he is not in the kitchen.

We keep track of which actions gave us which pieces of information about the world. This way, we can provide the user with the supporting facts when asking questions.

Clause

Facts about the world are expressed as clauses of the form (truth value, actor, action, arguments). For example (true, john, teleport, kitchen) means that John moved to the kitchen, while (false, john, drop, milk) means that John did not drop the milk. Note that because all information must be conveyed as actions, the sentence "John is in the garden" is represented as (true, god, set_property, is_in, garden).

Question

A question is represented as a tuple of the form (question type, clause, support).

1 John is in the garden.
2 Where is John? garden 1

This story is represented as a clause, clause = (true, god, set_property, john, is_in, garden), followed by a question, question = (evaluate, clause, {1}). A question like "Is john in the garden?" would instead be represented as question = (yes_no, clause, {1}).

Natural language generation

After the simulation is complete, a story (task) is nothing more but a list of clauses and questions. We turn this into text using the stringify function. This function performs a simple process: It repeatedly tries to find templates that can turn the next clause(s) or question(s) into text. It randomly samples a template from the matching ones, and goes on to the next clause that needs to be converted.

Templates can be selected further based on configuration (each task has a default configuration, but they can be passed through the command line as well). This enables turning on things like coreferences, conjunctions, etc.

References

• Jason Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov, Alexander M. Rush, Bart van Merriënboer, "Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks", arXiv:1502.05698 [cs.AI].
• Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus, "End-To-End Memory Networks", arXiv:1503.08895 [cs.NE].

babi-tasks's Issues

Inconsistent output in tasks 5 and 17

I am getting incorrect/inconsistent answers generated from tasks 5 and 17.

For task 5:

...
11 John journeyed to the kitchen.
12 Daniel gave John the milk.
13 Daniel moved to the bathroom.
14 Mary journeyed to the kitchen.
15 John gave Daniel the milk.
16 Who received the milk?	John	12
17 Sandra moved to the bedroom.
18 Sandra moved to the garden.
19 John moved to the hallway.
20 Sandra went to the office.
21 John gave Daniel the apple.
22 John journeyed to the bathroom.
23 Daniel gave John the apple.
24 Who received the apple?	John	23

Note that lines 12 and 15 both involve trading milk between Daniel and John. In this case the answer to "Who received the milk?" (16) is John. On lines 21 and 23, there is a trade of the apple, and in this case the answer to "Who received the apple?" (24) is also John. But the order of the trade is reversed between the two cases, so it seems like at least one of these answers is incorrect. (In particular, if the question is supposed to correspond to the most recent transfer, then the first answer, on line 16, is incorrect.)

For task 17:

1 The pink triangle is above of the square.
2 The red square is to the right of of the pink triangle.
...
10 Is the red square to the left of the pink triangle?	yes	1 2

This is wrong according to the facts given. (Also, I think there is an extra "of" in the generated facts.)

Since these are randomly generated, these exact examples might not be directly reproducible, but I was able to generate similar inconsistencies by repeatedly running the generation code for these two tasks.

Dataset unavailable

The dataset (The SimpleQuestions dataset) provided on the project page seems to be unavailable

https://research.fb.com/downloads/babi/

That is the broken link: https://www.dropbox.com/s/tohrsllcfy7rch4/SimpleQuestions_v2.tgz

Can you please make it available again?

Missed questions and answers lines in task 16

Example: qa16_basic-induction_train.txt

1 Julius is a rhino.
2 Brian is a swan.
3 Brian is white.
4 Greg is a frog.
5 Julius is yellow.
6 Greg is gray.
7 Bernhard is a lion.
8 Lily is a rhino.
9 Lily is green.
10 ???
1 Lily is a frog.
2 Bernhard is a frog.
3 Bernhard is green.
4 Brian is a lion.
5 Brian is white.
6 Julius is a swan.
7 Julius is green.
8 Lily is green.
9 Greg is a swan.
10 What color is Greg?	green	9 6 7

how could I run this project from source code?

if I want debug it, for example, I edit the source and run it immediately without reinstall it by luarocks, how can I do that

Conflicting facts in generated task data

I tried to generate some task data for question 19 with the following default command:

babi_tasks 19 1000 > test_19.txt

In the resulting file I got, I observed some conflicting facts, for example

1 The office is south of the garden.
2 The office is west of the hallway.
3 The kitchen is west of the garden.
4 The garden is south of the office.
5 What is the path from hallway to garden?      w,n     1 2

1 and 4 cannot hold at the same time. Another example is

1 The kitchen is west of the garden.
2 The hallway is north of the kitchen.
3 The office is south of the garden.
4 The office is east of the garden.
5 What is the path from kitchen to office?      e,s     3 1

Here 3 and 4 are confusing. And one more:

1 The bedroom is west of the bathroom.
2 The garden is north of the bedroom.
3 The hallway is west of the bedroom.
4 The hallway is north of the garden.
5 What is the path from bedroom to garden?      w,s     3 4

In this one there are two paths from bedroom to garden, (1) using fact 2; (2) using 3 and 4. The generated ground truth answer ignored the more direct solution, and the two solutions are actually conflicting, as fact 2 says garden is north of the bedroom but fact 3 and 4 together imply that garden is south of the bedroom.

Is this a common problem?

Three Supporting facts requiring info from 4 facts

Based on the Task name, this shouldn't be the case or am I wrong.
I am getting stories such as -:

1 Sandra travelled to the kitchen.
2 Sandra grabbed the milk.
3 Daniel journeyed to the bedroom.
4 Sandra dropped the milk.
5 Daniel journeyed to the hallway.
6 Sandra took the milk.
7 Sandra went to the bedroom.
8 Where was the milk before the bedroom? kitchen 1 2 6 7

Conflicting facts in generated task data

I tried to generate some task data for question 15
And I got some conflicting facts

babi/utilities.lua line 89: is this a bug?

function Grid:to_coordinates(i)
    return (i - 1) % self.width + 1, math.floor((i - 1) / 10) + 1
end

Should the number 10 here actually be self.width?

bug in logic?

Hi I generated a simple example for task 4:

asim@XX:~/bAbI-tasks/: babi-tasks 4 1

1 The office is south of the garden.

2 The hallway is south of the office.

3 What is south of the hallway? office 2

I consistently see such examples being generated with incorrect answers.

Loading Worlds from Text Files

What is the syntax for using world:load(filename) to add new worlds?

Let's say the file I wanted to load was world.txt.
I've tried world:load(world.txt), world:load world.txt, and other similar formats with varying spacing, but I only get syntax error near unexpected token '(' or syntax error near unexpected token 'world.txt'.

New Dataset

Hello , I was doing a project which requires to generate a conversations artificially, like the The Dialog-based Language Learning dataset of Facebook AI. Can I do that. Please Note I want the domain of my conversations to be realted to a student library.

Generated data problematic for task 17

Similar to Issue #1 , data generated from task 17 also seems problematic. With the following command:

babi-tasks 17 10

Here is one generated problematic example:

1 The rectangle is below of the red square.
2 The red square is below of the rectangle.
3 The rectangle is above of the red square.
4 The pink rectangle is to the left of of the red square.
5 Is the rectangle below the red square?    no  2 1
6 Is the pink rectangle to the left of the rectangle?   yes 2 1
7 Is the pink rectangle above the rectangle?    no  2 1
8 Is the rectangle to the right of the pink rectangle?  yes 2 1
9 Is the rectangle above the red square?    yes 2 1
10 Is the rectangle below the red square?   no  2 1
11 Is the red square below the rectangle?   yes 2 1
12 Is the pink rectangle to the right of the red square?    no  2 1

https://www.dropbox.com/s/tohrsllcfy7rch4/SimpleQuestions_v2.tgz is broken

The dataset (The SimpleQuestions dataset) provided on the project page seems to be unavailable

https://research.fb.com/downloads/babi/

That is the broken link: https://www.dropbox.com/s/tohrsllcfy7rch4/SimpleQuestions_v2.tgz

I can get simplequestion.txt from https://s3.amazonaws.com/fairdata/parlai/simplequestions/simplequestions.tar.gz,

but I can't get subset of freebase,fb2m, fb5M.
Maybe it's my problem, but can you provide a link like https://s3.amazonaws.com