maluuba / qgen-workshop

ImplementAI Workshop on Deep NLP for Question Generation

Multi-task Question and Answer Generation

With the goal of building an end-to-end model, we ended up building a multi-task model to generate (question, answer) pairs from a document. We combine a few core concepts for text processing using neural networks to build our model.

See the notebook for an explanation of the model with an overview of the code.

Setup

• Install Python 3.x (we recommend using Anaconda/Miniconda).
• If on Windows, it is highly recommended to install Numpy and SciPy built with MKL support. There's two ways to do this depending on your package manager:
  • conda: conda create --name q-gen python=3.5 h5py numpy pandas scipy to pull the packages maintained by Continuum through Anaconda.
  • pip: download NumPy + SciPy with MKL support from here. Install with pip install <path_to_whl>.
• Install requirements:
  • Non Windows: pip install -r requirements.txt.
  • Windows: pip install -r requirements.win.txt.
• Download GloVe 100 dim embeddings from here and extract to the root of this repo.
• Download NewsQA and process per these instructions. Put (dev|test|train).csv into the root of this repo.

Training

Prepare the data:

PYTHONPATH=".:$PYTHONPATH" python qgen/data.py

Train:

PYTHONPATH=".:$PYTHONPATH" python qgen/model.py

Loading in TensorBoard

tensorboard --logdir='log_dir'

qgen-workshop's Issues

Documentation about how to run the inference

can you please add details on how to run inference after finishing the training

automatic evaluation metrics

i want to Evaluation the question that i create with an automatic evaluation metrics.
can you reccomend me which one to use?
is it possoble at this project?
thanks

hidden layers

can someone tell me how many hidden layerss exist at this implementaion?
i think that 3 but not sure that i am right.
thanks

Bizarre questions generated.

After training the model the questions generated are very bizarre.
The model is trained and (test|train|dev).csv is generated as per the instructions.
Can you tell me what might be the reason for this?

Question: what did obama obama about ?
Answer: conscience wo n’t

Question: who was the the ? ?
Answer: lionel messi

Question: how how many the the the ? ?
Answer: third

Question: who is the the ? ?
Answer: cristiano

Question: what what is the to to ?
Answer: rail

Question: what is the the to to to ?
Answer: millimeter wave

Question: what is the the to ?
Answer: deny

Question: who is is the to ? ?
Answer: qaeda

Question: what is the the ? ?
Answer: safe haven

Question: who what is the to to ? ? ? ?
Answer: taliban

Question: what what the the the to to ? ?
Answer: momentum

Question: how many many the the the ? ? ?
Answer: six

Question: who is the the ? ?
Answer: maurice jarre

Question: who was the the ? ?
Answer: trade center

Question: what did the the ? ?
Answer: little eichmanns

Question: what was the the ? ?
Answer: 30-year-old dolphin at sea world

Question: what was the the ? ?
Answer: colliding

Question: what was the the the ? ?
Answer: dolphin

Question: where was the the ? ?
Answer: discovery cove area of

Question: what what was the the ? ? ?
Answer: 83

Question: what is the the to ?
Answer: innocent

Question: who is the the to ? ?
Answer: renters

Question: how many many the the ? ? ?
Answer: foreclosed

Question: who is the the ? ?
Answer: criticized mortgage

Question: what is the the ? ?
Answer: mortgage meltdown

Question: what does the say say
Answer: many renters are being evicted for landlords '

Question: who was the the ? ?
Answer: ronnie

Question generation from another story

Is there a way to use this code to generate the questions from a random story file? I mean that can this code be modified in any way so that I can use the trained model to generate questions on a new story file?

no module named 'qgen'

I'm trying to run this through Anaconda on windows.

When I run "python qgen\data.py I get the error:

no module named 'qgen'

What am I doing wrong?

Error

This is the error which I got after running the command to prepare the data
PYTHONPATH=".:$PYTHONPATH" python qgen/data.py

Error:
Traceback (most recent call last):
File "qgen/data.py", line 249, in
trim_embeddings()
File "qgen/data.py", line 227, in trim_embeddings
for data in [_load_training_stories().values(), _load_test_stories().values()]:
File "qgen/data.py", line 208, in _load_training_stories
_training_stories = _read_data("train.csv")
File "qgen/data.py", line 160, in _read_data
document_text = row[1]
IndexError: list index out of range

Output

I have successfully trained the model on the NewsQA dataset. How do I use the trained model to generate questions on other data or strings?

And what does this output indicate in the model.py file?
https://github.com/Maluuba/qgen-workshop/blob/master/qgen/model.py#L150-L154

Output:

Question: who is the the ? ?
Answer: customers

Question: how many the will the ? ? ?
Answer: than 2,000

Question: what what is the ? ? ?
Answer: international

Question: who is the the ? ?
Answer: sonia

Question: what did the the ? ?
Answer: meet its oil supply obligations in nigeria

Question: who is is the ? ? ?
Answer: mend

2019-04-18 13:38:31.348890: W tensorflow/core/framework/allocator.cc:108] Allocation of 4300832256 exceeds 10% of system memory.

During running this repo, I am not able to complete my model training due to this error. First this warning comes and then later my terminal stops executing this code by giving the error Segmentation fault: Core dumped. Any kind of help will be highly valuable. Kindly guide me. Thanks in advance.

Where is this "glove.6B.100d.trimmed.txt" file?

As in Glove.6B file i can see only glove.6B.100d.txt .
Can anyone help me in this?

dataset

hey which data set did you use at your project?

Error

Hi @surbhitkapoor ,
This is the error which I got after running the command to prepare the data
PYTHONPATH=".:$PYTHONPATH" python qgen/data.py

Error:
Traceback (most recent call last):
File "qgen/data.py", line 249, in
trim_embeddings()
File "qgen/data.py", line 227, in trim_embeddings
for data in [_load_training_stories().values(), _load_test_stories().values()]:
File "qgen/data.py", line 208, in _load_training_stories
_training_stories = _read_data("train.csv")
File "qgen/data.py", line 160, in _read_data
document_text = row[1]
IndexError: list index out of range
getting same issue can you please help me to resolve this issue

Thanks & Regards,
Manikantha Sekhar

Resource Exhausted Error

When I try to train the model of Google Colab, following all the steps stated, it shows the error

`ResourceExhaustedError Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1333 try:
-> 1334 return fn(*args)
1335 except errors.OpError as e:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
1318 return self._call_tf_sessionrun(
-> 1319 options, feed_dict, fetch_list, target_list, run_metadata)
1320

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata)
1406 self._session, options, feed_dict, fetch_list, target_list,
-> 1407 run_metadata)
1408

ResourceExhaustedError: OOM when allocating tensor with shape[145,16,400003] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node decoder/TensorArrayStack/TensorArrayGatherV3}} = TensorArrayGatherV3[dtype=DT_FLOAT, element_shape=[?,400003], _device="/job:localhost/replica:0/task:0/device:GPU:0"](decoder/TensorArray, decoder/TensorArrayStack/range, decoder/while/Exit_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[{{node loss/_129}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3300_loss", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

ResourceExhaustedError Traceback (most recent call last)
in ()
24 decoder_inputs: batch["question_input_tokens"],
25 decoder_labels: batch["question_output_tokens"],
---> 26 decoder_lengths: batch["question_lengths"],
27 })
28 print("Loss: {0}".format(loss_value))

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
927 try:
928 result = self._run(None, fetches, feed_dict, options_ptr,
--> 929 run_metadata_ptr)
930 if run_metadata:
931 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
1150 if final_fetches or final_targets or (handle and feed_dict_tensor):
1151 results = self._do_run(handle, final_targets, final_fetches,
-> 1152 feed_dict_tensor, options, run_metadata)
1153 else:
1154 results = []

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1326 if handle is None:
1327 return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1328 run_metadata)
1329 else:
1330 return self._do_call(_prun_fn, handle, feeds, fetches)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1346 pass
1347 message = error_interpolation.interpolate(message, self._graph)
-> 1348 raise type(e)(node_def, op, message)
1349
1350 def _extend_graph(self):

ResourceExhaustedError: OOM when allocating tensor with shape[145,16,400003] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node decoder/TensorArrayStack/TensorArrayGatherV3 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py:328) = TensorArrayGatherV3[dtype=DT_FLOAT, element_shape=[?,400003], _device="/job:localhost/replica:0/task:0/device:GPU:0"](decoder/TensorArray, decoder/TensorArrayStack/range, decoder/while/Exit_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[{{node loss/_129}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3300_loss", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'decoder/TensorArrayStack/TensorArrayGatherV3', defined at:
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/usr/local/lib/python3.6/dist-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelapp.py", line 477, in start
ioloop.IOLoop.instance().start()
File "/usr/local/lib/python3.6/dist-packages/tornado/ioloop.py", line 832, in start
self._run_callback(self._callbacks.popleft())
File "/usr/local/lib/python3.6/dist-packages/tornado/ioloop.py", line 605, in _run_callback
ret = callback()
File "/usr/local/lib/python3.6/dist-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 536, in
self.io_loop.add_callback(lambda : self._handle_events(self.socket, 0))
File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events
self._handle_recv()
File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
self._run_callback(callback, msg)
File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
callback(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell
handler(stream, idents, msg)
File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "/usr/local/lib/python3.6/dist-packages/ipykernel/ipkernel.py", line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/usr/local/lib/python3.6/dist-packages/ipykernel/zmqshell.py", line 533, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
if self.run_code(code, result):
File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2882, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 15, in
decoder_outputs, _, _ = seq2seq.dynamic_decode(decoder, scope="decoder")
File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py", line 328, in dynamic_decode
final_outputs = nest.map_structure(lambda ta: ta.stack(), final_outputs_ta)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py", line 381, in map_structure
structure[0], [func(*x) for x in entries])
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py", line 381, in
structure[0], [func(*x) for x in entries])
File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py", line 328, in
final_outputs = nest.map_structure(lambda ta: ta.stack(), final_outputs_ta)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/tensor_array_ops.py", line 856, in stack
return self._implementation.stack(name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/tensor_array_ops.py", line 289, in stack
return self.gather(math_ops.range(0, self.size()), name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/tensor_array_ops.py", line 303, in gather
element_shape=element_shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 6031, in tensor_array_gather_v3
flow_in=flow_in, dtype=dtype, element_shape=element_shape, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[145,16,400003] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node decoder/TensorArrayStack/TensorArrayGatherV3 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py:328) = TensorArrayGatherV3[dtype=DT_FLOAT, element_shape=[?,400003], _device="/job:localhost/replica:0/task:0/device:GPU:0"](decoder/TensorArray, decoder/TensorArrayStack/range, decoder/while/Exit_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[{{node loss/_129}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3300_loss", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
`

I tried reducing batch size and even reduced the training dataset.

overfitting

hey! how did you deal with overfitting at you project?

test the project with new test.csv

I trained the model and then created questions from the given test.csv file.
Then I created a new file test.csv with new data with the same foramt.
I had a bug. When I looked in the debug, I figured out that the matrix "answers" in the end of the following code is full with zeros:

answers = session.run(answer_tags, {
document_tokens: batch["document_tokens"],
document_lengths: batch["document_lengths"],
})
answers = np.argmax(answers, 2)

When I run the code with the given test.csv this matrix has also ones values and the project run successfully.
Do you have any idea how to solve this issue?
Thanks.

error of answers matrix at the model.py

#i run the code of the project... and stuck at the last part at the model of:

#- Now that we have a trained model, let's use it to generate some new questions and answers! First, #we predict some answers. Notice how we're only feeding the document itself to the model.

#all this line work fine:**

saver = tf.train.Saver()
session = tf.InteractiveSession()
saver.restore(session, "model-5")

batch = next(test_data())
batch = collapse_documents(batch)

answers = session.run(answer_tags, {
document_tokens: batch["document_tokens"],
document_lengths: batch["document_lengths"],
})

and the answers and batch matrix are looks like at the files that i upload.
the answers shape is:(15, 1156, 2)

#after that i run the line:
answers = np.argmax(answers, 2)
now the answers shape is (15, 1156)
#this matrix fool of zores:
[[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
...
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]
[0 0 0 ... 0 0 0]]

then i run:
batch = expand_answers(batch, answers)

and got the error:
in ()
1 print(batch)
----> 2 batch = expand_answers(batch, answers)
3
4 helper = seq2seq.GreedyEmbeddingHelper(embedding, tf.fill([batch["size"]], START_TOKEN), END_TOKEN)
5 decoder = seq2seq.BasicDecoder(decoder_cell, helper, encoder_state, output_layer=projection)
in expand_answers(batch, answers)
32 print(new_batch)
33
---> 34 return _prepare_batch(new_batch)
35
36
in _prepare_batch(batch)
27
28 batch_size = len(batch)
---> 29 max_document_len = max(len(document) for document in document_words )
30 max_answer_len = max(len(answer) for answer in answer_indices )
31 max_question_len = max(len(question) for question in question_input_words )
ValueError: max() arg is an empty sequence

**when i take a look at the function of the error :
def expand_answers(batch, answers):
new_batch = []

for i in range(batch["size"]):
split_answers = []
last = None
for j, tag in enumerate(answers[i]):
if tag:
if last != j - 1:
split_answers.append([])
split_answers[-1].append(j)
last = j

for answer_indices in split_answers:
    document_id = batch["document_ids"][i]
    document_text = batch["document_text"][i]
    document_words = batch["document_words"][i]
    answer_text = " ".join(document_words[i] for i in answer_indices)
    new_batch.append({
        "document_id": document_id,
        "document_text": document_text,
        "document_words": document_words,
        "answer_text": answer_text,
        "answer_indices": answer_indices,
        "question_text": "",
        "question_words": [],
    })

return _prepare_batch(new_batch)
##i found out that tag is alway 0 at the line:
for j, tag in enumerate(answers[i]):
because of it split_answers, and new_batch are [], are empty.

##may the answers is not as it should be and this is the problem.
hope that you can help me solve this issue,
tal
answers_matrix.txt
batch_matrix.txt

Generate Questions

How do I generate a set of questions from a piece of string i.e. STORY file?

size of entry document input

hey!
someone can explain me how many entries exist at the documnt input as i marked at the pic
(is it the number of words from 1 document?)

Implementation Next Steps

The notebook does a great job of enumerating all of the components important to QA generation but its not totally clear (or totally clear to me at least) what the next steps would be within this implementation to make a "functional" solution. For example based on the Joint Model for Question Answering and Question Generation paper we would probably need an attention mechanism, beam search, multiple encoder/decoder layers, etc. Maybe we could make a list?

attention mechanism

Hello, i'm trying to add an attention to the model but i can not figure out what is the correct batch_size i should pass to the initial state i tried this

 # Attention
  # attention_states: [batch_size, max_time, num_units]
attention_states = tf.transpose(encoder_outputs, [1, 0, 2])

  # Create an attention mechanism
attention_mechanism = tf.contrib.seq2seq.LuongAttention(
      encoder_cell.state_size, attention_states,
      memory_sequence_length=None)

decoder_cell = tf.contrib.seq2seq.AttentionWrapper(
      decoder_cell, attention_mechanism,
      attention_layer_size=encoder_cell.state_size)

batch = next(training_data())
batch = collapse_documents(batch)
batch_size = len(batch)
initial_state = decoder_cell.zero_state(batch_size, tf.float32).clone(cell_state=encoder_state)

decoder = seq2seq.BasicDecoder(decoder_cell, helper, initial_state, output_layer=projection)

**it gives me this error:

InvalidArgumentError: ConcatOp : Dimensions of inputs should match: shape[0] = [99,100] vs. shape[1] = [15,200]
	 [[Node: decoder/while/BasicDecoderStep/decoder/attention_wrapper/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](decoder/while/Identity_7, decoder/while/Identity_4, decoder/while/BasicDecoderStep/decoder/attention_wrapper/concat/axis)]]