andy-yangz / nlp_multi_task_learning_pytorch

A multitask learning architecture for Natural Language Processing of Pytorch implementation

nlp_multi_task_learning_pytorch

A basic multitask learning architecture for Natural Language Processing of Pytorch implementation.

The two tasks we use here is POS Tagging and Chunking.

As below, we build the model, normally several layers of RNN above embedding. Later, we train specific task on different layer according to its complexity. For example, we think Chunking is a higher task than POS Tagging, inspired by Deep multi-task learning with low level tasks supervised at lower layers.

Running Examples

You can check several running examples in run.sh. I will explain one here.

echo "Joint Training on the different level"
python main.py --data './data' \  # The directory put the training data
        --emsize 256 \	# embedding size
        --npos_layers 1 \	# number of POS tagging training layer
        --nchunk_layers 2 \	# number of chunking training layer
        --nhid 128 \	# number of hidden states for RNN
        --batch_size 128 \
        --seq_len 10 \	# sequence length
        --cuda \	# enbale GPU
        --train_mode 'Joint' \
        --epochs 300 \	
        --log_interval 20 \
        --save './result/joint_diff'

Current Done

• A basic architecture for POS tagging and chunking
• Explorate the best hyperparameters of NN

Todo

• Chunking exploration

nlp_multi_task_learning_pytorch's Issues

About MTL

Can I ask you some questions about MTL?

POS tag layer and two RNNs

I do not see why POS tag layer has two RNNs as it looks like in the diagram. Where is that in the code?

about version

hi~ I tried to run your code with pytorch(0.4.0) and python (3.5.2),but the program reported an error:
Loading corpus...
Begin training...
main.py:108: UserWarning: torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.
torch.nn.utils.clip_grad_norm(model.parameters(), args.clip)
Traceback (most recent call last):
File "main.py", line 200, in
loss_log = train(loss_log)
File "main.py", line 114, in train
cur_loss = cur_loss.cpu().numpy()[0]
IndexError: too many indices for array

I think the version is not compatible, so what's the versions of pytorch and python should be ?