louishenrifranc / attention

Attention is All You Need in Sonnet

Implementation of Attention is All You Need in Sonnet/Tensorflow

Architecture:

Paper: https://arxiv.org/abs/1706.03762

Usage

1. Install requirements pip install -r requirements.txt
2. Install sonnet
3. Run run_micro_services.sh

Organisation of the repository

Transformer's architecture is composed of blocks. The uses of Sonnet makes the implementation very modular, and reusable. I tried to keep the blocks as much decouple as possible, following the paper:

• attention/algorithms/transformers: Create an tf.contrib.learn.Experiment, and the tf.contrib.data.Dataset

• attention/modules/cores: Implementation of the core blocks of Transformer such as MultiHeadAttention, PointWiseFeedForward

• attention/modules/decoders: Implementation of a Decoder block, and a Decoder

• attention/modules/encoders: Implementation of an Encoder block, and an Encoder.

• attention/models: Implementation of a full Transformer Block. This Module is responsible to create the Encoder and the Decoder

• attention/services: Micro Services that create the dataset, or train the model

• attention/utils: Some classes uses as utility (recursive namespace, mocking object)

• attention/*/tests/: Test of the Module/Algorithm/MicroService

Training Task implemented

• Copy inputs
• Dialogue generation

Road Map

• Code modules
• Test modules
• Construct input function
• Build Estimator
• Run estimator
• Plug into a workflow
• Add validation queue
• Iterate over model improvements

attention's Issues

What's the LICENSE for this repository?

Could you consider releasing the code under open source licenses such as MIT? Thanks.

ModuleNotFoundError: No module named 'attention'

sudo apt install python-pip
pip install tensorflow
pip install dm-sonnet
pip install tqdm  
pip install numpy
git clone https://github.com/louishenrifranc/attention.git
cd attention
./run_micro_services.sh

Gives:

$ ./run_micro_services.sh 
Experiment will be run without saving current version of the code/config
Running Task Data Generation
Traceback (most recent call last):
  File "attention/services/create_copy_task/create_copy_task.py", line 1, in <module>
    from tqdm import trange
ModuleNotFoundError: No module named 'tqdm'
Run Task Data Generation
Run Attention Traininig
Traceback (most recent call last):
  File "attention/services/attention_train/attentiontrain.py", line 6, in <module>
    from attention.utils.config import AttrDict, RunConfig
ModuleNotFoundError: No module named 'attention'

So tried:

 sudo apt install python3-tqdm
 sudo apt install python3-numpy

But then get:

$ ./run_micro_services.sh 
Experiment will be run without saving current version of the code/config
Running Task Data Generation
Traceback (most recent call last):
  File "attention/services/create_copy_task/create_copy_task.py", line 9, in <module>
    from attention.utils.config import AttrDict
ModuleNotFoundError: No module named 'attention'
Run Task Data Generation
Run Attention Traininig
Traceback (most recent call last):
  File "attention/services/attention_train/attentiontrain.py", line 6, in <module>
    from attention.utils.config import AttrDict, RunConfig
ModuleNotFoundError: No module named 'attention'

OS: Ubuntu Bionic.
Any ideas?

Error when running

Experiment will be run without saving current version of the code/config
Running Task Data Generation
100%|######################################################################################| 200000/200000 [00:22<00:00, 8875.19it/s]
100%|########################################################################################| 30000/30000 [00:03<00:00, 8611.42it/s]
Run Task Data Generation
Run Attention Traininig
INFO:tensorflow:Using config: {'_model_dir': 'transformer_output/model', '_tf_random_seed': None, '_save_summary_steps': 500, '_save_checkpoints_steps': 500, '_save_checkpoints_secs': None, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7facaa92ce48>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, 'environment': None}
Traceback (most recent call last):
  File "attention/services/attention_train/attentiontrain.py", line 83, in <module>
    TrainAttention(**args).main()
  File "attention/services/attention_train/attentiontrain.py", line 56, in main
    self.train_and_evaluate(model=model)
  File "attention/services/attention_train/attentiontrain.py", line 71, in train_and_evaluate
    validation_answer_filename=os.path.join(self.datasets.valid_data_dir, "answer.txt"))
  File "/attention/attention/algorithms/transformer/transformer.py", line 68, in train_and_evaluate
    max_sequence_len=train_params["max_sequence_len"])
KeyError: 'max_sequence_len'