sudo apt-get update
sudo apt-get install bzip2 git libxml2-dev
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
rm Miniconda3-latest-Linux-x86_64.sh
source .bashrc
conda create --name <name_of_environment> python==3.7
conda activate <name_of_environment>
conda install pytorch=1.1.0 cudatoolkit=9.0 -c pytorch
pip install torchtext==0.4
pip install torchvision==0.3.0
pip install scikit-learn pandas jupyter ipython
pip install more-itertools
- Add your data folder to /data-to-text/data
- Make changes to
make-dataset.py
to read your data - Be sure to change
ENT_SIZE
- Max
ENT_SIZE
tested is 244. You can try higher but I can't say where exactly it'll break. - Change the ENT_SIZE in
onmt/__init.py__
python data/make-dataset.py --folder data/ --dataset <your data folder name here>
python create-experiment.py --name exp-1
python preprocess.py --config preprocess.cfg
python train.py --config train.cfg
(if you want to train for more than 30000 steps, modify lines 79 and 80 in train.cfg)
(for multiple GPUs modify line 64 in train.cfg)
python average-checkpoints.py --folder exp-1 --steps 26000 27000 28000 29000 30000
python translate.py --config translate.cfg
You can get the BLUE score by running:
cat experiments/exp-1/gens/test/predictions.txt | sacrebleu --force data/test_output.txt