epochx / commitgen

Code and data for the paper "A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes"

A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes

• Requirements

  • Torch, Cutorch (http://torch.ch/docs/getting-started.html)
  • Python packages unidiff, pygments: pip install unidiff pygments

• Setup environment

  1. Clone this repositoty: cd ~ git clone https://github.com/epochx/commitgen-dev.git
  2. Create data path: mkdir ~/data/preprocessing
  3. Export env variable: export env WORK_DIR=~/data (without trailing slash!)

• Download our paper data:

  1. Get the raw commit data used in our paper from https://osf.io/67kyc/?view_only=ad588fe5d1a14dd795553fb4951b5bf9 (click on "OSF Storage" and then on "Download as zip".) Unzip the file where convenient.
  2. Unzip the desired dataset zip and move the resulting folder to ~/data.

• Pre-process data

  1. Parse and filter commits and messages: cd ~/commitgen python ./preprocess.py FOLDER_NAME --language LANGUAGE, where FOLDER_NAME is the name of the folder from the previous step. Add the '--atomic' flag to keep only atomic commits. This will generate a pre-processed version of the dataset in a pickle file in ~/data/preprocessing. Try python ./preprocess.py --help for more details on additional pre-processing parameters.
  2. Generate training data: cd ~/commitgen ./buildData.sh PICKLE_FILE_NAME LANGUAGE (PICKLE_FILE_NAME with no .pickle).

• Train the model 1.- Run the model cd ~/commitgen ./run.sh PICKLE_FILE_NAME LANGUAGE (PICKLE_FILE_NAME with no .pickle)

You can also dowload additional github project data by using our crawler do cd ~/commitgen and run python crawl_commits.py --help for more details on how to do it.

commitgen's Issues

CUDA must be assembled in this program?

CUDA must be assembled in this program? with GPU?

dataset `keras` and some questions about preprocessing

Thanks for your wonderful work!

May I ask for the raw data which named keras which you mentioned in your paper.
When I try to preprocess the data, I find it is missing.
I had already downloaded other data from https://osf.io/67kyc/?view_only=ad588fe5d1a14dd795553fb4951b5bf9

By the way, when I try to preprocess (./preprocess.py FOLDER_NAME --language LANGUAGE), there are a lot of warnings.
For example, when I run python2 preprocess.py Theano_commits --language python,there are a lot of warnings like .../commitgen/commitgen/code.py:209: UserWarning: 'ascii' codec can't encode character u'\u2014' in position 61: ordinal not in range(128) warnings.warn(str(e))
Meantime, the number of commits parsed is less than the extracted.
Do they influenced the final results?
Could you tell me about your developing environment?

Looking forward to your reply!

Thanks a lot!

How are the .diff .json files generated from the repos?

Are you using any script?
It seems that I can use the line below to generate .diff file

git diff file1 file2 > {sha}.diff

How about .json? Are you using git log --pretty?

git log --pretty=format:{...}