hiroakimikami / mlprogram

PyTorch library for synthesizing programs from natural language

mlprogram

A Library of Deep Learning (Machine Learning) for Programming Tasks. It provides a toolbox for implementing and evaluating deep learning methods related to programming.

Purpose

The main purpose of this repository is making my experiments easy. Recently, many papers proposed deep learning methods for programming, such as programming by example and auto reparing. But many of them requires complex and error-prone implementations. For example, beam search decoding with a programming language grammar is complex and unique to this task. I want to create and maintain well-tested implementations of such algorithms.

Focuses

• Library for handling programming languages in deep learning tasks
• Utilities for benchmark datasets of various tasks
• Simple baseline solution for program generation

Now I do not place value on re-implementing exsiting papers. The machine learning for programming field is still immature. There are no de-fact benchmark tasks in this field (such as image classification w/ ImageNet and object detection w/ COCO in the image). Also, there are no de-fact model (such as ResNet in the image).

Feature Lists and Plans

• Benchmark dataset
  • Auto Reparing
    • DeepFix: the official repository
  • Program Synthesis from Natural Language
    • Hearthstone: Latent Predictor Networks for Code Generation
    • Django: Learning to Generate Pseudo-code from Source Code Using Statistical Machine Translation, ACE2-15
    • NL2Bash: nl2bash
    • (TODO) Spider: Spider 1.0 Yale Semantic Parsing and Text-to-SQL Challenge
  • Programming by Examples
    • 2D CSG
    • (TODO) DeepCoder
    • (TODO) ShapeNet
• Deep Learning Models
  • Attention Based LSTM
  • AST LSTM (based on NL2Code)
• ProgramSynthesis Methods
  • supervised training
  • reinforcement learning for programming by example
  • (TODO) interpreter arppoximated by DNN
• Other Papers
  • NL2Code: the official repository
  • TreeGen: the official repository
  • PbE with REPL: the official repository

Benchmark

NL2Prog (Hearthstone)

Method	#params [MiB]	training time [min]	max time per example [sec]	BLEU@top1	config name
tree LSTM	7.7	92	15	0.75020	hearthstone/baseline_evaluate_short
tree LSTM	7.7	92	180	0.76540	hearthstone/baseline_evaluate_long

Programming by Example without Inputs (CSG)

Method	#params [MiB]	training time [min]	max time per example [sec]	generation rate	config file
tree LSTM	16	75	30	18/30	csg/baseline_evaluate_short
tree LSTM	16	75	360	22/30	csg/baseline_evaluate_long
tree LSTM + REINFORCESynthesizer	16	75	30	18/30	csg/baseline_evaluate_rl_synthesizer_short
tree LSTM + REINFORCESynthesizer	16	75	360	22/30	csg/baseline_evaluate_rl_synthesizer_short

Auto Repair

TODO

Usage Examples

tools/launch.py is the launcher script and configs directory contains the examples.

Train/Evaluate NL2Code with Hearthstone Dataset

It requires CUDA enabled GPU.

$ python tools/launch.py --config configs/nl2code/nl2code_train.py
$ python tools/launch.py --config configs/nl2code/nl2code_evaluate.py

Warning

• The implementation is highly experimental, and I may change it significantly.
• The reproduced algorithms may be different from the authors' implementations. For example, the original implementation of NL2Code uses the grammar of Python 2.7.x while this repository uses the grammar of running Python version.

mlprogram's Issues

Reduce the vocab size of ActionSequenceEncoder

Some vocabs in the ActionSequenceEncoder are not used.
Special rules such as None -> Root and node types such as Root and None should be removed from the encoder.

Cache downloaded dataset

Make `ActionSequence.generate` function class

Add Parser base class

Use mypy to check type

Setup CI (GitHub Actions)

Add test case for NL2Bash config

• Add download script to CI

Rename nl2prog -> ml-program

Replace TopKModel with a pytorch-pfn-extras extension

Protocol does not exist in Python 3.7

Rename tokenize_query to extract_query

Add device arg to CommonBeamSearchSynthesizer

Add documentation

SampleFix: Learning to Correct Programs by Sampling Diverse Fixes

• Write an initial implementation
• Get the result similar to the paper

Add type hints to all elements

Reorganize modules

• Put all language-specific code into a languages/<lang> module
• Move utils/logging => logging
• Move utils/random => random
• Move random_split into utils/data
• Merge top_k_model into SaveTopKModel
• Move top_k_element into collections
• Move asts to languages/

Reorganize module

• entrypoint.parse => entrypoint.configs
• entrypoint.torch etc => entrypoint.types.torch
• ast_reference_sampler => SequentialProgramSampler

Reorganize utils

• Remove save/load
• Move Compose, Sequence, Map to functools
• Move other functions to toplevel

Predictor might have to output log_prob instead of prob.

Add custom_types when launching

• Add types when launching

DeepCoder: Learning to Write Programs

• Copy code from my another repository and replace chainer with pytorch
• Reproduce the result similar to the paper

Add types to GenerateToken

This change will remove get_token_type argument.

Naming convention

• AST or Ast
• n_xxx or n_xxxs or num_xxx or xxx_num, etc...
• type or kind
• reference or query
• value or token in ActionSequenceEncoder
• IXxx for interface/abstract classes

Add TreeGen

nn.Pick and utils.Pick should be merged

Refactor functions by using function compoition

Some functions (e.g., parse/unparse, code_to_action_sequence) is function composition.
I can simplify the implementation and configs with configurable function composition.

Enable multi-process evaluation

The evaluation will be CPU-intensive in many tasks handled by mlprogram.
Instead of using a GPU, using multiple CPUs for evaluation might be a good idea.

Rename C's Tokenizer to Lexer

Add Write, Execute, Assess: Program Synthesis with a REPL

• Write initial implementation
• Make RL training stable
• Reproduce the result similar to the paper

Add utilities for C/C++

• Tokenize string
• Try to compile

Add Deepfix dataset

refs #40

Create CommonBeamSearchSynthesizer from TrainModel

Add input to Interpreter

Currently, interpreters receive no input. But they should receive inputs before eval.

Merge parse and unparse into single class

Stop early if score exceeds threshold

TreeGen: A Tree-Based Transformer Architecture for Code Generation

• Write an initial implementation
• Get the result similar to the paper
• Make training stable (the loss becomes NaN after 40 epochs with Hearthstone dataset)

Make abort_if_loss_is_nan an extension

Fix typo in ActionOptions

Erase __list suffix

Remove Any from type hints

Dump JSON log after evaluation

Fix CommonBeamSearchSynthesizer.create (pass device to constructor)

Use PFRL in train_REINFORCE

I should use a reliable library for reinforcement learning because RL is a complex and error-prone method.
Now I plan to use PFRL.

Evaluate with (reduced) dataset during training

Refactoring

• Enable to use Colab
• Make nl2code module
• Fix API signature

Progress bar in nl2code_train

Hi, Is there any progress bar when running the python tools/launch.py -c configs/hearthstone/nl2code_train.gin ?
I can't tell whether it's running or the code hangs in my environment.

Reorganize testcases

load_state_dict and state_dict for BeamSearchSynthesizer

Monitor performance metrics during training

• Time required to create a minibatch
• Time required for forward/backward/optimizer.step
• GPU utilization rate

Use gin-config to launch training, evaluation, etc.

• Add external_configurables
• Add tools/launch.py
• Rename and change examples to configs