How to generate augmented js file about contracode

[parasj]

Hi @Vitvicky and @kambehmw! Thanks for trying out our code.

It takes quite a bit of time to preprocess the data, so we provide pre-augmented datasets at this location https://drive.google.com/file/d/1YfPTPPOv4evldpN-n_4QBDWDWFImv7xO/view?usp=sharing, as part of this dataset folder.

However, if you want to modify some of the augmentations and add a new transform, these are an example set of commands we used to generate the augmented pre-processed dataset:

$ split -l$((`wc -l < javascript_dedupe_definitions_nonoverlap_v2_train.jsonl`/136)) javascript_dedupe_definitions_nonoverlap_v2_train.jsonl javascript_dedupe_definitions_nonoverlap_v2_train.split.jsonl -da 3
$ mkdir -p $OUTDIR
$ find . -name "*split.jsonl*" | parallel --files -I% --max-args 1 -j137 wc -l %
$ find . -name "*split.jsonl*" | parallel --files -I% --max-args 1 -j137 node node_src/transform_jsonl.js % ../augmented/%.augmented

[kambehmw]

Thank you for sharing this code. I also would like to know how to perform pre-processing and compiler transforms. I would appreciate it if you could update the README on the procedure.

[QZH-eng]

Hi Parasj, thanks for publishing your code. I want to ask what are the parameters of the equipment used in your experiment? I found that 16G memory is not enough if using the javascript_augmented.pickle.gz file.

[parasj]

Hi @QZH-eng -- thank you for trying our repository!

Pretraining is memory hungry as contrastive learning benefits from large batch sizes (see https://arxiv.org/abs/2002.05709). Moreover, the transformer backbone we leverage uses significantly more memory than typical image classification architectures.

We generally performed pretraining over 2-4 16GB V100 GPUs. We provide pretrained checkpoints due to the large cost of pretraining. Finetuning is very cheap and was performed on 1 V100 GPU.

Some recommendations to reduce memory consumption:
(1) reducing the sequence length for the Transformer encoder
(2) decreasing the hidden dimension size of our model
(3) adding checkpoint annotations for gradient checkpointing (e.g. PyTorch gradient checkpointing)

[parasj]

@QZH-eng I created a new issue #17 -- please use this for further discussion