alibaba / sumn-universal-user-representation

SUMN-universal-user-representation

Exploiting Behavioral Consistence for Universal User Representation (AAAI 2021)

By Jie Gu*, Feng Wang*, Qinghui Sun, Zhiquan Ye, Xiaoxiao Xu, Jingmin Chen, Jun Zhang

arxiv: https://arxiv.org/abs/2012.06146

Introduction

In this paper, we focus on universal (general-purpose) user representation. The obtained universal representations are expected to contain rich information, and be applicable to various downstream applications without further modifications (e.g., user preference prediction and user profiling). Accordingly, we can be free from the heavy work of training task-specific models for every downstream task as in previous works.

Usage

The project is developed based on Alibaba Cloud: PAI tensorflow, MaxCompute (a data processing platform for large-scale data warehousing) and OSS (a storage service). The related APIs and functions are: tf.python_io.TableWriter in dumper.py, tf.data.TableRecordDataset in loader.py, tf.gfile in train.py, gfile in inference.py, gfile in helper.py, and set_dist_env in env.py. You can revise these APIs and functions if needed.

Script for training the representation model

rm model.tar.gz
tar -czf model.tar.gz ./data_dumper ./data_loader ./main ./model ./util

ITERATION=1000000
SNAPSHOT=100000
TARGET_LENGTH=500
LEARNING_RATE=0.001
DROPOUT=0.1
BATCH_SIZE=128
MODEL_DIM=256
WORD_DIMENSION=256
POOL=mhop
CHECKPOINT_PATH=your_own_oss_directory/drop${DROPOUT}_lr${LEARNING_RATE}_tarlen${TARGET_LENGTH}_pool${POOL}_dim256_mlp512_b${BATCH_SIZE}

odpscmd -e "use your_own_maxcompute_project; pai \
        -name tensorflow180 -project algo_public \
        -Dscript=\"file://`pwd`/model.tar.gz\" \
        -Dtables=\"odps://your_own_maxcompute_project/tables/your_own_maxcompute_table_for_model_training\" \
        -DentryFile=\"main/train.py\" \
        -DgpuRequired=\"100\"\
        -Dbuckets=\"your_own_oss_buckets\" \
        -DuserDefinedParameters='--max_steps=${ITERATION} --snapshot=${SNAPSHOT} --checkpoint_dir=${CHECKPOINT_PATH} \
            --target_length=${TARGET_LENGTH} --learning_rate=${LEARNING_RATE} --dropout=${DROPOUT} --batch_size=${BATCH_SIZE} \
            --model_dim=${MODEL_DIM} --word_emb_dim=${WORD_DIMENSION} --pooling=${POOL} \
            --max_query_per_week=900  --max_words_per_query=24' \
        "

Script for representation inference

rm model.tar.gz
tar -czf model.tar.gz ./data_dumper ./data_loader ./main ./model ./util

INPUT_TABLE=your_own_maxcompute_table_for_inferring_inputs
OUTPUT_TABLE=your_own_maxcompute_table_for_inferring_outputs
CHECKPOINT_PATH=your_own_oss_directory/drop0.1_lr0.001_tarlen1000_poolmax_dim256_mlp512_b128/
STEP=1000000

odpscmd -e "use your_own_maxcompute_project; pai \
        -name tensorflow180 -project algo_public \
        -Dscript=\"file://`pwd`/model.tar.gz\" \
        -Dtables=\"odps://your_own_maxcompute_project/tables/${INPUT_TABLE}\" \
        -Doutputs=\"odps://your_own_maxcompute_project/tables/${OUTPUT_TABLE}/version=drop0.1_lr0.001_tarlen1000_poolmax_dim256_mlp512_b128_100w\" \
        -DentryFile=\"main/inference.py\" \
        -Dbuckets=\"your_own_oss_buckets\" \
        -DuserDefinedParameters=\"--checkpoint_dir='$CHECKPOINT_PATH' --step=$STEP \" \
        -Dcluster='{\"worker\":{\"count\":64,\"cpu\":200,\"memory\":4096,\"gpu\":50}}' \
        "

License

See LICENSE for details.

Citation

If you find this repo useful in your research, please consider citing the paper:

@inproceedings{SUMN_user_representation,
  author    = {Jie Gu and Feng Wang and Qinghui Sun and Zhiquan Ye and Xiaoxiao Xu and Jingmin Chen and Jun Zhang},
  title     = {Exploiting Behavioral Consistence for Universal User Representation},
  booktitle = {Thirty-Fifth {AAAI} Conference on Artificial Intelligence, {AAAI}
               2021},
  pages     = {4063--4071},
  year      = {2021}
}

sumn-universal-user-representation's Issues

Nice written code! Some suggestions.

@m2408gj @JieGu

First of all, nice written code!
As I read the code, I saw parts that needed some modifications or would be good to be updated, so I suggest them.

1. Could you add some documents or README like Getting started or Quick start? Since there is no example data in code, it's quite hard to start over.
2. Continuing from question 1, can you post example data? If you look at the code for training with local data, it appears to read the local_data/universal_transformer_sample.txt file. Is it possible to upload this file? or some other.
3. Seems using PAI-tensorflow, such as tf.data.TableRecordDataset function which is not supported in plain TF. Could you decouple the dependency of PAI-tensorflow and change to plain TF runnable code?
4. Some codes are missing such as LocalFileDataLoader in loader, which is calling from train_locally.py.

Thanks a lot!