hemingkx / spec-bench

Spec-Bench: A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding

Spec-Bench: A Comprehensive Benchmark and Unified
Evaluation Platform for Speculative Decoding

| Paper | Blog | Leaderboard | Roadmap |

Speedup comparison of Speculative Decoding methods on Spec-Bench, evaluated by Vicuna-7B-v1.3.

Introduction

Spec-Bench is a comprehensive benchmark designed for assessing Speculative Decoding methods across diverse scenarios. Based on Spec-Bench, we aim to establish and maintain a unified evaluation platform for open-source Speculative Decoding approaches. This platform facilitates the systematic assessment of existing methods in the same device and testing environment, thereby ensuring fair comparisons.

Currently, Spec-Bench supports the evaluation of the following open source models:

• EAGLE
• Hydra
• Medusa
• Speculative Sampling
• Prompt Lookup Decoding
• REST
• Lookahead Decoding

Update

2024.3.12: We now support statistics for #Mean accepted tokens.

2024.3.11: We have integrated Hydra into Spec-Bench, check it out!

Installation

conda create -n specbench python=3.9
conda activate specbench
cd Spec-Bench
pip install -r requirements.txt

Model Weights

Download corresponding model weights (if required) and modify the checkpoint path in eval.sh.

• vicuna-v1.3
• EAGLE
• Hydra
• Medusa-1
• Speculative Sampling

Additonal Setup

REST (Optional)

Build DraftRetriever from source

cd model/rest/DraftRetriever
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
maturin build --release --strip -i python3.9 # will produce a .whl file
pip3 install ./target/wheels/draftretriever-0.1.0-cp39-cp39-linux_x86_64.whl

Create a datastore

cd model/rest/datastore
./datastore.sh # modify your own path

Inference

Select specific command line in eval.sh, the results will be stored in data/spec_bench/model_answer/.

./eval.sh

Speedup Report

Obtain the corresponding speedup compared to vanilla autoregressive decoding.

python evaluation/speed.py --file-path /your_own_path/eagle.jsonl --base-path /your_own_path/vicuna.jsonl

Result Comparison

Examine whether the generated results are equal to autoregressive decoding or not.

python evaluation/equal.py --file-path /your_own_path/model_answer/ --jsonfile1 vicuna.jsonl --jsonfile2 eagle.jsonl

Contributing

We warmly welcome contributions and discussions related to Spec-Bench! If you have any suggestions for improvements or ideas you'd like to discuss, please don't hesitate to open an issue. This will allow us to collaborate and discuss your ideas in detail.

More models are welcome! - If you're aware of any open-source Speculative Decoding methods not currently included in Spec-Bench, we encourage you to contribute by submitting a pull request. This helps ensure Spec-Bench remains a comprehensive and fair benchmarking platform for comparing existing methods. Please ensure that your changes are well-tested before submission.

Acknowledgments

This codebase is built from Medusa and EAGLE. We integrated code implementations of multiple open-source Speculative Decoding methods to facilitate unified evaluation.

Citation

If you find the resources in this repository useful, please cite our paper:

@misc{xia2024unlocking,
      title={Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding}, 
      author={Heming Xia and Zhe Yang and Qingxiu Dong and Peiyi Wang and Yongqi Li and Tao Ge and Tianyu Liu and Wenjie Li and Zhifang Sui},
      year={2024},
      eprint={2401.07851},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

spec-bench's Issues

REST methodology verification process

I noticed that in Table 3 you present a summary of different decoding methodologies.

In particular, you present REST as a methodology amenable to Nucleus sampling, but on reading the REST methodology paper, it's not clear how they establish that their methodology maintains the same output distribution from a LLM.

How to think a comparison is fair

Thanks for your work, I would like to ask, how do you think the comparison results shown by spec-bench are fair? For example, REST can control the size of the datastore that needs to be maintained; lookahead needs to control the length of N-grams and the size of the pool; how do you think the results provided by spec-bench are fair? I'm not quite sure, it would be greatly appreciated if you could provide further explanation.

A100的自回归tokens/s仅为40.24是否太慢了？

我们在4090和L40上进行了测试，自回归的tokens/s均达到了50以上，感觉不太合理。是否可能是cuda的版本不同造成的呢？

Add Hydra

Maybe consider adding Hydra to these? It's a very similar approach to EAGLE, so I'd be curious to see what the performance difference is.

PaSS methodology.

Thank you so much for your work in establishing this benchmark!

I wanted to ask if you would include PaSS (https://arxiv.org/pdf/2311.13581.pdf) to your benchmark analysis, since it is included in your references, and it allows nucleus sampling, greedy sampling, but it doe require the training of extra token embeddings.