microsoft / moonlit

This is a collection of our research on efficient AI, covering hardware-aware NAS and model compression.

Moonlit: Research for enhancing AI models' efficiency and performance.

Moonlit is a collection of our model compression work for efficient AI.

ToP (@KDD'23): Constraint-aware and Ranking-distilled Token Pruning for Efficient Transformer Inference

ToP is a constraint-aware and ranking-distilled token pruning method, which selectively removes unnecessary tokens as input sequence pass through layers, allowing the model to improve online inference speed while preserving accuracy.

SpaceEvo (@ICCV'23): SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inference

SpaceEvo is an automatic method for designing a dedicated, quantization-friendly search space for target hardware. This work is featured on Microsoft Research blog: Efficient and hardware-friendly neural architecture search with SpaceEvo

ElasticViT (@ICCV'23): ElasticViT: Conflict-aware Supernet Training for Deploying Fast Vision Transformer on Diverse Mobile Devices

ElasticViT is a two-stage NAS approach that trains a high-quality ViT supernet over a very large search space for covering a wide range of mobile devices, and then searches an optimal sub-network (subnet) for direct deployment.

LitePred (@NSDI'24): LitePred: Transferable and Scalable Latency Prediction for Hardware-Aware Neural Architecture Search

LitePred is a lightweight transferrable approach for accurately predicting DNN inference latency. Instead of training a latency predictor from scratch, LitePred is the first to transfer pre-existing latency predictors and achieve accurate prediction on new edge platforms with a profiling cost of less than 1 hour.

moonlit's Issues

Support for Compresso pruned weights removal

currently after merging pruning masks and LoRA weights, LLaMA-7B size is increasing from 15GB to 26GB. Please provide support to remove pruned weights from the model

Is the version of Transformer in the requirement the correct one?

My error message says：
File "/home/Moonlit/Compresso/train.py", line 235, in
main()
File "/home/Moonlit/Compresso/train.py", line 204, in main
from transformers.integrations import AzureMLCallback, ProgressCallback
ImportError: cannot import name 'ProgressCallback' from 'transformers.integrations' (/home/longyongliu/.local/lib/python3.10/site-packages/transformers/integrations/init.py)
It looks like there's a problem with the version of Transformer, but I'm building the environment as described in the requirement file.

ElasticViT: offline_models_dir is not found

Hi, thank you for your nice work of ElasticViT.
I'm trying to run the codes according to the instruction information, and I met this issue when I run the training command:
$ python -m torch.distributed.launch --nproc_per_node=2 train_eval_supernet.py configs/final_3min_space.yaml

The error occurs due to the lack of "offline_model_dir" and " lib_data_dir" in the yaml file "final_3min_space.yaml", so that the "select_min_arch" method of "class FuseSuperNet" cannot work. I was wondering if it is necessary to use the "offline_model_dir" from you, or should we edit the codes somewhere else? Thanks.

File "train_eval_supernet.py", line 261, in
main()
File "train_eval_supernet.py", line 226, in main
current_bank_id, direction, train_loss = train_one_epoch(
File "/home/jiy1rng/forclone/Moonlit/ElasticViT/process.py", line 205, in train_one_epoch
arch = model_without_ddp.arch_sampling(mode=mode, random_subnet_idx=uniform_idx,
File "/home/jiy1rng/forclone/Moonlit/ElasticViT/models/model.py", line 852, in arch_sampling
arch, _ = self.sample_random_subnet_from_range(
File "/home/jiy1rng/forclone/Moonlit/ElasticViT/models/model.py", line 758, in sample_random_subnet_from_range
min_arch = self.select_min_arch(flops=min_flops)
File "/home/jiy1rng/forclone/Moonlit/ElasticViT/models/model.py", line 735, in select_min_arch
min_archs = self.offline_archs['min']
KeyError: 'min'

compresso performance on wikitext2, ptb and c4 datasets

Hi,

I have read the paper of compresso and it is a great job.

I would like to know if there is any result for the datasets like wikitext2, ptb and c4?

Thanks

Action required: migrate or opt-out of migration to GitHub inside Microsoft

Migrate non-Open Source or non-External Collaboration repositories to GitHub inside Microsoft

In order to protect and secure Microsoft, private or internal repositories in GitHub for Open Source which are not related to open source projects or require collaboration with 3rd parties (customer, partners, etc.) must be migrated to GitHub inside Microsoft a.k.a GitHub Enterprise Cloud with Enterprise Managed User (GHEC EMU).

Action

✍️ Please RSVP to opt-in or opt-out of the migration to GitHub inside Microsoft.

❗Only users with admin permission in the repository are allowed to respond. Failure to provide a response will result to your repository getting automatically archived.🔒

Instructions

Reply with a comment on this issue containing one of the following optin or optout command options below.

✅ Opt-in to migrate

@gimsvc optin --date <target_migration_date in mm-dd-yyyy format>

Example: @gimsvc optin --date 03-15-2023

OR

❌ Opt-out of migration

@gimsvc optout --reason <staging|collaboration|delete|other>

Example: @gimsvc optout --reason staging

Options:

• staging : This repository will ship as Open Source or go public
• collaboration : Used for external or 3rd party collaboration with customers, partners, suppliers, etc.
• delete : This repository will be deleted because it is no longer needed.
• other : Other reasons not specified

Need more help? 🖐️

• Email [email protected]. ✉️
• Post your questions in GitHub inside Microsoft Team in Microsoft Teams. 🗨️