bnabis93 / vision-language-examples

Vision-lanugage model example code.

vision-language-examples

Vision-lanugage model example code.

Transformer

• 00-vit-huggingface
• 01-vit-torch-train
• 02-vit-export

Acceleration

• 00-vit-baseline
• 01-vit-tensorrt
• 02-vit-faster-transformer
• 03-vit-xformers

Embedding

• 00-openai-text-embedding
• 01-openai-text-embedding-fine-tuning
• 02-image-embedding-models
• 03-clip-similarity

Segmentation

• 01-sam-transformer-fine-tuning
• 02-sam-point-fine-tuning

Diffusion

• 01-diffusers-quick-start
• 02-hack-diffusers-pipeline

vision-language-examples's Issues

xFormers-ViT performance degradation in A100 GPU

Performance degradation in A100 GPU

• Vanilla Attention: 3.87ms
• Sparse Attention: 9.33ms
• Memory Efficient Attention: 6.34ms
• Sparse Attention is 2.4x slower than Vanilla Attention
• Memory Efficient Attention is 1.6x slower than Vanilla Attention

python reproduce.py

ViT Forward only
<torch.utils.benchmark.utils.common.Measurement object at 0x7fe7d59c9780>
profile
  Median: 3.87 ms
  IQR:    0.26 ms (3.76 to 4.02)
  514 measurements, 1 runs per measurement, 1 thread
Memory used: 28.87548828125 MB
Sparse ViT Forward only
<torch.utils.benchmark.utils.common.Measurement object at 0x7fe803ea64a0>
profile
  Median: 9.33 ms
  IQR:    0.21 ms (9.31 to 9.51)
  22 measurements, 10 runs per measurement, 1 thread
Memory used: 32.49267578125 MB
Mem efficient ViT Forward only
<torch.utils.benchmark.utils.common.Measurement object at 0x7fe803de7c10>
profile
  Median: 6.34 ms
  IQR:    0.08 ms (6.31 to 6.39)
  315 measurements, 1 runs per measurement, 1 thread
Memory used: 266.49072265625 MB
ViT average inference time : 3.186824321746826ms
ViT average inference time : 8.32324504852295ms
ViT average inference time : 5.165774822235107ms