kirillhiddleston / virtex

Code for the paper "VirTex: Learning Visual Representations from Textual Annotations"

VirTex: Learning Visual Representations from Textual Annotations

Karan Desai and Justin Johnson
University of Michigan

---

Preprint: arxiv.org/abs/2006.06666

Model Zoo, Usage Instructions and API docs: kdexd.github.io/virtex

VirTex is a pretraining approach which uses semantically dense captions to learn visual representations. We train CNN + Transformers from scratch on COCO Captions, and transfer the CNN to downstream vision tasks including image classification, object detection, and instance segmentation. VirTex matches or outperforms models which use ImageNet for pretraining -- both supervised or unsupervised -- despite using up to 10x fewer images.

Get the pretrained ResNet-50 visual backbone from our best performing VirTex model in one line without any installation!

import torch

# That's it, this one line only requires PyTorch.
model = torch.hub.load("kdexd/virtex", "resnet50", pretrained=True)

Usage Instructions

1. How to setup this codebase?
2. VirTex Model Zoo
3. How to train your VirTex model?
4. How to evaluate on downstream tasks?

These can also be accessed from kdexd.github.io/virtex.

Citation

If you find this code useful, please consider citing:

@article{desai2020virtex,
    title={VirTex: Learning Visual Representations from Textual Annotations},
    author={Karan Desai and Justin Johnson},
    journal={arXiv preprint arXiv:2006.06666},
    year={2020}
}

Build a Docker Container(Dockerfile)

FROM python:3.6-stretch
MAINTAINER Kozlov <[email protected]>

# устанавливаем параметры сборки
RUN apt-get update && \
	apt-get install -y gcc make apt-transport-https ca-certificates build-essential

# проверяем окружение python
RUN python3 --version
RUN pip3 --version

# задаем рабочую директорию для контейнера
WORKDIR  /usr/src/<virtex-d>

# устанавливаем зависимости python
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# копируем все файлы из корня проекта в рабочую директорию
COPY src/ /src/
RUN ls -la /src/*

# запускаем приложение Python
CMD ["python3", "/src/setup.py"]

структура файлов должна выглядеть следующим образом

virtex-d
      |-- src
          |-- main.py
          |-- other_module.py
      |-- requirements.txt
      |-- Dockerfile

Запускаем команду docker build . если хотите собрать в текущий проект или можно задать имя docker build NAME .

Проверям доке командой docker images и после запускаем docker run