Comments (1)
Hi Amit,
The README is up. Some of your questions may be addressed by the README, but a few would require extensions beyond what is implemented here. Hope the below helps:
- Patch size is set by the arg
--pixel-per-patch
- Scripts to train with the multilingual
pixel_translation_multi_simple_epoch
task can be found in thegrid_scripts
directory; for a single language pair without temperature sampling across languages, thevisual-text
task may be sufficient, scripts for which are shown ingrid_scripts
on the main branch - It is not implemented to read in image data from disk, only either raw text or binarized rendered images in the fairseq format, or to tokenize images with multiple rows and columns of patches. Here are where you'd minimally need to make modifications to do so:
- If you'd like to read in image files, you'd want to modify the
VisualTextDataset
to read in your images in place of text and tokenize them into format expected by the downstream tasks withimage_generator
. One option would be to have the source text files contain the image file paths per line aligned to the target text- The tensor functions in
image_generator
can be modified to appropriately tokenize your images both horizontally and vertically. When training with raw data, when constructing a batch, the code callsget_tensors()
to generate a sequence of tensors representing the tokenized image for a given text sequence. This and the 3 fns it calls (get_tensor
,get_image
,slice
) are what would be necessary to modify.
- The tensor functions in
- If the above have been modified to yield the same output format as ours, the data could be used directly for training or binarized the same way as our synthetic text images potentially without further modification
- You'd likely also want to modify the positional embeddings to be 2D
- If you'd like to read in image files, you'd want to modify the
from visrep.
Related Issues (4)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from visrep.