xqiprogramming / lumina-t2x

Lumina-T2X is a unified framework for Text to Any Modality Generation

$\textbf{Lumina-T2X}$: Transform Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers

   

[🤖️ Lumina-T2I Checkpoints]

📰 News

• [2024-04-29] 🔥🔥🔥 We released the 5B model checkpoint built upon it for text-to-image generation.
• [2024-04-25] 🔥🔥🔥 Support 720P video generation with arbitrary aspect ratio. Examples 🚀🚀🚀
• [2024-04-19] 🔥🔥🔥 Demo examples released.
• [2024-04-05] 😆😆😆 Code released for Lumina-T2I.
• [2024-04-01] 🚀🚀🚀 We release the initial version of Lumina-T2I for text-to-image generation.

🚀 Quick Start

For training and inference, please refer to Lumina-T2I README.md

📑 Open-source Plan

• Lumina-T2I (Training, Inference, Checkpoints)
• Lumina-T2V
• Training Code
• Web Demo
• Cli Demo

📜 Index of Content

• Lumina-T2X
  • 📰 News
  • 🚀 Quick Start
  • 📑 Open-source Plan
  • 📜 Index of Content
  • Introduction
  • 📽️ Demo Examples
    • Text-to-Image Generation
    • Text-to-Video Generation
    • Text-to-3D Generation
    • More examples
  • ⚙️ Diverse Configurations

Introduction

We introduce the $\textbf{Lumina-T2X}$ family, a series of text-conditioned Diffusion Transformers (DiT) capable of transforming textual descriptions into vivid images, dynamic videos, detailed multi-view 3D images, and synthesized speech. At the core of Lumina-T2X lies the Flow-based Large Diffusion Transformer (Flag-DiT)—a robust engine that supports up to 7 billion parameters and extends sequence lengths to 128,000 tokens. Drawing inspiration from Sora, Lumina-T2X integrates images, videos, multi-views of 3D objects, and speech spectrograms within a spatial-temporal latent token space, and can generate outputs at any resolution, aspect ratio, and duration.

🌟 Features:

• Flow-based Large Diffusion Transformer (Flag-DiT): Lumina-T2X is trained with the flow matching objective and is equipped with many techniques, such as RoPE, RMSNorm, and KQ-norm, demonstrating faster training convergence, stable training dynamics, and a simplified pipeline.
• Any Modalities, Aspect, and Duration within one framework:
  1. $\textbf{Lumina-T2X}$ can encode any modality, including mages, videos, multi-views of 3D objects, and spectrograms into a unified 1-D token sequence at any resolution, aspect ratio, and temporal duration.
  2. By introducing the nextline and nextframe tokens, our model can support resolution extrapolation, i.e., generating images/videos with out-of-domain resolutions not encountered during training.
• Low Training Resources: Despite increasing token length, which generally extends training time, our Large-DiT reduces the total number of training iterations needed, thus minimizing overall training time and computational resources. Moreover, by employing meticulously curated text-image and text-video pairs featuring high aesthetic quality frames and detailed captions, our $\textbf{Lumina-T2X}$ model is learned to generate high-resolution images and coherent videos with minimal computational demands. Remarkably, the default Lumina-T2I configuration, equipped with a 5B Flag-DiT and a 7B LLaMA as the text encoder, requires only 20% of the computational resources needed by Pixelart-$\alpha$.

📽️ Demo Examples

Text-to-Image Generation

Text-to-Video Generation

720P Videos:

Prompt: The majestic beauty of a waterfall cascading down a cliff into a serene lake.

video_720p_1.mp4

video_720p_2.mp4

Prompt: A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage. She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick. She walks confidently and casually. The street is damp and reflective, creating a mirror effect of the colorful lights. Many pedestrians walk about.

video_tokyo_woman.mp4

360P Videos:

video_360p.mp4

Text-to-3D Generation

multi_view.mp4

More examples

For more demos visit this website

⚙️ Diverse Configurations

We support diverse configurations, including text encoders, DiTs of different parameter sizes, inference methods, and VAE encoders. Additionally, we offer features such as 1D-RoPE, image enhancement, and more.