gyunggyung / mllmarxivtalk

[Google Meet] MLLM Arxiv Casual Talk

MLLMArxivTalk

최신 MLLM 관련 스터디. 기본 오후에 진행. 논문, 강의, 코드, 뉴스, 블로그 등 다양한 자료로 학습.

MLLM, LLM, NLG, Dialogue, Reinforcement learning, Distillation, Efficient, Sentence similarity, multiple tasks, multimodal, Stable diffusion, TTS, Text-To-Video, All-To-All, 우주, 생명, 지능, 윤리, 규제, 법, 노화, 의학, 투자, 개발, 인프라, 디자인, 경영, ETC...

유망 스타트업 C레벨, 국내외 탑티어 연구자, 국내외 탑티어 대학, 대학원 재학생과 졸업생, 석학, 교수 등 A급 인재들이 최신 논문, 강의 등 스터디 및 프로젝트 진행.

기본 매주 수요일 오후 7시반. 사전 학습 없이 논문 읽기 최대 20분, 토론 최대 40분. 한 번에 1 ~ 10개 논문, 강의 등 진행. 지금까지는 항상 3개. 주제 논문 선정은 자유. 탑티어 학회 논문 및 프로젝트 제작 예정.

주말을 포함하여, 거의 매일 추가 스터디 존재. 흥미로운 주제거나 참여 되는 날만 중간에 들어와서 중간에 나가도 무관. 모든 규칙은 협의 가능. 오프라인 모임도 예정. 자율 참여.

진행 사항 + 예정

2023-02-16 23:30 ~ 24:45 염기웅, 강수진, 고현웅

• GPT Understands, Too
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• Do Prompt-Based Models Really Understand the Meaning of their Prompts?

2023-02-19 23:30 ~ 24:30 염기웅, 박상준, 강수진

• ∞-former: Infinite Memory Transformer
• Improving language models by retrieving from trillions of tokens
• Augmented Language Models: a Survey

2023-02-22 19:30 ~ 21:00 염기웅, 박상준, 이웅기, 이현제

• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Structure and Content-Guided Video Synthesis with Diffusion Models
• MusicLM: Generating Music From Text

2023-02-23 23:00 ~ 24:00 염기웅, 박상준, 황명하

• InstructGPT : Training language models to follow instructions with human feedback
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

2023-02-24 17:00 ~ 19:00 염기웅

• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Constitutional AI: Harmlessness from AI Feedback
• Provable Copyright Protection for Generative Models
• What learning algorithm is in-context learning? Investigations with linear models
• A Path Towards Autonomous Machine Intelligence
• PAL: Program-aided Language Models
• Toolformer: Language Models Can Teach Themselves to Use Tools

2023-03-01 20:30 ~ 21:40 염기웅, 이대환

• LLaMA: Open and Efficient Foundation Language Models
• Improving alignment of dialogue agents via targeted human judgements
• Training Compute-Optimal Large Language Models

2023-03-04 22:00 ~ 23:30 염기웅, 황명하

• LLaMA-based ChatGPT training, ChatLLaMA
• RLHF: Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• BaGuaLu: Targeting Brain Scale Pretrained Models with over 37 Million Cores

2023-03-05 21:00 ~ 21:30 염기웅,

• LLaMA-7B, LLAMA Up-data, LLaMA: INT8 edition, UForm

2023-03-08 00:00 ~ 01:00 염기웅, 김미담

• Language Is Not All You Need: Aligning Perception with Language Models
• Flamingo: a Visual Language Model for Few-Shot Learning, Blog
• Multimodal Chain-of-Thought Reasoning in Language Models

2023-03-08 19:30 ~ 20:30 염기웅, 최재훈, 황지현, 김혜인

• How to use UForm
• How to create KoChatLLaMA

2023-03-09 20:00 ~ 22:00 염기웅, 윤상현, 신승욱

• Competition-Level Code Generation with AlphaCode
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• GPU and learning method required for KoChatLlaMA fine-tuning
• Advantages and Problems of UForm

2023-03-10 21:00 ~ 22:20 염기웅, 나요한, 최재훈, 외 청강 5인

• GPT-4 is coming next week – and it will be multimodal, says Microsoft Germany
• MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
• PaLM-E: An Embodied Multimodal Language Model

2023-03-10 20:00 ~ 21:00 염기웅, 황지현, 이대환, 나요한

• Language Is Not All You Need: Aligning Perception with Language Models
• Multimodal Chain-of-Thought Reasoning in Language Models

NEXT

• Tightly-Integrated Generative Encoder-Decoder Representation
• Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
• PaLM: Scaling Language Modeling with Pathways
• SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks
• LoRA: Low-Rank Adaptation of Large Language Models
• Language Models are Few-Shot Learners
• Low-rank Adaptation for Fast Text-to-Image Diffusion Fine-tuning
• huggingface-projects/diffusers-gallery
• huggingface-projects/diffusers-gallery-bot

규칙

1. 영어만 사용은 금지. 한국어 중심 사용. 특수 용어는 영어 사용.
2. 1주일에 논문 2개 이상 스터디. 되는 사람은 10개 이상.
3. 3분에서 20분 현장에서 논문 읽기. 5분에서 30분 토론.
4. 1시간 스터디 시, 바로 나가도 됨. 원할 때 10분 이하 참여도 무관. 자유롭게 진행. 2시간 매일도 가능.
5. 각자 더 뛰어난 게 있다는 것을 인지. 다들 대단한 분들이니 질문 많이 하고, 정보 공유 자주.
6. 본인이 하기로 한 일만은 수행. 한다고 말하고, 안 하는 것은 민폐다.
7. 기본적으로 녹화 후 내부 공유.
8. 정보를 혼자 알게 쓰지 말고, 다 같이 알게 말하기.
9. 개인 사정으로 스터디 탈퇴 시, 자기소개에 인사 작성.
10. 여러 기관 좋은 규칙 붙여넣기.
11. 팀에 도움이 된다고 판단하면, 위 규칙을 모두 무시하고 행동.
12. 추가.

후보

앞으로 할만한 논문, 코드, 강의 등.

papaer

• Improving language models by retrieving from trillions of tokens

• FLAN: Finetuned Language Models Are Zero-Shot Learners

• T0: Multitask Prompted Training Enables Zero-Shot Task Generalization

• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

• The Wisdom of Hindsight Makes Language Models Better Instruction Followers

• Exploring the Benefits of Training Expert Language Models over Instruction Tuning

• Unsupervised Imputation of Non-ignorably Missing Data Using Importance-Weighted Autoencoders

• The Power of Scale for Parameter-Efficient Prompt Tuning

• Constitutional AI: Harmlessness from AI Feedback

• Deep reinforcement learning from human preferences

• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

• Large Language Models with Controllable Working Memory

• Do Prompt-Based Models Really Understand the Meaning of their Prompts?

• Muse: Text-To-Image Generation via Masked Generative Transformers

• Structure and Content-Guided Video Synthesis with Diffusion Models

• Generative Pretraining from Pixels

• A hunt for the Snark: Annotator Diversity in Data Practices

• Accurate global machine learning force fields for molecules with hundreds of atoms

• Algorithms with More Granular Differential Privacy Guarantees

• Anomaly Clustering: Grouping Images into Coherent Clusters of Anomaly Types

• Are we cobblers without shoes? Making Computer Science data FAIR

• Code Generation for In-Place Stencils

• Creating, Calibrating, and Validating Large-Scale Microscopic Traffic Simulation

• Increasing Impact of Mobile Health Programs: SAHELI for Maternal and Child Care

• Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges

• Developer Productivity for Humans: A Human-Centered Approach to Developer Productivity

• Development of a Machine Learning Model for Sonographic Assessment of Gestational Age

• Drug Design on Quantum Computers

• Estimates of broadband upwelling irradiance from GOES-16 ABI

• Information Processing and Management

• Flake Aware Culprit Finding

• Flexible Budgets in Restless Bandits: A Primal-Dual Algorithm for Efficient Budget Allocation

• Helpful Neighbors: Leveraging Neighbors in Geographic Feature Pronunciation

• High-Performance GPU-to-CPU Transpilation and Optimization via High-Level Parallel Constructs

• Helpful Neighbors: Leveraging Neighbors in Geographic Feature Pronunciation

• Infrastructuring Care: How Trans and Non-Binary People Meet Health and Well-Being Needs through Technology

• KwikBucks: Correlation Clustering with Cheap-Weak and Expensive-Strong Signals

• Learning to Bid in Contextual First Price Auctions

• Machine Learning for Healthcare: A Bibliometric Study of Contributions from Africa

• Scalable Decision-Focused Learning in Restless Multi-Armed Bandits with Application to Maternal and Child Health

• Robust Planning over Restless Groups: Engagement Interventions for a Large-Scale Maternal Telehealth Program

• Recitation-Augmented Language Models

• RL4ReAl: Reinforcement Learning for Register Allocation

• Quantum Simulation of Exact Electron Dynamics can be more Efficient than Classical Mean-Field Methods

• Quantum simulation of exact electron dynamics can be more efficient than classical mean-field methods

• Propeller: A Profile Guided, Relinking Optimizer for Warehouse-Scale Applications

• Large Language Models are Zero-Shot Reasoners

• Deepmind: Improving language models by retrieving from trillions of tokens

• Deepmind: Building safer dialogue agents

• Deepmind: Competitive programming with AlphaCode

• Deepmind: Mastering Stratego, the classic game of imperfect information

• Deepmind: DeepMind’s latest research at NeurIPS 2022

• Deepmind: Building interactive agents in video game worlds

• Deepmind: Discovering novel algorithms with AlphaTensor

• Deepmind: AlphaFold reveals the structure of the protein universe

• Deepmind: Tackling multiple tasks with a single visual language model

• Deepmind: Exploring the beauty of pure mathematics in novel ways

• Deepmind: Nowcasting the next hour of rain

• Deepmind: Putting the power of AlphaFold into the world’s hands

• Google Research: Deciphering clinical abbreviations with privacy protecting ML

• Google Research: Google Research, 2022 & beyond: Language, vision and generative models

• Google Research: Google Research, 2022 & beyond: Responsible AI

• Google Research: Learning with queried hints

• Google Research: Open Source Vizier: Towards reliable and flexible hyperparameter and blackbox optimization

• Google Research: Google Research, 2022 & beyond: ML & computer systems

• Google Research: Real-time tracking of wildfire boundaries using satellite imagery

• Google Research: Breaching the 2 LMP Approximation Barrier for Facility Location with Applications to k-Median

• Google Research: Chimane-Mosetén

• Google Research: Differentially Private All-Pairs Shortest Path Distances: Improved Algorithms and Lower Bounds

• Google Research: Differentially Private Fair Division

• Google Research: DiffQG: Generating Questions on Paired Sentences

• Google Research: Assessment of Security Defense of Native Programs Against Software Faults

• Google Research: Adaptive mixing of auxiliary losses in supervised learning

• OpenAI: Multimodal Neurons in Artificial Neural Networks

• OpenAI: DALL·E: Creating Images from Text

• OpenAI: CLIP: Connecting Text and Images

• OpenAI: Image GPT

• OpenAI: Jukebox

• OpenAI: Solving Rubik’s Cube with a Robot Hand

• OpenAI: Multimodal Neurons in Artificial Neural Networks

• OpenAI: CLIP: Connecting Text and Images

• OpenAI: Image GPT

• OpenAI: MuseNet

• OpenAI: Emergent Tool Use from Multi-Agent Interaction

github

• DistilKoBiLSTM
• Donut: Document Understanding Transformer
• The RWKV Language Model (and my LM tricks)

youtube

• Study Playlist
• Improving Language Models by Retrieving from Trillions of Tokens | NLP Journal Club
• ECMLPKDD2021: WuDao: Pretrain the World, Keynote speaker talk by Jie Tang
• StrictlyVC in conversation with Sam Altman, part two (OpenAI)
• Are Bigger Language Models Better? | DeepMind Gopher and RETRO
• The Illustrated Retrieval Transformer

other

• CS224U: Natural Language Understandin
• Gen-1: The Next Step Forward for Generative AI
• DIFF-SVC FOR VOCAL SYNTH USERS
• Chat GPT detector by ZeroGPT: detect OpenAI text

mllmarxivtalk's Issues

참여 질문 리스트

스터디에 참여해도 될지 고민하는 분들께, 몇 가지 질문을 드려봅니다. 대답 가능하신 것만 해주세요.

검색이나 chatgpt나 무엇을 쓰든, 작성한 내용을 이해만 하시면 괜찮습니다.

기초

1. lstm, bert, gpt, t5 장단점 한계 차이 특징
2. 가장 흥미롭게 읽은 논문 5문장 요약
3. sdt를 쓰면 좋을 때와 아닐 때
4. 모든 것을 end to end로 하는 게 좋은가?
5. 파라미터 사이즈 1조 개 이상의 모델을 돌릴 때 필요한 서버는?
6. 현 최대의 파라미터 수를 가진 모델은?
7. P-tuning 에 대한 설명과 개선점. V3가 나온다면 개선점은?
8. 강화학습을 nlp task에 쓴 모델 설명 5개
9. 멀티 모달, 멀티 랭기지, 멀티 테스크가 항상 좋은가?
10. 형태소와 워드피스 등 토크나이저의 장단점
11. 논문을 20분 안에 읽거나 사전 학습 가능 여부
12. 스터디 시 항상 참여 가능한가?

심화

1. ∞-former는 이론상 1조 7500억개의 토큰을 손실 없이 생성할 수 있는가?
2. Gpt4의 파라미터 개수와 구조
3. Show Don’t Tell 을 한국어로 적용하는 방법
4. retro gpt3 비교 한계 분석
5. T5 T0 비교 한계 의미 가치 분석
6. You는 어떤 모델로 만들었는가?
7. 우주의 모양은?
8. 1경년 후에 무슨 일이 일어나는가?
9. 순간이동을 누가 언제 어떻게 왜 구현 했는가?
10. 영생하는 동물과 한계는?
11. 다음 빅뱅을 버티는 방법
12. 역대 최연소 교수의 최고의 논문은?
13. https://www.nature.com/articles/d41586-023-00340-6 https://www.nature.com/articles/d41591-023-00014-w https://www.nature.com/articles/s42256-022-00593-2 분석 요약
14. 현 스터디 멤버 분석
15. P-tuning 시 I love you로 고정해도 같은 성능이 나오는가?
16. https://arxiv.org/abs/2009.01325 https://arxiv.org/abs/2203.02155 분석 및 chatgpt 구조 분석
17. https://arxiv.org/pdf/2301.00704v1.pdf https://arxiv.org/pdf/2301.12597v1.pdf 비교 분석

참고로 저는 90프로 이상의 답변은 바로 됩니다. 나머지는 약 2, 3개는 검색이 필요합니다. 팀원 중 많은 분이, 검색 시 완벽한 답변이 가능합니다.

기초 답변이 완벽하거나, 심화에서 올바른 접근을 하면 합류. 전부 완벽하면 모시고, 애매하면 인터뷰, 아니면 탈락.

할 일

• 책 제목: 프로메터우스와 바드의 꿈.
• 논문과 프로젝트:
  • P-Tuning V3, 인공지능과 인간의 뇌의 한계,
  • 파라미터 1조개 이상 모델을 성능을 내는 100B 모델.
  • 영화 웹툰 등 시계열 이미지 생성.
  • LSTM, GPT, BERT, T5 초 경량화 및 각 테스크 별 성능 비교.
• 스터디: 최신 LLM 대화 경량화 강화학습 등 https://github.com/gyunggyung/LMMArxivTalk

네이처 사이언스 급의 무언가를 같이 만들고, 노벨상 급을 받고, 조 단위를 벌며 세상을 바뀝시다.

해당 스터디에 참여해도 될지 고민하는 분들께

몇 가지 질문을 드려봅니다. 대답 가능하신 것만 해주세요.
검색, ChatGPT나 사람들에게 물어봐도 됩니다.
무엇을 하셔도 작성한 내용만 이해하시면 괜찮습니다.

기초

1. 본인이 가장 잘 아는 논문을 5문장 내외로 서술.
2. word2vec 관련 정보 최대한 쉽게 서술.
3. 본인의 관심 task를 10 문장 내외로 서술.
4. lstm, bert, gpt, t5 장단점, 한계, 차이, 특징은?
5. 형태소와 워드피스 등 토크나이저의 장단점은?
6. 논문을 20분 안에 읽거나 사전 학습 가능한지?
7. 스터디 시 항상 참여 가능한지?

심화

1. P-tuning에 대한 설명. V3가 나온다면 개선점은?
2. ∞-former는 이론상 1조 7500억개의 토큰을 손실 없이 생성할 수 있는가?
3. 본인이 생각하는 GPT-4 예상 파라미터 개수와 구조는?
4. retro, chatgpt 등 빅테크에서 나온 모델들의 비교 한계 분석.
5. 현 스터디 멤버 분석.
6. 원하는 질문.

청강 링크

우리 스터디 청강을 하고 싶으신 분들은 여기에 오세요! 초대, 공유 해주셔도 됩니다. 참고로 세미나 발표식의 스터디를 할 때만 (1주일에 1회 정도) 청강을 진행할 거 같습니다. 더 자세한 것은 슬렉에서

https://join.slack.com/t/llm-edg7544/shared_invite/zt-1r65qi61c-7qyD4JON8gNbT1j89CNjYw

네이처 사이언스 논문 추가

https://www.nature.com/search?q=generative+AI%C2%A0&journal=

https://www.science.org/action/doSearch?AllField=generative+AI+

https://www.nature.com/articles/s41598-022-23656-1

https://www.nature.com/articles/d41586-023-00107-z

https://www.nature.com/articles/s41598-022-22222-z

https://www.nature.com/articles/s42256-019-0028-1

https://www.nature.com/articles/s42256-022-00539-8

https://www.nature.com/articles/s42256-022-00598-x

https://www.nature.com/articles/d41586-020-03416-9

https://www.nature.com/articles/s42256-022-00591-4

https://www.nature.com/articles/s42256-021-00310-5

https://www.nature.com/articles/s41587-023-01695-x

https://www.nature.com/articles/s42256-022-00593-2

https://www.nature.com/articles/s41598-022-23325-3

https://www.nature.com/articles/s41598-021-01295-2

https://www.nature.com/articles/s41598-022-20698-3

https://www.nature.com/articles/s41467-019-10933-3

https://www.nature.com/articles/s41587-020-0417-3

https://www.nature.com/articles/s41587-020-0418-2

https://www.nature.com/articles/s42256-022-00564-7

https://www.nature.com/articles/s41551-021-00689-x

https://www.nature.com/articles/s42256-023-00613-9

https://www.nature.com/articles/s41746-022-00634-5

https://www.nature.com/articles/s41551-021-00771-4

https://www.science.org/doi/10.1126/sciadv.abh1303

https://www.science.org/doi/10.1126/sciadv.abg3338

https://www.science.org/doi/10.1126/sciadv.aaz4169

https://www.science.org/doi/10.1126/sciadv.aat9004

https://www.science.org/doi/10.1126/sciadv.aav2761

https://www.science.org/doi/10.1126/science.aat2663

https://www.science.org/doi/10.1126/sciadv.abm6127

https://www.nature.com/articles/d41586-023-00423-4

https://www.science.org/doi/10.1126/science.adg7879

https://www.nature.com/articles/369719a0

https://www.science.org/doi/10.1126/science.222.4629.1224

https://www.nature.com/articles/d41586-023-00340-6

https://www.science.org/doi/10.1126/science.add8258

https://www.science.org/doi/10.1126/science.adg4246

https://www.science.org/content/article/dark-energy-supermassive-black-holes-physicists-spar-over-radical-idea

https://www.nature.com/articles/nature14539

https://www.nature.com/articles/nature16961

https://chrome.google.com/webstore/detail/arxivgpt/fbbfpcjhnnklhmncjickdipdlhoddjoh?hl=en&authuser=0

LLMArxivTalk

LMMArxivTalk에서 LLMArxivTalk으로 변경.

아마 홍보한 링크들이 의미를 잃고 취업을 완료한 후에 진행 예정.