yeongjoonju / occlusion-robust-3d-face-cfr-gan

Occlusion robust 3D face reconstruction model in CFR-GAN (WACV 2022)

Yeong-Joon Ju

😄 Profile

🎓 Affiliation: Pattern Recognition and Machine Learning Lab in Korea University (2020~)

I'm researching efficient and effective in-context augmentation for multimodal large language models (MLLMs).

Research direction: face generation (vision) $\rightarrow$ dialogue generation (NLP) $\rightarrow$ neural network debugging (XAI) $\rightarrow$ 🌞 language modeling for multimodalities (NLP)

📃 Publications

• "Neuron-based Debugging Framework for Neural Networks Using Class-conditional Visualizations", arXiv, 2023. | code

  Authors: Y-J. Ju, J-H. Park, and S-W. Lee

  History -> NeurIPS 2023: rejected - 3 borderline accept and 1 reject

• "Class-conditional Feature Visualizations in Realistic Image Space for Neuron-based Debugging", KCC XAI Workshop, 2024 (oral).

  Authors: Y-J. Ju, J-H. Park, and S-W. Lee

  Oral speaker: Yeong-Joon Ju

• "Complete Face Recovery GAN: Unsupervised Joint Face Rotation and De-Occlusion from a Single-View Image", WACV, 2022. | code

  Authors: Y-J. Ju, G-H. Lee, J-H. Hong, and S-W. Lee

• "CPR: Mitigating Large Language Model Hallucinations with Curative Prompt Refinement", SMC, 2024.

  Authors: J-W. Shim, Y-J. Ju, J-H. Park, and S-W. Lee

• "CIRF: Importance of Related Features for Plausible Counterfactual Explanations", Information Sciences, 2024.

  Authors: H-D. Kim, Y-J. Ju, J-H. Hong, and S-W. Lee

• "Explaining Generative Diffusion Models via Visual Analysis for Interpretable Decision-Making Process", Expert Systems with Applications, 2024. | code

  Authors: J-H. Park, Y-J. Ju, and S-W. Lee

• "Enhancing Robustness of Over-Parameterized Models via Feature Reweighting Using Logit-Wise Mixup", ICPRAI, 2024 (oral).

  Authors: W-S. Jo, Y-J. Ju, and S-W. Lee

  Oral speaker: Yeong-Joon Ju

• "Compensatory Debiasing for Gender Imbalances in Language Models", ICASSP, 2023 (oral). | code

  Authors: T-J. Woo, W-J. Nam, Y-J. Ju, and S-W. Lee

  Oral speaker: Yeong-Joon Ju

Co-authors: Seong-Whan Lee / Gun-Hee Lee / Woo-Jeoung Nam / Jung-Ho Hong / Ji-Hoon Park / Hee-Dong Kim / Tae-Jin Woo / Woo-Seok Jo / Jeong-Woo Shim

💻 Research Projects (national or industry-academic projects)

• Research on AI-based large-scale language modeling techniques for Military-related topic (hidden), Agency for Defense Development (ADD, 국방과학연구소), Korea (Aug. 2023 ~ Nov. 2023) Team leader (practitioner)

• Development of an AI assistant for elementary education teacher based on a large language model (LLM), i-Scream Edu (아이스크림에듀), Korea (May 2023 ~ Jan. 2024) Team leader (practitioner)

  Research of LLM for NLG

• Development of Artificial Intelligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation, Institute of Information & communications Technology Planning & Evaluation (IITP), Korea (Jun. 2022 ~ Dec. 2023) Team leader (practitioner)

  Research of plug-and-play debugging for the decision-making process of neural networks

• Development of facial skin change analysis and visualization techniques, Samsung Electronics (삼성전자), Korea (Jun. 2020 ~ Jun. 2021)

  Research of facial dense alignment technique based on optical flow

👍 Awards, etc.

• Reviewer at IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024

• Grand Prize in The 6th SW·AI Hackerton, Sejong University, Korea (Jul. 2019)

  Project: Android application that manages seat status for discussion lounge / Development of seat status detection model and API based on Flask (Jul. 2019 ~ Jul. 2019)

• Grand Prize in The 7th Creative Design Competition, Sejong University, Korea (Jun. 2019)

  Project: Web application that automatically blurs out license plates and faces / Development of web application based on React.js, detection models, and API. (Mar. 2019 ~ Jun. 2019)

occlusion-robust-3d-face-cfr-gan's Issues

About align in Preprocessing step

Thanks for your greate work! I want to train my model following the readme file , and I noticed you remind that You must align images with align.py in Preprocessing step.
So should I first execute align.py in origin datasets to get the aligned datasets, and then execute the create_train_stage1.py using the aligned datasets path as a parameter?
Thanks again!

convert tflite?

안녕하세요, face frontalization 일 때문에 논문하고 코드를 보고 있는데요... 모델 구성정보를 받을수 있을까요? training 하시는데 시간은 어느정도 걸리셨을까요? 이 모델을 조금 경량화한다음에 training 해서 tflite 로 변경하고 싶은데 조언 좀 받고싶습니다.

About alignment

Hello, I 'm trying to reproduce this good work, and I wonder if the multiPIE dataset needs to be aligned for the first training process?