bzantium / bert-aad

Adversarial Adaptation with Distillation for BERT Unsupervised Domain Adaptation

[2023.08 - Current] LLM Researcher @Kakaobrain

• Developing Language Foundation Model a.k.a. KoGPT2

[2020.03 - 2023.08] Machine Learning Engineer @SK Telecom & EleutherAI

• Developing Large Language Model for SK Telecom
• Developed Multimodal AI Service at SKT A.
  • AI Eraser (Object Removal with Image Inpainting)
    • role: project manager & implementation of pipeline algorithm (segmentation postprocessing & enhancement in inpainting performance).
• Developing polyglot and oslo project at EleutherAI
  • polyglot: Large Language Models of Well-balanced Competence in Multi-languages
    • role: distributed training of LM with Megatron LM & data crawling, preprocessing and model evaluaton. Published 1.3B, 3.8B, 5.8B, 12.8B polyglot-ko models.
  • oslo: Open Source for Large-scale Optimization
    • role: tensor parallel 1D, 2D, 3D implementation.

Interest

• Foundation Model / NLP / Multimodal AI

bert-aad's Issues

The adapt accuracies are not comparable to no adapt?

Hi,

Thanks for your code. It helps a lot. When I was trying to reproduce your results on the amazon review datasets, I found the BERT-ADD accuracies are worse than no adapt results? Have you encountered the same issue? Epoch [79/80] Step [195/200]: acc=0.5000 g_loss=0.6922 d_loss=0.6932 kd_loss=0.0000. I also noticed that when the adapt training converges, only the kd_loss descent to 0, but g_loss and d_loss didn't descend at all. Is it normal or maybe this is where the problem is? Or could you please release the hyperparameters for Bert-AAD?

Thanks,