aqweteddy / leveragejustafewkeywords

Unofficial implementation for EMNLP2019 <Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training>

Hi there 👋, this is Pin Zu Li's Github !

I'm a NLP Engineer. my works primarily on NLP, including developing large language models that integrate Taiwanese culture and addressing the challenge of fake news early detection.

Publication

Chat Vector: A Simple Approach to Equip LLMs With New Language Chat Capabilities Shih-Cheng Huang, Pin-Zu Li, Yu-Chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tzong-Han Tsai, Hung-yi Lee. We propose using a chat vector to transfer the instruction following capability from the English language LLM to the Traditional Chinese LLM without the need for any additional training.

leveragejustafewkeywords's Issues

way of the model work in train.py

As your method is like for each batch of the dataset, the teacher model would run first, then immediately apply the student model.

What we think is like we should apply the teacher model to the entire dataset first, after 1 epoch then we could apply the student model.

We would like to know your opinion about this.

Thank you.

Teacher model in model.py

In line 27-29, we changed

mask = bow.sum(1) == 0
result[mask, self.general_asp] = 1
result = torch.softmax(result, -1)

to
result = torch.softmax(result, -1)
result[mask, :] = 0
result[mask, self.general_asp] = 1

As apply softmax before or after the assign, actually makes a different

calc_z in train.py

Hi, we are a team recently working on the same paper and want to apply it to our own dataset. There is something that I want to discuss with you.

In train.py, calc_z function, line 168, we changed
bsum = r.sum(-1).view(-1, 1)
to
bsum = r.sum(0).view(1, -1)

As we think it changes the way of summation against the paper, in fact, we got better performance after changing applied to OPOSUM dataset.