sagorbrur / bangla-bert

Bangla-Bert is a pretrained bert model for Bengali language

Hi there 👋

This is Sagor Sarker

An enthusiastic NLP, AI, ML learner

bangla-bert's Issues

Some weights of the model checkpoint at sagorsarker/bangla-bert-base were not used when initializing BertModel

Some weights of the model checkpoint at sagorsarker/bangla-bert-base were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias']

• This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

CODE Portion:

class BertClassifier(th.nn.Module):

def __init__(self, pretrained_model='sagorsarker/bangla-bert-base', nb_class=20):
    super(BertClassifier, self).__init__()
    self.nb_class = nb_class
    self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model)
    self.bert_model = AutoModel.from_pretrained(pretrained_model)
    self.feat_dim = list(self.bert_model.modules())[-2].out_features
    self.classifier = th.nn.Linear(self.feat_dim, nb_class)

def forward(self, input_ids, attention_mask):
    cls_feats = self.bert_model(input_ids, attention_mask)[0][:, 0]
    cls_logit = self.classifier(cls_feats)
    return cls_logit

PRE_TRAINED_MODEL_NAME = 'sagorsarker/bangla-bert-base'

model = BertClassifier(pretrained_model=PRE_TRAINED_MODEL_NAME, nb_class=nb_class)