Comments (5)
roberta is bert,you can load roberta use pytorch transformer, select model as bert
from roberta_zh.
no. this roberta_zh is changed based on bert with main ideas from RoBERTa paper, but it may not compatible to roberta pytorch implementation from facebook ai project. so load from BertPreTrainedModel should be right way.
can you paste code here to show how you load roberta_zh using BertPreTrainedModel, and how you use it.
(anyway you can have a try, but it should be failed)
from roberta_zh.
It's the same with original pytorch-transformers
bert_config = MODEL_CLASSES[args.model_type][0].from_json_file(args.bert_config_file)
tokenizer = MODEL_CLASSES[args.model_type][2](vocab_file=args.vocab_file,
do_lower_case=args.do_lower_case)
model = MODEL_CLASSES[args.model_type][1].from_pretrained(args.model_name_or_path,
from_tf=bool('.ckpt' in args.model_name_or_path), config=bert_config)
However, for RobertaTokenizer
, you cannot use it to load roberta_zh
.
- It requires
merges_file
parameter which is not inroberta_zh
Traceback (most recent call last):
File "run_transformers.py", line 447, in <module>
main()
File "run_transformers.py", line 389, in main
tokenizer = MODEL_CLASSES[args.model_type][2](vocab_file=args.vocab_file, do_lower_case=args.do_lower_case)
TypeError: __init__() missing 1 required positional argument: 'merges_file'
- If you use
from_pretrained
, it complains an encoding problem ofroberta_zh
.
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils - Model name '/home/vimos/.pytorch_pretrained_bert/roberta_zh/pytorch_model.bin' nsubsubsectionot found in model shortcut name list (roberta-base, roberta-large, roberta-large-mnli). Assuming '/home/vimos/.pytorch_pretrained_bert/roberta_zh/pytorch_model.bin' is a path or url to a directory containing tokenizer files.
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils - Didn't find file /home/vimos/.pytorch_pretrained_bert/roberta_zh/added_tokens.json. We won't load it.
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils - Didn't find file /home/vimos/.pytorch_pretrained_bert/roberta_zh/special_tokens_map.json. We won't load it.
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils - Didn't find file /home/vimos/.pytorch_pretrained_bert/roberta_zh/tokenizer_config.json. We won't load it.
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils - loading file /home/vimos/.pytorch_pretrained_bert/roberta_zh/pytorch_model.bin
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils - loading file /home/vimos/.pytorch_pretrained_bert/roberta_zh/pytorch_model.bin
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils - loading file None
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils - loading file None
09/28/2019 14:23:53 - INFO - pytorch_transformers.tokenization_utils - loading file None
Traceback (most recent call last):
File "run_transformers.py", line 448, in <module>
main()
File "run_transformers.py", line 389, in main
tokenizer = MODEL_CLASSES['roberta'][2].from_pretrained(args.model_name_or_path)
File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/pytorch_transformers/tokenization_utils.py", line 293, in from_pretrained
return cls._from_pretrained(*inputs, **kwargs)
File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/pytorch_transformers/tokenization_utils.py", line 421, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/pytorch_transformers/tokenization_roberta.py", line 82, in __init__
mask_token=mask_token, **kwargs)
File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/site-packages/pytorch_transformers/tokenization_gpt2.py", line 118, in __init__
self.encoder = json.load(open(vocab_file, encoding="utf-8"))
File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/home/vimos/anaconda3/envs/rcqa/lib/python3.7/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
from roberta_zh.
yeah, it is.
from roberta_zh.
can you please suggest which pytorch bert package is compatible with this pre-trained model?
I had the same failure as Vimos when loading it with facebook transformers.
from roberta_zh.
Related Issues (20)
- XLNet其实不能稳压RoBERTa吧? HOT 1
- GPT vs BERT, under same computation and data resource, which one is better for downstream tasks like GLUE?
- 在pytorch模型上做post train HOT 2
- NaN probability sometimes when inference on GPU
- CMRC示例
- 是否可以开放语料,供其他模型对比
- 其中依赖的预训练模型是否和bert官方提供是一样的?
- 关于MLM中,中文全词掩盖的预测标签问题 HOT 7
- What are the pretrained-language-model that is obviously better than BERT and RoBERTa?
- 预处理数据丢失问题 HOT 1
- Unrelated parameters in the config
- resource文件夹下的vocab和代码不对应
- pytorch用BERT的加载方式加载roberta模型,呢么创建token时special token 是按照bert的方式还是roberta的方式呢
- 请问下,怎么进行GPU训练?
- Huggingface
- 利用roberta_zh的tokenizer来做中文NER任务时报错 HOT 2
- tensorboard可视化模型输出结果 train的masked_lm_loss和masked_lm_accuracy是空的,eval的图只有一个点
- Loss curve
- 下载问题和加载模型
- 加载的小问题求解答
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from roberta_zh.