Coder Social home page Coder Social logo

paddlepaddle / paddlehub Goto Github PK

View Code? Open in Web Editor NEW
12.5K 184.0 2.1K 206.69 MB

Awesome pre-trained models toolkit based on PaddlePaddle. (400+ models including Image, Text, Audio, Video and Cross-Modal with Easy Inference & Serving)

Home Page: https://www.paddlepaddle.org.cn/hub

License: Apache License 2.0

Python 99.94% Shell 0.01% Dockerfile 0.05%
deep-learning model awesome nlp vision text2image

paddlehub's Introduction

English | 简体中文


⭐Features

  • 📦400+ AI Models: Rich, high-quality AI models, including CV, NLP, Speech, Video and Cross-Modal.
  • 🧒Easy to Use: 3 lines of code to predict 400+ AI models.
  • 💁Model As Service: Easy to serve model with only one line of command.
  • 💻Cross-platform: Support Linux, Windows and MacOS.

💥Recent Updates

🌈Visualization Demo

Wenxin Big Models Stable_Diffusion series Disco Diffusion series
Include ERNIE-ViLG, ERNIE-ViL, ERNIE 3.0 Zeus, supports applications such as text-to-image, writing essays, summarization, couplets, question answering, writing novels and completing text。 Supports functions such as text_to_image, image_to_image, inpainting, ACGN external service, etc. Support Chinese and English input
  • Many thanks to CopyRight@ERNIELACDDParserfor the pre-trained models, you can try to train your models with them.
Input Audio Recognition Result

I knocked at the door on the ancient side of the building.

我认为跑步最重要的就是给我带来了身体健康。
Input Text Output Audio
Life was like a box of chocolates, you never know what you're gonna get.
  • Many thanks to CopyRight@PaddleSpeech for the pre-trained models, you can try to train your models with PaddleSpeech.

⭐ Thanks for Your Star

  • All the above pre-trained models are all open source and free, and the number of models is continuously updated. Welcome Star to pay attention.

🍻Welcome to join PaddleHub technical group

  • If you have any questions during the use of the model, you can join the official WeChat group to get more efficient questions and answers, and fully communicate with developers from all walks of life. We look forward to your joining.
  • please add WeChat above and send "Hub" to the robot, the robot will invite you to join the group automatically.

✈️QuickStart

🚁The installation of required components.

# install paddlepaddle with gpu
# !pip install --upgrade paddlepaddle-gpu

# or install paddlepaddle with cpu
!pip install --upgrade paddlepaddle

# install paddlehub
!pip install --upgrade paddlehub

🛫The simplest cases of Chinese word segmentation.

import paddlehub as hub

lac = hub.Module(name="lac")
test_text = ["今天是个好天气。"]

results = lac.cut(text=test_text, use_gpu=False, batch_size=1, return_tag=True)
print(results)
#{'word': ['今天', '是', '个', '好天气', '。'], 'tag': ['TIME', 'v', 'q', 'n', 'w']}

🛰️The simplest command of deploying lac service.

!hub serving start -m lac

📚License

The release of this project is certified by the Apache 2.0 license.

👨‍👨‍👧‍👦Contribution

We welcome you to contribute code to PaddleHub, and thank you for your feedback.

  • Many thanks to 肖培楷, Contributed to street scene cartoonization, portrait cartoonization, gesture key point recognition, sky replacement, depth estimation, portrait segmentation and other modules
  • Many thanks to Austendeng for fixing the SequenceLabelReader
  • Many thanks to cclauss optimizing travis-ci check
  • Many thanks to 奇想天外,Contributed a demo of mask detection
  • Many thanks to mhlwsk,Contributed the repair sequence annotation prediction demo
  • Many thanks to zbp-xxxp,Contributed modules for viewing pictures and writing poems
  • Many thanks to zbp-xxxp and 七年期限,Jointly contributed to the Mid-Autumn Festival Special Edition Module
  • Many thanks to livingbody,Contributed models for style transfer based on PaddleHub's capabilities and Mid-Autumn Festival WeChat Mini Program
  • Many thanks to BurrowsWang for fixing Markdown table display problem
  • Many thanks to huqi for fixing readme typo
  • Many thanks to parano cqvu deehrlic for contributing this feature in PaddleHub
  • Many thanks to paopjian for correcting the wrong website address #1424
  • Many thanks to Wgm-Inspur for correcting the demo errors in readme, and updating the RNN illustration in the text classification and sequence labeling demo
  • Many thanks to zl1271 for fixing serving docs typo
  • Many thanks to AK391 for adding the webdemo of UGATIT and deoldify models in Hugging Face spaces
  • Many thanks to itegel for fixing quick start docs typo
  • Many thanks to AK391 for adding the webdemo of Photo2Cartoon model in Hugging Face spaces

paddlehub's People

Contributors

1084667371 avatar 46319943 avatar adaxiadaxi avatar ak391 avatar akiirokaede avatar buchongyu2 avatar channingss avatar chunzhang-hub avatar d-danielyang avatar dependabot[bot] avatar desmonday avatar dxxxp avatar eepgxxy avatar haoyuying avatar houj04 avatar jm12138 avatar kinghuin avatar kpatr1ck avatar linjieccc avatar nepeplwu avatar paopjian avatar rainyfly avatar shenyuhan avatar shinichiye avatar sjtubinlong avatar steffy-zxf avatar wgm-inspur avatar will-jl944 avatar zbp-xxxp avatar zeyuchen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

paddlehub's Issues

senta_demo 执行有问题

用的1.5.1的paddlepaddle及1.1.0的paddlehub,显示报错:
AttributeError: module 'paddle' has no attribute 'batch'
查询版本更新里面并没有这方面的改动…想知道哪边出了问题

valueerror: the length of ret_tokens can't match with labels

训练序列标注任务,自己构建的数据集合,和msra_ner一样格式,怎么会出现这句错误valueerror: the length of ret_tokens can't match with labels。ret_tokens是什么意思?可我确定 label的长度和text_a的长度相同了啊。

tiny error in readme of sequence-labeling demo

feed_list in readme.md is:

feed_list = [
    inputs["input_ids"].name, inputs["position_ids"].name,
    inputs["segment_ids"].name, inputs["input_mask"].name, label.name
]

but in sequence_label.py:

feed_list = [
        inputs["input_ids"].name,
        inputs["position_ids"].name,
        inputs["segment_ids"].name,
        inputs["input_mask"].name,
    ]

maybe label.name in readme is a tiny mistake ?
because I see self.dataset.get_labels() will be run in BaseReader

PaddleHub图像分类迁移教程错误

首先感谢这份教程,十分的完善清楚
但是在最后一步预测的时候是不是漏了一句
run_states = task.predict(data=data)
results = [run_state.run_results for run_state in run_states]

Is there a pretrained lexical_analysis model?

I ran
hub run lexical_analysis --input_text "在搜索引擎输入.尚赫”就可以发现多篇.尚赫涉嫌传销”.虚假宣传”.跨区经营”等各类媒体公开报道天津尚赫保健品有限公司违规经营的报道"
and it says

Install Module lexical_analysis
Can't find module lexical_analysis

I understand that lexical_analysis is a model, not a module, is there a way to call a trained model?

网络中使用fluid.layers.resize_bilinear层之后,在模型预测阶段会将输出都resize到训练阶段的patch大小

如题,我的网络中使用了resize层进行上采样,在模型预测阶段,发现模型会将图像resize到训练时patch大小,模型输出图像不是原始图像分辨率。
但是模型中不使用该层,那么预测阶段模型输出就是原始图像分辨率。
请问fluid.layers.resize_bilinear该层为什么会是这样,是否是resize层的bug。

can't find model/module named lac

Hi, I'm using command line to input "hub download lac", but I just get "PaddleHub can't find model/module named lac". I also try "hub search lac" but only see a TypeError: 'NoneType' object is not iterable. Could you please help me solve this problem? Many thanks.

P.S. I modify the hub_server.py file by removing all self.request().

关于PaddleHub Module参数中的processor兼容问题

您好,我在近期尝试使用 paddlehub.Module("lac") 进行词性分析时发现有processor参数,在windows下开启4进程时速度提升明显,但在centos下开启时似乎并没有什么效果。想要请问一下processor参数在centos环境下是否支持开启多进程,是否有其他方式可以较为显著地提升词性分析的速度?
我目前采用的是在保证句子不被切断的前提下将文章尽可能切割成500字符一下的分段,在分词结束后再重新拼接。如果开启多进程的话,更多分段和较少分段应该如何取舍呢?

无法安装module

20190808181715
20190808181657

无法安装module,搜到也装不了。外网能连,paddle-gpu和hub都是最新的。

word2vec模型如何使用

想直接利用hub.Module(name="word2vec_skipgram")拿到word2vec的模型直接使用,但是没找到demo。能否补充一下这块。

现在的测试代码是这样的,但是运行会报错:

import paddlehub as hub

module = hub.Module(name="word2vec_skipgram")
inputs, outputs, program = module.context(trainable=False)

word_ids = inputs["word_ids"]
embedding = outputs["word_embs"]
print(embedding[1123])

ERNIE做NER,如何加入bilstm+crf ?

想用bilstm+crf作为ERNIE标注任务的任务层
是不是继承SequenceLabelTask类,重写_build_net()_add_loss()方法就可以,是否有可参考的demo呢,感谢🙏

我在百度大脑AI studio 运行了PaddleHub的实例三,目标检测,发现只有ssd可以正常运行,faster-rcnn和yolov3都提示没有发现模型。

示例三

使用目标检测模型 SSD/YOLO v3/Faster RCNN 对图片进行目标检测

$ wget --no-check-certificate https://paddlehub.bj.bcebos.com/resources/test_object_detection.jpg
$ hub run ssd_mobilenet_v1_pascal --input_path test_object_detection.jpg
$ hub run yolov3_coco2017 --input_path test_object_detection.jpg
$ hub run faster_rcnn_coco2017 --input_path test_object_detection.jpg




运行输出
运行耗时: 7秒751毫秒
Can't find module faster_rcnn_coco2017
Can't find module yolov3_coco2017
Module ssd_mobilenet_v1_pascal already installed in /home/aistudio/.paddlehub/modules/ssd_mobilenet_v1_pascal

predict.py代码疑问

在predict.py的第47行有dataset = hub.dataset.ChnSentiCorp(),那在其他的任务中,请问加载的是自己的数据还是这个ChnSentiCorp数据,如果是自己的那加载的是训练数据还是测试数据呢?谢谢大佬解答一下!

文本分类采用bert模型,在finetune_and_eval()报错。

数据集为chnsenticorp
module = hub.Module(name="bert_chinese_L-12_H-768_A-12")
.....
cls_task.finetune_and_eval()
运行报错:

File "/home/*****/.local/lib/python3.6/site-packages/paddle/fluid/backward.py", line 310, in append_backward_ops
op.desc, cpt.to_text(no_grad_dict[block.idx]), grad_sub_block_list)

EnforceNotMet: Input ShapeTensor cannot be found in Op reshape2 at [/paddle/paddle/fluid/framework/op_desc.cc:306]
PaddlePaddle Call Stacks:
0 0x7f63a310b650p void paddle::platform::EnforceNotMet::Init<char const*>(char const*, char const*, int) + 352

paddlehub安装不成功

使用pip install paddlehub-0.4.6b0-py2-none-any.whl 进行安装,安装成功后执行hub命令,出现以下错误,paddle版本为1.3

image

PaddleHub 进行图像分类迁移报错

运行图像分类的示例notebiok报错
58 # Step 4: load pretrained parameters
59 place = fluid.CUDAPlace(0)
AttributeError: 'ClassifierTask' object has no attribute 'inference_program

PaddleHub的使用指南

请问下有没有关于paddlehub的使用指南呢?比如module里都有哪些预训练模型和预训练模型的名字,不知道名字无法加载预训练模型。怎么加载自己的数据集?

LAC自定义词典干预分词问题请教

自定义词典 user.dict 中的格式如下:
单词\t单词词性\t单词词频

这个单词词频设置多少合适呢,有参考的大小吗?

例如句子:吉利新帝豪标配搪塑中控实际是注塑
默认情况下分成了
['吉利新帝豪标配搪塑中控', '实际', '是', '注塑']

在 user.dict 中添加
吉利新帝豪 n 30000
后,变成👇:
['吉利新帝豪', '标配搪塑中控', '实际', '是', '注塑']

然后再次添加
标配 vn 30000
就不起作用了

语义相似度匹配任务

在做语义相似度匹配任务时,如果数据很多的话,predict.py 里面data的添加是怎么一种策略呢,如何能够快速匹配相似语义。

运行predict时报错

Traceback (most recent call last):
File "predict.py", line 76, in
fluid.io.load_persistables(exe, args.checkpoint_dir)
File "/home/tianzhen/.local/lib/python3.7/site-packages/paddle/fluid/io.py", line 704, in load_persistables
filename=filename)
File "/home/tianzhen/.local/lib/python3.7/site-packages/paddle/fluid/io.py", line 575, in load_vars
filename=filename)
File "/home/tianzhen/.local/lib/python3.7/site-packages/paddle/fluid/io.py", line 610, in load_vars
executor.run(load_prog)
File "/home/tianzhen/.local/lib/python3.7/site-packages/paddle/fluid/executor.py", line 565, in run
use_program_cache=use_program_cache)
File "/home/tianzhen/.local/lib/python3.7/site-packages/paddle/fluid/executor.py", line 642, in run
exe.run(program.desc, scope, 0, True, True, fetch_var_name)
paddle.fluid.core.EnforceNotMet: Invoke operator load error.
Python Callstacks:
File "/home/tianzhen/.local/lib/python3.7/site-packages/paddle/fluid/framework.py", line 1654, in append_op
attrs=kwargs.get("attrs", None))
File "/home/tianzhen/.local/lib/python3.7/site-packages/paddle/fluid/io.py", line 596, in load_vars
attrs={'file_path': os.path.join(dirname, new_var.name)})
File "/home/tianzhen/.local/lib/python3.7/site-packages/paddle/fluid/io.py", line 575, in load_vars
filename=filename)
File "/home/tianzhen/.local/lib/python3.7/site-packages/paddle/fluid/io.py", line 704, in load_persistables
filename=filename)
File "predict.py", line 76, in
fluid.io.load_persistables(exe, args.checkpoint_dir)
C++ Callstacks:
Cannot open file CKPT_DIR/@HUB_ernie-stable@encoder_layer_6_multi_head_att_key_fc.w_0 for load op at [/paddle/paddle/fluid/operators/load_op.h:37]
PaddlePaddle Call Stacks:
0 0x7fe8da2e6830p void paddle::platform::EnforceNotMet::Init<char const*>(char const*, char const*, int) + 352
1 0x7fe8da2e6ba9p paddle::platform::EnforceNotMet::EnforceNotMet(std::exception_ptr::exception_ptr, char const*, int) + 137
2 0x7fe8da8b65a6p paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const + 774
3 0x7fe8da8b6813p std::Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CPUPlace, false, 0ul, paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, float>, paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, double>, paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, int>, paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, signed char>, paddle::operators::LoadOpKernel<paddle::platform::CPUDeviceContext, long> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::M_invoke(std::Any_data const&, paddle::framework::ExecutionContext const&) + 35
4 0x7fe8db492da6p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void
, boost::detail::variant::void
, boost::detail::variant::void
, boost::detail::variant::void
, boost::detail::variant::void
, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 662
5 0x7fe8db4938a4p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 292
6 0x7fe8db491a7bp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 267
7 0x7fe8da44c23ep paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 158
8 0x7fe8da44f21fp paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocatorstd::string > const&, bool) + 143
9 0x7fe8da2d74bep
10 0x7fe8da31620ep
11 0x5d8014p _PyMethodDef_RawFastCallKeywords + 548
12 0x5d9970p _PyObject_FastCallKeywords + 1744
13 0x54b0f1p
14 0x552197p _PyEval_EvalFrameDefault + 17495
15 0x54b9f2p _PyEval_EvalCodeWithName + 594
16 0x5d8c62p _PyFunction_FastCallKeywords + 1154
17 0x54af20p
18 0x54f0eep _PyEval_EvalFrameDefault + 5038
19 0x54b9f2p _PyEval_EvalCodeWithName + 594
20 0x5d8c62p _PyFunction_FastCallKeywords + 1154
21 0x54e3dcp _PyEval_EvalFrameDefault + 1692
22 0x54b9f2p _PyEval_EvalCodeWithName + 594
23 0x5d8c62p _PyFunction_FastCallKeywords + 1154
24 0x54af20p
25 0x54f0eep _PyEval_EvalFrameDefault + 5038
26 0x54b9f2p _PyEval_EvalCodeWithName + 594
27 0x5d8c62p _PyFunction_FastCallKeywords + 1154
28 0x54af20p
29 0x54f0eep _PyEval_EvalFrameDefault + 5038
30 0x54b9f2p _PyEval_EvalCodeWithName + 594
31 0x5d8c62p _PyFunction_FastCallKeywords + 1154
32 0x54af20p
33 0x552197p _PyEval_EvalFrameDefault + 17495
34 0x54b9f2p _PyEval_EvalCodeWithName + 594
35 0x54dd33p PyEval_EvalCode + 35
36 0x630f22p
37 0x630fd7p PyRun_FileExFlags + 151
38 0x631c3fp PyRun_SimpleFileExFlags + 383
39 0x65414ep
40 0x6544aep _Py_UnixMain + 46
41 0x7fe8f89eab6bp __libc_start_main + 235
42 0x5e04eap _start + 42

运行sequence_label的demo报错

EnforceNotMet: Input ShapeTensor cannot be found in Op reshape2 at [D:\1.5.1\release_cuda97\paddle\paddle\fluid\framework\op_desc.cc:306]

不清楚是为什么?尝试了很多办法都行不通。。。求助!!!

模型基准测试结果

您好,请问哪里可以看到模型基准测试结果呢?比如图像分类任务,我想知道通过paddle官方训练的某个模型的基准测试结果。在哪里可以看呢?

can't load module with: hub.Module(name="XXX")

in manager.py: name = search_result.get('name', None)

but the search_result does not return the "name" at all, 'cos in the hub_server.py:

            return {
                'url': self.resource_list_file['url'][index],
                'md5': self.resource_list_file['md5'][index],
                'version': version
            }

安装paddlehub后import卡住

您好!
我在linux系统上安装了paddlepaddle 1.5.1 和 paddlehub 1.0.2。但是我import paddlehub时会卡住,手动中断后显示卡在了connect某个socket一步。
请问这是否和 #69 是同样的错误?应如何解决?
4c07edf9cba683f0fbd0132aabdf756

运行流程问题请教

看官方给出的demo
在finetune_and_eval()之后,在预测的时候又把前面的所有流程走了一遍,最后这个finetune_and_val()改成了predict(), 那这时候predict使用的难道不是预训练的模型么。
看到代码中注释了一句finetune之后会自动保存,请问这个是保存到哪了呢?在使用新模型的时候还是写module = hub.Module(name="ernie")吗

paddlehub没有办法离线运行

训练机子不让上外网,我一个劲的docker save docker load 日的我不要不要的。已经折腾两周了,结果载入paddlehub还要求连外网。不连外网就用不了了吗?

!hub run senta_bilstm --input_text "今天难道不是个好日子么"

成功源码编译方式安装了paddlepaddle+CUDA 10+python 3.6之后,安装了paddlehub。简单测试下面的case,得到的结论是:
!hub run senta_bilstm --input_text "今天难道不是个好日子么"
[{'text': '今天难道不是个好日子么', 'sentiment_label': 0, 'sentiment_key': 'negative', 'positive_probs': 0.0669, 'negative_probs': 0.9331}]

比较尴尬。

建议修改 import paddlehub 访问网络的特性

因部署关系 ,希望能在运行时可以不依赖于网络是否通畅。

希望能解决以下两个场景使用问题:
1.有时候跑单元测试也会因为网络不好,造成测试用例失败。
2.在下载过相关的PaddleHub中的预训练模型和预置数据集,可以不依赖于网络运行。

squad数据集的模型怎么构造?

squad数据集的模型怎么构造?ernie的sequence_out输出后,我接了两层全连接,最后输出shape为[-1,2], 和label[23,28]计算cross_entropy,然后使用finetue_and_evl函数(“sequence_labeling”)训练,怎么报错? Invoke operator mul error.

C++部分的Enforce failed

您好,我在进行 PaddleHub LAC 的测试中出现了报错如下:

C++ Callstacks:
Enforce failed. Expected posix_memalign(&p, alignment, size) == 0, but received posix_memalign(&p, alignment, size):12 != 0:0.

因个人对程序本身和C++方面不太熟悉想向您请教问题出现的可能原因。
测试中使用了 Python 的 flask 框架做成了分词接口,在经过27w多次分词后发生的报错。
完整报错信息如下:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/site-packages/flask/compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functionsrule.endpoint
File "/root/PaddleLAC/lexical_analysis/infer_test.py", line 78, in tokenize
results = lac.lexical_analysis(data={"text": article_list}, batch_size=len(article_list))
File "/usr/local/lib/python3.6/site-packages/paddlehub/module/module.py", line 484, in call
return_numpy=False)
File "/usr/local/lib/python3.6/site-packages/paddle/fluid/executor.py", line 651, in run
use_program_cache=use_program_cache)
File "/usr/local/lib/python3.6/site-packages/paddle/fluid/executor.py", line 749, in run
exe.run(program.desc, scope, 0, True, True, fetch_var_name)
paddle.fluid.core_avx.EnforceNotMet: Invoke operator gru error.
Python Callstacks:
C++ Callstacks:
Enforce failed. Expected posix_memalign(&p, alignment, size) == 0, but received posix_memalign(&p, alignment, size):12 != 0:0.
Alloc 256333568 error! at [/paddle/paddle/fluid/memory/detail/system_allocator.cc:57]
PaddlePaddle Call Stacks:
0 0x2b13dc647b78p void paddle::platform::EnforceNotMet::Initstd::string(std::string, char const*, int) + 360
1 0x2b13dda11337p paddle::memory::detail::AlignedMalloc(unsigned long) + 295
2 0x2b13dda115f7p paddle::memory::detail::CPUAllocator::Alloc(unsigned long*, unsigned long) + 39
3 0x2b13dda0ea0ap paddle::memory::detail::BuddyAllocator::RefillPool(unsigned long) + 58
4 0x2b13dda0f21fp paddle::memory::detail::BuddyAllocator::Alloc(unsigned long) + 719
5 0x2b13dd9eff05p void* paddle::memory::legacy::Allocpaddle::platform::CPUPlace(paddle::platform::CPUPlace const&, unsigned long) + 181
6 0x2b13dd9f0c35p paddle::memory::allocation::LegacyAllocator::AllocateImpl(unsigned long) + 405
7 0x2b13dd9ec533p paddle::memory::allocation::AllocatorFacade::Alloc(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void
, boost::detail::variant::void
, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 227
8 0x2b13dd9ec72ep paddle::memory::allocation::AllocatorFacade::AllocShared(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 30
9 0x2b13dd9d066cp paddle::memory::AllocShared(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 44
10 0x2b13dd9c9554p paddle::framework::Tensor::mutable_data(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, paddle::framework::proto::VarType_Type, unsigned long) + 148
11 0x2b13dc8a983ap float* paddle::framework::Tensor::mutable_data(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, unsigned long) + 74
12 0x2b13dce35772p paddle::operators::GRUCPUKernel::BatchCompute(paddle::framework::ExecutionContext const&) const + 482
13 0x2b13dce37533p std::Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CPUPlace, false, 0ul, paddle::operators::GRUCPUKernel, paddle::operators::GRUCPUKernel >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::M_invoke(std::Any_data const&, paddle::framework::ExecutionContext const&) + 35
14 0x2b13dd98e547p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void
, boost::detail::variant::void
, boost::detail::variant::void
, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 375
15 0x2b13dd98ecb1p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 529
16 0x2b13dd98cb5bp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 267
17 0x2b13dc7c6c9ep paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 206
18 0x2b13dc7c9d1fp paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocatorstd::string > const&, bool) + 143
19 0x2b13dc639e9dp
20 0x2b13dc67697ep
21 0x4aadf7p _PyCFunction_FastCallKeywords + 775
22 0x541a15p
23 0x547191p _PyEval_EvalFrameDefault + 17665
24 0x5418c1p
25 0x541bbfp
26 0x54701dp _PyEval_EvalFrameDefault + 17293
27 0x5418c1p
28 0x541bbfp
29 0x54701dp _PyEval_EvalFrameDefault + 17293
30 0x5418c1p
31 0x54b116p _PyFunction_FastCallDict + 342
32 0x45224fp _PyObject_FastCallDict + 495
33 0x45234bp _PyObject_Call_Prepend + 203
34 0x452102p _PyObject_FastCallDict + 162
35 0x5a508ep
36 0x452102p _PyObject_FastCallDict + 162
37 0x452604p _PyObject_FastCallKeywords + 212
38 0x541a15p
39 0x54701dp _PyEval_EvalFrameDefault + 17293
40 0x5418c1p
41 0x5427bdp PyEval_EvalCodeEx + 109
42 0x482017p
43 0x451f20p PyObject_Call + 96
44 0x546f1cp _PyEval_EvalFrameDefault + 17036
45 0x540bb1p
46 0x541de7p
47 0x547191p _PyEval_EvalFrameDefault + 17665
48 0x540bb1p
49 0x541de7p
50 0x547191p _PyEval_EvalFrameDefault + 17665
51 0x540bb1p
52 0x541de7p
53 0x547191p _PyEval_EvalFrameDefault + 17665
54 0x540bb1p
55 0x54b1f6p _PyFunction_FastCallDict + 566
56 0x45224fp _PyObject_FastCallDict + 495
57 0x45234bp _PyObject_Call_Prepend + 203
58 0x451f20p PyObject_Call + 96
59 0x4c6093p
60 0x452102p _PyObject_FastCallDict + 162
61 0x541a15p
62 0x547191p _PyEval_EvalFrameDefault + 17665
63 0x5418c1p
64 0x54b116p _PyFunction_FastCallDict + 342
65 0x45224fp _PyObject_FastCallDict + 495
66 0x45234bp _PyObject_Call_Prepend + 203
67 0x451f20p PyObject_Call + 96
68 0x4c6093p
69 0x452102p _PyObject_FastCallDict + 162
70 0x541a15p
71 0x547191p _PyEval_EvalFrameDefault + 17665
72 0x540bb1p
73 0x541de7p
74 0x547191p _PyEval_EvalFrameDefault + 17665
75 0x475fecp _PyGen_Send + 124
76 0x549290p _PyEval_EvalFrameDefault + 26112
77 0x475fecp _PyGen_Send + 124
78 0x2b13b74db5bfp
79 0x451f20p PyObject_Call + 96
80 0x546f1cp _PyEval_EvalFrameDefault + 17036
81 0x540bb1p
82 0x541de7p
83 0x547191p _PyEval_EvalFrameDefault + 17665
84 0x540bb1p
85 0x541de7p
86 0x547191p _PyEval_EvalFrameDefault + 17665
87 0x540bb1p
88 0x541de7p
89 0x547191p _PyEval_EvalFrameDefault + 17665
90 0x540bb1p
91 0x541de7p
92 0x547191p _PyEval_EvalFrameDefault + 17665
93 0x5418c1p
94 0x542700p PyEval_EvalCode + 96
95 0x426738p PyRun_FileExFlags + 360
96 0x42691dp PyRun_SimpleFileExFlags + 221
97 0x43bb9ap Py_Main + 3402
98 0x41d922p main + 354
99 0x2b13aaecab35p __libc_start_main + 245

LAC是否支持多线程?

用hub使用lac有几个问题,望解答:

  1. lac是否支持多线程?没看到接口有描述,在业务实现多线程,报string index out of range错误,只能改为多进程,才开始正常
  2. lac的代码实现在哪里?比如lexical_analysis这个api接口。
  3. lac的句法分析是用哪套API呢?
  4. 原程序使用了logger做日志输出,指定到了文件里面。引入hub,加载lac模块后,日志既输出到文件,还输出到了控制台,不知道怎么关掉。

训练和测试使用一批相同的数据,训练误差和测试误差不一致

训练和测试使用一批相同的数据,训练误差0.03左右,但是测试误差3.4左右。

def train_net(args, pad_num=20, num_classes=100):
from _8conv_baseline_char_fc_attdecode_small_open.cnn_ctc_model_senet_8conv_pfc_attdecode import encoder_net_attdecode_return_conv_features
images = fluid.layers.data(name='img', shape=[1,32,256], dtype='float32')
images = fluid.layers.cast(x= images , dtype='float32')
mask = fluid.layers.data(name='mask', shape=[20, 16*128], dtype='float32')
mask = fluid.layers.cast(x= mask , dtype='float32')
#fluid.layers.Print(images,summarize = 10,message="images")
conv_features= encoder_net_attdecode_return_conv_features(images,word_vector_dim-1,mask)#fluid.layers.data(name='conv_feature', shape=[20,64], dtype='float32')
print("conv_features is ",conv_features)
#content = fluid.layers.data(name='word', shape=[1], dtype='int32', lod_level=1)

pos = fluid.layers.data(name="pos", shape=[pad_num, 4],append_batch_size=False, dtype='float32')
pos = fluid.layers.cast(x= pos , dtype='float32')
_tmp_pos = fluid.layers.unsqueeze(pos, axes=[0])
_tmp_pos = fluid.layers.expand(_tmp_pos, expand_times=[pad_num, 1, 1])
_tmp_pos_transp = fluid.layers.transpose(_tmp_pos, perm=[1, 0, 2])
print("_tmp_pos is ",_tmp_pos)
print("_tmp_pos_transp is ",_tmp_pos_transp)
adjacency = _tmp_pos - _tmp_pos_transp #????# fluid.layers.data(name='adja', shape=[pad_num, pad_num, 6], append_batch_size=False, dtype='float32')

node_label = fluid.layers.data(name='node_label', shape=[pad_num, 1], append_batch_size=False, dtype='int32')
node_label = fluid.layers.cast(x=node_label, dtype='int64')

node_mask = fluid.layers.data(name='node_mask', shape=[pad_num, 1], append_batch_size=False, dtype='int32')
node_mask = fluid.layers.cast(x=node_mask, dtype='float32')

edge_label = fluid.layers.data(name='edge_label', shape=[pad_num, pad_num], append_batch_size=False, dtype='int32')
edge_label = fluid.layers.cast(x=edge_label, dtype='float32')

edge_balance = fluid.layers.data(name='edge_mask', shape=[pad_num, pad_num], append_batch_size=False, dtype='float32')
edge_balance = fluid.layers.cast(x=edge_balance, dtype='float32')
tmp = fluid.layers.fill_constant([pad_num, pad_num], value=0.5, dtype='float32')
edge_mask = fluid.layers.less_than(tmp, edge_balance)
edge_mask = fluid.layers.cast(x=edge_mask, dtype='float32')

edge_out, node_out = encoder_net(conv_features, adjacency, node_mask, edge_mask, pad_num, num_classes)
print('edge out is ',edge_out)
print('node_out is ',node_out)
print(" ok  encoder  ")
node_cost = fluid.layers.softmax_with_cross_entropy(node_out, node_label)
node_cost = fluid.layers.reduce_sum(node_cost * node_mask)/fluid.layers.reduce_sum(node_mask)

#edge_cost = fluid.layers.sigmoid_cross_entropy_with_logits(edge_out, edge_label)
edge_cost = focal_loss_sigmod(edge_out, edge_label)
edge_cost = fluid.layers.reduce_sum(edge_cost * edge_balance)
edge_cost *= args.lamb

sum_cost = node_cost + edge_cost

node_index = fluid.layers.argmax(node_out, axis=1)
edge_score = fluid.layers.sigmoid(edge_out)

_**inference_program = fluid.default_main_program().clone(for_test=True)**_
if args.learning_rate_decay == "piecewise_decay":
    learning_rate = fluid.layers.piecewise_decay([args.stepvalue], [args.lr, args.lr * 0.1])
else:
    learning_rate = args.lr

optimizer = fluid.optimizer.Adam(learning_rate=learning_rate)
#optimizer = fluid.optimizer.Momentum(learning_rate=learning_rate, momentum=0.9)
_, params_grads = optimizer.minimize(sum_cost)
model_average = None
if args.average_window > 0:
    model_average = fluid.optimizer.ModelAverage(
        args.average_window,
        min_average_window=args.min_average_window,
        max_average_window=args.max_average_window)
return node_cost, edge_cost, inference_program, model_average, node_index, edge_score,sum_cost

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.