Coder Social home page Coder Social logo

怎样不冻结bert参数？ about bertner HOT 5 CLOSED

VioletJKI avatar VioletJKI commented on September 28, 2024
怎样不冻结bert参数？

from bertner.

Comments (5)

yumath avatar yumath commented on September 28, 2024

@VioletJKI 模型中已经冻结了参数

from bertner.

airship-explorer avatar airship-explorer commented on September 28, 2024

我把这儿 grads = tf.gradients(self.loss, train_vars) 的train_vars改成了tvars，但是出来的结果全是0

我采取相同的方式想更新BERT参数，出来结果也全是0，请问你解决了吗？

from bertner.

yumath avatar yumath commented on September 28, 2024

@Guohai93 确实是这样，我的理解是bert的参数过多，无法在如此小的NER数据集上收敛。你如果需要更新bert参数的话，建议先对bert参数在自己数据集上进行训练，再拿来做NER

from bertner.

airship-explorer avatar airship-explorer commented on September 28, 2024

@Guohai93 确实是这样，我的理解是bert的参数过多，无法在如此小的NER数据集上收敛。你如果需要更新bert参数的话，建议先对bert参数在自己数据集上进行训练，再拿来做NER

做了一点实验，发现采用训练LSTM的1e-3学习速率太大了，导致底层预训练的BERT被破坏了，如果采用小的学习速率比如1e-4或更小，结果不再是0了。

from bertner.

yumath avatar yumath commented on September 28, 2024

@Guohai93 👍

from bertner.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.