Coder Social home page Coder Social logo

Layer variable names about lbann HOT 2 CLOSED

llnl avatar llnl commented on August 25, 2024
Layer variable names

from lbann.

Comments (2)

timmoon10 avatar timmoon10 commented on August 25, 2024

I propose the following name replacements for the following matrices in lbann_layer.hpp:

Matrix Description Proposed Name
WB Weight-bias matrix m_weights
WB_D Gradient w.r.t. weight-bias matrix m_weights_gradient
Zs Output of forward pass linear transform m_preactivations
Acts Layer activations m_activations
Ds Error signal from "previous" layer m_prev_error_signal
Ds_temp Error signal to "next" layer m_error_signal

from lbann.

timmoon10 avatar timmoon10 commented on August 25, 2024

Honestly, I think we should split up fully-connected layers, activation layers, and dropout layers so that we don't have to deal with the awkward Zs/m_preactivations matrix. In my proposed scheme, every layer will have four matrices: m_prev_activations, m_activations, m_prev_error_signal, and m_error_signal. m_prev_activations and m_prev_error_signal will point to the data in the "previous" layer if the data distribution is the same and will make a copy otherwise (thus, they will usually take up little memory and will save us the cost of doing multiple data redistributions). The current organization of lbann_layer.hpp requires that we pass an optimizer as an input, which limits us to optimize just one matrix. I suggest that we pass an optimizer factory to the layer and instantiate the optimizer internally. This will be useful if we want multiple optimizers (e.g. for filters and biases in convolutional layers). Finally, I don't like having an extra row of ones on the activations (so every activation matrix has NumNeurons+1 rows), but I haven't thought of a clean way to deal with this.

from lbann.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.