Comments (2)
I propose the following name replacements for the following matrices in lbann_layer.hpp:
Matrix | Description | Proposed Name |
---|---|---|
WB | Weight-bias matrix | m_weights |
WB_D | Gradient w.r.t. weight-bias matrix | m_weights_gradient |
Zs | Output of forward pass linear transform | m_preactivations |
Acts | Layer activations | m_activations |
Ds | Error signal from "previous" layer | m_prev_error_signal |
Ds_temp | Error signal to "next" layer | m_error_signal |
from lbann.
Honestly, I think we should split up fully-connected layers, activation layers, and dropout layers so that we don't have to deal with the awkward Zs/m_preactivations matrix. In my proposed scheme, every layer will have four matrices: m_prev_activations, m_activations, m_prev_error_signal, and m_error_signal. m_prev_activations and m_prev_error_signal will point to the data in the "previous" layer if the data distribution is the same and will make a copy otherwise (thus, they will usually take up little memory and will save us the cost of doing multiple data redistributions). The current organization of lbann_layer.hpp requires that we pass an optimizer as an input, which limits us to optimize just one matrix. I suggest that we pass an optimizer factory to the layer and instantiate the optimizer internally. This will be useful if we want multiple optimizers (e.g. for filters and biases in convolutional layers). Finally, I don't like having an extra row of ones on the activations (so every activation matrix has NumNeurons+1 rows), but I haven't thought of a clean way to deal with this.
from lbann.
Related Issues (20)
- Build Issue On Lassen HOT 9
- Nonconst reference to locked view buffer with in-place HOT 2
- Strong scalability of LBANN CosmoFlow
- LBANN WAE tests are failing
- gaussian_fill test failing on pascal
- ci_test/unit_tests/test_unit_inplace_distconv.py skipped on Corona
- ci_test/unit_tests/test_unit_layer_convolution_distconv.py is skipped on Tioga
- Data_* Cleanup
- PROBIES integration test lost files from vast
- test_unit_algo_ltfb_trunc_selection.py - fails intermittently HOT 1
- Old driver functionality
- Variability in ResNet integration test
- Spack development branch build issue HOT 1
- openmpi fork() issue with python datareader
- error: invalid operands to binary expression ('const lbann::callback::(anonymous namespace)::MemUsage' and 'const lbann::callback::(anonymous namespace)::MemUsage') HOT 5
- Multiple build errors: error: static_cast from 'const lbann::l2_weight_regularization *' to 'const lbann::objective_function_term *', which are not related by inheritance, is not allowed, etc HOT 4
- Weight demodulation HOT 9
- Potential bug in lbann.Scale HOT 8
- cuTENSOR error (CUTENSOR_STATUS_NOT_SUPPORTED) HOT 1
- Expired link in the readthedocs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lbann.