Comments (9)
I ran some performance comparisons of the elemental and block matrix layouts doing GEMMs on Catalyst. This used 32-bit floats and Elemental's default blocksize.
1 node:
mats1.pdf
2 nodes:
mats2.pdf
4 nodes:
mats4.pdf
16 nodes:
mats16.pdf
Once we get beyond small matrices, the block distribution is better. The exact cross-over point depends on the number of nodes. It probably makes sense to switch to it for fully-connected layers.
from lbann.
An issue with this: Several Elemental methods that the documentation lists as taking AbstractDistMatrix
arguments in fact are only implemented for ElementalMatrix
types and do not support BlockMatrix
types. Ones I have found this to be the case for:
Hadamard
Dot
HilbertSchmidt
(needed becauseDot
calls it)ColumnTwoNorms
from lbann.
from lbann.
Probably not too hard, especially if the functions don't actually need the Elemental layout. If that's the case, then they should just take AbstractDistMatrix
types.
We also need to update our Elemental extensions (ColumnSum
and ColumnMax
).
from lbann.
I implemented the above operations and will be making a pull request to Elemental soon.
However, Jack Poulson was skeptical about the block matrix distribution being better in El::Gemm (due to its implementation, plus the fact that even if we fixed the implementation, there would be little difference in performance). I updated my benchmark and re-ran some tests this morning, and I now find that the element distribution is ~60% faster for the largest matrices. I'm not sure what led to the performance we saw in the results above.
We may still get an improvement on convolutional kernels, I have not looked into that. Once the PR is in Elemental, we could test that end-to-end in LBANN.
from lbann.
Elemental as of commit 776b805f0131f39ceeec8943f19a7803fa950d43
now supports block versions of Hadamard
, Dot
, ColumnTwoNorms
, and ColumnMaxNorms
. (And has tests to confirm that they're correct.)
from lbann.
from lbann.
I believe that we can close this issue because the Elemental library has a very efficient implementation of the Element-wise matrix operation. Nikoli, can you please document your conversation with Jack so that we can put this to rest.
from lbann.
Can read this comment for full details, but basically: the GEMM in the elemental distribution still does local GEMMs using the BLAS-3 acceleration, and the only difference is a slight change in communication pattern (MPI_Allgather
instead of MPI_Bcast
).
The current Elemental implementation of the GEMM for block distributions actually internally converts from a block to an elemental distribution.
from lbann.
Related Issues (20)
- Build Issue On Lassen HOT 9
- Nonconst reference to locked view buffer with in-place HOT 2
- Strong scalability of LBANN CosmoFlow
- LBANN WAE tests are failing
- gaussian_fill test failing on pascal
- ci_test/unit_tests/test_unit_inplace_distconv.py skipped on Corona
- ci_test/unit_tests/test_unit_layer_convolution_distconv.py is skipped on Tioga
- Data_* Cleanup
- PROBIES integration test lost files from vast
- test_unit_algo_ltfb_trunc_selection.py - fails intermittently HOT 1
- Old driver functionality
- Variability in ResNet integration test
- Spack development branch build issue HOT 1
- openmpi fork() issue with python datareader
- error: invalid operands to binary expression ('const lbann::callback::(anonymous namespace)::MemUsage' and 'const lbann::callback::(anonymous namespace)::MemUsage') HOT 5
- Multiple build errors: error: static_cast from 'const lbann::l2_weight_regularization *' to 'const lbann::objective_function_term *', which are not related by inheritance, is not allowed, etc HOT 4
- Weight demodulation HOT 9
- Potential bug in lbann.Scale HOT 8
- cuTENSOR error (CUTENSOR_STATUS_NOT_SUPPORTED) HOT 1
- Expired link in the readthedocs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lbann.