Comments (3)
Let me first elaborate a bit on the cuTENSOR side of things. As far as I understand, cuTENSOR allows for a dynamic way of changing the floating point accuracy, which is allowed to be of higher precision then the output array. The exact details are mostly mentioned in the docs page you linked, but here are some important notes:
- the specified accuracy is applicable to intermediate results. For example, when doing matrix multiplication, you get expressions of the form
a_11 * b_11 + a_12 * b_21 + ...
, where the specified accuracy will be used for the multiplications and summations. This means thata_11*b11
will be computed as the specified accuracy, even though the final result may still evolve some truncation if the output array is of lower accuracy. - the specified accuracy is a lower bound, i.e. if you specify
Float32
, cuTENSOR may decide to increase the accuracy toFloat64
if all input/output arrays and scalars support this. However, it will never decrease it toFloat16
.
Over to the TensorOperations side of things, in principle we do not expose the additional option to have increased intermediate accuracy when the output array is of lower accuracy. TensorOperations (not only for CuArrays) chooses an output type that has the precision based on promoting the type of all input arrays, and this is also the guarantee we then ask of cuTENSOR.
Finally, if you are using Julia, be careful with the conversion from Array
to CuArray
. cu(A)
is defined to silently change the precision to Float32
from whatever it was before, as this is typically the most optimized precision for your GPU, but this may not necessarily be what you want, in which case you should CuArray(A)
.
Thus, answering your first question, cuTENSOR will in general not lower the floating point accuracy, but it may increase it if all input/output arrays and scalars support this. TensorOperations only asks the guaranteed precision of the output array from cuTENSOR, which is determined in function of the inputs.
For your second question, this is rather application-dependent. In generic cases, there really is no way of knowing if 1e-2
is actually a floating point effect or rather a value being zero. This is just inherent to working with floating point numbers. As a pathological example, the following holds:
julia> 1e-1 + 1e20 - 1e20 - 1e-1
-0.1
Nevertheless, when dealing with addition and multiplication only, this mostly occurs when your input floats are of vastly different scales. This is precisely what eps
will tell you, it is the smallest number you can add which will still be representable, thus any precision smaller than this is lost. If your use-case has some bounds on the scales of input floats, you could possibly exclude these pathological cases, and decide that anything larger than some predefined limit is definitely not zero, and possibly anything smaller than this is actually zero. Often people use sqrt(eps)
or eps^(3/4)
for this, but this is mostly phenomenological and you should probably experiment and see what works for you. However, unless you abolutely require this, you could just leave the ones that are probably zero and continue, as these should only contribute about as much as the floating point errors you are making to the final result.
from tensoroperations.jl.
I don't understand the question very well, but if you are working with Float32
, the expected precision is order 1e-7. Check:
julia> eps(Float32)
1.1920929f-7
Hence, entries of the arrays that should be zero, can easily end up to be order 1e-7 due to numerical precision.
from tensoroperations.jl.
Yeah I understand the signficant digits thing. I guess two questions I have:
- Does cuTENSOR ever change the floating point accuracy when calling cuTENSOR? There's some things on this docs page from CUDA that are unclear to me about the accuracy guarantees.
- Not a TensorOperations specific question necessarily. But is there a recommended way to differentiate between a value being zero and a value just being small when output by @tensor? I have values all the way from 1e-2 to 1e-15. Some of those definitely should just be zero but are not due to precision issues, but some should be non-zero and just small.
from tensoroperations.jl.
Related Issues (20)
- Certain contractions with matrices resulting in zeros HOT 2
- Uncontracted indices error when summing two contracted objects HOT 3
- `TensorOperations.jl` does not seem to support 32-bit Julia? HOT 7
- How can @tensor macro be used in variable length systems? HOT 2
- Does TensorOperations.jl not support Zygote.jl? HOT 1
- Cryptic `IndexError` messages HOT 2
- Specify output type HOT 4
- Update Cuda.jl to v4? HOT 5
- NCon cannot do partial trace? HOT 1
- Unexpected behavior when there is a global minus sign HOT 2
- Sums are not commutative HOT 2
- Question: @cutensor not defined HOT 5
- possible memory leak with metaprogramming
- Why drop caching Tensors? HOT 3
- Is TensorOperations able to take advantage of symmetry in the output? HOT 8
- Manual allocation strategy HOT 2
- Enable multithreads when doing the permutedims in the TTGT algorithms HOT 2
- Unexpected `DimensionMismatch` (v4.0.2 -> v4.0.3) HOT 3
- Wrong result with subnetworks with equal labels HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensoroperations.jl.