molssi / tensor-interfaces

A place to store information for the tensor discussions and possible specifications.

tensor-interfaces

A place to store information for the tensor discussions and possible specifications.

Terminology/Notation

Classifying and Specifying Tensor Operations

Interface Issues

Proposals

• Proposal 1 (Paul Springer)

  This is an initial proposal for a mixed-precision tensor contraction interface.

• Proposal 2a (Devin Matthews)

  Fixed-type tensor contraction interface using Einstein notation.

• Proposal 2b (Devin Matthews)

  Fixed-type tensor contraction interface using BLAS-like arguments.

tensor-interfaces's Issues

Question: Does it make sense to support 0-dimensional tensors (scalars)?

Question: Should tensor elements be required to be distinct?

Using the general stride layout, it is often assumed that the strides are set such that no two elements share the same location. In some cases, strides are also assumed to be positive, so that stride[i+1] >= stride[i]*shape[i] e.g.

But, allowing more flexible representations may make sense in some circumstances, e.g. one can broadcast a lower-dimensional tensor into a higher-dimensional one by adding modes with 0 strides. Of course, it would be very difficult to support writing to such tensors.

Low-level Tensor Interface I

This issue collects the status of the various discussion topics for defining a tensor interface that is:

• low-level
• dense
• shared-memory
• synchronous
• preferably CPU/GPU portable

As sub-issues are resolved, summaries will be added here.

Sub-issues:

• Terminology (#1)
• Tensor structure
  • Stride requirements/restrictions
  • Standard layouts
  • Alignment
  • Scalars as tensors
• Naming
  • Arguments
  • Functions
• Types
  • Integral arguments
  • Scalars
• Operations
  • Classification
  • Definition
  • Naming
  • Batching
• Threading
• Algorithm selection

Tensor terminology

Let's discuss and decide on some terminology to use consistently for dense tensors:

1. A tensor which is addressed by n indices is:

   • a: n-dimensional
   • b: n-mode
   • c: order-n
   • d: rank-n
   • e: n-way
   • f: n-adic
   • g: n-fold
   • h: n-index

2. The tensor A[ijkl] has four:

   • a: dimensions
   • b: modes
   • c: ranks
   • d: indices
   • e: ways
   • f: subspaces

3. The number of distinct values that i can have in A[ijkl] is the first:

   • a: length
   • b: extent
   • c: dimension

4. The ordered set of extents (3.) (one for each mode (2.)) is the:

   • a: shape
   • b: size
   • c: structure

5. i, j, k, and l in A[ijkl] are:

   • a: 0-based
   • b: 1-based

6. i in A[ijkl] is a(n):

   • a: index
   • b: label

7. The operation B[jkli] <- A[ijkl] is a:

   • a: tensor transpose
   • b: mode permutation
   • c: reshaping
   • d: shuffle

8. The operation B[jkli] <- A[ijkl] can be denoted in shorthand as:

   • a: (1,2,3,0) "comes-from"
   • b: (3,0,1,2) "goes-to"

9. In the operation C[ijkn] <- A[ijlmn] * B[lmkn], l and m are:

   • a: contracted
   • b: internal
   • c: bound

10. ...and i, j, k, and n are:

    • a: uncontracted
    • b: external
    • c: free

Aligned ISO/C++ standardization proposal

There is an ISO/C++ standard proposal in progress which is well-aligned with this effort.
https://github.com/ORNL/cpp-proposals-pub/tree/master/P0009