add/sub and mul/div are all slightly different, and they are kinda hard to read.

A device should be in charge of allocating tensors/arrays, and doing operations on them. This should resolve stack overflow from #16, and make adding gpu device (#9) easier in the future.

For safety & clarity reasons. If you clone a tensor for backprop, more often than not you want that to be a different tensor and for it to be treated separately during backprop.

For cases where you do want to keep the id the same, .duplicate() should be used.

The only place this really occurs is in kl_div_with_logits_loss where target_probs is cloned sicne it's used twice.

Needed for transformers #34

Ideally we'd have p be a const parameter. unfortunately f32 cannot be const in stable.

Many uses cases make p 1 / N, where N is just an integer.

Dropout1In<N> would set p to be 1.0 / N as f32 for now.

Current only works for actual probability distributions. hard cross entropy only has 1 non zero entry in inner dimension, so sum across that before taking mean

This is for #34 , and is also actually related to #33 (since 33 requires transpose matrix multiplication, which this issue does as well)

Related to #6

E.g. for xavier uniform initialization you need to know the in size & out size.

This will likely require a different trait than Randomize, and I'm still inclined to keep randomize. It'll also be slightly easier to use since the user won't have to pass in a distribution.

Options:

• model.reset_params(&mut rng);
• model.init_params(&mut rng);
• model.randomize_params(&mut rng);

This should use Tensor::randomize() under the hood.

This would reduce last dim to the maximum value in that dimension. It can use T::Device::reduce_last_dim(..., &mut f32::max) (see logsumexp for example using that).

Example:

let t: Tensor2D<2, 3> = Tensor2D::new([[1.0, 2.0, 3.0], [-1.0, -2.0, -3.0]]);
let r: Tensor1D<2> = max_last_dim(t);
assert_eq!(r.data(), &[3.0, -1.0]);

This will also slightly reduce the required movement of tape when using these functions.

Needed for transformers #34

Needed for #34 . In multi head attention, you concatenate the output of all the single attention heads.

Note that this may require nightly access similar to #1 because we can't do expressions with const generics yet.

This will be another generic parameter of all tensors. Most existing operations will likely require float generic.

Related to #9 since it involves an additional generic parameter

Something that takes a usize length of the dataset, and you can:

1. Sample batches of a const known size
2. Iterate shuffled batches of const known size

Each of these would return a [usize; M] where M is a const M: usize

This will need:

• Write single tensor to .npy file
• Create a zip with multiple files from a struct
• Ability to read a single np array from file into a tensor
• Ability to read a collection of np arrays into a arbitrarily nested struct of tensors

While this would force functions to allocate space for derivatives inside, it would be cleaner from an api perspective.

E.g.

fn matmul_ref(a: &..., b: &..., tape: H) {}

pytorch's adam page has psuedo code for this: https://pytorch.org/docs/stable/generated/torch.optim.Adam.html?highlight=adam#torch.optim.Adam

Since everywhere its used is with data the callee function owns for both. This will also allow it to reduce allocations for derivatives

something like

fn select<const S: usize>(self, inds: &[usize, N]) -> Self<S, ...>;

I imagine the gradients for this would just be 1 if i is in inds, otherwise 0

• Linear batched forward (matmul & broadcast add)
• Backprop algorithm
• Optimizer updates
• forward with tape & without tape

This function would stop using alloc_zeroed() and Box::from_raw() https://github.com/coreylowman/dfdx/blob/main/src/devices/allocate.rs#L11:

        let layout = Layout::new::<T>();
        debug_assert_eq!(layout.size(), T::NUM_BYTES);
        unsafe {
            let ptr = alloc_zeroed(layout) as *mut T;
            Box::from_raw(ptr)
        }

E.g.

• const generic sizes
• clean/safe/hackable implementation
• Gradients not stored directly on parameters, makes updating & writing optimizers much cleaner

It currently:

1. Copies rhs data
2. Creates an empty version of rhs to store derivative

It'd be nice to reuse one, but because of the order of operations it may be impossible; to compute lhs derivative you need rhs & result, and to compute rhs derivative you need lhs & result.

This would be variable sized head where the input to the module is duplicated and the same input is passed to all sub modules.

Unclear how this would work since we are already using tuples. Perhaps something like:

impl Module<I> for MultiHead<(A, B)> {}
impl Module<I> for MultiHead<(A, B, C)> {}
impl Module<I> for MultiHead<(A, B, C, D)> {}
...

?

This would accept an array of T::Reduced::ArrayType, where Dtype is usize, and select the items from last dimension that match up. It would return a Tensor::Reduced.

Example:

let t: Tensor2D<2, 3> = Tensor2D::new([[1.0, 2.0, 3.0], [-1.0, -2.0, -3.0]]);
let r: Tensor1D<2> = gather_last_dim(t, [0, 1]);
assert_eq!(r.data(), &[1.0, -2.0]);

Examples where this would remove an allocation:

1. kl_divergence_with_logits_loss where target_probs is duplicated
2. binary_cross_entropy_with_logits_loss in b calculation where max_value is duplicated.

One of the arguments can be reused as the storage for the gradient.

Currently using the matrixmultiply crate, but I think performance could be much improved with using the actual BLAS library. Unclear how compiling/including that works since it has to be compiled per machine.

There's a lot of work to be done here. Very rough list of todos:

• Preparation

  • Move map functions to devices #199
  • Move conv to devices #198
  • Add where clauses for map functions to make partial progress on kernels possible (so we can start using cuda without all ops implemented)

• Devices

  • Add Cuda device that wraps cudarc::CudaDevice and an rng
  • Add StdRng to Cpu
  • Add rng seed to device construction
  • Add two GATs to device trait: DeviceArc and DeviceRng
    • Add CpuRc which contains Arc<T> and Arc<Cpu>

• Tensors

  • Add Device to all tensor structs
  • TensorCreator should accept &Device as parameter, and remove Rng since that will be accessed through device
  • Move Device to generic argument of Tensors
  • Enable moving tensors between devices

• nn

  • Add trait ModuleCreator
    • Add ModuleCreator::zeros(Device)
    • Add ModuleCreator::default(Device) which calls zeros & reset params
  • Remove implementations for Default
  • Remove rng parameter from ResetParams, should use tensor's devices

• Kernels

  • Add trait LaunchKernel<K, Args>
  • Move all Cpu traits to a combo of impl LaunchKernel<...> for Cpu and trait <Kernel>CpuImpl/impl <Kernel>CpuImpl for <Kernel>. See cudarc/examples/kernels.rs
  • (In a separate crate) proc macro that wraps around kernels and maps them to something usable for ptx compiling (e.g. kernel!(|a, b, c| { *a = b + c }) (#185)
  • Look into when/how to build the kernels (compile time hopefully??) (#184)

• Testing

  • Add feature based device construction in all tests (something like #[cfg(feature="test-cuda"]) that when specified uses cuda instead of cpu?
  • Add macro build_test_device!() to use that uses testing features to create the device

Done:

• Is it even possible to compile a rust closure to a cuda kernel? Assuming very small set of supported operations. Is this worth the maintainability?
  • If we go the fixed set of functions route, how many different generic closures does dfdx use currently?
  • ANSWER: Yes it is possible (the rust cuda project does it), but it will take some work. Automatic closure conversion to kernel is probably the direction i'll be trying to go since hand building all the cuda kernels next to the cpu closures seems too much work.
• What functionality does nvidia provide for deep learning already? Assuming matmul & conv forward/backward. How to use these?
  • ANSWER: cudnn, all tensors are 4d, supports base set of operations. probably not what we want to depend on tbh since it doesn't support everything we would need on GPU (e.g. optimizer kernels)

pytorch's sgd page has pseudocode for this: https://pytorch.org/docs/stable/generated/torch.optim.SGD.html

0.9.0 - nightly conv nets & transformers

• #34
• #55
• #1
• #67
• #123
• #115
• #111
• #96

Comparison against pytorch (patch version bump)

• #20
• #22

Misc other generic const exprs functions (patch version bump)

• #43
• #14

Released v0.5.1 - Mnist example with linear MLP

• #19
• #23
• #24
• #27
• #26
• #7

Released v0.5.2 - RL examples & save/load

• #13
• #29
• #30
• #31
• #6
• #32

Released v0.6.0 - transformers prep & other additions

• #36
• (minor breaking change) #41
• #40
• #38
• (patch breaking change) #35
• #33
• #37
• #51
• (breaking) #44
• (breaking) #46
• (breaking) #47
• (breaking) #41
• #53

Would like to add an small example of using a transformer architecture. This will likely involve new features such as batch mat mul and maybe some others.