Add CudaBytes about cudarc

[coreylowman]

I'm envisioning a single workspace held in Cuda, that all conv ops shared. It would be re-allocated if a bigger workspace is needed. So after the first forward pass it would be the size of the biggest workspace, and then any subsequent passes wouldn't have to re-allocate

[M1ngXU]

what's the difference to CudaSlice?

[coreylowman]

Not much, only that it wouldn't have a type generic on it. Although now that I think about it, we probably don't need this. What we really need is a way to reinterpret a CudaSlice<T> into another type.

What i'd like to do for workspaces is store a CudaSlice<u8> on Cuda, and then grab a view of those bytes as another type.

So maybe something like:

let mut workspace: CudaSlice<u8> = ...;
let _: CudaView<'_, f32> = workspace.transmute()

[M1ngXU]

ah ok; why not directly store it as f32 (dtype)?

[coreylowman]

Because it needs to support f16, f32, and f64 dtypes, and the device won't necessarily know which one since it doesn't have a dtype generic on it (and I'd prefer not to add it).

[M1ngXU]

Because it needs to support f16, f32, and f64 dtypes, and the device won't necessarily know which one since it doesn't have a dtype generic on it (and I'd prefer not to add it).

but doesn’t e.g. conv2d has a dtype? and the workspace would have the same dtype, wouldn’t it? or where do you plan to use it

[coreylowman]

coreylowman/dfdx@main...workspace

[M1ngXU]

hmm, ok, so everything runs on a single stream/one cudnn handle?