Random failing to associate CUDA stream issues? about tensorflow-nufft

[jmontalt]

I'm afraid I haven't seen this one before. I would start by tracking what is happening to fft_plan_. Did cufftPlanMany return an invalid plan? If so, why wasn't the error caught? Or was the plan valid at first but then we broke it? If so, where? Is this the first use of the plan?

Just throwing some ideas around. If you can reproduce it I'll also try to have a look.

[chaithyagr]

I seem to encounter this issue only in Graph mode!
@jmontalt do you expect anything to be in terms of execution cycles with graph vs eager execution?

[chaithyagr]

I guess we need to use the stream_executor wrapper to rightly use FFT.
It is available here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/stream_executor/fft.h
WDYT?

[jmontalt]

Hi @chaithyagr , I'm not sure where the problem originates, as I haven't had the chance to take a look yet. But yes, I think using the stream executor is the right thing to do. A while ago I considered doing just that, but in the end I decided against it simply because time is in short supply and things seemed to work anyway.

I think we should be able to pretty much replicate what the FFT ops do: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/fft_ops.cc