Unstable performance about cudart.jl

[timholy]

This is probably the exact same asynchronous/synchronous issue that explains #30---in other words, it's an artifact of how you're testing it.

[dfdx]

Indeed, adding device_synchronize() fixed differences in timings. However, I see that it also made computations on GPU 3 times slower:

# CPU 
 julia> @time for i=1:1000 gemm!('N', 'N', 1., A, Im, 0., A) end
   1.972331 seconds

# GPU
!julia> device_synchronize(); @time (for i=1:1000 CUBLAS.gemm!('N', 'N', 1., d_A, d_Im, 0., d_A) end; device_synchronize())
   5.733445 seconds (25.00 k allocations: 937.500 KB)

Is it something expected too?

[timholy]

Yes. (Not necessarily the precise numerical factor, but the general slowdown.) You're inhibiting the GPU from getting started on the next job as resources get freed from the last one.

[dfdx]

That makes sense. Thank you!