[QST] How to make Main loop fusion for floats about cutlass

[thakkarV]

@hwu36

[hwu36]

True fp32 is not supported by tensor core. Only tf32 can use tensor core. Do you want to convert fp32 to tf32 before the computation?

Do you want to support fprop or wgrad or anything else?

The inline ptx in scale_bias_relu_transform.h is hard coded for fp16x2, not for fp32. you don't have to write inline ptx, but just write cuda. Something like

if (input != special_nan) {  // we use a special nan to mark out of bound data.  we use 0x7eff for fp16 special nan.
  float res = input > float(0) ? input : input * leaky_alpha;
}

[satyabhagavan]

@hwu36 I implemented in the same way by defining leaky_alpha as float(0.1), by doing that I am getting nans at the output. Do I need to change the 'MmaElements' and 'MmaCols' in the scale_bias_relu_transform.h file in converting to floats.

[hwu36]

Yes. You'd better dump the value of matrix, bias, scale first to see if every thread owns the right data. You can use 1,2,3,4... to initialize a small matrix to do that.

Mainloop fusion is the most difficult one. If possible, you'd better do the fusion in the previous kernel epilogue, which is easier and has better performance.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.