one_hot_encoding uses too much gpu memory about cudf

[sklam]

Upon initial investigation using nvidia-smi, I don't find any evidence that is one_hot_encoding overallocating.

[sklam]

This may be a issue with auto printing in ipython notebook (i.e. last line non-assignment line in cell). If the dataframe df appears on the last line of a cell:

In[1]: df

It will be printed and also stored in the Out dictionary. So, a later del df will not fully remove the dataframe from memory because Out has a reference to it. A del Out[1] will work here but it is tedious to deal with that in every cell. Another workaround is to explicitly print(df) or str(df) to avoid Out acquiring a reference on any large object.

[sklam]

The issue is most likely due to how gpu dataframes are used in the notebook. Closing.