clean data from failed job about lithops

[gilv]

to clean data for failed jobs we can always do

try:
   pw.map(func, ...)
   results = pw.get_result()
   print("Execution completed")
except Exception as e:
  print (e)
finally:    
  pw.clean()

But this need to be done automatically

[JosepSampe]

@gilv This is fixed in #122, as long as you have data_cleaner = True in config.
https://github.com/pywren/pywren-ibm-cloud/blob/master/pywren_ibm_cloud/wren.py#L318

[gilv]

@JosepSampe data_cleaner = True will delete all data from successfull and failed jobs.. I wonder if cleaning partial results of failed jobs is the same as cleaning data from successfull jobs. Is there somethign different we can do with data from failed and successfull jobs?
I think to generate graphs requires data cleaner to be false, right? this is why we have default data_cleaner = false. Am i correct?

[JosepSampe]

@gilv I don't know why default data_cleaner = false... In fact it can be true since plots can be created anyway. Only in that case you use rabbitmq you need data_cleaner = False, becasue then you need to keep statuses in COS in order to print plots, since statuses are not downloaded trough rabbitmq. But rabbitmq is not the default usage of PyWren, so I think we can move default data_cleaner = True.

Also, I think there is nothing different we can do with failed jobs. PyWren raises the exception and then data_cleaner can be automatically executed.

[gilv]

@JosepSampe so should we make the default data_cleaner=True ?

[JosepSampe]

@gilv I think so

[JosepSampe]

@gilv I think this one can be already close with #233 . Now the data_cleaner() is always executed from any failed job