Time of getting result increase significantly when result few MB about lithops

[JosepSampe]

@idoyehe looking at the plot, did you create it with the python notebook?

[idoyehe]

@JosepSampe yes, just took a screenshot for the graph so it looks a little bit blur...

[JosepSampe]

Are you running your app code from a python notebook or from a .py file?

I say this because if you use a .py file, I integrated into PyWren a method that generates the plots. So you can avoid some code and simply use this method. Read this: #50

Also, it is preferable to upgrade PyWren to last commit, as I slightly changed the way to retrieve results.

[JosepSampe]

@idoyehe so, when you have everything updated, can you paste here the log of the reduce function when result is 5 MB?

[idoyehe]

Are you running your app code from a python notebook or from a .py file?

I say this because if you use a .py file, I integrated into PyWren a method that generates the plots. So you can avoid some code and simply use this method. Read this: #50

Also, it is preferable to upgrade PyWren to last commit, as I slightly changed the way to retrieve results.

amazing just tried it it works great!

[idoyehe]

@JosepSampe
When returning an empty dictionary:
returned empty dict.txt

When returning the 5MB list (inside dictionary):
returned list indside dict.txt

just want to explain that this job use partioner on a file in COS so the log contains all invocations, for the reducer please look the last one....

[JosepSampe]

@idoyehe These logs are from partitioner. I need logs from reducer ;)
Activation ID: 2921787e26814dcea1787e2681edceee and 9ccdc448d766425f8dc448d766925f16

[idoyehe]

returned empty dict.txt

returned list indside dict.txt

[JosepSampe]

Ok... as I thought, this issue is related to cloudpickle. It takes 1 minute to pickle 5MB of data... that it is converted to 36MB of pickled information.

I was already aware that cloudpickle takes long to pickle MB information. I think @gilv already did tests on this.

As I don't have the code of your app and the related data in COS, you can try to modify (locally in your laptop) the code of PyWren, just one line, and test if there is some improvement or not.

Just comment this line: https://github.com/pywren/pywren-ibm-cloud/blob/master/pywren/pywren_ibm_cloud/action/jobrunner.py#L28

and add: import pickle

[gilv]

@idoyehe thanks for reporting this. @JosepSampe may be we move to pickle instead of cloudpickle?

[idoyehe]

Ok... as I thought, this issue is related to cloudpickle. It takes 1 minute to pickle 5MB of data... that it is converted to 36MB of pickled information.

I was already aware that cloudpickle takes long to pickle MB information. I think @gilv already did tests on this.

As I don't have the code of your app and the related data in COS, you can try to modify (locally in your laptop) the code of PyWren, just one line, and test if there is some improvement or not.

Just comment this line: https://github.com/pywren/pywren-ibm-cloud/blob/master/pywren/pywren_ibm_cloud/action/jobrunner.py#L28

and add: import pickle

@JosepSampe I did what you suggested change the line, reinstall PyWren, and redeploy again...
but I could see improvement: the runtime is ~47 sec when return the list and ~16 sec when returning an empty dict....

here the log file of the reducer:
returned_list_inside_dict.txt

by the way there is no other packages that you can use to that ? because pickle is not very good recommended from what I search online...

[JosepSampe]

@idoyehe Seems now pickle only takes 7 seconds (Instead of 1 minute from cloudpickle).I changed pickle package everywhere in the project, so when @gilv merge #52, try againg becasue you can get major improvement.

Also, in the return statement of your reduce function, you don't return futures, right?

[idoyehe]

@idoyehe Seems now pickle only takes 7 seconds (Instead of 1 minute from cloudpickle).I changed pickle package everywhere in the project, so when @gilv merge #52, try againg becasue you can get major improvement.

Also, in the return statement of your reduce function, you don't return futures, right?

@JosepSampe
Tooks me arround 44 sec without returning run_statuses and invoke_statuses and arround 50 sec with retruning them

[JosepSampe]

@idoyehe ok
Just wanted to understand why if you return 5MB of data, output.pickle file takes 56.9MB... You can see this at the end of the log.

[idoyehe]

@JosepSampe
if I execute this process locally on my computer and export the retuned list to a text file the text file size is arround 24 MB and the size of the python object is 4.8 MB (calculating it using sys.getsizeof(result_list))
Then I executed the code using pywren just returning my list (without run_statuses and invoke_statuses) and after reviewing the log of the reducer the output.pickle size is 52MB

Can we say the pickle overheads is 28 MB ? sounds like too much?

[JosepSampe]

In my experience, don't take sys.getsizeof into account as it does not return correct size number when using over dicts or lists, or complex objects.

Yep, seems the overhead of pickle is 28MB

[idoyehe]

@JosepSampe So it is more than 100% overhead is sounds way too much...

There is no other methods to preform that job?

[JosepSampe]

@idoyehe What are you pickling? becasue as stated here: https://stackoverflow.com/questions/31304006/why-is-there-a-large-overhead-in-pickling-numpy-arrays , pickling is just a binary copying. So if you store the list in a text file, it only stores the numbers, however, when pickling, it stores the complet object.

[idoyehe]

@JosepSampe
You right i found a way to make my object much more simple without array and the overhead reduced significantly and the runtime decrease also!

Thank you!