`record`ing non-JSON'able objects about papermill

[rgbkrk]

Great question!

"arrays tend to be too big, so store them somewhere else and record the path to it"

That would be my recommendation and I'd like to make it a simple built-in if it makes sense across users. I think it requires some infrastructure as well as opinion. The stance I'd likely take inside Netflix as a paved path would be to record the array (or pandas dataframe) likely using Arrow to store it to HDFS or S3. Then the field we store in a papermill record is likely the full path to the record (which can be an S3 path, or an HDFS path). What object storage would you end up using?

How lossy would it be for you to use Pandas' JSON Table Schema + row oriented JSON option?

An alternative would be to add a step to pickle/serialize types that can't be stored as JSON in record before dumping them into the notebook.

As much as possible we want to keep the data types language agnostic (as there are R bindings for papermill as well). I think it would be ok to serialize them, so long as you're the one expecting it on the other side of reading them (where you == you and your collaborators).

[rgbkrk]

@charsmith -- any thoughts here?

[mpacer]

If you are going the pickle route, it's probably better to use dill.

[rgbkrk]

Shameless plug - have you tried cloudpickle?

[lukasheinrich]

as mentioned in my PR #56 there is some usecase of persisting the data that is recorded.. right not it assumes it's jsonable via panda's to_json() but perhaps other persistency approaches could be made pluggable

[betatim]

I think pickle (and friends) aren't the right way to go because you'd want to be able to load the data in R (and maybe other languages).

There are docs on how to write pandas dataframes to files but not simple numpy arrays. This is nice because you don't have to use an object store.

How lossy would it be for you to use Pandas' JSON Table Schema + row oriented JSON option?

The usecase I have in mind is storing scikit-learn classifiers which are mostly (very) big numpy arrays. You could turn them into a dataframe first ... having said that scikit-learn classifiers mean nothing to R so using a python only storage format might be Ok.

Maybe the common denominator here is: a method for recording a large "bunch of bytes" which records the path where they were stored in the papermill metadata.

[rgbkrk]

a method for recording a large "bunch of bytes" which records the path where they were stored in the papermill metadata.

That seems reasonable, even if I don't see what the right way to do that would be.