cPickle.PicklingError: Can't pickle <type 'ellipsis'>: attribute lookup __builtin__.ellipsis failed about gensim

[Dieterbe]

I'm currently working around this issue by using http://jsonpickle.github.com/

diff --git a/src/gensim/utils.py b/src/gensim/utils.py
index 817f3b7..3d797a9 100644
--- a/src/gensim/utils.py
+++ b/src/gensim/utils.py
@@ -1,4 +1,4 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright (C) 2010 Radim Rehurek <[email protected]>
@@ -13,6 +13,7 @@ from __future__ import with_statement
 import logging
 import re
 import unicodedata
+import jsonpickle
 import cPickle
 import itertools
 from functools import wraps # for `synchronous` function lock
@@ -421,16 +422,24 @@ def chunkize(corpus, chunks, maxsize=0):
         for chunk in chunkize_serial(corpus, chunks):
             yield chunk


 def pickle(obj, fname, protocol=-1):
     """Pickle object `obj` to file `fname`."""
-    with open(fname, 'wb') as fout: # 'b' for binary, needed on Windows
-        cPickle.dump(obj, fout, protocol=protocol)
+    with open(fname, 'w') as fout:
+       fout.write(jsonpickle.encode(obj))


 def unpickle(fname):
     """Load pickled object from `fname`"""
-    return cPickle.load(open(fname, 'rb'))
+    with open(fname, 'r') as fin:
+       return jsonpickle.decode(fin.read())

[piskvorky]

Can jsonpickle handle very large objects (reasonably memory efficient during save/load)? Dedan had another issue with cPickle, see #31 , so perhaps completely switching from pickle to json would solve both at the same time...

[Dieterbe]

Radim, your question triggered this little experiment:
http://dieter.plaetinck.be/poor_mans_pickle_implementations_benchmark.html
I shall check out your numpy-based approach, it is probably better than my jsonpickle approach.

[piskvorky]

Nice! I like benchmarks :)

How about the standard json package? (simplejson in python <2.6)

[Dieterbe]

What do you mean? what about it?
the jsonpickle page says "The standard Python libraries for encoding Python into JSON, such as the stdlib’s json, simplejson, and demjson, can only handle Python primitives that have a direct JSON equivalent (e.g. dicts, lists, strings, ints, etc.). jsonpickle builds on top of these libraries"

http://jsonpickle.github.com/

[piskvorky]

Oh, I didn't know it builds on json. In that case its performance is prolly nearly identical, no need to test.

Btw I remembered reading about json speed on metaoptimize some time ago, I managed to googled it up: http://metaoptimize.com/blog/2009/03/22/fast-deserialization-in-python/

[Dieterbe]

Well, the article explicitly discourages using it because it's buggy and unmaintained.

[piskvorky]

?? It's part of the standard python library. You probably mean cjson.

[Dieterbe]

Yes, I meant cjson. Anyway I don't feel the need to test more things right now, as the numpy native persistency thing you did is probably best anyway. Or am I missing something?

[piskvorky]

For numpy arrays, I think you're right :) Numpy is also very actively developed/maintained, so there's a good chance potential bugs will be fixed quickly. The core numpy guys are very good engineers.