Python pickler unable to pickle object the native pickler can about cpython

[encukou]

Thank you @apmorton for the report, and @picnixz for the fix!

[picnixz]

I'd say it's not a bug because pickle._Pickler is private and should not be used directly. Why do you need the private one?

[aisk]

I'd say it's not a bug because pickle._Pickler is private and should not be used directly. Why do you need the private one?

pickle.Pickler may be provided by pickle._Pickler if the native extension _pickle is not presented:

cpython/Lib/pickle.py

Lines 1770 to 1784 in 42b25dd

	try:
	from _pickle import (
	PickleError,
	PicklingError,
	UnpicklingError,
	Pickler,
	Unpickler,
	dump,
	dumps,
	load,
	loads
	)
	except ImportError:
	Pickler, Unpickler = _Pickler, _Unpickler
	dump, dumps, load, loads = _dump, _dumps, _load, _loads

[picnixz]

Oh! yes, my bad! I'll hide my comment to reduce the noise!

[picnixz]

The ZeroCopyBytearray buffer appears to need the following implementation (it's actually in Lib/test/pickletester) but I confirmed that the issue remains but only if the buffer is empty (otherwise it's fine). The implementation is roughly

class ZeroCopyBytearray(bytearray):
    def __reduce_ex__(self, protocol):
        if protocol >= 5:
            return type(self)._reconstruct, (pickle.PickleBuffer(self),), None
        else:
            return type(self)._reconstruct, (bytes(self),)

    @classmethod
    def _reconstruct(cls, obj):
        with memoryview(obj) as m:
            obj = m.obj
            if type(obj) is cls:
                # Zero-copy
                return obj
            else:
                return cls(obj)

So, the __reduce_ex__ method makes more things than what you provided in your example, but it does not fix the issue at all.

EDIT: I'm investigating the issue in picnixz@a8c1464

[apmorton]

Correct, I minimized the example to only the relevant parts for reproduction.

The issue only happens if you use protocol 5 and end up with two objects having different IDs produce a PickleBuffer with the same underlying buffer ID.

[picnixz]

Yes, the issue is that this specific test case is not covered (I've added one and I'll try to fix it), which is why we did not see the issue (by specific case, I mean the two different instances that are empty, not that it still works if I use the same non-empty buffer values).