Unpickling Exceptions with keyword arguments in multiprocessing throws an error about cpython

[sobolevn]

Welcome 👋
And thanks a lot for the detailed bug report and reproducer.

I suspect that this might be a very tricky case to solve.

[maksbotan]

Actually, the reproducer can be simplified even further. Error is triggered without multiprocessing:

import pickle

class Foo(Exception):
    foo: int
    bar: str

    def __init__(self, foo, bar):
        self.foo = foo
        self.bar = bar

if __name__ == '__main__':
    p = pickle.dumps(Foo(foo=42, bar="a"))
    f = pickle.loads(p)
    print(f)

Traceback:

Traceback (most recent call last):
  File "/data/check_mp.py", line 22, in <module>
    f = pickle.loads(p)
TypeError: Foo.__init__() missing 2 required positional arguments: 'foo' and 'bar'

Removing Exception or using positional arguments fixes the problem.

[maksbotan]

It's documented in the pickle docs that classes which require keyword arguments for their constructor must implement __getnewargs_ex__ to work with pickle: https://docs.python.org/3/library/pickle.html#object.__getnewargs_ex__.

Perhaps we can improve the docs and/or the error message, however.

First of all, how am I supposed to discover that if I'm using multiprocessing? I forgot that it uses pickle under the hood.

Second, docs talk about __new__:

You should implement this method if the __new__() method of your class requires keyword-only arguments.

My class does not have __new__ explicitly defined, so reading the docs I do not assume that this sentence is relevant to my case 🤔

[AlexWaygood]

It's documented in the pickle docs that classes which require keyword arguments for their constructor must implement __getnewargs_ex__ to work with pickle: https://docs.python.org/3/library/pickle.html#object.__getnewargs_ex__.

Perhaps we can improve the docs and/or the error message, however.

[AlexWaygood]

Oh, but your constructor doesn't require keyword arguments. You just use keyword arguments. Hm.

[picnixz]

Possibly related: #89275 and #89275 (comment).

I don't have a setup to check the current implementation now, but maybe Irit's comment is still valid:

The default __reduce__ method of Exception returns the arg you pass to the Exception constructor.

So it might happen that keyword arguments are simply ignored (or not remembered as such?).

[Eclips4]

Yes, the keyword arguments are simply ignored.
BaseException.args - what is it? Doc says: "The tuple of arguments given to the exception constructor.", However:

class Foo(Exception):
    def __init__(self, x, y):
        self.x = x
        self.y = y


a = Foo(1, y=2)
print(a.args)

> (1,)

So, should we also have a BaseException.kwargs or BaseException.args should include all passed arguments: positional, positional-only, keyword, keyword only?

[picnixz]

Alternatively, I'd suggest changing the docs to handle those cases, by saying something like "if your custom class exception supports keyword-arguments in the constructor (required or not), then you must implement a custom reduction". That's one way to easily solve the issue (and say that .args are only the positional arguments (at the moment of the call, not in the signature)).

Or your idea of having .kwargs would also make sense (so that .args and .kwargs would reflect the call). But I'm not sure if this could break something.

[Eclips4]

Alternatively, I'd suggest changing the docs to handle those cases, by saying something like "if your custom class exception supports keyword-arguments in the constructor (required or not), then you must implement a custom reduction". That's one way to easily solve the issue (and say that .args are only the positional arguments (at the moment of the call, not in the signature)).

We still need to update the existing exceptions, like ImportError:

>>> a = ImportError(name="foo")
>>> a.args
()
>>> b = ImportError("foo")
>>> b.args
('foo',)
>>>

[Eclips4]

So, we need to decide whether we should add a BaseException.kwargs and support it or, easier and preferable (in my opinion), state this behavior in the docs and adapt existing exceptions to work well with the existing behavior.

[picnixz]

Formally speaking, "The tuple of arguments given to the exception constructor" is correct since "argument" is not used when talking about a signature (where we talk about parameters), so it should indeed refer to what was given to the constructor call (thought we could specify that args only store the positional arguments of the call).

I also prefer first changing the docs before introducing something else.

[tyshchuk]

I encountered this issue along with @maksbotan, and in my opinion, updating the Exception documentation doesn't really help here. My reasoning is as follows:

The error message we received from pickle indicated that some constructor lacked certain arguments but did not specify why. In our real-world scenario, the code was more complex than the simplified example we provided, so initially, I had no idea what was causing the problem. There is no situation where my first instinct would be to scrutinize the Exception documentation. For example, we suspected that the dataclass decorator (which our Exceptions also had) might be causing the issue, so we spent some time refactoring it. After running some tests, we saw that this approach didn't help. We went through several other guesses similarly. Even if this note were in the documentation, it would still take us a considerable amount of time to find it.

In my opinion, this behavior is neither natural nor expected. Sure, there may be a deep technical explanation for why this happens, but it doesn't make sense to argue that this should happen. I would prefer that Exception-derived classes are not treated as exceptions (no pun intended) by the pickling mechanism, rather than simply having this noted somewhere in the documentation.