locals() does not include local variables set in exec() statement in 3.13b2 about cpython

[gaogaotiantian]

This is due to PEP 667, but the part that matters is not the print(locals()), it's the exec() part. This is a dup of #118888 and you can check the detailed discussion there.

In short - we never encourage the users to use the side effects on the current scope with exec(), we explicitly told users not to. This is a breaking change but it's intentional and technically did not break any of our promise (it changed a long existing undefined behavior). The latest 3.13 docs lists this out in exec section.

[JelleZijlstra]

Confirmed this change:

% python ~/py/tmp/execbug.py  # 3.12
{'sys': <module 'sys' (built-in)>, 'b': 2}
% ./python.exe ~/py/tmp/execbug.py  # 3.14 (main)
{}

Possibly related to PEP-668; cc @markshannon @gaogaotiantian. Note that the backward compatibility section of the PEP says that certain corner cases may change: https://peps.python.org/pep-0667/#id1.

[nineteendo]

This is a dup of #118888

Can you close this then?

[gaogaotiantian]

Let me know if you have more questions @jpe .

[jpe]

I think I see what's going on and can work around it (I've already rewritten the code where I first ran into it).

I'm not convinced that it's ok for exec() to work with a copy of locals because there's probably a fair amount of code that relies on the changes being propagated to the frame locals. The following seems to work, though will it fail if the function is optimized?

import sys
imp = 'import sys'
assign = 'b = 2'
def f():
    frame = sys._getframe()
    exec(imp, globals(), frame.f_locals)
    exec(assign, globals(), frame.f_locals)
    print(locals())
f()

[gaogaotiantian]

The function is optimized in your example. This should work.

As for the code relying on this, yes there will be some. However, that's a fragile behavior from the beginning and we have warned the users not to use it for a long time.

modifications to the default locals dictionary should not be attempted.

This is an explicitly discouraged behavior and it should not be a big surprise that it stopped working one day. The current implementation provides a clearer boundary and semantics.

[jpe]

I see; thanks

[gaogaotiantian]

Notice that

assign1 = 'a = 2'
assign2 = 'b = 2'
def f():
    a = 1
    exec(assign1)
    exec(assign2)
    print(locals())
f()

will print {'a': 1, 'b': 2} in the previous versions, which is super confusing. If you do frame.f_locals, it will print {'a': 2, 'b': 2} which is still a different behavior. It's impossible to keep full backwards compatibility (because it just does not make sense), so might as well make it easy to explain.

[gaogaotiantian]

Actually, let me discuss this with more core devs. We'll see if this is the best behavior. It won't be the same as before - that's for sure, but maybe the frame.f_locals is the more expected one.

[gaogaotiantian]

Okay I made a mistake here. exec() actually behaves exactly as before, but when you do print(locals()), 3.12 will update the existing locals dict and 3.13 will create a new dict based on the fast variables. You probably should not use the frame.f_locals in this case. Why are you importing sys with exec? It's not like that you can use that in the local scope (unless you keep passing the dict as locals).

[jpe]

The original code is not code that I think is particularly well written and may have worked by accident, but it did work all the way to 3.13. I've already changed it. The code was getting a reference to the imported module via locals() and then returned it. Again, this was't the best way to do it and I would not recommend writing it that way.

There's been at least 1 other bug report on this so this might not be a rarely used code pattern.

I guess I don't follow why locals() is only returning a dict with the fast variables. If extra variables are added to the frame, shouldn't they be included?

[gaogaotiantian]

If extra variables are added to the frame, shouldn't they be included?

It should, but no variable is added to the frame. When you do exec('b = 2'), it executes in a different frame and the variable is added to that local scope. A simple example, this won't work:

assign = 'b = 2'
def f():
    exec(assign)
    print(b)
f()

Because b is not added to the frame.

This is what actually happened on 3.13:

assign = 'b = 2'
def f():
    d = locals()
    exec(assign, globals(), d)
    exec('print(b)', globals(), d)
f()

b is added to the scope d above so locals() in f won't have it. Every locals() returns a new dict in 3.13 now, whereas in 3.12 they update and return the same dict so the code happened to work. In no cases, a new variable is added to the frame.

[jpe]

After some digging, it looks like the way to add extra variables to the frame is through frame.f_locals. We have other code that does that and I'm glad I won't need to change it. Is the fact that extra variables can be set documented anywhere? It's implied in PEP 667, but PEPs are proposals and not necessarily reference documentation. I'm hoping that the extra variable storage won't disappear from the frame in some future version, though we may be able to work around that if it were to happen.

[gaogaotiantian]

it looks like the way to add extra variables to the frame is through frame.f_locals

Technically, it's impossible to add extra variables in the optimized frames (function scope frames) in CPython. It's impossible to make the following code work even with f_locals.

import sys
def f():
    exec("a = 1", locals=sys._getframe().f_locals)
    print(a)
f()

You should not expect that there is a long-term support for the "extra variable"-like feature. What you might feel like the extra variable might be:

import sys
def f():
    exec("a = 1", locals=sys._getframe().f_locals)
    exec("print(a)", locals=sys._getframe().f_locals)
f()

This is a completely different story because both a = 1 and print(a) executes in a different frame with a customized local scope. You can achieve that with a self-defined scope like I mentioned above. That's always supported.

I'm not sure what you are trying to do (or have done with some workaround), but under the current mechanism of Python frames, really adding a variable is impossible.

[eryksun]

Why does FrameLocalsProxy support assigning extra 'locals' via __setitem__()? It doesn't support removing them via __delitem__(). What's the point of extra 'locals' that aren't actually local variables? With f_locals for an optimized scope, I'd expect to be able to change only the values of existing variables. It seems like it would send a clearer message if trying to set a new name in an instance of FrameLocalsProxy raised KeyError.

[gaogaotiantian]

Why does FrameLocalsProxy support assigning extra 'locals' via __setitem__()?

Because the debugger needs it, otherwise it would make the life of the developers of the debugger miserable. pdb will not work (as it is) for sure.

[jpe]

I maintain a debugger which sets & retrieves extra variables when users execute arbitrary code in a stack frame. I imagine this is what pdb does as well, though it's been years since I looked at pdb.

I think the extra variables in a frame were originally added way-back-when to support exec or from ... import *, maybe when fast locals were introduce (I've worked with the python core code since the 1.5.2 days). Now that exec no longer even appears to work in a function frame, I worry that the extra variables dict may be removed at some point.

[gaogaotiantian]

I maintain a debugger which sets & retrieves extra variables when users execute arbitrary code in a stack frame. I imagine this is what pdb does as well, though it's been years since I looked at pdb.

I think the extra variables in a frame were originally added way-back-when to support exec or from ... import *, maybe when fast locals were introduce (I've worked with the python core code since the 1.5.2 days). Now that exec no longer even appears to work in a function frame, I worry that the extra variables dict may be removed at some point.

So wildcard import (from ... import *) is not supported in function scope, that does not work. If you are familiar with the Python core code, then you'll know that the "extra variables" never really exist on the frame. They exist on an arbitrary scope, which is not the frame variables.

exec works in a function frame, the thing changed is the semantics of locals(). When you do exec(code), you are implicitly doing exec(code, globals(), locals()) - this did not change at all.

Again, your code is not executing in your function frame (I think that would be obvious as you are familiar with how CPython really works), it executes in a new frame with global scope globals() and local scope locals(). When you do assignments, you are creating/editing variables in the dict provided by locals() - which is, again, not the frame.

In 3.12 and before, that's a cached dict of the frame local variables. You can write anything to it, and in some cases (FastToLocals, LocalsToFast) it writes back to the frame fast variables or vice versa. Every call to locals() returns the same dict and that's why some of your code works. This is very obscure so we changed that.

Now locals() is a snapshot of the current frame locals which never writes back, and that's why your code stopped working.

frame.f_locals is the new thing the debugging should use, it's a proxy so it's write through. But, a new variable to frame.f_locals is still not an extra variable on the frame. It's a variable that can be accessed through frame.f_locals. Consider it belongs to the scope frame.f_locals, instead of the frame.

That being said, I don't think it's promised that this mechanism will live forever. The thing about working on a debugger is that you had to use some black magic that could be removed in the future. For now, I don't think there's any plan to remove the extra dict in frame.f_locals because pdb still relies on that. The thing I think can be guaranteed is, there will always be a way to execute custom user code from a debugger.

[jpe]

from ... import * once worked in function frames, class scopes, and other places -- I brought it up since I think it was one of the motivations for adding a dictionary to the frame object in addition to the fast locals array.

You are correct that when using exec(), a new frame is created. It is the frame object that I was referring to; the frame object is what contains the "PyObject *f_extra_locals;" field.

Anyway, I think my questions are pretty much answered. Thanks for your help.

[gaogaotiantian]

You are correct that when using exec(), a new frame is created. It is the frame object that I was referring to; the frame object is what contains the "PyObject *f_extra_locals;" field.

Not really. f_extra_locals is actually in the original frame. The newly created is an unoptimized frame, which means it does not have fast variables - it'll be like executing in a class definition. The code writes directly to the scope provided by locals argument in exec(). In the debuggers case, it could be frame.f_locals which is the proxy for the original frame.

I hope that I answered all of your questions.

[eryksun]

Because the debugger needs it, otherwise it would make the life of the developers of the debugger miserable.

I suppose the alternative would be to operate on a copy of f_locals. After each exec(), update the local variables in f_locals from their current values in the copy. I see how it's more convenient to simply let the FrameLocalsProxy instance support extra locals for such cases. Though I still think it invites misunderstanding.

[jpe]

Yes, with

f = sys._getframe()
exec(code, globals(), f.f_locals)

a new frame (and c frame object) is created, but the locals (both the fast locals and f_extra_locals) are retrieved from and stored in the frame object f. It's likely that the frame that the code is exec'd in has no local variables stored.

I think I confused the issue by switching from talking about the exec(" ") with no globals or locals arguments use case to the use case of running arbitrary code is a given stack frame in a debugger (or at least appearing to by using the locals from the frame).

Thanks again.

[jpe]

@eryksun Yes, the workaround for a debugger if support for f_extra_locals is dropped from the frame object is for the debugger to essentially manage the extra locals, though this complicates thing a bit -- the debugger probably wants to treat functions differently than module scopes. I'm fairly sure there were other uses for f_extra_locals in older python versions (from ... import *, exec, maybe others) so it wasn't only for debuggers.

[gaogaotiantian]

I'm fairly sure there were other uses for f_extra_locals in older python versions (from ... import *, exec, maybe others) so it wasn't only for debuggers.

I hightly doubt that considering f_extra_lcoals was added a month ago with PEP 667.

[jpe]

There has been support for storing the equivalent of extra locals back as far as I can remember. The dictionary may have been moved from somewhere else recently.

[gaogaotiantian]

I suppose the alternative would be to operate on a copy of f_locals. After each exec(), update the local variables in f_locals from their current values in the copy. I see how it's more convenient to simply let the FrameLocalsProxy instance support extra locals for such cases. Though I still think it invites misunderstanding.

This is basically exactly how it works before PEP 667 (converting fast locals to dict and vice versa), and that's a pretty bad experience.

[gaogaotiantian]

There has been support for storing the equivalent of extra locals back as far as I can remember. The dictionary may have been moved from somewhere else recently.

The extra locals were stored in f_locals with the "locals" converted from fast variables. They were not stored separately. f_locals was a cached dict after FastToLocals and code that executes on the frame never takes a look at it.

[jpe]

The extra locals were stored in f_locals with the "locals" converted from fast variables. They were not stored separately. f_locals was a cached dict after FastToLocals and code that executes on the frame never takes a look at it.

Yes, that sounds right. So the extra locals were in f_locals along with the value of fast locals as of the last FastToLocals call. I just checked and in python 2.7, the following worked:

def f():
  exec('b = 2')
  print b

Note that I'm not trying to argue that this should work now or that it's good code, only that there once were other uses for the equivalent of extra locals.

[eryksun]

Why does FrameLocalsProxy support assigning extra 'locals' via __setitem__()? It doesn't support removing them via __delitem__()

Why doesn't FrameLocalsProxy support __delitem__()? One can del a variable in an optimized scope. Additionally, one should be able to del any extra local variable that was added. As is, an exec() in a FrameLocalsProxy locals can nonsensically raise NameError.

>>> def f():
...     x = 42
...     exec('del x', locals=sys._getframe().f_locals)
...     
>>> f()
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    f()
    ~^^
  File "<python-input-2>", line 3, in f
    exec('del x', locals=sys._getframe().f_locals)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 1, in <module>
NameError: name 'x' is not defined

>>> def f():
...     exec('x = 42; del x', locals=sys._getframe().f_locals)
...     
>>> f()
Traceback (most recent call last):
  File "<python-input-6>", line 1, in <module>
    f()
    ~^^
  File "<python-input-5>", line 2, in f
    exec('x = 42; del x', locals=sys._getframe().f_locals)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 1, in <module>
NameError: name 'x' is not defined

[jpe]

The inability to del is a change in behavior from prior versions. The following raises an UnboundLocalError. on the print() line in 3.11, sets n to None in 3.12, and raises a NameError on the exec() line in 3.13:

import ctypes
import sys

PyFrame_LocalsToFast = ctypes.pythonapi.PyFrame_LocalsToFast
PyFrame_LocalsToFast.restype = ctypes.c_int
PyFrame_LocalsToFast.argtypes = [ctypes.py_object, ctypes.c_int]

def func():
    frame = sys._getframe()
    n = 1
    exec('del n', globals(), frame.f_locals)
    PyFrame_LocalsToFast(frame, 1)
    print(n)
    
func()

Should I open a new bug for this?

[gaogaotiantian]

Should I open a new bug for this?

No this is intentional.

First of all, frame.f_locals returns a new type. By definition, it won't keep everything backwards compatible. So anything involving frame.f_locals that behaves differently from previous versions does not become a bug automatically.

PyFrame_LocalsToFast is a no-op now BTW.

The reason we chose to disallow users to delete from frame.f_locals is that the compiler wouldn't know it.

The difference between 3.11 and 3.12 is that we don't check if the local value is NULL anymore in LOAD_FAST, so we can't set any local variable to NULL arbitrarily. The solution is to set them to None, but that's just not correct. It's very inconsistent - how could del n set n to None?

So for the new proxy, we simply disallow users to delete variables, because there's no reasonable result coming out of it. Maybe deleting "extra variable" would make sense, because they are not there anyway. But at least for now, for consistency, we simply don't allow that - so the users don't need to care about whether the variable is "extra".

[eryksun]

The difference between 3.11 and 3.12 is that we don't check if the local value is NULL anymore in LOAD_FAST, so we can't set any local variable to NULL arbitrarily.

Sigh. 😞

[jpe]

I understand. Thanks for the explanation.