ximport improvements (and possibly import) about shoebot

[karstenw]

I'm currently experiencing a cold so my thought might be squishy...

ximport should allow for .bot files and distingish between .py and .bot.

if extension == .bot
add SB/NB functions

Perhaps we run into another problem of compiled libs. Are .bot files compiled and if yes, how?

[stuaxo]

There's no compilation of .bot files (beyond py->pyc, and I'd need to double check that).

I need to think about the other bit; it sounds reasonable ?

Apart from .bot extensions, it makes me wonder if we should support shebangs and have that as a way of telling something is a nodebox/shoebot file, and what they'd look like, for instance

#!/bin/env nodebox

#!/bin/env sbot

[karstenw]

I've just looked at the objects in Context.namespace, all the commands, constants etc that are loaded into the script namespace. That's about 230 entries. About 150 if I ignore the special "_" stuff.

In my opinion that's too much to include in a lib automatically.

namespace-full.txt

[stuaxo]

Started building a test for this + realised loading .bot files + fixing up the namespace should probably be implemented with importlib + a finder and loader.

[karstenw]

In my opinion that's too much to include in a lib automatically.

But there should be an option to deliberately do so.

[stuaxo]

I wonder if it's possible to add them lazily ?

Ultimately it's something like dict.update() once a frame so it may not be the worst.

I'll experiment with it in shoebot, with .py import at first (since that doesn't need extra work compared to .bot to import), and benchmark.

import in python can be a little heavy anyway, let's see if this adds anything noticeable.

[stuaxo]

Realised a lot of that won't be needed, it should matter of passing the right dict to the importer.

[stuaxo]

Made a rough proof of concept -

#441

This uses a finder / loader - but would need some issues resolved

• caching the module in sys.modules needs to be implemented
• no pyc compilation happens; this is the same for shoebot code itself, where exec(...) is used.

[karstenw]

I am new to importlib so I used the smallest change that would make it run.

def ximport(self, libName):
    lib = importlib.__import__( libName )
    for name in self._ns:
        if name[0] != '_':
            lib.__dict__[name] = self._ns[name]
    self._ns[libName] = lib
    lib._ctx = self
    return lib

If that proves useful, I'll add a param loadContext=False to default to old behaviour. For testing I copied the Supershape lib and removed the "_ctx.".

[karstenw]

Problems:

• libraries defining functions with names from the namespace. Currently these get overwritten.
• checking if a name already exists collides with module caching on following scripts.

[stuaxo]

Doesn't the library get exec'd during import though ?

So may calls to the API in the body of the module it will fail I think?

I thought that passing globals() to import would help, but it doesn't - the importer just uses that to grab __path__ and a couple of other variables.

You might be able to use __dict__.setdefault to work around the issue of overwriting.

[karstenw]

I filtered for names. That's only half the work. Must check for the right instance of Context and before copying check the target lib for conflicts.

[karstenw]

Doesn't the library get exec'd during import though ?

That step happens probably in importlib.import()

[stuaxo]

Yep...

I guess it means that you can't have a library with this fill and friends in the body, since that would fail during the __import__ line ?

I wonder if some of this would go away if we had a step that transformed the code in the same way as processing does ?
https://happycoding.io/tutorials/java/processing-in-java#processing-is-java

Under the hood adding imports and changing fill etc to context.fill.

[karstenw]

I guess it means that you can't have a library with this fill and friends in the body, since that would fail during the __import__ line ?

It does not fail. Since the lib is already imported, copying the attributes overwrites the lib.fill with Context.fill.

I think the problem is that I try to manipulate attributes in a highly caching environment.

I am looking for a solution that does not involve storing namelists from first import to distinguish between imported and attribute copied names in the lib.

Perhaps my first idea is better: from nodebox.bowels import line,transform,path.

[stuaxo]

The first idea is definitely the most sensible one, in the context of ordinary python.

I think something is wrong in my assumptions, I need to poke this with a debugger, I thought __import__ execs everything in the body, so fill and friends don't exist yet.

Though - to be fair, maybe there are enough differences in shoebot + nodebox to make this tricky.

I'm not in front of my own computer right now so can't test anuthing.

[karstenw]

After some contemplating:

• I let the replace _ctx idea drop. _ctx is ugly and cheap and does the job
• perhaps I'll revive it by introducing an extension for NB/SB Library files like .nblib which won't have a _ctx but a selection of the NodeBox grammar included. Deciding behaviour on import.

[stuaxo]

Only populating the namespace for some file extension seems sane (implementing a custom file encoding is another option)

I'm going to see if I can replace the shoebot core before coming back to this, the architecture makes it too hard to do this in a nice way at the moment.