flecs-hub / flecs-lua

Lua script host for flecs

flecs-lua

This is a Lua binding for Flecs, it can be used to extend an application or as a standalone module.

Dependencies

• Flecs v3.2.3 with most addons enabled - backward and forward compatibility is limited due to dependencies on semi-private API's

• Lua 5.3 or later (requires 64-bit integers)

Build

Meson >= 0.55.0

meson build # -Dtests=enabled
cd build
ninja
ninja test

Lua API

Usage

Scripts are hosted by the FlecsLua module for the world it's imported into.

Using this library as a standalone Lua module is possible but is not the main focus of the project.

/* Creates a script host for the world */
ECS_IMPORT(world, FlecsLua);

/* Get a pointer to the VM */
lua_State *L = ecs_lua_get_state(world);

/* Execute init script, the world for all API calls is implicit */
luaL_dofile(L, argv[1]);

/* Lua systems will run on the main thread */
while(ecs_progress(world, 0))

Most of the functions are bound to their native counterparts, components are (de-)serialized to- and from Lua, it is not designed to be used with LuaJIT where FFI is preferred.

Components defined from the host with the meta module and from Lua are handled the same way and can be mixed in systems, queries, etc.

The script executed on init should be similar to the host's main().

main.lua

local ecs = require "ecs"
local m = require "module"

local ents = ecs.bulk_new(m.Position, 10)

for i in 1, #ents do
    ecs.set(ents[i], m.Position, { x = i * 2, y = i * 3})
end

Modules can be stuctured like normal Lua modules.

Importing native modules or other Lua scripts is left to the host application. ecs.import() does not load flecs modules, instead it returns a table with all components and named entities from an already imported flecs module.

Flecs modules are defined and imported with ecs.module(), it takes an import callback which is called immediately and returns the imported module ID. All components and entities should be registered inside the callback function for proper scoping.

module.lua

local ecs = require "ecs"
local m = {}

function m.system(it)

    for p, e in ecs.each(it) do
        print("entity: " .. e)
        p.x = p.x + 1
        p.y = p.y + 1
    end
end

ecs.module("name", function()

    m.Position = ecs.struct("Position", "{float x; float y;}")

    ecs.system(m.system, "Move", ecs.OnUpdate, "Position")

end)

return m

Debugging

For debug builds (#ifndef NDEBUG) most API functions will retrieve the current file, source line and expose them through ar.short_src and ar.currentline. Set watches for these variables to track where API functions are called from in Lua.

Lua state lifecycle

• A default Lua state is created when the module is imported

• To set a custom lua_State use ecs_lua_set_state(), this will destroy the default state.

• In both cases lua_close() is called on ecs_fini() to trigger __gc metamethods before the world is destroyed.

flecs-lua's Issues

Custom de/serializer API

This would be useful for components that are exposed as userdata/tables with metamethods.

Threading

Lua has a GIL which means calling into a VM from multiple threads will always result in locking. To get around this most threading implementation e.g. effil create their own OS threads with separate VM's.

Worker threads will have limitations e.g. require/ecs.import can't be used to reinitialize shared variables in a module because import functions are not idempotent, the modules have to be copied to/shared with the worker states.

Torch threads seem to be more efficient for this.

Cleanup

• system metadata (including the param reference) stored in the lua state should be registered as collectible to fix resource leaks with ecs.init()

• ecs_iter_t should be userdata

• make iterators with only shared terms usable

Systems

My thoughts so far on how systems should work:

• ecs.system() wraps the flecs function like the existing APIs we have so far

• There has to be a dummy ecs_lua_entrypoint(ecs_iter_t *it) between the VM and flecs

• When the system is called from flecs it pushes each "argument" as an array of components, then copies it backs into flecs when it's finished

• flecs-lua probably shouldn't "own" the lua_State, so it has be kept in mind that the VM can persist across frames and:

• if a system has a Init() we definitely have to persist that VM until DeInit() anyway

Some things that are not so obvious:

1. If there are multiple systems registered from a state, how do we figure out which Lua system is supposed to run?

2. Do we need a lua_State per system?

3. Can we make multithreading work?

4. How does it interact with host scripts that aren't modules/systems?

5. Can we still make "simple" scripts work where sleep() is possible?

6. Reloading script/systems is kinda important, what changes in flecs-lua would that require?

Latest stable version

Hi!

I'll open an issue to ask, in case this helps other people too.

What is the latest stable version of flecs supported?

The last commit 895d3cb says support for 3.2.0 is not quite ready and there are some bugs.

The previous commit on the README file mentions 2.4 70c9b9d, but there's been a lot of work in between these too commits.

What is a the latest stable commit that we can use and what flecs version does it target?

Deferred column serialization

Should be safe to do this in all cases.

How are lua systems forced to run on the main thread?

I saw that in system.c there is an assert to make sure that lua systems are run on the main thread:

ecs_assert(stage_id == 0, ECS_INTERNAL_ERROR, "Lua callbacks must run on the main thread");

But i can't find any code that actually disables multithreading while initializing lua systems or for the world as a whole. Am i missing something?

Build error

Hi, I'm new to meson and ninja building but I was trying to see if these lua bindings work.
It fails at "ninja" step with this error:

Is this binding still being worked on or it's discontinued? I saw that the porting to 3.0.X is not done yet.

Thanks and sorry to bother you!

v3 migration

Migration to v3 is a work in progress, the majority of the v3 api is already exposed in v2.4, those should be finished first:

• Pair API (replaces trait API)

• event/event[] instead of phase for systems/triggers

• ecs.observer()

• ecs.term_iter(), ecs.term_next(), etc.

• Update ecs.query() to accept a filter or more of the ecs_query_desc_t fields

• Update ecs.system() the same as way as above

• Update ecs.trigger() to accept a term

• Update ecs.observer() to accept a filter

Breaking changes

flecs-lua will target v2.4 until v3 becomes stable (or at least until a beta release)

• Remove/replace deprecated functions, ecs_iter_t fields

• add_owned() -> override()

• Switch to the new builtin meta module

• Update constants / builtin components

Misc

• ecs_lua_query_next() -> ecs_lua_iter_next()

• ecs.iter_next()

• Consider dropping all other ecs.*_next() functions

• Consider re-adding the range checks to flecs-lua from the old meta module

• Expose JSON module

• System status callback

• Consider v2.4-style API for systems, entities, etc

Tests failed on MacOS

stderr:
warning: ../test/main.c(118): FlecsMonitor may impact benchmark results
fatal: ../subprojects/flecs/src/datastructures/vec.c(278): assert: size == v->elem_size INVALID_PARAMETER
TEST: ecs_os_abort() was called!
Assertion failed: (size == v->elem_size), function ecs_vec_get, file vec.c, line 278.
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――


Summary of Failures:

 4/13 iter        FAIL            0.20s   killed by signal 6 SIGABRT
 5/13 pipeline    FAIL            0.20s   killed by signal 6 SIGABRT
13/13 snapshot    FAIL            0.46s   killed by signal 6 SIGABRT

API discussion

I have no idea how to approach this yet.

Lua ECS implementations for reference:

entitas-lua
tiny-ecs
ecs-lua (tiny-ecs fork)

Feature coverage

What's still missing:

• Iterator columns that aren't just [inout] or [in]

• An each function for iterating columns that is faster than the current solution

• Traits

• Triggers

• Full Query API: sorting, etc

• Subqueries - SanderMertens/flecs#258

• Snapshots

• Most functions that take a filter

• All path stuff e.g. ecs.get_fullpath()

• Scopes e.g. ecs.set_scope()

• ecs.array() - flecs-meta does not detect size and alignment fixed: flecs-hub/flecs-meta#17

What's already done:

• Most entity, type functions

• De/serialization (excl. maps)

• Struct, alias components

• Tags

• Prefabs

• Modules

• Systems

• Pipelines and all pipeline funtions

• Singleton API

• Timers

• Logging, tracing, assert API

For consideration:

• More components types?

• Deferred operations?

• Component actions

Features that can't be implemented without threading support:

• Systems that aren't locked to the main thread

• Triggers on host components These are always called on the ecs_progress() thread

• Using script components from the host with a trigger or action callback implemented in Lua same as above

Cmake build

How about adding cmake support? 🙂

Pointer comparisons

There are some comparisons of unrelated pointers in de/init code which isn't reliable, get rid of them.

Bugs to fix

• Iterator lifecycle (free the iterator if it isn't iterated all the way)

• Filter lifecycle

• Query lifecycle (ecs,each())

• ecs.pipeline() is broken

• ecs.emmy_class() is broken

• ecs.world_stats() is broken

• Missing argument checks where flecs got more strict with entity values, tldr turn flecs asserts into lua errors

Potential improvements

• make deserialize_type() non-recursive(?) and use depth variable to make error messages more descriptive, e.g. foo.bar.array[2]: invalid type

• Expose JSON module?

• System status callback?

• Consider v2.4-style API for systems, entities, etc

• Move from fragile macros to using the internal import_handles() function for exposing builtin flecs entities/components