Incompatibility of pointers to base and derived classes in Lua function calls about sol2

[congard]

Complete test case that reproduces the original issue

So, I continued my research regarding the original issue - and finally have found out all circumstances for the original issue aka "strange problem" to occur.

Final results

1. It doesn't matter if any environments are created
2. The issue appears when sol::state is being used in the external library
3. The issue occurs when the library is built as shared, but only on Windows
4. It seems that The Original Issue is unrelated to #1523
5. Tested using both sol v3.2.3 and v3.3.0

Demo

Demo and more detailed results can be found here: https://github.com/congard/sol2-issue-1532

[congard]

So, I've created a test case that reproduces this issue demonstrates UB of #1523. It seems that we are dealing with undefined behavior here - the test case produces slightly different output on each run.

Short test case description:

1. Create 2 classes: base (Foo) and derived (Bar)
2. Create n environments with the global environment as the parent
3. For each environment:
   1. Register types Foo and Bar for the current environment
   2. Bind a function that takes base class (Foo) as an argument
   3. Create a new Lua function func that takes an argument arg and calls the aforementioned function with the argument arg
4. For each environment:
   1. Call function func with a pointer to Bar instance as an argument

Code:

#include <iostream>
#include <sol/sol.hpp>

//#define VERBOSE

#ifdef VERBOSE
#define LOG_V(...) printf(__VA_ARGS__)
#else
#define LOG_V(...)
#endif

// base class
class Foo {
public:
    Foo() {
        LOG_V("ctor\n");
    }

    ~Foo() {
        LOG_V("dtor\n");
    }
};

// derived class
class Bar: public Foo {};

template<typename T> static void registerType(sol::table &table);

template<> void registerType<Foo>(sol::table &table) {
    if (table["Foo"].valid())
        return;
    auto foo = table.new_usertype<Foo>("Foo");
    LOG_V("Foo registered\n");
}

template<> void registerType<Bar>(sol::table &table) {
    if (table["Bar"].valid())
        return;
    auto bar = table.new_usertype<Bar>("Bar", sol::base_classes, sol::bases<Foo>());
    LOG_V("Bar registered\n");
}

static bool testEnvCount(int envCount) {
    sol::state state;

    std::vector<sol::environment> envs(envCount);

    // create environments
    for (auto &env : envs)
        env = sol::environment(state, sol::create, state.globals());

    auto cfgEnv = [&state](sol::environment &env) {
        registerType<Bar>(env);
        registerType<Foo>(env);

        state.script("function func(foo) doSomethingWithFoo(foo) end", env);

        env["doSomethingWithFoo"] = [](Foo *foo) {
            LOG_V("[doSomethingWithFoo] Foo: %ti\n", (ptrdiff_t) foo);
        };
    };

    // configure environments
    for (auto &env : envs)
        cfgEnv(env);

    Bar bar;

    auto callFunc = [bar_ptr = &bar](sol::environment &env) {
        auto result = env["func"].get<sol::function>()(bar_ptr);

        if (!result.valid()) {
            sol::error error = result;
            LOG_V("%s\n", error.what());
        }

        return result.valid();
    };

    // call Lua function "func"
    for (auto &env : envs) {
        if (!callFunc(env)) {
            return false;
        }
    }

    return true;
}

int main() {
    constexpr int testFrom = 1;
    constexpr int testTo = 30;

    for (int i = testFrom; i <= testTo; ++i) {
        if (testEnvCount(i)) {
            std::cout << "OK for envCount = " << i << "\n";
        } else {
            std::cout << "Failed for envCount = " << i << "\n";
        }
    }

    return 0;
}

Output:

1st run

OK for envCount = 1
Failed for envCount = 2
Failed for envCount = 3
OK for envCount = 4
Failed for envCount = 5
Failed for envCount = 6
OK for envCount = 7
Failed for envCount = 8
OK for envCount = 9
Failed for envCount = 10
OK for envCount = 11
Failed for envCount = 12
OK for envCount = 13
Failed for envCount = 14
Failed for envCount = 15
OK for envCount = 16
OK for envCount = 17
Failed for envCount = 18
OK for envCount = 19
OK for envCount = 20
OK for envCount = 21
OK for envCount = 22
Failed for envCount = 23
OK for envCount = 24
OK for envCount = 25
OK for envCount = 26
Failed for envCount = 27
OK for envCount = 28
OK for envCount = 29
Failed for envCount = 30

2nd run

OK for envCount = 1
Failed for envCount = 2
Failed for envCount = 3
OK for envCount = 4
Failed for envCount = 5
Failed for envCount = 6
OK for envCount = 7
Failed for envCount = 8
OK for envCount = 9
Failed for envCount = 10
Failed for envCount = 11  <-- difference here, for example
Failed for envCount = 12
OK for envCount = 13
Failed for envCount = 14
Failed for envCount = 15
OK for envCount = 16
OK for envCount = 17
Failed for envCount = 18
OK for envCount = 19
OK for envCount = 20
OK for envCount = 21
OK for envCount = 22
Failed for envCount = 23
OK for envCount = 24
OK for envCount = 25
OK for envCount = 26
Failed for envCount = 27
OK for envCount = 28
OK for envCount = 29
Failed for envCount = 30

Sol error message:

stack index 1, expected userdata, received sol.Bar *: value at this index does not properly reflect the desired type (ba
d argument into 'void(Foo *)')
stack traceback:
        [C]: in global 'doSomethingWithFoo'
        [string "function func(foo) doSomethingWithFoo(foo) en..."]:1: in function <[string "function func(foo) doSometh
ingWithFoo(foo) en..."]:1>

In the example above, switching to v3.2.3 helps, as it described in #1523. However, unfortunately, in my project it didn't help. The only available "workaround" is the one described at the end of the first comment.

[congard]

Some additional debug info from my project (tested with sol v3.2.2):

algine::Object registered
algine::Scene (extends public algine::Object) registered
stack_check_unqualified.hpp:520: has_derived == false  <------ error happens because of this
stack_check_unqualified.hpp:520: detail::demangle<T>() == "void) [T = algine::Object]"
stack_check_unqualified.hpp:536: success == false => calling `handler(L, index, type::userdata, indextype, "value at this index does not properly reflect the desired type")`

Error message:

stack index 1, expected userdata, received sol.void) [T = algine::Scene *]: value at this index does not properly reflect the desired type (bad argument into 'void) [T = void](void) [T = algine::Object *])')
stack traceback:
        [C]: in function 'func'
        [string "loadType('algine::Object') loadType('algine::..."]:1: in function 'f'

Code:

Lua lua;
auto &state = *lua.state();
sol::environment env = state.globals();
env["func"] = [](Object *obj) { printf("Global state\n"); };
state.script("loadType('algine::Object') loadType('algine::Scene') function f(o) func(o) end", env);
if (auto res = env["f"].get<sol::function>()((algine::Scene*) this); !res.valid()) {
    sol::error err = res;
    printf("%s\n", err.what());
}

Hope it will help to fix this issue