Preserving ordering in sol2 tables and converting to ordered JSON about sol2

[Rochet2]

Sol tables are just lua tables. And lua tables are not guaranteed to be ordered, so the assumption in step 1 that the code creates the table with specific order is incorrect. As I understand, tables are basically hash maps.

From table constructor docs

The order of the assignments in a constructor is undefined.

The for_each documentation tells that it does not guarantee order.

The iterators you use to walk through a sol::table are NOT guaranteed to iterate in numeric order

Furthermore, in lua you can only A) iterate through a table by specifying the keys to iterate yourself and accessing that specific key in a loop, B) by iterating from 1 up to the first absent index (ipairs), C) by looping through all keys in the table with pairs which does not guarantee order (pairs calls next). for_each in sol is the same as pairs so it will never have guaranteed order even if the data would be ordered, and from the documentation of next which pairs uses:

The order in which the indices are enumerated is not specified, even for numeric indices.

To collect all of this together, what you can do is create an ordered list containing the data instead and iterate through it using ipairs (or in sol, just get length and iterate from 1 to length):

// myTable = {
// { first = 1 },
// { second = 2 },
// { third = 3 },
// }
sol::table myTable = lua.create_table();
myTable[1] = lua.create_table_with(1, "first", 2, 1);
myTable[2] = lua.create_table_with(1, "second", 2, 2);
myTable[3] = lua.create_table_with(1, "third", 2, 3);

for (int i = 1, count = myTable.size(); i <= count; ++i) {
    lua["print"](i, myTable[i], myTable[i][1], myTable[i][2]);
}

This is a lot less efficient and inconvenient overall but it guarantees order. There are of course other options, like creating your own data structure, or exposing some kind of C++ data structure to lua depending on what you need. On that vein, if you want an easy way for ordering while having a regular map, you could track the insertions and deletions from the table in a separate list. However, seems that for_each in sol does not check the metamethod __pairs 🤔. We could expose the order table and iterate through it instead.

-- Define the table with metatable
function createTrackedTable()
    local data = {}
    local order = {}

    local mt = {
        __newindex = function(t, k, v)
            if v == nil then
                -- Remove key from data and order if the value is nil
                rawset(data, k, nil)
                for i, key in ipairs(order) do
                    if key == k then
                        table.remove(order, i)
                        break
                    end
                end
            else
                -- Add new key-value pair to data and key to order
                if rawget(data, k)  == nil then
                    table.insert(order, k)
                end
                rawset(data, k, v)
            end
        end,

        __index = data,

        __pairs = function(t)
            local i = 0
            local function iter(data, oldk)
                i = i + 1
                local k = order[i]
                if k then
                    return k, data[k]
                end
            end
            return iter, data, nil
        end,
    
        order = order,
    }

    return setmetatable({}, mt)
end

-- Usage
local trackedTable = createTrackedTable()

-- Insert elements
trackedTable["first"] = 1
trackedTable["second"] = 2
trackedTable["third"] = 3

-- Print elements in their insertion order
for k, v in pairs(trackedTable) do
    print(k, v)
end

-- Remove element
trackedTable["second"] = nil
-- add it back in
trackedTable["second"] = 2

-- Print elements in their insertion order
for k, v in pairs(trackedTable) do
    print(k, v)
end

    // Create a tracked table
    sol::function createTrackedTable = lua["createTrackedTable"];
    sol::table trackedTable = createTrackedTable();

    // Insert elements into the tracked table
    trackedTable["first"] = 1;
    trackedTable["second"] = 2;
    trackedTable["third"] = 3;

    // Access the order table
    sol::table mt = lua["getmetatable"](trackedTable);
    sol::table order = mt["order"];

    // Iterate over the order table to print elements
    for (int i = 1, count = order.size(); i <= count; ++i) {
        sol::object key = order[i];
        lua["print"](key, trackedTable[key]);
    }

In general only lists, stacks, queues and similar data structures have insertion order preserved. Maps and sets order the data on its own, based on the keys or values usually, so they may not suit your needs if you require insertion order being preserved. I would say that JavaScript is quite unique in its implementation of a map style structure with insertion order when iterated (ref). Not sure I have seen anything similar elsewhere yet. 👀

[gsisinna]

@Rochet2 thank you very much for the technical explanation and support. I have continued to read the documentation of both Lua and Sol2 and will try to implement one of your proposed methods to achieve separate sorting. Basically, even if I do not respect the insertion order I need the keys to always be in a certain order before conversion to Json.
In general, I am well aware of the pattern to be followed for the final result in terms of key sorting, but I am trying to get something that scales well for future software modifications and customization of results 👍🏻