[Enhancement] Proper distinction between "failures" and "errors" about luaunit

[n1tehawk]

I like the idea in general, and would consider this a very useful feature. However, upon closer investigation I notice some potential problems:

• Changing the error type of failed LuaUnit tests might impact the usefulness of 'standalone' LuaUnit, i.e. scenarios where users would rely on calling assertions directly - like lu.assertEquals(true, false). If we return a failure "object" (non-string error) there, and without error protection, some Lua versions will no longer report anything useful and display (error object is not a string) instead.
• As you have already indicated above, LuaUnit's own internal testing also relies on string comparisons, especially when verifying error messages.

A possible solution to both these problems could be introducing a __tostring metamethod. My experiments suggest that Lua 5.3 and current LuaJIT respect/use this, but unfortunately Lua 5.1 and 5.2 do NOT. (See http://lua-users.org/lists/lua-l/2011-11/msg00693.html)

My impression is that we would probably be better off (and avoid these kind of pitfalls) if we just add a prefix (string) to the error message to indicate failures. If the prefix is specific enough (e.g. "LuaUnit test FAILURE: "), it can be filtered and acted upon accordingly within execOneFunction(); while keeping the error still valid as a standard string.

Regards, NiteHawk

[n1tehawk]

For a 'sneak preview' of what I have in mind, look at 0434c73 and dbf98d5 ...

[bluebird75]

Hi Nitehawk,

Le 14/12/2015 22:42, NiteHawk a écrit :

I like the idea in general, and would consider this a very useful
feature. However, upon closer investigation I notice some potential
problems:

• Changing the error type of failed LuaUnit tests might impact the
  usefulness of 'standalone' LuaUnit, i.e. scenarios where users
  would rely on calling assertions directly - like
  |lu.assertEquals(true, false)|. If we return a failure "object"
  (non-string error) there, and without error protection, some Lua
  versions will no longer report anything useful and display |(error
  object is not a string)| instead.

I suppose that it would actually display a table ref but yes, I got the
point. However, this is not a use case that I am aware of.

• As you have already indicated above, LuaUnit's own internal
  testing also relies on string comparisons, especially when
  verifying error messages.

Test suites are here to evolve...

Pulling information from an object might actually make the test suites
more robust instead of relying on text. The TestError object would
actually contain in attributes the errore message, the stack trace, the
type of failure, and any additional information we might want to put
there in the future.

A possible solution to both these problems could be introducing a
|__tostring| metamethod. My experiments suggest that Lua 5.3 and
current LuaJIT respect/use this, but unfortunately Lua 5.1 and 5.2 do
/NOT/. (See http://lua-users.org/lists/lua-l/2011-11/msg00693.html)

My impression is that we would probably be better off (and avoid these
kind of pitfalls) if we just add a prefix (string) to the error
message to indicate failures. If the prefix is specific enough (e.g.
/"LuaUnit test FAILURE: "/), it can be filtered and acted upon
accordingly within |execOneFunction()|; while keeping the error still
valid as a standard string.

I already do this for pulling the stracktrace, so why not...

See :

function M.LuaUnit:protectedCall( classInstance , methodInstance,
prettyFuncName)

    local function err_handler(e)
        return debug.traceback(e..SPLITTER, 3)
    end

    local ok=true, fullErrMsg, stackTrace, errMsg, t
        ok, fullErrMsg = xpcall( function () methodInstance() end,

err_handler )
if ok then
return ok
end

    t = strsplit( SPLITTER, fullErrMsg )
    errMsg = t[1]
    stackTrace = string.sub(t[2],2)
    ...

I would like to use this same mechanism to also provide the ability to
skip test, to mark an early test success, to force a failure without an
assertion.

So ... up to you, feel free to experiment.

Cheers,

Philippe

[AbigailBuccaneer]

It would also be nice to somehow distinguish between user test errors and internal errors - they both current lead to a failure in the exact same way. (I've just encountered this in #32.)

[n1tehawk]

Thanks for merging the above PR! I'll now prepare another pull request that should address this issue.

[bluebird75]

Well done !