ers35 / luastatic Goto Github PK

I've got a Lua script which doesn't end in .lua, but when I try to use luastatic on it, it refuses to build:

$ luastatic fennel /usr/lib/x86_64-linux-gnu/libluajit-5.1.a -I /usr/include/luajit-2.0
luastatic 0.0.9
usage: luastatic main.lua[1] require.lua[2] liblua.a[3] library.a[4] -I/include/lua[5] [6]
  [1]: The entry point to the Lua program
  [2]: One or more required Lua source files
  [3]: The path to the Lua interpreter static library
  [4]: One or more static libraries for a required Lua binary module
  [5]: The path to the directory containing lua.h
  [6]: Additional arguments are passed to the C compiler

If I rename the file, it will run, but that conflicts with other files I have in my repo. It would be nice to be able to specify an output file and accept an input which does not end in .lua.

I could prepare a patch to do this if you agree it's a good idea.

(I promise, that's the last one for today)

A cursory strace (or a package loading error) reveals the final binary is still looking on disk for modules if it doesn't find them in the bundle. How can I disable all external module loading? In a way that a CPATH env variable can't re-enable it?

Thanks

Why do you have to set CC to "" for the last usage-example

CC="" lua luastatic.lua main.lua

to make it work ?

I use luastatic with openresty/LuaJIT2 and static compilation. One thing I'd like to see is the automatic loading of supplied C modules during the initial state construction.

For example, I want to include int64 support, which is povided by PUC's lint64.c module. I have monkeypatched luastatic like this (following added to template out):

void luaL_requiref (lua_State *L, char const* modname,
                    lua_CFunction openf) {
  luaL_checkstack(L, 3, "not enough stack slots");
  lua_pushcfunction(L, openf);
  lua_pushstring(L, modname);
  lua_call(L, 1, 1);
  lua_getglobal(L, "package");
  lua_getfield(L, -1, "loaded");
  lua_replace(L, -2);
  lua_pushvalue(L, -2);
  lua_setfield(L, -2, modname);
  lua_pop(L, 1);
}

int luaopen_int64(lua_State *L);

int main(int argc, char *argv[])
{
  lua_State *L = luaL_newstate();
  luaL_openlibs(L);
  createargtable(L, argv, argc, 0);
  luaL_requiref(L, "int64", luaopen_int64);

(the luaL_requiref polyfill comes from Kepler's compat-5.2 C API, one may wish to exclude that for Lua > 5.1)

I then invoke luastatic like this:

luastatic con.lua lint64-101/lint64.c src/libluajit.a -I./src/ -I./lint64-101 -static

This all works because lint64-101/lint64.c is passed as a compiler argument. My request is that luastatic parses .c files in it's arguments and emits the luaL_requiref calls automatically. The user would still need to supply include dirs used by the C module, but this simplifies static compilations.

./luastatic.lua lpm.lua src/reader.lua src/parser.lua -I/usr/include/lua5.3/

cc -Os lpm.lua.c -rdynamic -lm -o lpm -I/usr/include/lua5.3/
/usr/bin/ld: /tmp/ccvKVuNz.o: in function lstop': lpm.lua.c:(.text+0x32): undefined reference to lua_sethook'
/usr/bin/ld: /tmp/ccvKVuNz.o: in function msghandler': lpm.lua.c:(.text+0x56): undefined reference to lua_tolstring'
/usr/bin/ld: lpm.lua.c:(.text+0x6f): undefined reference to luaL_callmeta' /usr/bin/ld: lpm.lua.c:(.text+0x80): undefined reference to lua_type'
/usr/bin/ld: lpm.lua.c:(.text+0x8a): undefined reference to lua_typename' /usr/bin/ld: lpm.lua.c:(.text+0x9e): undefined reference to lua_pushfstring'
/usr/bin/ld: lpm.lua.c:(.text+0xab): undefined reference to lua_type' /usr/bin/ld: lpm.lua.c:(.text+0xc7): undefined reference to lua_getglobal'
/usr/bin/ld: lpm.lua.c:(.text+0xd9): undefined reference to lua_getfield' /usr/bin/ld: lpm.lua.c:(.text+0xe9): undefined reference to lua_rotate'
/usr/bin/ld: lpm.lua.c:(.text+0xf6): undefined reference to lua_settop' /usr/bin/ld: lpm.lua.c:(.text+0x101): undefined reference to lua_pushstring'
/usr/bin/ld: lpm.lua.c:(.text+0x111): undefined reference to lua_rotate' /usr/bin/ld: lpm.lua.c:(.text+0x11e): undefined reference to lua_settop'
/usr/bin/ld: lpm.lua.c:(.text+0x12b): undefined reference to lua_pushinteger' /usr/bin/ld: lpm.lua.c:(.text+0x142): undefined reference to lua_callk'

Any idea how to solve this?

How would I go about adding these to a static binary, compiled with musl?

Just playing around with luastatic, and I wanted to compile a file together with luaossl.
After compiling luaossl, I have the Lua bindings, a openssl.o file and an openssl.so file.

So far I've tried creating an archive using ar rcs openssl.a openssl.so, which gives me a static archive.
Then, doing lua luastatic.lua test.lua ../luaossl/src/openssl.lua ../luaossl/src/5.3/openssl.a /usr/lib/lua5.3/liblua.a -I/usr/include/lua5.3 -s -static

This results in module '_openssl' not found. require "openssl" works fine, because that is included in the luastatic bundle.
But the actual openssl.so is not, so it throws that error.

Cheers in advance!

The test.lua file:

local ssl = require "openssl"
print(ssl.version())
print(_VERSION)

Both the most recent script from the git and the LuaRocks version work fine using 5.3 Lua, but always give an error when trying to use the other versions said to support (5.2, 5.1 and LuaJIT)

$ luastaic file.lua /usr/lib/libluajit-5.1.so /usr/bin/luajit-2.0.5
nm: /usr/lib/libluajit-5.1.so: no symbols
cc -Os file.lua.c  /usr/lib/libluajit-5.1.so -rdynamic -lm  -o input /usr/bin/luajit-2.0.5
/usr/bin/ld: /usr/bin/luajit-2.0.5: stderr: invalid version 2 (max 0)
/usr/bin/ld: /usr/bin/luajit-2.0.5: error adding symbols: bad value
collect2: error: ld returned 1 exit status

$ luastatic thing.lua /usr/lib/liblua5.2.so /usr/bin/lua5.2
nm: /usr/lib/liblua5.2.so: no symbols
cc -Os thing.lua.c  /usr/lib/liblua5.2.so -rdynamic -lm  -o thing /usr/bin/lua5.2
/usr/bin/ld: /tmp/ccq8PIjP.o: in function `msghandler':
thing.lua.c:(.text+0xe9): undefined reference to `lua_rotate'
/usr/bin/ld: thing.lua.c:(.text+0x111): undefined reference to `lua_rotate'
/usr/bin/ld: /tmp/ccq8PIjP.o: in function `main':
thing.lua.c:(.text.startup+0x105): undefined reference to `lua_rotate'
/usr/bin/ld: thing.lua.c:(.text.startup+0x151): undefined reference to `lua_rotate'
collect2: error: ld returned 1 exit status

I may be just doing something wrong

Tested on Manjaro(/Arch) x86_64

Hi! Luastatic looks really cool — I already just finished hacking my way through the same problem space when I was told about about luastatic. (it did help me out teaching me some nm tricks when I got into cross-compiling!)

The use-case I had at hand was building a single-binary for LuaRocks, which includes a bunch of dependencies. The overall goal was described here, it's something I'd like to see as a goal for LuaRocks 3.1.

This was the solution I ended up with so far:

https://github.com/luarocks/luarocks/tree/master/binary

There's one big all_in_one Lua script which does mostly the same as luastatic, but also a static-gcc wrapper script and config file which fools LuaRocks into building static libraries even though it doesn't (yet) have support for it (the make binary target in the root LuaRocks makefile launches the whole thing).

We would like to add first-class support for static libraries in LuaRocks eventually (there was an unfinished Google Summer of Code project for that), and the above static-gcc hack looks promising.

Given that LuaRocks knows about dependencies (and third-party library dependencies as well), the idea of adding a command to LuaRocks that would produce a single-binary out of a script sounds compelling: it would essentially take a rock with a bin/ script and statically-compile it along with all its dependencies (essentially doing the equivalent of figuring out the luastatic command-line flags). At that point, luastatic could even be part of LuaRocks itself.

How does that idea sound to you?

cross compiling for windows

CC=x86_64-w64-mingw32-gcc luastatic main.lua /usr/x86_64-w64-mingw32/lib/liblua5.2.a -I/usr/x86_64-w64-mingw32/include/lua5.2/

cross compiling for windows is good but creating a cross platform code will be more convenient for end users.

I have created a variable is_windows which check wheter this script is running on windows or not. Most of the windows user use mingw so i used gcc instead of cc.

-- The boolean check for building on windows platform
local is_windows = _G.package.config:sub(1, 1) == "\\"

-- The C compiler used to compile and link the generated C source file.
local CC = os.getenv("CC") or (is_windows and "gcc" or "cc")

uname is not available in windows which causes this error.

'uname' is not recognized as an internal or external command,
operable program or batch file.

to solve this error I used this code.

local UNAME = "Unknown"

if is_windows then
	UNAME = "Windows"
else
	UNAME = (shellout("uname -s") or "Unknown"):match("%a+") or "Unknown"
end

also we cannot check a command by /dev/null in windows. so I just replaced previous code with this.

if not execute(CC .. " --version" .. (is_windows and "" or " 1>/dev/null 2>/dev/null")) then
	io.stderr:write("C compiler not found.\n")
	os.exit(1)
end

HI,

I have a Lua(-only) project that I'm trying to compile with luastatic. Here's the command it generates:

cc -Os scratch.luastatic.c ./vendor/luajit2/dbin/lib/libluajit-5.1.a -rdynamic -lm -ldl -o scratch2 -I./vendor
/luajit2/dbin/include/luajit-2.1/ -static

I only added -static and provided paths to a static build of LuaJIT.

This succeeds in building a static binary, however when I run it, I get this error:

[string "moot.signal"]:5: Symbol not found: signal
stack traceback:
	[C]: in function '__index'
	[string "moot.signal"]:5: in main chunk

Which traces back to a simple foo = ffi.C.signal in the moot.signal module. It seems like either the signal symbols aren't being linked or LuaJIT/FFI can't access them.

If I compile it without -static, it works.

Looks like compiler receives argument -s which means optimize size.
Maybe we should strip resulting executable by default also?

serg@mosolov  ~/projects/test/luafltk/win  CC=/home/serg/crscmpl/mxe/usr/bin/i686-w64-mingw32.static-g++ luastatic main.lua libmoonfltk.a /home/serg/crscmpl/mxe/usr/i686-w64-mingw32.static/lib/liblua.a -I/home/serg/crscmpl/mxe/usr/i686-w64-mingw32.static/include /home/serg/crscmpl/mxe/usr/i686-w64-mingw32.static/lib/libfltk.a -lgdi32 -lole32 -lcomctl32 -luuid -lws2_32 -lcomdlg32 -mwindows
/home/serg/crscmpl/mxe/usr/bin/i686-w64-mingw32.static-g++ -Os main.luastatic.c libmoonfltk.a /home/serg/crscmpl/mxe/usr/i686-w64-mingw32.static/lib/liblua.a /home/serg/crscmpl/mxe/usr/i686-w64-mingw32.static/lib/libfltk.a -lm -o main.exe -I/home/serg/crscmpl/mxe/usr/i686-w64-mingw32.static/include -lgdi32 -lole32 -lcomctl32 -luuid -lws2_32 -lcomdlg32 -mwindows
serg@mosolov  ~/projects/test/luafltk/win  ls -l  ✔  6879  10:42:15
total 3760
-rwxrwxr-x 1 serg serg 556 мар 24 15:14 crscmpl
-rw-rw-r-- 1 serg serg 850450 мар 24 13:47 libmoonfltk.a
-rw-rw-r-- 1 serg serg 1852 мар 24 14:55 main.cpp
-rwxrwxr-x 1 serg serg 2644259 апр 23 10:42 main.exe
-rw-rw-r-- 1 serg serg 1275 апр 23 09:18 main.lua
-rw-rw-r-- 1 serg serg 17394 апр 23 10:42 main.luastatic.c
-rw-rw-r-- 1 serg serg 17394 апр 23 10:11 main.luastatic.c_after
-rw-rw-r-- 1 serg serg 17472 апр 23 10:11 main.luastatic.c_fixed
-rw-rw-r-- 1 serg serg 278148 апр 23 09:23 out.txt
serg@mosolov  ~/projects/test/luafltk/win  ~/crscmpl/mxe/usr/bin/i686-w64-mingw32.static-strip main.exe  ✔  6880  10:42:16
serg@mosolov  ~/projects/test/luafltk/win  ls  ✔  6881  10:42:30
crscmpl libmoonfltk.a main.cpp main.exe main.lua main.luastatic.c main.luastatic.c_after main.luastatic.c_fixed out.txt
serg@mosolov  ~/projects/test/luafltk/win  ls -l  ✔  6882  10:42:30
total 2700
-rwxrwxr-x 1 serg serg 556 мар 24 15:14 crscmpl
-rw-rw-r-- 1 serg serg 850450 мар 24 13:47 libmoonfltk.a
-rw-rw-r-- 1 serg serg 1852 мар 24 14:55 main.cpp
-rwxrwxr-x 1 serg serg 1557504 апр 23 10:42 main.exe
-rw-rw-r-- 1 serg serg 1275 апр 23 09:18 main.lua
-rw-rw-r-- 1 serg serg 17394 апр 23 10:42 main.luastatic.c
-rw-rw-r-- 1 serg serg 17394 апр 23 10:11 main.luastatic.c_after
-rw-rw-r-- 1 serg serg 17472 апр 23 10:11 main.luastatic.c_fixed
-rw-rw-r-- 1 serg serg 278148 апр 23 09:23 out.txt

before strip:
-rwxrwxr-x 1 serg serg 2644259 апр 23 10:42 main.exe
after strip:
-rwxrwxr-x 1 serg serg 1557504 апр 23 10:42 main.exe

Currently luastatic loader prints syntax error and completely exits if it fails to load a bundled module. It's more convenient to use standard loader interface and return some message including the syntax error from the loader to let require include it into its nicely formatted report and throw a normal Lua error.

Of course it's strange to bundle modules with syntax errors but there are some usecases, e.g. it's reasonable to have a module with Lua 5.3 bit operation syntax and attempt to load it with pcall(require, "module_name") and fallback to some other implementation if require fails.

I user 0.0.12, It works normally when running in the current directory, but runs abnormally when copied to other directories.


How to solve it ?

Nevermind, just didn't read enough.

If you can't find lauxlib.h, look it up with

locate lauxlib.h

If you don't have it it's in liblua5.2-dev (apt get install liblua5.2-dev)

Then tweak the readme's command

luastatic main.lua /usr/lib/x86_64-linux-gnu/liblua5.2.a -I<lauxlib.h parent folder>

Hi,
Great work on luastatic, it works very well. I have a problem with my project which has the source file with the main entry point named ir2net.lua, and the project directory named ir2net.

During compilation luastatic overwrites the main script (ir2net.lua) with what appears to be an intermediate elf file.

This behavior destroys my main source file and I loose all work since my last git commit. luastatic should check if a source file exists before overwriting it, or at the very least there should be a warning in the readme to alert users that they could loose their work in this configuration.

Hi !
This is a very useful tool, and I'd like to use it from a Lua application. Perhaps you could split the API (as a module) from the main script ? That would allow to simply do something like:

local static = require 'luastatic'
static.compile("main.lua", "require1.lua")

Because for now I have to call the script with os.execute(), and that's not very convenient.

Thank you !

When there is foo/init.lua module, it can be loaded using standard Lua both using require "foo" and require "foo.init". However, when bundling using luastatic it seems that only require "foo.init" works.

Arguments are only stored in the arg table, they are not pushed on the stack as the lua stand-alone interpreter does:

local t = {...}
print(#arg, #t)

$ lua args.lua abc def
2	2
$ ./luastatic.lua args.lua -llua
cc -Os args.lua.c   -rdynamic -lm  -o args -llua
$ ./args abc def
2	0

On some systems the executable nm in the current $PATH is not necessarily the only nm(1) utility.

For example, Debian Jessie nm is distinct from gcc-nm. The latter is GCC plugins aware, using the former would produce errors when enabling LTO during compilation.

Proposing the following patch:

--- a/luastatic.lua
+++ b/luastatic.lua
@@ -10,6 +10,7 @@ local dep_library_files = {}
 local otherflags = {}

 local CC = os.getenv("CC") or "cc"
+local NM = os.getenv("NM") or "nm"

 local function file_exists(name)
   local f = io.open(name, "r")
@@ -72,7 +73,7 @@ for i, name in ipairs(arg) do
       extension == "dylib"
     then
       -- the library either a Lua module or a library dependency
-      local nmout = shellout("gcc-nm " .. info.path)
+      local nmout = shellout(NM .. " " .. info.path)
       if not nmout then
         print("nm not found")
         os.exit(1)

Is it possible to do use luastatic for modular programming?

Usecase-example:
Let's take

1. a string.c-file which contains some helper functions concerning strings and
2. the actual main.c which forms the standalone program and which calls functions from string.c.

Now the special thing about string.c is, that it actually consists of lua-code, which has been converted to C-code using luastatic.

This would be very exciting.

@daveyc asks

This looks like a promising tool but IMO would be better with proper command line parsing. I want to include code that's installed in /usr/local//lua/ directories but found that I have to create symlinks to do so. Command options like -p /usr/local/share/lua/5.n and -L /usr/local/lib/lua/5.n would make it more usable.

from #2

This is a very interesting library! I feel it would be great if it could be quickly installed through LuaRocks! :D

I tried to build a static binary using the following command.

luastatic hw.lua pl/pretty.lua pl/utils.lua pl/compat.lua pl/lexer.lua socket/core.a /usr/local/lib/liblua.a -I/u/lua/static/lua-5.1.5/src

The socket/core.a library was not included. I had a quick look at the code and it seems this is because luastatic uses "basename" to strip of the directory name.

This looks like a promising tool but IMO would be better with proper command line parsing. I want to include code that's installed in /usr/local//lua/ directories but found that I have to create symlinks to do so. Command options like -p /usr/local/share/lua/5.n and -L /usr/local/lib/lua/5.n would make it more usable.

Could this be statically linked and ran in kernel space?
Maybe also something like compiling lua to iso?

Hello,

In my main Lua module, I use some thirdparty modules kept in a separate directory like

local argp = require 'thirdparty/argparse'
local colors = require 'thirdparty/ansicolors'

While building my exe with Luastatic, I am passing these also as arguments and I get the
required exe, but when I run the exe from another directory (than were the code resides), I get the following err:

[string "yang"]:2: module 'thirdparty/argparse' not found:
no field package.preload['thirdparty/argparse']
no module 'thirdparty/argparse' in luastatic bundle

Is this something to do with the way I used require or something wrong with the
way I am passing the modules to luastatic ?
This is my command:

luastatic.lua main.lua thirdparty/argparse.lua ... -I $LUAJIT_INC_DIR -L $LUAJIT_LIB_DIR -lluajit -lm -ldl

Is this OK ? or do I have to first compile the modules to .a and then link ?

Thanks

--
Ani

I have tried to build a static executable from gtk-demo from https://github.com/pavouk/lgi.

$ cd lgi/samples/gtk-demo
$ luastatic main.lua /usr/local/share/lua/5.3/lgi.lua
$(find /usr/local/share/lua/5.3/lgi -type f)
*.lua
/usr/local/lib/liblua.a /home/vm/lgi/lgi/lgi.a
-I/usr/local/include $(pkg-config --libs
gobject-introspection-1.0 gmodule-2.0 libffi)
cc -Os main.lua.c /home/vm/lgi/lgi/lgi.a /usr/local/lib/liblua.a -rdynamic -lm -ldl -o main -I/usr/local/include -lgirepository-1.0 -lgobject-2.0 -Wl,--export-dynamic -lgmodule-2.0 -pthread -lglib-2.0 -lffi

In the build vm, the resulting executable works fine, but in another vm:

$ ./main
[string "main"]:30: module 'lgi' not found:
no field package.preload['lgi']
no module 'lgi' in luastatic bundle
no file '/usr/local/share/lua/5.3/lgi.lua'
no file '/usr/local/share/lua/5.3/lgi/init.lua'
no file '/usr/local/lib/lua/5.3/lgi.lua'
no file '/usr/local/lib/lua/5.3/lgi/init.lua'
no file './lgi.lua'
no file './lgi/init.lua'
no file '/usr/local/lib/lua/5.3/lgi.so'
no file '/usr/local/lib/lua/5.3/loadall.so'
no file './lgi.so'
stack traceback:
[C]: in function 'require'
[string "main"]:30: in local 'func'
[string "main"]:36: in main chunk

It cannot find /usr/local/share/lua/5.3/lgi.lua, even though it was specified as a dependency. Could it be a problem with nested modules? lgi.lua requires lgi.init which comes from a subdirectory in /usr/local/lib. Can luastatic handle these subdirectories too? Dunno, something seems to go wrong ...

The output of luac -s some.lua works already (luac.out needs to be renamed as luac.lua) and gives much smaller binaries.

Some more streamlined option handling which allows precompiling lua scripts would be nice.

Hi,

Is it possible to supply LuaJIT object files to luastatic instead of scripts? I don't necessarily want my Lua script to appear as a string in the final binary. I ran a quick test by just adding the object file to the luastatic command but the main script failed to load the linked module - though it's likely I messed up the luastatic invokation.

Thanks!

As a newcomer to lua, I found this documentation much better than other lua-related projects' documentation.

Also, it's my first time submitting an issue to an open-source project, so please tell me if this sounds a bit over-the-top.

Anyways, I think we should add the following for people who installed lua through homebrew:

/opt/homebrew/Cellar/lua/5.4.4_1/include/lua/lauxlib.h = whenever laxulib.h is needed

/opt/homebrew/Cellar/lua/5.4.4_1/lib/liblua.a - liblua.a = whenever liblua is needed

/opt/homebrew/Cellar/lua/5.4.4_1/include/lua/ = directory to folder containing lua.h

Luastatic main.lua /opt/homebrew/Cellar/lua/5.4.4_1/lib/liblua.a -I/opt/homebrew/Cellar/lua/5.4.4_1/include/lua/ - complete lua compiler script for "main.lua"

Normally, directory paths aren't that hard to get, but they don't appear on any homebrew documentation and hence I had to dig deep to find where homebrew had put these files.

I just think this will make luastatic more convenient to use if we add this documentation. Thanks for reading.

It would be nice if luastatic could mimic standard Lua interpreter behaviour when handling errors:

• Attempt to convert non-string error objects to string if they have __tostring metamethod, replace them with (error objects is a %type% value) otherwise.
• Append traceback to string errors.
• Print error message to stderr instead of stdout.

For reference see handle_script function in recent versions of lua.c.

It seems like the current luastatic.lua only does dynamic linking:

cc -Os -std=c99 luastatic.lua.c liblua.a -rdynamic -lm -ldl -Ilua-5.2.4/src -o luastatic

I prefer to use musl-gcc which only supports static linking.
Is it possible to make luasttic.lua provide option for static linking like the following?

cc -Os -std=c99 luastatic.lua.c liblua.a -static -lm -Ilua-5.2.4/src -o luastatic