Make better dependency management about libmem

[rdbo]

Now both Python and Rust dynamically fetch the library, avoiding annoyances from building and linking libmem.
The only thing left to do is make this more friendly for C/C++
On Linux, it's pretty good already due to the CMakeLists thing that dynamically fetches and links libmem.
But on Windows, most people will use Visual Studio projects, that's probably what should be aiming for
Either way, a lot of progress on this field 💯

[MolotovCherry]

I completely understand the reasons. I still think that having a ready to go build.rs file in the docs at least with some steps would be really helpful (and thankfully, this build.rs won't mess with the one already in the crate on crates.io)

E.g.

1. Clone libmem repo to a subfolder in your project with git clone --recurse-submodules --remote-submodules <YourGitHubUrl>
2. Drop this build.rs script in your project to build it. If you want, you can implement additional functionality to save/cache the generated libs and hardcode the path to them so they don't recompile all the time

Or you can go about it this other way with these other steps, and install it globally to make it easy on yourself later, with this other build script which is simple and hardcoded

(One bonus the former one has is when it comes to github actions builds, which can also be cached, though I should note that github actions will fail to build this unless you set CARGO_TARGET_DIR to something like D:\target, since cmake runs out of path space on the build otherwise)

(I got this building on github actions, and it's quite convenient)

By the way, thanks for your library and the hard work you put into it! It's really nice! 😄

[rdbo]

I'm working on other projects right now, but I see that this is definitely something to worry about. I'll pin this issue so (hopefully) I don't forget. Thanks for your suggestion 👍

[nathan818fr]

About pre-built binaries, I've made a repo that creates them for multiple platforms (the script can be used standalone or via GitHub Actions to create GitHub Releases): https://github.com/nathan818fr/libmem-build
Feel free to fork or use parts of this repo.

There is one difference regarding linux: it provide the static libraries independently (so it use liblibmem_partial.a instead of the one created by makebundle.sh). This makes the artifacts more similar between Windows and Linux.

PS: Thanks for libmem

[rdbo]

I actually thought about that for a while (and I had implemented it even), but I decided not to implement it simply because you can't ship the whole repository to crates.io (too big) and I wouldn't want the build to be reliable on GitHub.
An alternative to this could be a script that downloads pre-built binaries for every platform and puts them in the correct path. But I have to take time to make binary builds for every single platform.

[rdbo]

@nathan818fr
That's impressive! We could definitely use that here 💯
For the static library, I think one might have linking issues with it. If I remember correctly, the linker does not link multiple static libraries together into a single one, so the third party dependencies will be left off. And that's what makebundle.sh is supposed to solve (it generates a big binary though).
With that said, I will be taking a better look at the repo, and PRs are welcome if you wish to contribute directly
Having that implemented in the main repo will save a lot of time here, thanks for your efforts 👍

[nathan818fr]

Yes, with my way, all libraries must be linked to the target program.
That's what I'm doing here on a project using libmem: https://github.com/nathan818fr/ts3-server-hook/blob/9ed27aaea3b4edb15fe9a984c0d5709d9e1f99b6/cmake/libmem-config.cmake

It's true that it's probably simpler to distribute just one big static binary (probably easier for users, etc.).
In that case, would it be nice to do the same thing on Windows (so it would give a similar result between the different platforms with only: liblibmem.so|libmem.dll and liblibmen.a|libmem.lib)? I'll take a look.

Are you OK with the current artifact structure?

• include/libmem/ - directory with libmem headers
• lib/ - directory with libmem dynamic and static libraries
• licenses/ - directory containing all licenses (libmem and dependencies)
• GLIBC_VERSION.txt|MUSL_VERSION.txt|MSVC_VERSION.txt|[...] - single-line text file providing essential version information for the platform (e.g. the glibc version used at compile-time generally indicates the minimum version required by the runtime - that's why I compile on debian buster, the oldoldstable)

I will submit a PR soon.

[rdbo]

@nathan818fr IIRC, MSVC is able to ship all the libraries into one without any additional steps, so that's why the makebundle.sh script doesn't run for Windows.
As for the structure, I think it's pretty good. Everything important seems to be included 💯

[nathan818fr]

Maybe I'm missing something, but when I build with MSVC I don't get all shipped into one library. I get:

• libmem.lib (946 KiB)
• injector.lib (76 KiB)
• capstone.lib (2 426 KiB)
• keystone.lib (3 5801 KiB)
• LIEF.lib (172 023 KiB)
• llvm.lib (4 156 KiB)

And if I try to compile a demo that uses LM_HookCode, it doesn't work without explicitly including the other libraries.

libmem build logs

git clone --recurse-submodules https://github.com/rdbo/libmem.git

cmake -S . -B build-static -DLIBMEM_BUILD_STATIC=ON
cmake --build build-static


**********************************************************************
** Visual Studio 2022 Developer PowerShell v17.8.1
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
PS C:\Users\user\Tmp> git clone --recurse-submodules https://github.com/rdbo/libmem.git
[...]
PS C:\Users\user\Tmp> cd libmem
PS C:\Users\user\Tmp\libmem> cmake -S . -B build-static -DLIBMEM_BUILD_STATIC=ON
-- The C compiler identification is MSVC 19.38.33130.0
-- The CXX compiler identification is MSVC 19.38.33130.0
-- The ASM compiler identification is MSVC
-- Found assembler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.38.33130/bin/Hostx64/x64/cl.exe
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.38.33130/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.38.33130/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- [*] Platform: Windows
-- [*] Build tests: OFF
-- [*] Build static library: ON
-- [*] Build for 32 bits: OFF
-- CMAKE_C_FLAGS: /DWIN32 /D_WINDOWS
-- CMAKE_CXX_FLAGS: /DWIN32 /D_WINDOWS /EHsc
-- Configuring done (4.2s)
-- Generating done (0.1s)
-- Build files have been written to: C:/Users/user/Tmp/libmem/build-static
PS C:\Users\user\Tmp\libmem> cmake --build build-static
[...]
[100%] Linking CXX static library libmem.lib
[100%] Built target libmem
PS C:\Users\user\Tmp\libmem> dir -s build-static "*.lib"

    Directory: C:\Users\user\Tmp\libmem\build-static

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        05/12/2023     10:10         968908 libmem.lib

    Directory: C:\Users\user\Tmp\libmem\build-static\external

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        05/12/2023     10:10          77874 injector.lib


    Directory: C:\Users\user\Tmp\libmem\build-static\external\capstone-engine-prefix\src\capstone-engine-build

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        05/12/2023     10:06        2483872 capstone.lib

    Directory: C:\Users\user\Tmp\libmem\build-static\external\keystone-engine-prefix\src\keystone-engine-build\llvm\lib

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        05/12/2023     10:07       36660216 keystone.lib

    Directory: C:\Users\user\Tmp\libmem\build-static\external\lief-project-prefix\src\lief-project-build

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        05/12/2023     10:10      176151074 LIEF.lib

    Directory: C:\Users\user\Tmp\libmem\build-static\external\llvm

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        05/12/2023     10:10        4255850 llvm.lib

test compile logs (non-working + working)

demo.c

#include <libmem/libmem.h>
#pragma comment(lib, "shell32.lib")

void a() { printf("a\n"); }
void b() { printf("b\n"); }

int main(int argc, char **argv) {
  LM_HookCode((lm_address_t) &a, (lm_address_t) &b, 0);
  a();
  return 0;
}

It don't compile when I only link libmem.lib:

PS C:\Users\user\Tmp\libmem> cl demo.c /nologo /MDd /Iinclude /link build-static/libmem.lib
   Creating library demo.lib and object demo.exp
libmem.lib(asm.c.obj) : error LNK2019: unresolved external symbol cs_open referenced in function LM_DisassembleEx
libmem.lib(asm.c.obj) : error LNK2019: unresolved external symbol cs_close referenced in function LM_DisassembleEx
libmem.lib(asm.c.obj) : error LNK2019: unresolved external symbol cs_disasm referenced in function LM_DisassembleEx
libmem.lib(asm.c.obj) : error LNK2019: unresolved external symbol cs_free referenced in function LM_DisassembleEx
libmem.lib(asm.c.obj) : error LNK2019: unresolved external symbol ks_open referenced in function LM_AssembleEx
libmem.lib(asm.c.obj) : error LNK2019: unresolved external symbol ks_close referenced in function LM_AssembleEx
libmem.lib(asm.c.obj) : error LNK2019: unresolved external symbol ks_asm referenced in function LM_AssembleEx
libmem.lib(asm.c.obj) : error LNK2019: unresolved external symbol ks_free referenced in function LM_AssembleEx
demo.exe : fatal error LNK1120: 8 unresolved externals

It compile when I link libmem.lib, capstone.lib, keystone.lib and llvm.lib (LIEF.lid and injector.lib not required here):

PS C:\Users\user\Tmp\libmem> cl demo.c /nologo /MDd /Iinclude /link build-static/libmem.lib build-static/external/capstone-engine-prefix/src/capstone-engine-build/capstone.lib build-static/external/keystone-engine-prefix/src/keystone-engine-build/llvm/lib/keystone.lib build-static/external/llvm/llvm.lib
demo.c
   Creating library demo.lib and object demo.exp
PS C:\Users\user\Tmp\libmem> ./demo.exe
b

Edit: OP also link with all libraries:

Edit 2: It works well when also bundling libraries manually on Windows: nathan818fr@d4314b5

[rdbo]

@nathan818fr That's interesting. I really thought MSVC resolved the linking problems for me, apparently not. Good to know.
I'll try your manual bundling for Windows later 👍

[rdbo]

In Python, the setup.py script will fetch a binary release from the GitHub if it has not found a libmem installation in the OS. The same could be done for Rust.