cloudlinux / libcare

libcare -- Patch Userspace Code in Live Processes

LibCare -- Patch Userspace Code on Live Processes

Welcome to LibCare --- Live Patch Updates for Userspace Processes and Libraries.

LibCare delivers live patches to any of your Linux executables or libraries at the runtime, without the need for restart of your applications. Most frequently it is used to perform critical security updates such as glibc's GHOST (aka CVE-2015-0235, see how we deal with it in GHOST sample) and QEMU's CVE-2017-2615, but can also be used for serious bug fixes on the fly to avoid interruption of service (see server sample).

See https://kernelcare.com for live Linux Kernel updates also.

FAQ

How the live patches are generated?

We use the same code generating procedure we used in production for years in kernelcare.com:

1. both original and patched source code are translated to assembler,
2. corresponding assembler files are compared and new instrumented assembler code is generated, where patches are stored into special ELF sections,
3. instrumented assembler code is compiled using target project's build system while patch ELF sections are collected in binaries',
4. binary patch files are extracted from the ELF sections.

The libcare-patch-make script is a handy script for the patch generation for a makeable project. Just do:

$ src/libcare-patch-make some_serious_bug.patch

And find binary patches for all the deliverables of the project in the patchroot directory. See our simple server for usage sample.

For more details follow to the patch preparation chapter of the internals.

How the live patches are applied?

It is a lot like loading a shared library into another process' memory:

1. our binary libcare-ctl (the doctor) attaches to a patient via ptrace(2),
2. patient's objects are examined by the doctor,
3. doctor puppets the patient to allocate patch memory near the original object,
4. references in the patch are resolved by the doctor, the patch code is ready to be executed now,
5. doctor rewrites targeted original functions with unconditional jumps to the appropriate patched versions, ensuring that no thread of patient is executing them first.

Now the patient executes patched versions of the functions.

For more details follow to the Patching chapter of the internals.

Will my patches re-apply if I restart the process?

Not at the moment. We only track start of the new processes for the tests, see here.

Does live patching affect performance?

Negligibly. Since code patches simply redirect execution from original code functions to the new ones the overhead is small and comparable to additional “jmp” instruction.

Installation and dependencies

All the Linux-distros with available libunwind, elfutils and binutils packages are supported.

However, the libcare is only tested on Ubuntu from 12.04 to 16.04 and on CentOS from 6.8 to 7.x.

Dependencies

To install the dependencies on RHEL/CentOS do the following:

$ sudo yum install -y binutils elfutils elfutils-libelf-devel libunwind-devel

To install the dependencies on Debian/Ubuntu do the following:

$ sudo apt-get install -y binutils elfutils libelf-dev libunwind-dev

Building libcare

To build libcare emit at project's root dir:

$ make -C src
...

This should build all the utilities required to produce a patch out of some project's source code.

It is highly recommended to run the tests as well, enabling Doctor libcare-ctl to attach ptracecles to any of the processes first:

$ sudo setcap cap_sys_ptrace+ep ./src/libcare-ctl
$ make -C tests && echo OK
...
OK

Now all the required tools are built and we can build some patches. Skip to server sample for that.

How does it work?

Internals are quite confusing and are described here.

libcare's Issues

Pack DWARF info for libunwind

Some applications are built without frame pointers. Investigate whether libunwind can work with these.

Why can't apply a different patch to the same target process which was already patced

When applying patch, libcare-ctl will return immediately if it finds that the target process was already patched, adn i cann't figure out why the sanity check is needed. I would appreciate it if somebody can explain why?

Any release plan for this project?

Hi, I am interested in this project by applying live patch for glibc, qemu, etc.
I saw the latest version was updated three year ago though there are many commits after that.
Is there any formal release plan for this software?

Regards
Chuan

if i have three changed .c file, use git diff create a *.diff file, how can i make a patch ?

1 diff --git a/dpdkxxxxxxx/xxxxxx/vhost.c b/dpdkxxxxxxx/xxxxx/vhost.c
2 index 3c3f6a4..36fdfb5 100644
3 --- a/dpdkxxxxx/xxxxxx/vhost.c
4 +++ b/dpdkxxxxxx/xxxxxxx/vhost.c
5
14 diff --git a/dpdkxxxxx/xxxxxxx/vhost.h b/dpdkxxxxxx/xxxxxxx/vhost.h
15 index 399b7bc..4c0aa80 100644
16 --- a/dpdkxxxxxx/xxxxxx/vhost.h
17 +++ b/dpdkxxxxx/xxxxxxx/vhost.h

27 diff --git a/dpdkxxxxxx/xxxxxxxxxx/vhost_user.c b/dpdkxxxxxx/xxxxxxx/virtio_net.c
28 index 395febb..5b957a0 100644
159 --- a/dpdkxxxxx/xxxxxx/virtio_net.c
160 +++ b/dpdkxxxxxx/xxxxxxx/virtio_net.c

How to patch QEMU using libcare?

Hi,
I got the "KPCC: cant't find output file, passing through" when using libcare-patch-make to make demo patch for qemu. It seems like libcare-cc requires obj must be compiled using “cc -o output.o output.c“，however qemu Makefile is not doing compilation in this way. I cannot make qemu demo patch using libcare-patch-make.
Could please guide us by writting a brief tutorial on how to patch QEMU using libcare ?

@paboldin @olshanov @loyfan
Thanks.
Ying

During unpatch stack can't be unwinded above patches

Due to the kpatch load as a generic object we can't unwind stack for above the code from patches.
Fix that either by providing our own unwinder or using shared object to load the patch.

add test that dynamically loads/unloads objects

first this will help to fix races in the inspecting code,
then this will help to test code that intercepts dlopen once it is done

making patch a shared object

Make patch a shared object rather then a common ELF. Load it via __libc_dlopen_mode.

print function names during unwind

Print function names during unwind

In multithreaded programs，some threads "Verifying safety for pid xxxxx" FAILED !

Verifying safety for pid 88521...
Stacktrace to verify safety for pid 88521:
[0x7fa5c32aea3d] __poll_nocancel+0x24
[0x7fa5c9d3a11f] fdset_event_dispatch+0x6f
[0x7fa5c9d3b270] rte_vhost_driver_session_start+0x10
[0x559bb0d2860b] _init+0x176513
kpatch_patch.c(201): safety check failed for 7fa5c9d39fb0

kpatch_patch.c(497): Patching xxxx.so failed, unapplying partially applied patch

Finished ptrace detaching.Failed to apply patch './libshared.kpatch'
kpatch_patch.c(588): Failed to apply patch './libshared.kpatch'

write full log to a buffer, only show it on failure

By default only log to a buffer, show full log only after failure.

Provide option to switch to verbose output.

tests should run without additional permissions

Make libcare-ctl able to run tests as children, so that no extra permissions required for testing.

can't attach to a parent 'su'

Somewhy we can't attach to a parent 'su' as this hangs us in wait4().

QEMU blocked by patching for over 1 minute

I am testing libcare with QEMU, and run into some problem.

The target is a QEMU VM with a 1TB rbd disk. The QEMU process contains 783 threads.
The patch file contains one modified function.
While patching, the QEMU process blocked for over 1 minute.
From console output, I find it's because the patch_ensure_safety() function takes too much time.

Is ensure_safety necessary when we only modify the entry point of original function with just one jump call? I think the modification is too small to influence the safety.

Can you explain how the jump call influences the safety?
Thank you!

read aux vector from /proc/auxv

It is possible to read aux vector (that contains starting address) from the /proc/auxv.

Read error when apply QEMU patch

I made QEMU patch by libcare-patch-make successfully. But After apply patch for QEMU, following error logs are reported

Copying 0x5 bytes from 0x4ceda0 to 0x555555db2240 in target... FAIL
kpatch_patch.c(495): Patching qemu-system-x86_64 failed, unapplying partially applied patch
Verifying safety for pid 32695...

I have checked kpatch_process_mem_read return is 5(I/O error) and patch info->daddr is 0x4ceda0. But qemu-system-x86_64 text section is as below

Object 'qemu-system-x86_64' (806:643262840), patch: yes
VM areas:
inmem: 555555554000-555555d5461c r-e, ondisk: 00000000-0080061c r-e
inmem: 555555f54d80-55555610c000 r--, ondisk: 00800d80-009b8000 r--
inmem: 55555610c000-5555565b4300 rw-, ondisk: 009b8000-00a16b28 rw-

0x4ceda0 (info->daddr) is invalid in qemu‘s process，Does it need add some offset to info->daddr when reading it? Or, is there something wrong when making patch but logs are not reported?

use pkgbuild make patch, failed at kp_sanity_check

when I use pkgbuild make a patch for my own project, kp_sanity_check failed. I check the symlist file, the original and patched file are different at most of the symbols at the offset value. Then I start with gilibc sample, but still failed at same step. Please see below. Can you please give some clues for this issue?
nscd.origin.symlist ------------------------------------------------------- nscd.patched.symlist
0000000000005488 0000000000000000 _init | 0000000000005420 0000000000000000 _init
0000000000006180 0000000000000674 main | 00000000000060e0 0000000000000674 main
00000000000067f4 0000000000000000 _start | 0000000000006754 0000000000000000 _start
0000000000006820 0000000000000000 deregister_tm_clones | 0000000000006780 0000000000000000 deregister_tm_clones
0000000000006850 0000000000000000 register_tm_clones | 00000000000067b0 0000000000000000 register_tm_clones
00000000000068d0 0000000000000000 frame_dummy | 0000000000006830 0000000000000000 frame_dummy
0000000000006910 000000000000007e print_version | 0000000000006870 000000000000007e print_version
0000000000006990 000000000000007f more_help | 00000000000068f0 000000000000007f more_help
0000000000006a10 0000000000000166 termination_handler | 0000000000006970 0000000000000166 termination_handler
0000000000006b80 000000000000050c parse_opt | 0000000000006ae0 000000000000050c parse_opt
0000000000007090 00000000000000aa nscd_open_socket | 0000000000006ff0 00000000000000aa nscd_open_socket
0000000000007140 0000000000000144 do_exit | 00000000000070a0 0000000000000144 do_exit
0000000000007290 000000000000003c notify_parent | 00000000000071f0 000000000000003c notify_parent
00000000000072d0 0000000000000213 nscd_run_prune | 0000000000007230 0000000000000213 nscd_run_prune
00000000000074f0 0000000000000137 fd_ready | 0000000000007450 0000000000000137 fd_ready
0000000000007630 000000000000065e restart | 0000000000007590 000000000000065e restart
0000000000007c90 0000000000000682 main_loop_epoll | 0000000000007bf0 0000000000000682 main_loop_epoll
0000000000008320 0000000000000653 main_loop_poll | 0000000000008280 0000000000000653 main_loop_poll
0000000000008980 0000000000000079 writeall | 00000000000088e0 0000000000000079 writeall
0000000000008a00 0000000000000d1e nscd_run_worker | 0000000000008960 0000000000000d1e nscd_run_worker
0000000000009720 0000000000000083 sendfileall | 0000000000009680 0000000000000083 sendfileall
00000000000097b0 000000000000150e nscd_init | 0000000000009710 000000000000150e nscd_init
000000000000acc0 0000000000000160 register_traced_file | 000000000000ac20 0000000000000160 register_traced_file
000000000000ae20 000000000000000b close_sockets | 000000000000ad80 000000000000000b close_sockets
000000000000ae30 00000000000002b9 start_threads | 000000000000ad90 00000000000002b9 start_threads
000000000000b0f0 0000000000000a01 cache_addpw | 000000000000b050 0000000000000a01 cache_addpw
000000000000bb00 0000000000000273 addpwbyX | 000000000000ba60 0000000000000273 addpwbyX
000000000000bd80 0000000000000025 addpwbyname | 000000000000bce0 0000000000000025 addpwbyname
000000000000bdb0 000000000000006a readdpwbyname | 000000000000bd10 000000000000006a readdpwbyname
000000000000be20 00000000000000de addpwbyuid | 000000000000bd80 00000000000000de addpwbyuid
000000000000bf00 000000000000009c readdpwbyuid | 000000000000be60 000000000000009c readdpwbyuid
000000000000bfa0 0000000000000222 getpwnam_r | 000000000000bf00 0000000000000222 getpwnam_r
000000000000c1d0 0000000000000222 getpwuid_r | 000000000000c130 0000000000000222 getpwuid_r
000000000000c400 0000000000000b93 cache_addgr | 000000000000c360 0000000000000b93 cache_addgr
000000000000cfa0 000000000000025d addgrbyX | 000000000000cf00 000000000000025d addgrbyX
000000000000d200 0000000000000025 addgrbyname | 000000000000d160 0000000000000025 addgrbyname
000000000000d230 0000000000000075 readdgrbyname | 000000000000d190 0000000000000075 readdgrbyname
000000000000d2b0 00000000000000de addgrbygid | 000000000000d210 00000000000000de addgrbygid
000000000000d390 000000000000009c readdgrbygid | 000000000000d2f0 000000000000009c readdgrbygid
000000000000d430 0000000000000222 getgrnam_r | 000000000000d390 0000000000000222 getgrnam_r
000000000000d660 0000000000000222 getgrgid_r | 000000000000d5c0 0000000000000222 getgrgid_r
000000000000d890 0000000000000903 cache_addhst | 000000000000d7f0 0000000000000903 cache_addhst
000000000000e1a0 00000000000003d9 addhstbyX | 000000000000e100 00000000000003d9 addhstbyX
000000000000e580 0000000000000019 addhstbyname | 000000000000e4e0 0000000000000019 addhstbyname
000000000000e5a0 000000000000006d readdhstbyname | 000000000000e500 000000000000006d readdhstbyname
000000000000e610 0000000000000019 addhstbyaddr | 000000000000e570 0000000000000019 addhstbyaddr
000000000000e630 000000000000006d readdhstbyaddr | 000000000000e590 000000000000006d readdhstbyaddr

improve logging

rework logging levels and messages

gcc __LINE__ make patch in mess by kpatch_gensrc

When I am going to make patch for QEMU, I found that a lot of unwanted assemble differences are generated by kpatch_gensrc due to gcc LINE;
That is, if I add some code in func A, the funB which is belowed under funA in the file could be recongized as changed because LINE is used and changed in funB.
There are a lot of LINE uses in qemu like error_setstage() which triggers a big patch while only a small part of it is truely needed.

Is there any good idea to solve this problem?
Maybe we should undefine LINE or just filter the changed function we want by kpatch_gensrc?

Thanks,
Chuan

Why don't you install kpatch_gensrc kpatch_strip and kpatch_make to system?

As the makefile wrote, the 3 tools have not been installed to system. So I have to install them manually or export KPATCH_PATH to where they are.
I think this is not good.
Will you improve this please?

[BUG] cannot find GOT entry for '0xb0' for two type of TLS reference

There maybe not support for two type of TLS reference which causes make patch failure.

1. Global TLS variable reference
   A simple sample is that we changed code in tls_shared as follow:
   cat libtls_shared.c
   #include <stdio.h>

__thread int tls_abc = 10;

void print_second_greetings(void)
{
tls_abc = 10;
printf("Hello from UNPATCHED shared library\n");
}

void print_third_greetings(void)
{
printf("Hello from PATCHED shared library!\n");
}

void print_greetings(void)
{
print_second_greetings();
}

IIUC, the GOT offset should be the same as static global TLS variable.
However, the code as follow seems not vaild:
if (ELF64_R_SYM(rela->r_info) == 0 &&
rela->r_addend == tls_offset)
return rela->r_offset;

2. Global TLS variable reference which is reference as extern and defined in other c file
   A simple sample is that we changed code in tls_simple as follow:

`tls_simple.c:
#include <stdio.h>
#include <time.h>
#include <stdlib.h>
#include <my.h>

int *p;

void print_greetings(void)
{
printf("TLS UNPATCHED\n");
}

int main()
{
v = 0xDEADBEAF;
p = &v;

while (1) {
	print_greetings();
	sleep(1);
}
return 0;

}

my.h
extern __thread v;

my.c:
#include <stdio.h>

__thread v = 0;`

Since TLS variable finish relocations in static link, the .rela.dyn should not include TLS variable.
As for now, libcare still to do --rel-fixup and find this TLS variable in relocation table which results in make patch failure.
IIUC, we should just simply not to do --rel-fixup as same as R_X86_64_TPOFF32?

Fail to build patch for GHOST sample

When trying to build patch for glibc in GHOST sample, there is an error on step ./scripts/pkgbuild packages/rhel7/glibc/glibc-2.17-55.el7:

eu-unstrip: cannot find matching section for [40] '.symtab'

Patch file does not appear in /kcdata.

Used docker container from repository and all previous steps from readme were done.