jxy-s / vfdynf

Application Verifier Dynamic Fault Injection

Application Verifier Dynamic Fault Injection

vfdynf.dll is an application verifier provider that implements unique-stack based systematic fault injection to simulate low resource scenarios.

The integration also works with the command line (the TEST type is DynFault):

appverif DynFault ... -for TARGET ... [-with [TEST.]PROPERTY=VALUE ...]

"Dynamic Fault Injection" (DynFault) is a replacement for "Low Resource Simulation" (LowRes) tests. LowRes is a probability-based (randomized) fault injection. In contrast, DynFault tracks stack hashes when determining where to inject faults. This provides better coverage when simulating low resource scenarios. DynFault injects failures for wait, heap, virtual memory, registry, file, event, section, and OLE string APIs. These are the same APIs as LowRes.

The ability to exclude modules in LowRes is limited. DynFault, in contrast, enables you to exclude stacks containing symbols matched by a set of regular expressions. Why is this helpful? I'll provide an example, which was the impetus for me reversing the undocumented parts of verifier to implement this library. MSVC implemented debug iterators which are valuable to identify bugs but break noexcept contracts. For example, the default std::string constructor is marked noexcept but with debug iterators enabled an allocation could occur within it and throw an exception. The cpp exception handling then can't locate a handler past noexcept. The contract is such that if an exception would cross that boundary the implementation should terminate the program. Hopefully you can see the problem with the limited functionality of LowRes (you can't use it with debug iterators). To solve this DynFault has a property that allows you to define a list of regular expressions. When DynFault encounters a stack matching any expression in this list, that stack hash is excluded from fault injection. As an example, this regular expression tries to isolate stacks containing std::basic_string's default constructor:

\s.*!.*_Alloc_proxy<.*>\s.*!std::basic_string<.*>::basic_string<.*>\s

The above regular expression will match on this stack:

testdynf.exe!heap_alloc_dbg_internal
testdynf.exe!heap_alloc_dbg
testdynf.exe!_malloc_dbg
testdynf.exe!malloc
testdynf.exe!operator new
testdynf.exe!std::_Default_allocate_traits::_Allocate
testdynf.exe!std::_Allocate<16,std::_Default_allocate_traits,0>
testdynf.exe!std::allocator<std::_Container_proxy>::allocate
testdynf.exe!std::_Container_base12::_Alloc_proxy<std::allocator<std::_Container_proxy> >
testdynf.exe!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::basic_string<char,std::char_traits<char>,std::allocator<char> >
testdynf.exe!main
testdynf.exe!invoke_main
testdynf.exe!__scrt_common_main_seh
testdynf.exe!__scrt_common_main
testdynf.exe!mainCRTStartup
KERNEL32.dll!BaseThreadInitThunk
ntdll.dll!RtlUserThreadStart

Enabling the best of both worlds - debug iterators and fault injection!

DynFault Properties (Options)

Name	Type	Description
GracePeriod	DWORD	Delays fault injection until after this period, in milliseconds.
SymbolSearchPath	String	Symbol search path used for dynamic fault injection and applying exclusions.
ExclusionsRegex	MultiString	Excludes stack from fault injection when one of these regular expression matches the stack.
DynamicFaultPeriod	DWORD	Clears dynamic stack fault injection tracking on this period, in milliseconds, zero does not clear tracking.
EnableFaultMask	QWORD	Mask of which fault types are enabled. Bit 1=Wait, 2=Heap, 3=VMem, 4=Reg, 5=File, 6=Event, 7=Section, 8=Ole, 9=InPage.
FaultProbability	DWORD	Probability that a fault will be injected (0 - 1000000).
FaultSeed	DWORD	Seed used for fault randomization. A value of zero will generate a random seed.
WaitExclusionsRegex	MultiString	Excludes stack from wait fault injection when one of these regular expression matches the stack.
HeapExclusionsRegex	MultiString	Excludes stack from heap fault injection when one of these regular expression matches the stack.
VMemExclusionsRegex	MultiString	Excludes stack from virtual memory fault injection when one of these regular expression matches the stack.
RegExclusionsRegex	MultiString	Excludes stack from registry fault injection when one of these regular expression matches the stack.
FileExclusionsRegex	MultiString	Excludes stack from file fault injection when one of these regular expression matches the stack.
EventExclusionsRegex	MultiString	Excludes stack from event fault injection when one of these regular expression matches the stack.
SectionExclusionsRegex	MultiString	Excludes stack from section fault injection when one of these regular expression matches the stack.
OleExclusionsRegex	MultiString	Excludes stack from OLE fault injection when one of these regular expression matches the stack.
InPageExclusionsRegex	MultiString	Excludes stack from section in-page fault injection when one of these regular expression matches the stack.

Installation

At this time there is no installer/script to automate installation. Here are the instructions to manually install the library:

1. copy vfdynf.dll to C:\Windows\System32 (or SysWOW64 for x86 support on an x64 OS)
2. add vfdynf.dll to the "Application Verifier Global Settings" "Verified Providers" list (again WOW6432Node when appropriate)
   • HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\{ApplicationVerifierGlobalSettings}
   • VerifierProviders

At this point vfdynf.dll is "registered" with application verifier and it should show up in the options via the command line or in the user interface.

Building

The repo uses submodules, after cloning be sure to init and update the submodules.

git clone https://github.com/jxy-s/vfdynf
cd .\vfdynf\
git submodule update --init --recursive
MSBuild .\vfdynf.sln

Credits

The following are used without modification. Credits to their authors.

• System Informer Native API Headers Collection of Native API header files. Gathered from Microsoft header files and symbol files, as well as a lot of reverse engineering and guessing.

• PCRE2 - Perl-Compatible Regular Expressions The PCRE2 library is a set of C functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5. PCRE2 has its own native API, as well as a set of wrapper functions that correspond to the POSIX regular expression API. The PCRE2 library is free, even for building proprietary software. It comes in three forms, for processing 8-bit, 16-bit, or 32-bit code units, in either literal or UTF encoding.

And, Grandfather Derpington ;)

vfdynf's Issues

Fault REGEX Suppresion Based on Fault Type

We want a way to suppress memory allocation failure injection during initialization (preferably without also suppressing other fault injection). That can be done today by a regex matching one of the stack frames present during agent startup, but it looks like there's not a way to have separate exclusion lists for each type of fault. That means we're stuck either opting out of memory allocation failure faults entirely, or we have to exempt the init path from all faults.

Fault Injection for In-page Operations for File Mappings

A nice to have feature would be fault injection for the actual in-page operations for file mappings. We could do something as simple as randomly changing the page protection of the mapping to PAGE_NOACCESS in the vfdynf hook to cause a fault on the first access, or something as advanced as creating a duplicate anonymous mapping, copying the data from the real mapped region, then randomly selecting a page in the mapped region to change protection to PAGE_NOACCESS.

CRT failure

This issue was reported privately to me:

2: kd> k
 # Child-SP          RetAddr               Call Site
00 00000014`a251da30 00007fff`2bf851b3     vrfcore!VerifierStopMessageEx+0x807
01 00000014`a251dd90 00007fff`2bf83ae4     vfbasics!AVrfpFreeMemLockChecks+0xff
02 00000014`a251ddf0 00007fff`2bf92f4a     vfbasics!AVrfpFreeMemNotify+0x38
03 00000014`a251de20 00007fff`2b9ad3d4     vfbasics!AVrfpRtlFreeHeap+0x9a
04 00000014`a251ded0 00007fff`2b9b5c70     vfdynf!_free_base+0x1c [minkernel\crts\ucrt\src\appcrt\heap\free_base.cpp @ 105] 
05 00000014`a251df00 00007fff`2b9ae7ed     vfdynf!__acrt_lowio_destroy_handle_array+0x40 [minkernel\crts\ucrt\src\appcrt\lowio\osfinfo.cpp @ 59] 
06 00000014`a251df30 00007fff`2b9b2038     vfdynf!__acrt_uninitialize_lowio+0x21 [minkernel\crts\ucrt\src\appcrt\lowio\ioinit.cpp @ 261] 
07 00000014`a251df60 00007fff`2b99ff9e     vfdynf!__acrt_execute_uninitializers+0x30 [minkernel\crts\ucrt\src\appcrt\internal\shared_initialization.cpp @ 58] 
08 00000014`a251df90 00007fff`2b99f9f7     vfdynf!__scrt_uninitialize_crt+0x1a [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\utility\utility.cpp @ 221] 
09 00000014`a251dfc0 00007fff`2b99faca     vfdynf!dllmain_crt_process_detach+0x67 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp @ 195] 
0a (Inline Function) --------`--------     vfdynf!dllmain_crt_dispatch+0xb [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp @ 220] 
0b 00000014`a251e000 00007fff`2c283d2a     vfdynf!dllmain_dispatch+0xb6 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp @ 288] 
0c 00000014`a251e060 00007fff`37a6a610     verifier!AVrfpStandardDllEntryPointRoutine+0xda
0d 00000014`a251e0e0 00007fff`2bf87549     vrfcore!VfCoreStandardDllEntryPointRoutine+0x180
0e 00000014`a251e170 00007fff`41075d37     vfbasics!AVrfpStandardDllEntryPointRoutine+0xe9
0f 00000014`a251e1f0 00007fff`410785e9     ntdll!LdrpCallInitRoutine+0x6f
10 00000014`a251e260 00007fff`41078854     ntdll!LdrpInitializeNode+0x1c1
11 00000014`a251e3b0 00007fff`4112bc30     ntdll!LdrpInitializeGraphRecurse+0x80
12 00000014`a251e3f0 00007fff`41126362     ntdll!AVrfInitializeVerifier+0x258
13 00000014`a251f500 00007fff`410c841b     ntdll!LdrpInitializeProcess+0x18de
14 00000014`a251f950 00007fff`410c82a3     ntdll!LdrpInitialize+0x15f
15 00000014`a251f9f0 00007fff`410c824e     ntdll!LdrpInitialize+0x3b
16 00000014`a251fa20 00000000`00000000     ntdll!LdrInitializeThunk+0xe

After investigation it appears that there are still paths that can cause vfdynf to not play nicely with the CRT. There was a previous effort for #12 via #13. The fix in #13 did help things substantially but it isn't a complete fix, in rare cases under stress scenarios the problem can still present itself, it seems. This change tried to avoid CRT initialization entirely. But, looking at the CRT items the std lib appears to still do some initialization that isn't immediately obvious even tho there are none from vfdynf itself:

The best fix for vfdynf would be to move away from any reliance on cpp or (more specifically) the MSVC standard library. The following is the only context that vfdynf needs to keep track of:

vfdynf/vfdynf/vrf_fault.cpp

Lines 15 to 25 in e49a057

	struct GlobalContext
	{
	DWORD TypeBase = MAXDWORD;
	std::mutex Lock;
	uint64_t LastClear = 0;
	std::unordered_map<uint32_t, StackEntry> StackTable;
	bool ExclusionsRegexInitialized = false;
	std::vector<std::wregex> ExclusionsRegex;
	};
	static GlobalContext* g_Context = nullptr;

std::mutx, std::vector, and std::unordered_map could be very easily avoided, the core functionality of regex exclusions relies on std::wregx. In order to completely move away from the standard library here a suitable replacement would need to be chosen. Alternatively there might be some compiler options to avoid it, but IIRC the regex in the standard library relies on the locale support - which is present in the CRT tables.

A fix for this will take some research and tinkering, but I'll attend to it when I can.

regex can result in recursive SRW lock acquire through apphelp

APPLICATION_VERIFIER_SRWLOCK_RECURSIVE_ACQUIRE (253)
The SRW lock is being acquired recursively by the same thread.
This stop is generated if the SRW lock (Param1) is being acquired
recursively by the same thread.
This will result in a deadlock and the thread would block indefinitely.
Recursive acquisition of an SRW lock in exclusive mode will cause a deadlock.
Recursive acquisition of an SRW lock in shared mode will cause a deadlock when
there is a thread waiting for exclusive access. Consider the example below:
- Thread A acquires the SRW lock in shared mode
- Thread B tries to acquire the SRW lock in exclusive mode and waits
- Thread A tries to acquire the SRW lock in shared mode recursively. This will
  be successful as long as there is no exclusive waiter (in this case B). Since
  SRW locks do not have writer starvation, thread A waits behind thread B.
  Now, Thread B is waiting for Thread A which is inturn waiting for Thread B
  causing a circular wait and hence a deadlock.
$ kb - to get the current stack trace. This is where the SRW lock is being
acquired recursively. 
$ dps Param2 - to get the stack trace for the first acquire. 
Arguments:
Arg1: 0a14afd8, SRW Lock 
Arg2: 02747b64, Address of the first acquire stack trace. Use dps <address> to see where the SRW lock was acquired. 
Arg3: 00000000, Not used 
Arg4: 00000000, Not used 


vrfcore!VerifierStopMessageEx+0x5b8
vfbasics!AVrfpVerifySRWLockAcquire+0xb8
vfbasics!AVrfpRtlAcquireSRWLockShared+0x3a
vfcuzz!VfCuzzRtlAcquireSRWLockShared+0x36
apphelp!SE_GetProcAddressForCaller+0x3af
ntdll!LdrGetProcedureAddressForCaller+0x36b
KERNELBASE!GetProcAddressForCaller+0x4d
KERNEL32!GetProcAddressStub+0x14
vfdynf!try_get_function+0x52
vfdynf!try_get_CompareStringEx+0x16
vfdynf!__acrt_eagerly_load_locale_apis+0xa
vfdynf!_lock_locales+0x5
vfdynf!std::_Lockit::_Lockit+0x14
vfdynf!std::locale::_Init+0x17
vfdynf!std::_Regex_traits<wchar_t>::_Regex_traits<wchar_t>+0x60
vfdynf!std::vector<std::basic_regex<wchar_t,std::regex_traits<wchar_t> >,std::allocator<std::basic_regex<wchar_t,std::regex_tra+0x15d
vfdynf!fault::InitExclusionsRegex+0x106
vfdynf!fault::IsStackOverriddenByRegex+0x6f
vfdynf!fault::ShouldFaultInject+0x7e0
vfdynf!Hook_NtOpenFile+0x5b
apphelp!SeUtilsIsSystem+0x80
apphelp!SepRouterHookIAT+0x112
apphelp!SE_DllLoaded+0x8c
ntdll!LdrpSendPostSnapNotifications+0x119
ntdll!LdrpNotifyLoadOfGraph+0x44
ntdll!LdrpPrepareModuleForExecution+0x4f
ntdll!LdrpLoadDllInternal+0x11e
ntdll!LdrpLoadDll+0x7e
ntdll!LdrLoadDll+0x97
vfbasics!AVrfpLdrLoadDll+0x52
KERNELBASE!LoadLibraryExW+0x14f
[REDACTED]
KERNEL32!BaseThreadInitThunk+0x19
ntdll!__RtlUserThreadStart+0x2b
ntdll!_RtlUserThreadStart+0x1b

I did not realize that regex relies on local support, this code in vfdyn should not be done "once" on first invocation. Rather, it should be done during process attach. So this:
https://github.com/jxy-s/vfdynf/blob/main/vfdynf/vrf_fault.cpp#L100

Should be moved to here:
https://github.com/jxy-s/vfdynf/blob/main/vfdynf/vrf_fault.cpp#L448

And the init_once logic should be removed. I don't remember if there was a reason that I put the regex initialization downstream. I will investigate and apply an appropriate patch.

Some hooks are not equivalent to LowRes

Some of the hook implementations in vfdynf are not producing the same result that the original "Low Resource Simulation" (LowRes) verification does. For example, the wait object hooks do not honor the timeout value in the same manner. And heap allocation paths don't raise exceptions when HEAP_GENERATE_EXCEPTIONS is specified. The hooks in vfdynf should present faults in the same manner as the low resource simulation that is out of the box from verifier.