mozilla / masche

Deprecated - MIG Memory Forensic library

MASCHE

MIG Memory Forensic library

⚠️ Deprecation Notice ⚠️

Mozilla is no longer maintaining the Mozilla InvestiGator (MIG) project.

Mozilla is also no longer making use of this code internally.

You are welcome to use this code as is with no warranty. Please fork it to continue development.

MASCHE stands for Memory Analysis Suite for Checking the Harmony of Endpoints. It is being developed as a project for the Mozilla Winter of Security program.

It works on Linux, Mac OS and Windows.

These are the current features:

• listlibs: Searches for processes that have loaded a certain library.
• pgrep: Has the same functionallity as pgrep on linux.
• memaccess/memsearch: Allows access and search into a given process memory.

You can find examples under the examples folder.

Compiling

You need golang installed.

Linux

You need glibc for 64 and 32 bits installed. On Fedora, the packages are:

• glibc-devel.i686
• glibc-devel.x86_64
• glibc-headers.i686
• glibc-headers.x86_64
• glibc.i686
• glibc.x86_64

Windows

In order to compile and run masche in windows you will need a gcc compiler. You can use mingw if you are running a 32 bits version of Windows or mingw-64 if you are running a 64 bits one. Just run go build on the package/example that you want.

It's possible to cross-compile from linux. And this is the recommended way.

• Install a cross compiler (for example, mingw-w64)
• Enable cross compiling in your go toolchain (run GOOS=windows ./all.bash inside your $GOROOT/src folder)

After that you should be able to cross compile masche without problems, just make sure to export the correct global variables: GOOS=windows CGO_ENABLED=1 CC=<your-cross-compiler> (for example: CC=x86_64-w64-ming32-gcc )

masche's Issues

WalkMemory sliding window

memaccess should provide some sliding window capability based on WalkMemory function. This will allow us to match strings that cross the WalkMemory's buffers boundaries.

Field test: find python programs that call socket.recvfrom_into

@gdestuynder and I were discussing security issue CVE-2014-1912 today, and he asked an interested question: could masche find in-memory python programs that call socket.recvfrom_into ?

Travis CI free usage ends Dec 3; mozilla repos should switch to other CI platforms

We're opening this issue because your project has used Travis CI within the last 6 months. If you have already migrated off it, you can close and ignore this issue.

Travis CI is ending free builds on public repositories. travis-ci.com stopped providingthem in early November, and travis-ci.org will stop after December 31, 2020. To avoid disruptions to your workflows, you must migrate to another CI service.

For production use cases, we recommend switching to CircleCI. This service is already widely used within Mozilla. There is a guide to migrating from Travis CI to CircleCI available here.

For non production use cases, we recommend either CircleCI or Github Actions. There is a guide to migrating from Travis CI to Github Actions available here. Github Actions usage within Mozilla is new, and you will have to work with our github administrators to enable specific actions following this process.

If you have any questions, reach out in #github-admin:mozilla.org on matrix.

Parsing maps file should support whitespaces in paths

Memory mapped files may contain spaces in their paths, and the current parsing mechanism uses whitespaces as separators.

For example, this may happen in a maps file:

08048000-083ba000 r-xp 00000000 08:01 529344 /home/user/program with spaces/mylibrary.so.4

Hey, this project seems a bit stalled, isn't it?

Thanks for aclaration

Change MEMORY_BASIC_INFORMATION to it's corresponding value.

The MEMORY_BASIC_INFORMATION structure is mapped to the default one in my OS, so if I want to see a 32bit process memory it will return a MEMORY_BASIC_INFORMATION_32 instead of a MEMORY_BASIC_INFORMATION_64 (See the remarks section on http://msdn.microsoft.com/en-us/library/windows/desktop/aa366775%28v=vs.85%29.aspx ).

This should be done calling the IsWow64Process function before getting the memory information. Probably the best way to do this is creating a new struct that can hold both values.

Memory Grepper for Mac OS

Be able to grep a process memory in Mac Os searching for given strings.

Provide pgrep functionality in Mac OS

YARA rules in MASCHE

I am going to begin adding the capability to match YARA rules against memory content. I will draw on the file scanning YARA capability (pull 335) for inspiration.

Permission Denied is being reported as a softerror instead of harderror

memaccess reports a permission denied as a softerror instead of a harderror

This is the softerror:

"Retries exceeded on reading 2048 bytes starting at 8048000: Error while reading 2048 bytes starting at 8048000: open /proc/22470/mem: permission denied"

Also, the error description is very redundant. Should we do something about it?

Memory Grepper for Windows

Be able to grep a process memory in Windows searching for given strings.

Add `offset` and `maxlength` options to WalkMemory

WalkMemory currently supports starting at a given address. It would be nice to also support stopping after X number of bytes are read. This should be exported in memsearch as well.
I'd also suggest renaming the starting point address into offset to match terms commonly used for this purpose.

OS X listlibs implementation is missing

api mismatch

masche/process/process_windows.go

Line 31 in dbff04b

func getAllPids() (pids []uint, harderror error, softerrors []error) {

the func signature for windows does not match those for others (and cause a build failure)

It should has the same return type as:

masche/process/process.go

Line 44 in dbff04b

func OpenAll() (ps []Process, softerrors []error, harderror error) {

And in the same file, in func Name(), line 22

masche/process/process_windows.go

Line 22 in dbff04b

harderror, softerrors = cresponse.GetResponsesErrors(unsafe.Pointer(r))

It gets softerrors and harderror misplaced (again) as the cresponse show the following:

masche/cresponse/cresponse.go

Line 27 in dbff04b

func GetResponsesErrors(responsePointer unsafe.Pointer) (softerrors []error, harderror error) {

Cross compiling darwin from linux

Howdy gents,
I think we covered this before, but I can't remember the outcome: is there a way to cross-compile Masche from linux to macos?

Here's the output I'm currently getting:

$ make CGO_ENABLED=1 mig-agent OS=darwin
mkdir -p bin/darwin/amd64
echo building mig-agent for darwin/amd64
building mig-agent for darwin/amd64
if [ ! -r conf/mig-agent-conf.go.inc ]; then echo "conf/mig-agent-conf.go.inc configuration file does not exist" ; exit 1; fi
# test if the agent configuration variable contains something different than the default value
# and if so, replace the link to the default configuration with the provided configuration
if [ conf/mig-agent-conf.go.inc != "conf/mig-agent-conf.go.inc" ]; then rm mig-agent/configuration.go; cp conf/mig-agent-conf.go.inc mig-agent/configuration.go; fi
GOOS=darwin GOARCH=amd64 GO15VENDOREXPERIMENT=1 go build  -o bin/darwin/amd64/mig-agent-20160427-0.ffb37c9.dev"" -ldflags "-X mig.ninja/mig.Version=20160427-0.ffb37c9.dev" mig.ninja/mig/mig-agent
# net
could not determine kind of name for C.AI_MASK
# mig.ninja/mig/vendor/github.com/mozilla/masche/process
could not determine kind of name for C.close_process_handle
could not determine kind of name for C.open_process_handle
could not determine kind of name for C.process_handle_t

gcc errors for preamble:
In file included from vendor/github.com/mozilla/masche/process/process_c_wrapper.go:5:0:
process.h:46:45: error: unknown type name 'process_handle_t'
 response_t *open_process_handle(pid_tt pid, process_handle_t *handle);
                                             ^
process.h:53:34: error: unknown type name 'process_handle_t'
 response_t *close_process_handle(process_handle_t process_handle);
                                  ^

Makefile:56: recipe for target 'mig-agent' failed
make: *** [mig-agent] Error 2

Global Makefile shouldn't run 64-bits tests

Running on Fedora 21, with glibc-headers x86_64 and i686 installed, I get the following:

$ make all
go get -u github.com/mozilla/masche/process
go get -u github.com/mozilla/masche/memsearch
go get -u github.com/mozilla/masche/memaccess
go get -u github.com/mozilla/masche/listlibs
make -C test/tools test64
make[1]: Entering directory '/home/ulfr/git/masche/test/tools'
gcc -Wall -Wextra -pedantic -std=c99 -O0 known_byte_sequences.c -o test
make[1]: Leaving directory '/home/ulfr/git/masche/test/tools'
go test ./memaccess ./memsearch ./process
ok      _/home/ulfr/git/masche/memaccess    0.139s
ok      _/home/ulfr/git/masche/memsearch    0.076s
ok      _/home/ulfr/git/masche/process  0.024s
make -C test/tools test32
make[1]: Entering directory '/home/ulfr/git/masche/test/tools'
gcc -Wall -Wextra -pedantic -std=c99 -O0 known_byte_sequences.c -m32 -o test
In file included from /usr/include/features.h:389:0,
                 from /usr/include/stdlib.h:24,
                 from known_byte_sequences.c:2:
/usr/include/gnu/stubs.h:7:27: fatal error: gnu/stubs-32.h: No such file or directory
 # include <gnu/stubs-32.h>
                           ^
compilation terminated.
Makefile:11: recipe for target 'test32' failed
make[1]: *** [test32] Error 1
make[1]: Leaving directory '/home/ulfr/git/masche/test/tools'
Makefile:22: recipe for target 'testbin32' failed
make: *** [testbin32] Error 2

Move list_libs to a new package

It should be in a new package so it can be used as a MIG Module.

Export Status() in Process interface

Would there be a way to return a system specific struct that contains detailed status information about a running process? I'd like to export additional information about the processes found in the MIG module, specifically the ppid, uid and gid. Here's what linux exposes:

Name:   firefox
State:  S (sleeping)
Tgid:   3038
Ngid:   0
Pid:    3038
PPid:   2514
TracerPid:      0
Uid:    1000    1000    1000    1000
Gid:    1000    1000    1000    1000
FDSize: 512
Groups: 10 1000 
VmPeak:  4819864 kB
VmSize:  3911816 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:   3274432 kB
VmRSS:   1500900 kB
VmData:  2536664 kB
VmStk:       356 kB
VmExe:       108 kB
VmLib:    130480 kB
VmPTE:      6780 kB
VmPMD:        32 kB
VmSwap:        0 kB
Threads:        68
SigQ:   0/31010
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 0000000f820044ef
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
Seccomp:        0
Cpus_allowed:   ff
Cpus_allowed_list:      0-7
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        10113106
nonvoluntary_ctxt_switches:     75792

go: Command not found

make

go get -u github.com/mozilla/masche/process
make: go: Command not found
Makefile:7: recipe for target 'get' failed
make: *** [get] Error 127

What is go?
Where is in Readme?
Go must die, Rewrite on C++ :-P

listlib fails on macos

Invoking listlib on macos results in a panic that seems to be related to C code. This issue surfaced in MIG modules tests, but can be reproduced using the listlib example:

julien@Juliens-Mac-mini[16:45:54]:examples [master -]$ sudo ./listlibs -r libssl
Password:
listlibs(27868,0x7fff77910180) malloc: *** error for object 0xe000000000000000: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
SIGABRT: abort
PC=0x7fff8ef25212
signal arrived during cgo execution

goroutine 1 [syscall, locked to thread]:
runtime.cgocall_errno(0x4000ef0, 0xc2080559c8, 0xc200000000)
        /usr/local/Cellar/go/1.4.2/libexec/src/runtime/cgocall.go:130 +0xf5 fp=0xc2080559a8 sp=0xc208055980
github.com/mozilla/masche/listlibs._Cfunc_free_loaded_libraries_list(0x4600040, 0xe000000000000000)
        /Users/julien/.go/src/github.com/mozilla/masche/listlibs/:32 +0x45 fp=0xc2080559c8 sp=0xc2080559a8
github.com/mozilla/masche/listlibs.listLoadedLibraries(0x42679f8, 0xc20802a2b0, 0xc208074000, 0x92, 0x93, 0x0, 0x0, 0x41e2e30, 0x0, 0x0)
        /Users/julien/.go/src/github.com/mozilla/masche/listlibs/listlibs_darwin.go:56 +0x4ef fp=0xc208055ad0 sp=0xc2080559c8
github.com/mozilla/masche/listlibs.ListLoadedLibraries(0x42679f8, 0xc20802a2b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/julien/.go/src/github.com/mozilla/masche/listlibs/listlibs.go:11 +0x89 fp=0xc208055b28 sp=0xc208055ad0
github.com/mozilla/masche/listlibs.GetMatchingLoadedLibraries(0x42679f8, 0xc20802a2b0, 0xc20805e0a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/julien/.go/src/github.com/mozilla/masche/listlibs/listlibs.go:18 +0xa0 fp=0xc208055c00 sp=0xc208055b28
main.findProcWithLib(0xc20805e0a0, 0xc208072000, 0x63, 0x80, 0xc20800a360, 0x0, 0x0, 0x41e2e30, 0x0, 0x0)
        /Users/julien/Code/masche/examples/listlibs.go:61 +0x180 fp=0xc208055d30 sp=0xc208055c00
main.main()
        /Users/julien/Code/masche/examples/listlibs.go:35 +0x443 fp=0xc208055f98 sp=0xc208055d30
runtime.main()
        /usr/local/Cellar/go/1.4.2/libexec/src/runtime/proc.go:63 +0xf3 fp=0xc208055fe0 sp=0xc208055f98
runtime.goexit()
        /usr/local/Cellar/go/1.4.2/libexec/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208055fe8 sp=0xc208055fe0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/Cellar/go/1.4.2/libexec/src/runtime/asm_amd64.s:2232 +0x1

rax     0x0
rbx     0x6
rcx     0x7fff5fbffba8
rdx     0x0
rdi     0x707
rsi     0x6
rbp     0x7fff5fbffbd0
rsp     0x7fff5fbffba8
r8      0x7fff7790f278
r9      0x0
r10     0x20000000
r11     0x206
r12     0x4225000
r13     0x4255000
r14     0x7fff77910180
r15     0x1
rip     0x7fff8ef25212
rflags  0x206
cs      0x7
fs      0x0
gs      0x0

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

warning in init() if the lib is compiled for 32 bits and running on 64 bits OS

Make a better test for listlibs

Our current test doesn't work. We are testing which processes have libc loaded, and that's almost all of them. We should create a dummy library that only a process known by us loads.

I think the current tests isn't spotting a bug on linux.

Process test failing on windows

(copied from issue #27 )
@Brinsky wrote:

I'm not sure how much it matters, but some of the tests are failing if I run make (via mingw-w64). The tests for memaccess and memsearch run fine, but the process test seems to run into some privilege issues. This happens whether or not I run the test with elevated permissions. And the common test doesn't even seem to exist (there is only common_linux_test.go). Perhaps all of these tests were designed specifically for Linux and I shouldn't be trying to run them? Here is the output of the test/build process:

C:\Users\Brian\Documents\Git Projects\Clones\masche>mingw32-make
go get -u github.com/mozilla/masche/process
go get -u github.com/mozilla/masche/memsearch
go get -u github.com/mozilla/masche/memaccess
go get -u github.com/mozilla/masche/listlibs
mingw32-make -C test/tools test64
mingw32-make[1]: Entering directory C:/Users/Brian/Documents/Git Projects/Clone s/masche/test/tools' gcc -Wall -Wextra -pedantic -std=c99 -O0 known_byte_sequences.c -o test mingw32-make[1]: Leaving directoryC:/Users/Brian/Documents/Git Projects/Clones
/masche/test/tools'
go test ./memaccess ./memsearch ./process ./common
ok /C/Users/Brian/Documents/Git_Projects/Clones/masche/memaccess 0.264s
ok /C/Users/Brian/Documents/Git_Projects/Clones/masche/memsearch 0.325s
--- FAIL: TestProcessName (0.00s)
process_test.go:44: System error number 299: Only part of a ReadProcessM
emory or WriteProcessMemory request was completed.

Pid: 328 failed to Open. Error: System error number 5: Access is denied.

. . .

FAIL
FAIL /C/Users/Brian/Documents/Git_Projects/Clones/masche/process 0.148s
? /C/Users/Brian/Documents/Git_Projects/Clones/masche/common [no test
files]
mingw32-make: *** [run_tests64] Error 1

Create Memory Dump for a Process in Windows

Given a Process ID we should be able to create a process dump programmatically.

Create a common interface for memory access

Most code could be reused if we have a common interface for reading processes' memory for the different operating systems.

Some initial work has been pushed to a feature branch. The main file is: https://github.com/mozilla/masche/blob/memory_reading_interface/memsearch/memory_access.h

Create Memory Dump for a Process in Linux

Given a Process ID we should be able to create a process dump programmatically.

#include statements for a missing process_windows.h

I am trying to build MASCHE on Windows 7 using the make tool included with MinGW-W64. Unfortunately, I'm getting stuck on errors about a missing process_windows.h. These errors seem to stem from #include statements within process_windows.c and process_windows.go. All of this is happening in the $GOPATH\src\github.com\mozilla\masche\process directory, which was pulled by the command go get -u github.com/mozilla/masche/process in the makefile. It is worth noting that process_windows.h definitely does not exist within that directory.

Any help would be much appreciated, I am looking forward to trying MASCHE out. Thanks!

Provide pgrep functionality in Linux

Better error handling on listlibs package

If there's an error while processing a specific pid it should be reported back instead of just logged.

Fix stylistic issues

Go Report Card has a thing or two to say about the code. It would be nice to address them: https://goreportcard.com/report/github.com/mozilla/masche

Add multiple encodings support

Masche only support UTF-8 now.

We want to support: utf-8, utf-16, utf-32, and windows-1252.

Add a Makefile

The project should have a Makefile at the root that can build the various examples and run the tests.

Memory Grepper for Linux

Be able to grep a process memory in Linux searching for given strings.

Create Memory Dump for a Process in Mac OS

Given a Process ID we should be able to create a process dump programmatically.

Test Fails for Linux 32 bit binaries.

--- FAIL: TestSearchInOtherProcess (0.02 seconds)
grep_mem_test.go:53: memoryGrep failed for case 1, the following buffer should be found: [13 14 10 13 11 14 14 15]

I think the problem is that the maps file is parsed incorrectly. This doesn't have anything to do with issue #17 (it was was before commit c90734e )

memaccess.WalkMemory doesn't respect the starting address parameter

Right now memsearch searches just for the first occurrence and stops there, and if you call the function again from the previous address it may return the same address because of how it works (there may be two occurences of the same string in the same memory page).

We should provide a FindAllOccurrences that returns a list with all occurences.

Provide pgrep functionality in Windows