hillu / go-yara

Go bindings for YARA

go-yara

Go bindings for YARA, staying as close as sensible to the library's C-API while taking inspiration from the yara-python implementation.

Build/Installation

On Unix-like systems, libyara version 4.3, corresponding header files, and pkg-config must be installed. Adding go-yara v4 to a project with Go Modules enabled, simply add the proper dependency…

import "github.com/hillu/go-yara/v4"

…and rebuild your package.

If libyara has been installed to a custom location, the PKG_CONFIG_PATH environment variable can be used to point pkg-config at the right yara.pc file.

For anything more complicated, refer to the "Build Tags" section below. Instructions for cross-building go-yara for different operating systems or architectures can be found in README.cross-building.md.

To build go-yara on Windows, a GCC-based build environment is required, preferably one that includes pkg-config. The 32-bit and 64-bit MinGW environments provided by the MSYS2 provide such an environment.

Build Tags

Static builds

The build tag yara_static can be used to tell the Go toolchain to run pkg-config with the --static switch. This is not enough for a static build; the appropriate linker flags (e.g. -extldflags "-static") still need to be passed to the go tool.

Building without pkg-config

The build tag yara_no_pkg_config can be used to tell the Go toolchain not to use pkg-config's output. In this case, any compiler or linker flags have to be set via the CGO_CFLAGS and CGO_LDFLAGS environment variables, e.g.:

export CGO_CFLAGS="-I${YARA_SRC}/libyara/include"
export CGO_LDFLAGS="-L${YARA_SRC}/libyara/.libs -lyara"
go install -tags yara_no_pkg_config github.com/hillu/go-yara

If libyara has been linked against other libraries (e.g. libcrypto, libmagic) and a static build is performed, these libraries also need to be added to CGO_LDFLAGS.

YARA 4.1.x vs. earlier versions

This version of go-yara can only be used with YARA 4.3 or later.

Version of go-yara compatible with YARA 4.1.x are available via the v4.2.x branch or tagged v4.2.* releases.

Version of go-yara compatible with YARA 4.1.x are available via the v4.1.x branch or tagged v4.1.* releases.

Version of go-yara compatible with YARA 4.0.x are available via the v4.0.x branch or tagged v4.0.* releases.

Versions of go-yara compatible with YARA 3.11 are available via the v3.x branch or tagged v3.* releases.

Versions of go-yara compatible with earlier 3.x versions of YARA are available via the v1.x branch or tagged v1.* releases.

License

BSD 2-clause, see LICENSE file in the source distribution.

Author

Hilko Bengen <[email protected]>

go-yara's Issues

could not determine kind of name for C.ERROR_INSUFICIENT_MEMORY

It appears that there is a typo in error.go...
C.ERROR_INSUFICIENT_MEMORY: "insufficient memory",

should be
C.ERROR_INSUFFICIENT_MEMORY: "insufficient memory",

This does not make it work for me yet, but at least now I am getting errors that make some sense...

Defining the same variable twice results in incorrect behaviour

I found that if I define the same variable twice in the compiler, scanning gives incorrect results.

Here's a minimal testcase:

package main

import (
	"fmt"
	"github.com/hillu/go-yara"
)

func errcheck(err error) { if err != nil { panic(err) } }

func main() {
	c, err := yara.NewCompiler()
	errcheck(err)
	err = c.DefineVariable("category", "")
	errcheck(err)
	err = c.DefineVariable("category", "")
	errcheck(err)
	err = c.AddString(`rule category_is_empty { condition: category == ""}`, "")
	errcheck(err)
	r, err := c.GetRules()
	errcheck(err)
	r.DefineVariable("category", "http")
	z, err := r.ScanMem([]byte{}, 0, 0)
	fmt.Printf("%#v\n", z)
}

One would expect that the rule category_is_empty wouldn't be matched, since category was defined to be equal to "http". However, the rule does get matched.

If you remove the second definition, the code works as intended.

Error when getting go-yara

I have a fresh Debian box with the latest version of Yara (3.11) as well as Golang which is 1.14.1. When running go get github.com/hillu/go-yara I get the following output:

go install github.com/hillu/go-yara
# github.com/hillu/go-yara
../go/src/github.com/hillu/go-yara/rule.go:144:8: could not determine kind of name for C.META_TYPE_NULL
../go/src/github.com/hillu/go-yara/rule.go:193:27: could not determine kind of name for C.RULE_GFLAGS_GLOBAL
../go/src/github.com/hillu/go-yara/rule.go:188:27: could not determine kind of name for C.RULE_GFLAGS_PRIVATE
cgo: 
gcc errors for preamble:
../go/src/github.com/hillu/go-yara/rule.go: In function 'string_matches':
../go/src/github.com/hillu/go-yara/rule.go:81:36: error: macro "yr_string_matches_foreach" requires 3 arguments, but only 2 given
  yr_string_matches_foreach(s, match) {
                                    ^
../go/src/github.com/hillu/go-yara/rule.go:81:2: error: 'yr_string_matches_foreach' undeclared (first use in this function); did you mean 'string_matches'?
  yr_string_matches_foreach(s, match) {
  ^~~~~~~~~~~~~~~~~~~~~~~~~
  string_matches
../go/src/github.com/hillu/go-yara/rule.go:81:2: note: each undeclared identifier is reported only once for each function it appears in
../go/src/github.com/hillu/go-yara/rule.go:81:27: error: expected ';' before '{' token
  yr_string_matches_foreach(s, match) {
                           ^          ~
                           ;

By running pkg-config I get the following:

pkg-config --libs yara
-L/usr/local/lib -lyara

pkg-config --cflags yara
-DUSE_LINUX_PROC -pthread -DMAGIC_MODULE -DHASH_MODULE -I/usr/local/include

Any ideas?

Add auto finding includes and libs

At least on Mac os x when using go-yara with a Yara being installed using brew trhough an error.

# github.com/hillu/go-yara
../../../github.com/hillu/go-yara/compiler.go:15:10: fatal error: 'yara.h' file not found
#include <yara.h>

I found a solution for this error, but I am not sure whether it will work on others systems as well.

The solution is adding this lines on compiler.go

diff --git a/compiler.go b/compiler.go
index 4c5a493..30ddbf8 100644
--- a/compiler.go
+++ b/compiler.go
@@ -5,6 +5,8 @@
 package yara

 /*
+#cgo pkg-config: --libs-only-L yara
+#cgo pkg-config: --cflags-only-I yara
 #ifdef _WIN32
 #define fdopen _fdopen
 #define dup _dup

Hope this helps.

segmentation violation when match found

I've run into a weird issue while playing with go-yara in a test project. After further testing I've found that the same issue occurs in the simple-yara test thats provided with this repo.

The yara rule I'm using in my testing is a very basic detection for Mimikatz:

rule MimikatzMem
{
	strings:
	    $s1 = "sekurlsa::msv" fullword ascii
	    $s2 = "sekurlsa::wdigest" fullword ascii
	    $s4 = "sekurlsa::kerberos" fullword ascii
	    $s5 = "sekurlsa::tspkg" fullword ascii
	    $s6 = "sekurlsa::livessp" fullword ascii
	    $s7 = "sekurlsa::ssp" fullword ascii
	    $s8 = "sekurlsa::logonPasswords" fullword ascii
	    $s9 = "sekurlsa::process" fullword ascii
	    $s10 = "sekurlsa::minidump" fullword ascii
	    $s11 = "sekurlsa::pth" fullword ascii
	    $s12 = "sekurlsa::tickets" fullword ascii
	    $s13 = "sekurlsa::ekeys" fullword ascii
	    $s14 = "sekurlsa::dpapi" fullword ascii
	    $s15 = "sekurlsa::credman" fullword ascii
	condition:
		1 of them
}

As this rule will match on itself the error I'm seeing can be triggered by scanning the rule itself (i.e. ./simple-yara -rule rules.yar rules.yar)

Doing so produces the following crash:

2020/01/02 14:43:37 Scanning file rules.yar... 
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x4b4687]

runtime stack:
runtime.throw(0x4fcbd1, 0x2a)
	/home/user/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
	/home/user/usr/local/go/src/runtime/signal_unix.go:378 +0x47c

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x4b4650, 0xc0000bd678, 0x1)
	/home/user/usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc0000bd648 sp=0xc0000bd610 pc=0x405beb
github.com/hillu/go-yara._Cfunc_string_matches(0x27fa400, 0x0, 0xc00009c0bc)
	_cgo_gotypes.go:1070 +0x45 fp=0xc0000bd678 sp=0xc0000bd648 pc=0x4a4e85
github.com/hillu/go-yara.(*String).Matches(0xc0000bd7a0, 0x0, 0x0, 0x0)
	/home/user/go/src/github.com/hillu/go-yara/rule.go:225 +0x6b fp=0xc0000bd710 sp=0xc0000bd678 pc=0x4a82eb
github.com/hillu/go-yara.(*Rule).getMatchStrings(0xc00009e050, 0x0, 0x0, 0x0)
	/home/user/go/src/github.com/hillu/go-yara/rule.go:244 +0x2c2 fp=0xc0000bd818 sp=0xc0000bd710 pc=0x4a8742
github.com/hillu/go-yara.(*MatchRules).RuleMatching(0xc0000940c0, 0xc00009e050, 0xc0000940c0, 0x7f507c7e6fa8, 0xc0000940c0)
	/home/user/go/src/github.com/hillu/go-yara/rules_callback.go:151 +0x224 fp=0xc0000bd978 sp=0xc0000bd818 pc=0x4a95b4
github.com/hillu/go-yara.scanCallbackFunc(0x1, 0x27f9020, 0x278e7e0, 0x40e4b6)
	/home/user/go/src/github.com/hillu/go-yara/rules_callback.go:90 +0x220 fp=0xc0000bda58 sp=0xc0000bd978 pc=0x4a8fc0
github.com/hillu/go-yara._cgoexpwrap_08a63f7e5a95_scanCallbackFunc(0x1, 0x27f9020, 0x278e7e0, 0xee341bf499c3200)
	_cgo_gotypes.go:1570 +0x3d fp=0xc0000bda88 sp=0xc0000bda58 pc=0x4a595d
runtime.call32(0x0, 0x7ffeaaa3b570, 0x7ffeaaa3b600, 0x20)
	/home/user/usr/local/go/src/runtime/asm_amd64.s:539 +0x3b fp=0xc0000bdab8 sp=0xc0000bda88 pc=0x455abb
runtime.cgocallbackg1(0x0)
	/home/user/usr/local/go/src/runtime/cgocall.go:314 +0x1b7 fp=0xc0000bdba0 sp=0xc0000bdab8 pc=0x405f97
runtime.cgocallbackg(0x0)
	/home/user/usr/local/go/src/runtime/cgocall.go:191 +0xc1 fp=0xc0000bdc08 sp=0xc0000bdba0 pc=0x405d41
runtime.cgocallback_gofunc(0x405c0f, 0x4b4860, 0xc0000bdc98, 0xc0000bdc88)
	/home/user/usr/local/go/src/runtime/asm_amd64.s:793 +0x9b fp=0xc0000bdc28 sp=0xc0000bdc08 pc=0x45708b
runtime.asmcgocall(0x4b4860, 0xc0000bdc98)
	/home/user/usr/local/go/src/runtime/asm_amd64.s:640 +0x42 fp=0xc0000bdc30 sp=0xc0000bdc28 pc=0x456f22
runtime.cgocall(0x4b4860, 0xc0000bdc98, 0xc0000bdc90)
	/home/user/usr/local/go/src/runtime/cgocall.go:131 +0x7f fp=0xc0000bdc68 sp=0xc0000bdc30 pc=0x405c0f
github.com/hillu/go-yara._Cfunc_yr_rules_scan_file(0x27f8ed0, 0x27cc090, 0x0, 0x4b3f00, 0x278e7e0, 0x0, 0x0)
	_cgo_gotypes.go:1455 +0x4d fp=0xc0000bdc98 sp=0xc0000bdc68 pc=0x4a542d
github.com/hillu/go-yara.(*Rules).ScanFileWithCallback.func2(0xc00009e038, 0x27cc090, 0x0, 0x278e7e0, 0x0, 0xc0000bddb8)
	/home/user/go/src/github.com/hillu/go-yara/rules.go:121 +0xe6 fp=0xc0000bdcf0 sp=0xc0000bdc98 pc=0x4aade6
github.com/hillu/go-yara.(*Rules).ScanFileWithCallback(0xc00009e038, 0x7ffeaaa3d520, 0x9, 0x0, 0x0, 0x4d3240, 0xc0000940c0, 0x0, 0x0)
	/home/user/go/src/github.com/hillu/go-yara/rules.go:121 +0x1ce fp=0xc0000bddf8 sp=0xc0000bdcf0 pc=0x4a89de
github.com/hillu/go-yara.(*Rules).ScanFile(...)
	/home/user/go/src/github.com/hillu/go-yara/rules.go:100
main.main()
	/home/user/go-yara/_examples/simple-yara/simple-yara.go:117 +0x729 fp=0xc0000bdf60 sp=0xc0000bddf8 pc=0x4b3889
runtime.main()
	/home/user/usr/local/go/src/runtime/proc.go:203 +0x21e fp=0xc0000bdfe0 sp=0xc0000bdf60 pc=0x42f0be
runtime.goexit()
	/home/user/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000bdfe8 sp=0xc0000bdfe0 pc=0x4577d1

As best I can tell this is crashing when trying to get the matches from some C binding related to getting string matches.

Any ideas on how to fix this?

Thanks

Import 'github.com/hillu/go-yara/internal/callbackdata" makes forking harder

When someone clones this repository the path github.com/hillu/go-yara/internal/callbackdata imported from utils.go causes the following error:

util.go:10:2: use of internal package not allowed

As the path contains internal it can only be imported from code inside the same module (ie: github.com/hillu/go-yara) the cloned repo has a different path and can't use this import path.

All users cloning this repository for experimentation will be faced with this error and may think that the code in the original repository is broken.

Errors cross compiling go-yara

I am trying to cross compile go-yara for x86 and x64 and I am getting some errors.
The tool set for cross compiling is xgo.

I can cross compile correctly for darwin or windows, but I am unable to do it for linux.

x64

/usr/local/lib/libyara.a(math.o): In function `string_entropy':
math.c:(.text+0xecc): undefined reference to `log2'
/usr/local/lib/libyara.a(math.o): In function `data_entropy':
math.c:(.text+0x1150): undefined reference to `log2'
collect2: error: ld returned 1 exit status

x86

/usr/bin/ld: skipping incompatible /usr/local/lib/libyara.a when searching for -lyara
/usr/bin/ld: skipping incompatible //usr/local/lib/libyara.a when searching for -lyara
/usr/bin/ld: cannot find -lyara
collect2: error: ld returned 1 exit status

I am not sure that the errors came either from go-yara or yara itself.

If you need more datils, please ask them.

Completely wipes .yar file && "Could not open file"

So, I made a very rough draft of a program. And I have the following code:

func printMatches(m []yara.MatchRule, err error) {
	if err == nil {
		if len(m) > 0 {
			for _, match := range m {
				log.Printf("- [%s] %s ", match.Namespace, match.Rule)
			}
		} else {
			log.Print("no matches.")
		}
	} else {
		panic(err.Error())
	}
}

func main() {
	compiler, err := yara.NewCompiler()
	if err != nil {
		panic(err.Error())
	}
	file, err := os.Create("/home/user/Documents/Golang/code/Go-Away-AV/ShadowTech.yar")
	if err != nil {
		panic(err.Error())
	}
	compiler.AddFile(file, "Infinity.yar")
	scanner, err := compiler.GetRules()
	if err != nil {
		panic(err.Error())
	}
	scanned, err := scanner.ScanFile("/home/user/Documents/SecurityUSB - Backup/Windows/Anti malware", 0, 0)
	printMatches(scanned, err)
}

However, It for some reason completely deletes everything in the .yar file, while keeping the file itself. It is also throwing the error: "could not open file" at the 'ScanFile()' section

Scanner: Returns and empty "matches" list

I download and installed the library and wrote a quick script to test if it works.

package main

import (
	"os"
	"path"
	"path/filepath"
	log "github.com/Sirupsen/logrus"
	"github.com/hillu/go-yara"
)

func CompileAllRules() (*yara.Compiler, *yara.Scanner, error) {
	log.Info("Start")
	var rule_count = 0
	var invalid_rules = 0

	compiler, err := yara.NewCompiler()
	if err != nil {
		log.Info(err)
		os.Exit(0)
	}

	current_path, cerr := os.Executable()
	if(cerr != nil){
		log.Info(cerr)
		os.Exit(0)
	}
	rules_path := filepath.Dir(current_path)

	log.Info("[COMPILER] Looking for Rules in ", rules_path)
	_ = filepath.Walk(rules_path, func(filePath string, fileObj os.FileInfo, ferr error) error {
		fileName := fileObj.Name()
		if (path.Ext(fileName) == ".yar") || (path.Ext(fileName) == ".yara") {
			rulesObj, _ := os.Open(filePath)
			defer rulesObj.Close()
			if(compiler.AddFile(rulesObj, "") != nil){
				compiler.Destroy()
				a, ferr := yara.NewCompiler()
				compiler = a
				invalid_rules+=1
				if ferr != nil {
					log.Info(ferr)
					os.Exit(0)
				}
			}else{
				rule_count+=1
			}
		}
			return nil
	})

	rules, err := compiler.GetRules()
	if(err != nil || rules == nil){
		log.Info("[COMPILER] Could not get the rules")
		os.Exit(0)
	}

	scanner, err := yara.NewScanner(rules)
	if(err != nil){
		log.Info("[SCANNER] Could not generate a scanner")
		os.Exit(0)
	}

	log.Info("[COMPILER] Compiled: ", rule_count, " Invalid: ", invalid_rules)
	log.Info("[SCANNER] New scanner initiated: ", scanner)
	return compiler, scanner, cerr
}

var (
	compiler *yara.Compiler
	scanner *yara.Scanner
)

func main() {
	

	compiler, scanner, _ = CompileAllRules()
	
	
	var matches []yara.MatchRule
	_ = filepath.Walk("/root", func(filePath string, fileObj os.FileInfo, ferr error) error {
		fileName := fileObj.Name()
		if (path.Ext(fileName) == ".yar") || (path.Ext(fileName) == ".yara") {
			//log.Info("[scanner] Scanning file: ", fileName)
			matches, _ = scanner.ScanFile(fileName)
			if (len(matches) != 0) {
				log.Info("[SCANNER] Mathes found: ", len(matches))
			}
		}
			return nil
	})
}

The code simply loops through the current directory looking for valid rules, Compiles them and returns a scanner to scan the files with. However when I try to scan the files which I know for a fact that the YARA rules have a match on, it returns an empty list.

Sample output:

INFO[0000] Start                                        
INFO[0000] [COMPILER] Looking for Rules in /root/yarax-scanner 
INFO[0001] [COMPILER] Compiled: 465 Invalid: 67         
INFO[0001] [SCANNER] New scanner initiated: &{0x7ffaa842be50 0xc0000a1360 <nil>}

It might be my mistake rather than the library, nevertheless I wanted to ask.

Instructions to build statically linked Linux binaries

I'm trying to build a statically linked Linux binary using on Go 1.11:

GOOS=linux GOARCH=amd64 go build --ldflags '-s -w -extldflags "-static"' -o agent

But I'm having some difficulties. I have compiled Yara from the sources with --without-crypto, but I get the following errors:

/usr/bin/ld: /tmp/go-link-968020938/000011.o: in function `unixDlOpen':
/home/user/.gvm/pkgsets/go1.11/global/pkg/mod/github.com/mattn/[email protected]/sqlite3-binding.c:37945: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000019.o: in function `mygetgrouplist':
/home/user/.gvm/gos/go1.11/src/os/user/getgrouplist_unix.go:16: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000018.o: in function `mygetgrgid_r':
/home/user/.gvm/gos/go1.11/src/os/user/cgo_lookup_unix.go:38: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000018.o: in function `mygetgrnam_r':
/home/user/.gvm/gos/go1.11/src/os/user/cgo_lookup_unix.go:43: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000018.o: in function `mygetpwnam_r':
/home/user/.gvm/gos/go1.11/src/os/user/cgo_lookup_unix.go:33: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000018.o: in function `mygetpwuid_r':
/home/user/.gvm/gos/go1.11/src/os/user/cgo_lookup_unix.go:28: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000016.o: in function `_cgo_18049202ccd9_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:49: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: i386 architecture of input file `/tmp/go-link-968020938/000000.o' is incompatible with i386:x86-64 output
/usr/bin/ld: //usr/local/lib/libyara.a(math.o): in function `string_entropy':
math.c:(.text+0xea7): undefined reference to `log2'
/usr/bin/ld: //usr/local/lib/libyara.a(math.o): in function `data_entropy':
math.c:(.text+0x1126): undefined reference to `log2'

(*yara.Rules).Destroy() Memory Leak

Hi hillu,

It seems that the (*yara.Rules).Destroy() function does not free all the memory allocated in C.

I wrote an example that reproduces the problem:

package main

import (
	"fmt"
	"os"
	"runtime"
	"strconv"
	"time"

	yara "github.com/hillu/go-yara"
)

const YaraRule = `rule MemoryLeak_%d {
strings:
	$s1 = "memory"
	$s2 = "leak"
condition:
	1 of them
}`

var variables = map[string]interface{}{
	"filename":  "",
	"filepath":  "",
	"filetype":  "",
	"extension": "",
}

func memoryLeak(i int64) {
	defer runtime.GC()
	r, err := yara.Compile(fmt.Sprintf(YaraRule, i), variables)
	if err != nil {
		panic(err)
	}
	defer r.Destroy()
	if _, err := r.ScanMem([]byte(fmt.Sprintf("memory leak %d", i)), 0, time.Second); err != nil {
		panic(err)
	}
}

func main() {
	n, err := strconv.ParseInt(os.Args[1], 10, 64)
	if err != nil {
		panic(err)
	}
	var i int64
	for ; i < n; i++ {
		memoryLeak(i)
	}
	time.Sleep(time.Second)
}

This program compiles rule by rule and destroys the compiled rule before initializing the next one. If Destroy() would work correctly, the memory usage for n=1 should be approximately the same as for n=10000.

Here is an program that uses the above compiled program to measure the memory usage peek:

package main

import (
	"fmt"
	"os/exec"
	"syscall"
)

func main() {
	for _, n := range []int64{
		1, 10, 100, 1000,
	} {
		cmd := exec.Command("./memoryleak", fmt.Sprint(n))
		if err := cmd.Run(); err != nil {
			panic(err)
		}
		kb := cmd.ProcessState.SysUsage().(*syscall.Rusage).Maxrss / 1024
		fmt.Printf("%.4d: %.5d KB\n", n, kb)
	}
}

This produces the following output on macOS:

0001: 02612 KB
0010: 04704 KB
0100: 06156 KB
1000: 06872 KB

Do you know, weather this is a memory leak or not? I tried the same with a similar program that uses strings.Contains(...) instead of YARA to exclude that this is a Golang problem:

package main

import (
	"fmt"
	"os"
	"runtime"
	"strconv"
	"strings"
	"time"
)

const ContainsString = "No memory leak %d"

func noMemoryLeak(i int64) {
	defer runtime.GC()
	if !strings.Contains(fmt.Sprintf(ContainsString, i), fmt.Sprintf("leak %d", i)) {
		panic("assertion error")
	}
}

func main() {
	n, err := strconv.ParseInt(os.Args[1], 10, 64)
	if err != nil {
		panic(err)
	}
	var i int64
	for ; i < n; i++ {
		noMemoryLeak(i)
	}
	time.Sleep(time.Second)
}

and the measurements were:

0001: 01328 KB
0010: 01480 KB
0100: 01488 KB
1000: 01484 KB

Kind Regards,

Marcel Gebhardt

Assertion failed: compiler->errors == 0

When a rule fails to compile it causes a crash

ERRO[0000] failed to add rule to compiler: /rules/yara/yara_mixed_ext_vars.yar: undefined identifier "filename"  animal=walrus
Assertion failed: compiler->errors == 0 (compiler.c: yr_compiler_get_rules: 825)
SIGABRT: abort
PC=0x7fe370f11f1b m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7fe370f11f1b
stack: frame={sp:0x7ffe3a5bee58, fp:0x0} stack=[0x7ffe3a5ac0f8,0x7ffe3a5bf120)
00007ffe3a5bed58:  00353238203a7365  0000564c0e1b74a0
00007ffe3a5bed68:  00007ffe3a5bf070  0000564c0d6e60c5 <runtime.gentraceback+6117>
00007ffe3a5bed78:  0000564c0df763e0  0000564c0e1b74a0
00007ffe3a5bed88:  0000564c0d6f0da1 <runtime.goexit+1>  00007ffe3a5bee80
00007ffe3a5bed98:  0000002200000000  00000000ffffffff
00007ffe3a5beda8:  00007ffe3a5bedf8  0000564c0d6b03f3 <runtime.sysMap+211>
00007ffe3a5bedb8:  000000c41fff0000  0000000000000001
00007ffe3a5bedc8:  0000002200000003  0000564c0d9bfbcc <github.com/maliceio/malice-yara/vendor/github.com/pkg/errors.Wrapf+236>
00007ffe3a5bedd8:  000000c42005ffe0  0000000000000000
00007ffe3a5bede8:  0000000000000000  00007ffe3a5bee38
00007ffe3a5bedf8:  0000564c0d6ac107 <runtime.persistentalloc1+439>  0000000000000007
00007ffe3a5bee08:  000000c42005f7e8  00007ffe3a5bf0a0
00007ffe3a5bee18:  ffffffffffffffff  000000000000015e
00007ffe3a5bee28:  000000000000015d  0000000000000200
00007ffe3a5bee38:  00007fe370f1456f  0000564c0d6eb647 <runtime.persistentalloc.func1+71>
00007ffe3a5bee48:  0000003000000010  0000000000000006
00007ffe3a5bee58: <00007fe370f12035  0000000000000000
00007ffe3a5bee68:  0000000000000000  0000000000000000
00007ffe3a5bee78:  00007fe370cbd628  00007fe370cbdb10
00007ffe3a5bee88:  0000000000000339  0000564c0d6c660a <runtime.main+538>
00007ffe3a5bee98:  0000005000048176  0000564c0dae699e <main.main.func2+382>
00007ffe3a5beea8:  000001f00027dd40  0000000000000000
00007ffe3a5beeb8:  0000000000000000  00007fe37115a2c0
00007ffe3a5beec8:  0000000000000000  ffffffffffffffff
00007ffe3a5beed8:  00007fe370f138fc  000000c42005f7e8
00007ffe3a5beee8:  00007fe370f1543c  ffffffffffffffff
00007ffe3a5beef8:  000000c42005f7e8  00007ffe3a5bf0a0
00007ffe3a5bef08:  00007fe370ee9fb4  ffffffffffffffff
00007ffe3a5bef18:  00007fe370eea004  0000564c0d6f0da1 <runtime.goexit+1>
00007ffe3a5bef28:  00007fe370c95163  0000000000000000
00007ffe3a5bef38:  0000000000000000  0000564c0dad2410 <github.com/maliceio/malice-yara/vendor/github.com/urfave/cli.(*App).Run+1520>
00007ffe3a5bef48:  000001d000271e03  0000000000000000
runtime: unknown pc 0x7fe370f11f1b
stack: frame={sp:0x7ffe3a5bee58, fp:0x0} stack=[0x7ffe3a5ac0f8,0x7ffe3a5bf120)
00007ffe3a5bed58:  00353238203a7365  0000564c0e1b74a0
00007ffe3a5bed68:  00007ffe3a5bf070  0000564c0d6e60c5 <runtime.gentraceback+6117>
00007ffe3a5bed78:  0000564c0df763e0  0000564c0e1b74a0
00007ffe3a5bed88:  0000564c0d6f0da1 <runtime.goexit+1>  00007ffe3a5bee80
00007ffe3a5bed98:  0000002200000000  00000000ffffffff
00007ffe3a5beda8:  00007ffe3a5bedf8  0000564c0d6b03f3 <runtime.sysMap+211>
00007ffe3a5bedb8:  000000c41fff0000  0000000000000001
00007ffe3a5bedc8:  0000002200000003  0000564c0d9bfbcc <github.com/maliceio/malice-yara/vendor/github.com/pkg/errors.Wrapf+236>
00007ffe3a5bedd8:  000000c42005ffe0  0000000000000000
00007ffe3a5bede8:  0000000000000000  00007ffe3a5bee38
00007ffe3a5bedf8:  0000564c0d6ac107 <runtime.persistentalloc1+439>  0000000000000007
00007ffe3a5bee08:  000000c42005f7e8  00007ffe3a5bf0a0
00007ffe3a5bee18:  ffffffffffffffff  000000000000015e
00007ffe3a5bee28:  000000000000015d  0000000000000200
00007ffe3a5bee38:  00007fe370f1456f  0000564c0d6eb647 <runtime.persistentalloc.func1+71>
00007ffe3a5bee48:  0000003000000010  0000000000000006
00007ffe3a5bee58: <00007fe370f12035  0000000000000000
00007ffe3a5bee68:  0000000000000000  0000000000000000
00007ffe3a5bee78:  00007fe370cbd628  00007fe370cbdb10
00007ffe3a5bee88:  0000000000000339  0000564c0d6c660a <runtime.main+538>
00007ffe3a5bee98:  0000005000048176  0000564c0dae699e <main.main.func2+382>
00007ffe3a5beea8:  000001f00027dd40  0000000000000000
00007ffe3a5beeb8:  0000000000000000  00007fe37115a2c0
00007ffe3a5beec8:  0000000000000000  ffffffffffffffff
00007ffe3a5beed8:  00007fe370f138fc  000000c42005f7e8
00007ffe3a5beee8:  00007fe370f1543c  ffffffffffffffff
00007ffe3a5beef8:  000000c42005f7e8  00007ffe3a5bf0a0
00007ffe3a5bef08:  00007fe370ee9fb4  ffffffffffffffff
00007ffe3a5bef18:  00007fe370eea004  0000564c0d6f0da1 <runtime.goexit+1>
00007ffe3a5bef28:  00007fe370c95163  0000000000000000
00007ffe3a5bef38:  0000000000000000  0000564c0dad2410 <github.com/maliceio/malice-yara/vendor/github.com/urfave/cli.(*App).Run+1520>
00007ffe3a5bef48:  000001d000271e03  0000000000000000

goroutine 1 [syscall]:
runtime.cgocall(0x564c0dae81c5, 0xc42005f7e8, 0x564c0de01b60)
	/usr/lib/go/src/runtime/cgocall.go:128 +0x66 fp=0xc42005f7b8 sp=0xc42005f780 pc=0x564c0d69c756
github.com/maliceio/malice-yara/vendor/github.com/hillu/go-yara._Cfunc_yr_compiler_get_rules(0x564c10139020, 0xc42000ec80, 0xc400000000)
	_cgo_gotypes.go:1015 +0x4f fp=0xc42005f7e8 sp=0xc42005f7b8 pc=0x564c0d9aed1f
github.com/maliceio/malice-yara/vendor/github.com/hillu/go-yara.(*Compiler).GetRules.func1(0x564c10139020, 0xc42000ec80, 0x0)
	/go/src/github.com/maliceio/malice-yara/vendor/github.com/hillu/go-yara/compiler.go:192 +0xd9 fp=0xc42005f820 sp=0xc42005f7e8 pc=0x564c0d9b57a9
github.com/maliceio/malice-yara/vendor/github.com/hillu/go-yara.(*Compiler).GetRules(0xc42006a840, 0x0, 0x0, 0x0)
	/go/src/github.com/maliceio/malice-yara/vendor/github.com/hillu/go-yara/compiler.go:192 +0x51 fp=0xc42005f860 sp=0xc42005f820 pc=0x564c0d9b1001
main.scanFile(0xc420024500, 0x16, 0x564c0daeb7b3, 0x6, 0x3c, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/maliceio/malice-yara/yara.go:116 +0x8c fp=0xc42005f930 sp=0xc42005f860 pc=0x564c0dae4f1c
main.main.func2(0xc4200c4580, 0x0, 0x0)
	/go/src/github.com/maliceio/malice-yara/yara.go:296 +0x17e fp=0xc42005fb28 sp=0xc42005f930 pc=0x564c0dae699e
github.com/maliceio/malice-yara/vendor/github.com/urfave/cli.HandleAction(0x564c0de355a0, 0xc420050df0, 0xc4200c4580, 0x0, 0x0)
	/go/src/github.com/maliceio/malice-yara/vendor/github.com/urfave/cli/app.go:490 +0xca fp=0xc42005fb50 sp=0xc42005fb28 pc=0x564c0dad415a
github.com/maliceio/malice-yara/vendor/github.com/urfave/cli.(*App).Run(0xc42009e820, 0xc42001e180, 0x3, 0x3, 0x0, 0x0)
	/go/src/github.com/maliceio/malice-yara/vendor/github.com/urfave/cli/app.go:264 +0x5f0 fp=0xc42005fd28 sp=0xc42005fb50 pc=0x564c0dad2410
main.main()
	/go/src/github.com/maliceio/malice-yara/yara.go:344 +0x89b fp=0xc42005ff88 sp=0xc42005fd28 pc=0x564c0dae64fb
runtime.main()
	/usr/lib/go/src/runtime/proc.go:198 +0x21a fp=0xc42005ffe0 sp=0xc42005ff88 pc=0x564c0d6c660a
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42005ffe8 sp=0xc42005ffe0 pc=0x564c0d6f0da1

rax    0x0
rbx    0x6
rcx    0x7fe370f11f1b
rdx    0x0
rdi    0x2
rsi    0x7ffe3a5bee70
rbp    0x7ffe3a5bee70
rsp    0x7ffe3a5bee58
r8     0x2000
r9     0x3
r10    0x8
r11    0x246
r12    0xffffffffffffffff
r13    0x15e
r14    0x15d
r15    0x200
rip    0x7fe370f11f1b
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

Assertion failed: r1.o != NULL, file exec.c, line 889

I am scanning files against a large collection of compiled yara rules. I am using yara-4.0.0

I am encountering this error when I try to run the cross-compiled version of the same go code on windows OS.

When I scan the same files with the same yara ruleset on linux, everything goes smoothly. I already searched a bit for this issue and I was hoping getting rid of the hash import in yara would solve the problem. Unfortunately it did not.

Any idea what might be causing this?

Assertion `match_offset + match_length <= callback_args->data_size' failed.

I ran some fuzz testing and uncovered this assertion fail:
scan.c:521: int _yr_scan_match_callback(const uint8_t *, int32_t, int, void *): Assertion match_offset + match_length <= callback_args->data_size failed.

This is technically libyara related, but I felt like posting here too.

Fuzzer code:

package main

// commit as of file creation (13.02.2019): 4cc0204f70e49688c0f42637dda8326344c64325
// https://github.com/hillu/go-yara/tree/4cc0204f70e49688c0f42637dda8326344c64325
import "github.com/hillu/go-yara"

//we need import c!

import "C"

//replace original main with dummy main()
func main(){}

//export Fuzz
func Fuzz(data []byte){
	c, err := yara.NewCompiler()
	if err != nil {
		return
        }
        defer c.Destroy()
        err = c.AddString(string(data[:]), "")
        if err != nil {
                return
        }
	r, err := c.GetRules()
	if err != nil {
		return
	}
        _, _ = r.ScanFile("testfile", 0, 0)
}

Testfile is a PNG file.

Crasher file is attached. crash-0de6903a06f42da60003232dd862790a7bd5ade3.zip

Meta variables with duplicate name are lost

It is common to add hashes to test rules on in the meta variables like so:

rule test
{
    meta:
        // The files that should match this rule. It will be tested against these.
        hash = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
        hash = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
    condition:
        true
}

The current API loses the first hash, because the result is a map keyed by the variable name.

Possible fix, which will put []interface{} into the metas map[string]interface{} map:

diff --git a/rule.go b/rule.go
index e0b7e7d..ef0966c 100644
--- a/rule.go
+++ b/rule.go
@@ -132,15 +132,30 @@ func (r *Rule) Metas() (metas map[string]interface{}) {
        for _, m := range mptrs {
                var id, str *C.char
                C.meta_get(m, &id, &str)
+               goId := C.GoString(id)
+
+               var val interface{}
                switch m._type {
                case C.META_TYPE_NULL:
-                       metas[C.GoString(id)] = nil
+                       val = nil
                case C.META_TYPE_STRING:
-                       metas[C.GoString(id)] = C.GoString(str)
+                       val = C.GoString(str)
                case C.META_TYPE_INTEGER:
-                       metas[C.GoString(id)] = int(m.integer)
+                       val = int(m.integer)
                case C.META_TYPE_BOOLEAN:
-                       metas[C.GoString(id)] = m.integer != 0
+                       val = m.integer != 0
+               default:
+                       continue
+               }
+
+               if oldval, prs := metas[goId]; prs {
+                       if sl, ok := oldval.([]interface{}); ok {
+                               metas[goId] = append(sl, val)
+                       } else {
+                               metas[goId] = []interface{}{oldval, val}
+                       }
+               } else {
+                       metas[goId] = val
                }
        }
        return

I'm not too happy with it though, too many interface{} :/

Can't import module

Can you try to compile this rules? I get an error unknown module "magic", and also undefined identifier "filepath"

import "magic"

rule EXE_cloaked_as_TXT {
	meta:
		description = "Executable with TXT extension"
		license = "https://creativecommons.org/licenses/by-nc/4.0/"
		author = "Florian Roth"
	condition:
		uint16(0) == 0x5a4d 					// Executable
		and magic.mime_type() matches /\.txt$/is   // TXT extension (case insensitive)
}

rule Cloaked_RAR_File {
	meta:
		description = "RAR file cloaked by a different extension"
		license = "https://creativecommons.org/licenses/by-nc/4.0/"
		author = "Florian Roth"
	condition:
		uint32be(0) == 0x52617221							// RAR File Magic Header
		and not magic.mime_type() matches /(rarnew.dat|\.rar)$/is	// not the .RAR extension
		and not filepath contains "Recycle" 				// not a deleted RAR file in recycler
}

I installed yara with all the flags needed, and file library is present in my system (Fedora XFCE 28)

Bug in the way that data is passed down to callback functions

While investigating some other issue I accidentally hit a bug that seems related to the mechanism for passing data to callback functions. By adding the following test case to rules_test.go you should be able to trigger the bug:

func TestBug(t *testing.T) {
	ch := make(chan bool, 0)
	go func() {
		for i := 0; i < 10000; i++ {
			makeRules(t, "rule test { strings: $a = /a.*a/ condition: $a }")
			time.Sleep(1 * time.Millisecond)
		}
		ch <- true
	}()

Loop:
	for {
		select {
		case <-ch:
			break Loop
		default:
			runtime.GC()
			fmt.Println("GC")
		}
	}
}

The bug can manifest itself in two ways, by panicing in line 44 of callback-util.go or by raising the following error:

runtime: nelems=512 nfree=501 nalloc=11 previous allocCount=10 nfreed=65535
fatal error: sweep increased allocation count

runtime stack:
runtime.throw(0x4174515, 0x20)
	/usr/local/go/src/runtime/panic.go:596 +0x95 fp=0x70000e1b3cd8 sp=0x70000e1b3cb8
runtime.(*mspan).sweep(0x44042f8, 0x4404200, 0x70000e1b3d00)
	/usr/local/go/src/runtime/mgcsweep.go:288 +0x786 fp=0x70000e1b3da8 sp=0x70000e1b3cd8
runtime.sweepone(0x0)
	/usr/local/go/src/runtime/mgcsweep.go:110 +0x1ad fp=0x70000e1b3e10 sp=0x70000e1b3da8
runtime.gcSweep(0x2)
	/usr/local/go/src/runtime/mgc.go:1786 +0xd4 fp=0x70000e1b3e40 sp=0x70000e1b3e10
runtime.gcMarkTermination.func2()
	/usr/local/go/src/runtime/mgc.go:1261 +0x62 fp=0x70000e1b3e60 sp=0x70000e1b3e40
runtime.systemstack(0xc42001e600)
	/usr/local/go/src/runtime/asm_amd64.s:327 +0x79 fp=0x70000e1b3e68 sp=0x70000e1b3e60
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1132 fp=0x70000e1b3e70 sp=0x70000e1b3e68

In my case commenting out the line fmt.Println("GC") switch from one error to the other, but your mileage may vary. It was the panic in callback-util.go what lead me find a possible explanation for what's happening.

Take a look at this:

C.yr_compiler_set_callback(c.cptr, C.YR_COMPILER_CALLBACK_FUNC(C.compilerCallback), unsafe.Pointer(&id))

This passes to the callback the address of id, as an unsafe.Pointer. My theory is that the garbage collector is moving id around before compilerCallback is called. By the time compilerCallback is called the address where id was previously stored contains different data.

Thoughts?

I couldn't use with the new yara 3.5

+ export CGO_CFLAGS=-I/usr/local/includeBuilding info Go binary...

+ export CGO_LDFLAGS=-L/usr/local/lib
+ go version
go version go1.6.3 linux/amd64
+ go get
# github.com/hillu/go-yara
../../hillu/go-yara/rules-callback.c: In function 'rules_callback':
../../hillu/go-yara/rules-callback.c:40:76: error: 'YR_MATCH {aka struct _YR_MATCH}' has no member named 'length'
         addString(user_data, string->identifier, m->offset, m->data, (int)m->length);

The storage control block address is invalid.

Hey,

I managed to successfully compile go-yara for windows and I am trying to run the example script that was provided. However, when I attempt to read and parse in the rules from a file, I would get Could not parse rule file yara.yar: The storage control block address is invalid. The rule file yara.yar contains a sample rule: rule dummy { condition: true }

I am calling my script using go run scanner.go -rule "yara.yar" test.go

Is it possible that I compiled the go-yara binary wrongly or am I using the example script wrongly?

Thank you.

Compile simple-yara.go when it's copied to another directory with yara 3.9.0

I copied _examples/simple-yara/simple-yara.go to another directory for testing.

uname -a
Linux test 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) x86_64 GNU/Linux

apt-get install libyara3
apt-get install libyara-dev

ls /usr/lib/x86_64-linux-gnu/libyara* -l
-rw-r--r-- 1 root root 802980 Feb 23  2019 /usr/lib/x86_64-linux-gnu/libyara.a
lrwxrwxrwx 1 root root     16 Feb 23  2019 /usr/lib/x86_64-linux-gnu/libyara.so -> libyara.so.3.9.0
lrwxrwxrwx 1 root root     16 Feb 23  2019 /usr/lib/x86_64-linux-gnu/libyara.so.3 -> libyara.so.3.9.0
-rw-r--r-- 1 root root 373496 Feb 23  2019 /usr/lib/x86_64-linux-gnu/libyara.so.3.9.0

export CGO_CFLAGS="-I/usr/include/yara"
export CGO_LDFLAGS="-L/usr/lib/x86_64-linux-gnu/ -lyara -lm"

go build simple-yara.go 
# github.com/hillu/go-yara
In file included from /usr/include/yara.h:33,
                 from ../../../go/pkg/mod/github.com/hillu/[email protected]/compiler.go:17:
/usr/include/yara/types.h:530:27: error: 'CHAR_BIT' undeclared here (not in a function)
   unsigned char tidx_mask[YR_BITARRAY_NCHARS(YR_MAX_THREADS)];
                           ^~~~~~~~~~~~~~~~~~
/usr/include/yara/types.h:530:27: note: 'CHAR_BIT' is defined in header '<limits.h>'; did you forget to '#include <limits.h>'?
/usr/include/yara/types.h:41:1:
+#include <limits.h>
 
/usr/include/yara/types.h:530:27:
   unsigned char tidx_mask[YR_BITARRAY_NCHARS(YR_MAX_THREADS)];
                           ^~~~~~~~~~~~~~~~~~

/usr/include/limits.h exists, I checked /usr/include/yara.h, it includes #include "yara/utils.h"
/usr/include/yara.h includes #include <limits.h>

I can't find the reason.

Using go-yara in Google Cloud Functions

Hey!

We're trying to use go-yara in Google Cloud Functions. Currently it does not work due to pkg_config not being available in Google Cloud Functions

Build failed: # pkg-config --cflags -- yara
Package yara was not found in the pkg-config search path.
Perhaps you should add the directory containing `yara.pc'
to the PKG_CONFIG_PATH environment variable
No package 'yara' found
pkg-config: exit status 1

My guess is that we need to pre-compile Yara and ship it while deploying for it to work?
What do you think?

Need to update bindings and API for upcoming YARA 4

A lot of the yr_?_foreach() macros that are used in the C interface have changed in recent YARA commits expecting an additional YR_SCAN_CONTEXT parameter.

The same is true for the callback functions given to the yr_?_scan_mem() functions:

int callback_function(
    YR_SCAN_CONTEXT* context,
    int message,
    void* message_data,
    void* user_data);

As it is, go-yara does not compile with VirusTotal/yara@6c91c5d.

exception.h:118: exception_handler: Assertion `FALSE' failed.

My project ran fine with Go 1.10.2. Since upgrading to Go 1.11.2, I see random crashes, after 30-60 minutes of runtime (several million scans):

exception.h:118: exception_handler: Assertion `FALSE' failed.
SIGABRT: abort
PC=0x80d248 m=16 sigcode=18446744073709551610

Using Yara 3.7.1. Line 118 reads:

    assert(FALSE);  // We should not reach this point.

Shall I report this with Yara instead? My build script:

export YARA_SRC=/home/willem/src/yara-3.7.1
export CGO_CFLAGS="-I${YARA_SRC}/libyara/include"
export CGO_LDFLAGS="-L${YARA_SRC}/libyara/.libs -lyara -lm" # 
go build --ldflags '-w -s -extldflags "-static"' -tags 'yara_static yara3.7 no_pkg_config'

Thanks!

Go mod and V4 release

Hi,

Currently we want to use Go-Yara V4 in a Go Modules project. However, due to the go.mod in the master branch not having v4 in the file the tooling is fetching v1.3.0. It can't fetch v4.0.0 as this is disallowed (since the go.mod doesn't have v4, but the fallback to v0.0 or v1.x ..).

Could you bump the go.mod file to v4?

Thanks!

New cgo pointer rules break calls to yara [go 1.6]

go test fails with the following panic

panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 1 [running]:
panic(0x5913e0, 0xc82000a630)
        /usr/lib/go/src/runtime/panic.go:464 +0x3e6
github.com/hillu/go-yara._cgoCheckPointer0(0x553360, 0xc820010980, 0x0, 0x0, 0x0, 0x540060)
        ??:0 +0x4d
github.com/hillu/go-yara.NewCompiler(0x0, 0x0, 0x0)
        /home/alexander/go/src/github.com/hillu/go-yara/compiler.go:76 +0x2f7
github.com/hillu/go-yara.Compile(0x60fca0, 0x5d, 0x0, 0x0, 0x0, 0x0)
        /home/alexander/go/src/github.com/hillu/go-yara/compiler.go:189 +0x44
github.com/hillu/go-yara.TestMain(0xc820053ef8)
        /home/alexander/go/src/github.com/hillu/go-yara/main_test.go:9 +0x39
main.main()
        github.com/hillu/go-yara/_test/_testmain.go:100 +0x114
exit status 2
FAIL    github.com/hillu/go-yara        0.006s

Apparently go 1.6 enforces the rules documented at golang .

undefined: makecbPool on cross compile from OS X to Linux.

Trying to cross compile from OS X to Linux using Yara 3.8.1. Following appears on build:

[REDACTED]/go/src/github.com/hillu/go-yara/util.go:9:20: undefined: makecbPool

Compiles fine for OS X.

Build error on Ubuntu Xenial with yara3.4

Any ideas? Thanks for making this available!

$ go install -tags yara3.4
# github.com/hillu/go-yara
./rules_callback.go:89:7: could not determine kind of name for C.CALLBACK_MSG_MODULE_IMPORTED

$ dpkg -l | grep yara
ii  libyara-dev:amd64                           3.4.0+dfsg-2build1                           amd64        YARA development libraries and headers
ii  libyara3:amd64                              3.4.0+dfsg-2build1                           amd64        YARA shared library
ii  yara                                        3.4.0+dfsg-2build1                           amd64        Pattern matching swiss knife for malware researchers

$ go version
go version go1.10.1 linux/amd64

stack-overflow with relatively long strings

I'm trying to use go-yara for a project but found an issue with rules that have long strings, like this one:

rule longstring
{
strings:
  $a = { 558bec81ec58010000a1[1]00410033c58945fc568b75085733ff5757576a018d85b4feffff68[1]d24000c785c4feffff0000408489bdbcfeffffc785b4feffff2a2f2a0089bdacfeffffc685c8feffff008985a8feffffff1544d140008985b8feffff3bc775125f33c05e8b4dfc33cde8[2]00008be55dc35357576a0368[1]d2400068[1]d24000575650ff1548d140008bd83bdf0f84010100008b8dc4feffff57518d95a8feffff5268[1]d2400068[1]d240008d85c8feffff5068[1]d2400053ff154cd140008bf03bf70f84cb000000575757578b3d50d1400056ffd785c07524818dc4feffff80f100006a048d8dc4feffff516a1f56ff1554d1400085c00f84960000006a006a006a006a0056ffd785c00f84830000006a008d95c0feffff528d85b0feffff50681300002056c785c0feffff04000000ff1558d1400085c0745981bdb0feffffc800000075268b3d5cd140008d8dc0feffff516a328d55c85256ffd783bdc0feffff0075e7bf01000000eb068bbdbcfeffff568b3560d14000ffd685c0741453ffd685c0740d8b85b8feffff50ffd685c075135b5f33c05e8b4dfc33cde8[1]0e00008be55dc38b4dfc33c083ff015b0f94c05f33cd5ee8[1]0e00008be55dc3 } 
condition:
  $a
}

Although the string is accepted by YARA command-line tool, in go-yara it causes a stack overflow:

$ go run main.go 
ASAN:DEADLYSIGNAL
=================================================================
==5131==ERROR: AddressSanitizer: stack-overflow on address 0x70000711cb80 (pc 0x00000427c60e bp 0x70000711d3d0 sp 0x70000711cb80 T4)
    #0 0x427c60d in _yr_re_emit (libyara.3.dylib+0xe460d)
    #1 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #2 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #3 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #4 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #5 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #6 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #7 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #8 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #9 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #10 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #11 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #12 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #13 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #14 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #15 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #16 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #17 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #18 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #19 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #20 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #21 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #22 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #23 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #24 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #25 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #26 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #27 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #28 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #29 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #30 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #31 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #32 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #33 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #34 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #35 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #36 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #37 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #38 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #39 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #40 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #41 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #42 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #43 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #44 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #45 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #46 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #47 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #48 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #49 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #50 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #51 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #52 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #53 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #54 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #55 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #56 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #57 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #58 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #59 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #60 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #61 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #62 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #63 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #64 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #65 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #66 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #67 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #68 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #69 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #70 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #71 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #72 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #73 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #74 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #75 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #76 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #77 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #78 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #79 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #80 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #81 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #82 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #83 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #84 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #85 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #86 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #87 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #88 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #89 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #90 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #91 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #92 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #93 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #94 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #95 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #96 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #97 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #98 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #99 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #100 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #101 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #102 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #103 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #104 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #105 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #106 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #107 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #108 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #109 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #110 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #111 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #112 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #113 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #114 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #115 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #116 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #117 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #118 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #119 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #120 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #121 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #122 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #123 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #124 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #125 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #126 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #127 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #128 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #129 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #130 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #131 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #132 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #133 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #134 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #135 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #136 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #137 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #138 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #139 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #140 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #141 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #142 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #143 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #144 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #145 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #146 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #147 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #148 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #149 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #150 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #151 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #152 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #153 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #154 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #155 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #156 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #157 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #158 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #159 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #160 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #161 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #162 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #163 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #164 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #165 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #166 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #167 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #168 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #169 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #170 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #171 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #172 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #173 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #174 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #175 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #176 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #177 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #178 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #179 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #180 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #181 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #182 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #183 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #184 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #185 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #186 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #187 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #188 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #189 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #190 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #191 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #192 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #193 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #194 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #195 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #196 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #197 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #198 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #199 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #200 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #201 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #202 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #203 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #204 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #205 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #206 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #207 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #208 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #209 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #210 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #211 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #212 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #213 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #214 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #215 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #216 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #217 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #218 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #219 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #220 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #221 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #222 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #223 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #224 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #225 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #226 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #227 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #228 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #229 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #230 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #231 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #232 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
    #233 0x4276142 in yr_re_ast_emit_code (libyara.3.dylib+0xde142)
    #234 0x426d9c8 in _yr_parser_write_string (libyara.3.dylib+0xd59c8)
    #235 0x426ee81 in yr_parser_reduce_string_declaration (libyara.3.dylib+0xd6e81)
    #236 0x41eac60 in yara_yyparse (libyara.3.dylib+0x52c60)
    #237 0x425f735 in yr_lex_parse_rules_string (libyara.3.dylib+0xc7735)
    #238 0x4218357 in yr_compiler_add_string (libyara.3.dylib+0x80357)
    #239 0x409cff4 in _cgo_01fa3c9c21b0_Cfunc_yr_compiler_add_string (main+0x409cff4)
    #240 0x405265f in runtime.asmcgocall (main+0x405265f)

Apparently the problem has to do with the stack size of golang programs. Have you experienced this issue? Do you know how to increase the stack size?

Can seem to build static linked binaries

Not sure if this is supported but the documentation suggests that it should be possible:

The build tag yara_static can be used to tell the Go toolchain to run pkg-config with the --static switch.

Apparently this is not enough:

mic@elite:~/go/src/github.com/hillu/go-yara$ go build -tags yara_static  _examples/simple-yara/simple-yara.go 
mic@elite:~/go/src/github.com/hillu/go-yara$ ldd simple-yara 
        linux-vdso.so.1 (0x00007fffa8162000)
        libyara.so.3 => /usr/local/lib/libyara.so.3 (0x00007f24916c3000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f24914a4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f24910b3000)
        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f2490c3b000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f249089d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f249192c000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2490699000)

Should I be able to build without having to ship the shared object?

FEATURE: Build Rules from Rule

It would be useful to have a way to convert or get a Rules object from a Rule object or a set of Rule objects.

Something like this:

func LoadRule(rule *Rule) (*Rules, error)

In the same line, adding Rule objects to a already Rules object, like this:

func (r *Rules) AddRule(rule *Rule) error 

fatal error: unexpected signal during runtime execution

Using the thor-webshells.yar rules file from Neo23x0 rules repository I got the fatal error when there is a match.

fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x31 pc=0x7c53d7]

However, if I comment line no.123 keepAlive(r), it works okay, can u explain what keepAlive function is needed for?

I'm not good at understanding usage of C in Golang.

warning during go get on alpine:3.6

Not an error, just a warning.

# github.com/hillu/go-yara
../../hillu/go-yara/rules-callback.c: In function 'rules_callback':
../../hillu/go-yara/rules-callback.c:44:61: warning: passing argument 4 of 'addString' discards 'const' qualifier from pointer target type [-Wdiscarded-qualifiers]
         addString(user_data, string->identifier, m->offset, m->data, (int)m->data_length);
                                                             ^
In file included from ../../hillu/go-yara/rules-callback.c:10:0:
cgo-gcc-export-header-prolog:54:13: note: expected 'void *' but argument is of type 'const uint8_t * {aka const unsigned char *}'

https://github.com/malice-plugins/yara/blob/master/Dockerfile

thank you for all your great work!

Error during compilation on Linux

I use Go v1.5.3 and YARA v3.4 on Ubuntu Linux. I compiled YARA from source (current github master). I also used the --without-crypto --disable-cuckoo and --disable-magic options.
However when I try to install the module I get some errors. (its not a missing yara.h file)

I am new to golang, so it's difficult for me to understand this compiler error.

root@Ronin:/mnt/go-projects/src/github.com/hillu/go-yara# go install github.com/hillu/go-yara
# github.com/hillu/go-yara
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_add_file':
./compiler.go:106: undefined reference to `yr_compiler_add_file'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_add_string':
./compiler.go:118: undefined reference to `yr_compiler_add_string'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_create':
./compiler.go:128: undefined reference to `yr_compiler_create'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_define_boolean_variable':
./compiler.go:140: undefined reference to `yr_compiler_define_boolean_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_define_float_variable':
./compiler.go:152: undefined reference to `yr_compiler_define_float_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_define_integer_variable':
./compiler.go:164: undefined reference to `yr_compiler_define_integer_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_define_string_variable':
./compiler.go:176: undefined reference to `yr_compiler_define_string_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_destroy':
./compiler.go:185: undefined reference to `yr_compiler_destroy'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_get_error_message':
./compiler.go:197: undefined reference to `yr_compiler_get_error_message'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_get_rules':
./compiler.go:208: undefined reference to `yr_compiler_get_rules'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_set_callback':
./compiler.go:219: undefined reference to `yr_compiler_set_callback'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc__yr_rules_scan_fd':
./rules.go:60: undefined reference to `yr_rules_scan_fd'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_initialize':
./rules.go:69: undefined reference to `yr_initialize'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_define_boolean_variable':
./rules.go:81: undefined reference to `yr_rules_define_boolean_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_define_float_variable':
./rules.go:93: undefined reference to `yr_rules_define_float_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_define_integer_variable':
./rules.go:105: undefined reference to `yr_rules_define_integer_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_define_string_variable':
./rules.go:117: undefined reference to `yr_rules_define_string_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_destroy':
./rules.go:127: undefined reference to `yr_rules_destroy'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_load':
./rules.go:138: undefined reference to `yr_rules_load'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_load_stream':
./rules.go:149: undefined reference to `yr_rules_load_stream'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_save':
./rules.go:160: undefined reference to `yr_rules_save'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_save_stream':
./rules.go:171: undefined reference to `yr_rules_save_stream'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_scan_file':
./rules.go:186: undefined reference to `yr_rules_scan_file'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_scan_mem':
./rules.go:202: undefined reference to `yr_rules_scan_mem'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_scan_proc':
./rules.go:217: undefined reference to `yr_rules_scan_proc'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules-callback.o: In function `rules_callback':
./rules-callback.c:39: undefined reference to `yr_get_tidx'
collect2: error: ld returned 1 exit status

Thanks, Florian

Compiling for Windows instructions

I have been struggling to get this compiled for windows using MinGW.

It would be good to have more instructions on how to proper compile the Yara project and generate libyara for windows 32 and 64bit. Even thought Yara has the Visual Studio solutions and I can easily generate libyara32.lib and libyara64.lib.... Not sure if these are compatible with MinGW and CGO

And how to have this package use these libs with CGO.

LoadRules fails with any Yara rule

I'm testing your library, and at this attempt:

rules, err := yara.LoadRules(rules_path)

if err != nil {
    fmt.Println("Something failed loading rules!");
    panic(err)
}

I always get:

Something failed loading rules!
panic: invalid file

goroutine 1 [running]:
panic(0x4a79a0, 0xc420012310)
    /usr/lib/go-1.7/src/runtime/panic.go:500 +0x1a1
main.main()
    /whatever/whatever/main.go:19 +0x5a7
exit status 2

I am using Yara 3.5.0 and I have tested with pretty much whatever rule.
Am I doing something wrong?

Warning when cross-compiling for Windows on Go 1.11

I'm not sure if this is of any interesting, but this warning is raised when cross-compiling for Windows on Go 1.11:

GOOS=windows GOARCH=amd64 CGO_ENABLED=1 CC=x86_64-w64-mingw32-gcc PKG_CONFIG_PATH=${YARA_SRC}/x86_64-w64-mingw32/lib/pkgconfig go install -ldflags '-extldflags "-static"' github.com/hillu/go-yara                
# github.com/hillu/go-yara
../../.gvm/pkgsets/go1.11/global/src/github.com/hillu/go-yara/rules_yara34.go: In function ‘_yr_rules_scan_fd’:
../../.gvm/pkgsets/go1.11/global/src/github.com/hillu/go-yara/rules_yara34.go:30:34: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
   return yr_rules_scan_fd(rules, (YR_FILE_DESCRIPTOR)fd, flags, callback, user_data, timeout);

Cross-compiling for Darwin or FreeBSD fails because of build constraints

While cross-compiling from a Debian dev machine to other platforms, building for Darwin and FreeBSD raises this issue instead:

go build github.com/hillu/go-yara/internal/callbackdata: build constraints exclude all Go files in $GOPATH/pkg/mod/github.com/hillu/[email protected]/internal/callbackdata

I've not been able to triage the issue entirely, although it sounds like it could be related to this? golang/go#24068

Segmentation fault apparently triggered during garbage collection.

I've detected a strange issue that seems related to the freeing of resources during garbage collection. For reproducing the issue run the following program:

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"

	yara "github.com/hillu/go-yara"
)

func compile() {
	for i := 0; ; i++ {
		rules, err := yara.Compile("rule t {condition:true}", nil)
		if err == nil {
			log.Printf("%d\n", i)
			rules.Destroy()
		} else {
			fmt.Println(err)
			os.Exit(1)
		}
	}
}

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		log.Println("ok")
	})
	go compile()
	log.Fatal(http.ListenAndServe(":8080", nil))
}

Once it's running do a bunch of requests to the HTTP server:

while true ; do curl localhost:8080 ; done

It can take a while before failing, but after a 5-10 minutes I get an error like this:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x7fff9d067f44]

runtime stack:
runtime.throw(0x42a4e05, 0x2a)
	/usr/local/go/src/runtime/panic.go:596 +0x95
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:274 +0x2db

goroutine 5 [syscall, locked to thread]:
runtime.cgocall(0x420ce10, 0xc420221e38, 0xc4204ca000)
	/usr/local/go/src/runtime/cgocall.go:131 +0xe2 fp=0xc420221e08 sp=0xc420221dc8
github.com/hillu/go-yara._Cfunc_yr_compiler_get_rules(0x30f95e00, 0xc4204ca000, 0xc400000000)
	github.com/hillu/go-yara/_obj/_cgo_gotypes.go:790 +0x4d fp=0xc420221e38 sp=0xc420221e08
github.com/hillu/go-yara.(*Compiler).GetRules.func1(0x30f95e00, 0xc4204ca000, 0x0)
	/Users/vmalvarez/go/src/github.com/hillu/go-yara/compiler.go:181 +0xfc fp=0xc420221e80 sp=0xc420221e38
github.com/hillu/go-yara.(*Compiler).GetRules(0xc4204c85c0, 0x429f4b5, 0x17, 0x0)
	/Users/vmalvarez/go/src/github.com/hillu/go-yara/compiler.go:181 +0x5b fp=0xc420221ec8 sp=0xc420221e80
github.com/hillu/go-yara.Compile(0x429f4b5, 0x17, 0x0, 0x1, 0x1, 0x0)
	/Users/vmalvarez/go/src/github.com/hillu/go-yara/compiler.go:205 +0x183 fp=0xc420221f78 sp=0xc420221ec8
main.compile()
	/Users/vmalvarez/go/src/github.com/VirusTotal/poc/main.go:12 +0x4a fp=0xc420221fe0 sp=0xc420221f78
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420221fe8 sp=0xc420221fe0
created by main.main
	/Users/vmalvarez/go/src/github.com/VirusTotal/poc/main.go:26 +0x5e

goroutine 1 [runnable]:
net.runtime_pollWait(0x4769f70, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005a148, 0x72, 0x0, 0xc4202d5060)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005a148, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005a0e0, 0x0, 0x43a53c0, 0xc4202d5060)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e030, 0xc4201f3200, 0x4247dc0, 0xffffffffffffffff)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e030, 0xc42003ae18, 0xc42003ae20, 0xc42003ae10)
	/usr/local/go/src/net/tcpsock.go:215 +0x49
net/http.tcpKeepAliveListener.Accept(0xc42000e030, 0x42a95e8, 0xc4201f3180, 0x43a9240, 0xc420018e10)
	/usr/local/go/src/net/http/server.go:3044 +0x2f
net/http.(*Server).Serve(0xc4200a0370, 0x43a8dc0, 0xc42000e030, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http.(*Server).ListenAndServe(0xc4200a0370, 0xc4200a0370, 0xc42004ff38)
	/usr/local/go/src/net/http/server.go:2585 +0xb0
net/http.ListenAndServe(0x429a220, 0x5, 0x0, 0x0, 0x29, 0xc420018c90)
	/usr/local/go/src/net/http/server.go:2787 +0x7f
main.main()
	/Users/vmalvarez/go/src/github.com/VirusTotal/poc/main.go:27 +0x89

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 4 [runnable, locked to thread]:
github.com/hillu/go-yara._Cfunc_yr_compiler_destroy(0x154dd200)
	github.com/hillu/go-yara/_obj/_cgo_gotypes.go:762 +0x41
github.com/hillu/go-yara.(*compiler).finalize.func1(0x154dd200)
	/Users/vmalvarez/go/src/github.com/hillu/go-yara/compiler.go:80 +0x60
github.com/hillu/go-yara.(*compiler).finalize(0xc4204c1ee0)
	/Users/vmalvarez/go/src/github.com/hillu/go-yara/compiler.go:80 +0x2e

I was able to solve this issue by explicitly destroying the compiler in yara.Compile. I inserted defer c.Destroy() in https://github.com/hillu/go-yara/blob/master/compiler.go#L196. However I still don't know why this happens. I'm completely new to Go internals, but I've studied your code and everything looks to me, specifically everything related to finalizers.

I think that destroying the compiler explicitly makes sense anyways, but I don't want to send a PR without fully understanding the issue and its possible ramifications.

I'm using go1.8.3 darwin/amd64 with the head version of go-yara.

Cannot 'Go get' or 'Go install'

Following the steps on the readme:

# pkg-config --cflags yara
Package yara was not found in the pkg-config search path.
Perhaps you should add the directory containing `yara.pc'
to the PKG_CONFIG_PATH environment variable
No package 'yara' found
pkg-config: exit status 1

Docker Deployment and Cgo Cfunc_string_matches

Problem overview

I have been attempting to deploy a web service in Go within a docker container (ubuntu 19.04) that accept a file and returns the names for each rule that matches. When testing this on my local machine (macOS) with yara 3.11 and yarac 3.11 I am able to scan the wannaCry sample [ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa](https://www.virustotal.com/gui/file/ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa/detection). Although when I go deploy this service within the docker container I get the following errors.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x2f pc=0xcd3e97]

runtime stack:
runtime.throw(0xf1dee4, 0x2a)
	/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:378 +0x47c

goroutine 61 [syscall, locked to thread]:
runtime.cgocall(0xcd3e60, 0xc000344d20, 0x1)
	/usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000344cf0 sp=0xc000344cb8 pc=0x40b96b
github.com/hillu/go-yara._Cfunc_string_matches(0x7fb6c544ec05, 0x0, 0xc0004c9bfc)
	_cgo_gotypes.go:1121 +0x45 fp=0xc000344d20 sp=0xc000344cf0 pc=0xb07415
github.com/hillu/go-yara.(*String).Matches(0xc000344e48, 0x0, 0x0, 0x0)
	/go/src/github.com/hillu/go-yara/rule.go:225 +0x6b fp=0xc000344db8 sp=0xc000344d20 pc=0xb0c2ab
github.com/hillu/go-yara.(*Rule).getMatchStrings(0xc0000a8ce0, 0x0, 0x0, 0x0)
	/go/src/github.com/hillu/go-yara/rule.go:244 +0x2c2 fp=0xc000344ec0 sp=0xc000344db8 pc=0xb0c722
github.com/hillu/go-yara.(*MatchRules).RuleMatching(0xc000324220, 0xc0000a8ce0, 0xc000324220, 0x7fb6ec1219f0, 0xc000324220)
	/go/src/github.com/hillu/go-yara/rules_callback.go:151 +0x224 fp=0xc000345020 sp=0xc000344ec0 pc=0xb0e3e4
github.com/hillu/go-yara.scanCallbackFunc(0x1, 0x7fb6c3d6916d, 0x21bb7e0, 0xf395a8)
	/go/src/github.com/hillu/go-yara/rules_callback.go:90 +0x220 fp=0xc000345100 sp=0xc000345020 pc=0xb0ddf0
github.com/hillu/go-yara._cgoexpwrap_08a63f7e5a95_scanCallbackFunc(0x7fb600000001, 0x7fb6c3d6916d, 0x21bb7e0, 0x0)
	_cgo_gotypes.go:1621 +0x3d fp=0xc000345130 sp=0xc000345100 pc=0xb08aad
runtime.call32(0x0, 0x7fb6cdffa3f0, 0x7fb6cdffa480, 0x20)
	/usr/local/go/src/runtime/asm_amd64.s:539 +0x3b fp=0xc000345160 sp=0xc000345130 pc=0x461f3b
runtime.cgocallbackg1(0x0)
	/usr/local/go/src/runtime/cgocall.go:314 +0x1b7 fp=0xc000345248 sp=0xc000345160 pc=0x40bd17
runtime.cgocallbackg(0x0)
	/usr/local/go/src/runtime/cgocall.go:191 +0xc1 fp=0xc0003452b0 sp=0xc000345248 pc=0x40bac1
runtime.cgocallback_gofunc(0x40b98f, 0xcd40b0, 0xc000345340, 0xc000345330)
	/usr/local/go/src/runtime/asm_amd64.s:793 +0x9b fp=0xc0003452d0 sp=0xc0003452b0 pc=0x46350b
runtime.asmcgocall(0xcd40b0, 0xc000345340)
	/usr/local/go/src/runtime/asm_amd64.s:640 +0x42 fp=0xc0003452d8 sp=0xc0003452d0 pc=0x4633a2
runtime.cgocall(0xcd40b0, 0xc000345340, 0x10279e1)
	/usr/local/go/src/runtime/cgocall.go:131 +0x7f fp=0xc000345310 sp=0xc0003452d8 pc=0x40b98f
github.com/hillu/go-yara._Cfunc_yr_rules_scan_mem(0x7fb6d4000eb0, 0xc001c5d19a, 0x35a000, 0x1, 0xcd3790, 0x21bb7e0, 0x0, 0x0)
	_cgo_gotypes.go:1524 +0x4d fp=0xc000345340 sp=0xc000345310 pc=0xb0855d
github.com/hillu/go-yara.(*Rules).ScanMemWithCallback.func1(0xc0000a8960, 0xc001c5d19a, 0xc001c5d19a, 0x35a000, 0xd4a656, 0x1, 0x21bb7e0, 0x0, 0x7523cf)
	/go/src/github.com/hillu/go-yara/rules.go:91 +0xfc fp=0xc0003453a0 sp=0xc000345340 pc=0xb1099c
github.com/hillu/go-yara.(*Rules).ScanMemWithCallback(0xc0000a8960, 0xc001c5d19a, 0x35a000, 0xd4a656, 0x1, 0x0, 0xdc17a0, 0xc000324220, 0x0, 0x0)
	/go/src/github.com/hillu/go-yara/rules.go:91 +0x1b7 fp=0xc000345488 sp=0xc0003453a0 pc=0xb0cb27
github.com/hillu/go-yara.(*Rules).ScanMem(...)
	/go/src/github.com/hillu/go-yara/rules.go:68
github.secureserver.net/threat/util/yara.(*Rules).ScanBuffer(0xc0001559e0, 0x1054b20, 0xc000022080, 0xc001c5d19a, 0x35a000, 0xd4a656, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.secureserver.net/threat/util/yara/yara.go:84 +0x184 fp=0xc000345660 sp=0xc000345488 pc=0xb142d4
project/classification.(*Plugin).classifySamples(0xc0002b6c80, 0x1054b20, 0xc000022080, 0xc0001f5000, 0xc00014c660, 0x0, 0x17, 0xc000575960)
	/go/src/project/classification/classification.go:176 +0x1dd fp=0xc000345800 sp=0xc000345660 pc=0xb16b0d

This stack trace basically shows that we're taking in a sample, attempting to classify it, using ScanBuffer as the mechanism for scanning and returning the matching rules. The top function within the stack trace points to github.com/hillu/go-yara._Cfunc_string_matches so I've been trying to figure out how this function could cause issues within a docker a container but nowhere else.

As for the ruleset, I have been using a relatively large ruleset for testing and when using rules that just check for the PE header or the Cannot Run in DOS mode string, there are no errors so this could also be the source of error here.

With regards to how I am installing Yara in the docker container, following is a snippet for how I am installing yara on the host that contains the go built application as well

# steps required to install Yara and remove the source files
RUN git clone https://github.com/VirusTotal/yara.git /tmp/yara
WORKDIR ${YARA_SRC_PATH}
RUN \
    apt update && \
    apt install automake libtool make pkg-config libyara-dev libc-dev -y && \
    ./bootstrap.sh && \
    ./configure && \
    make && \
    make install && \
    rm -rf ${YARA_SRC_PATH}

Things I have tried so far

• compiling the rules with yarac and compiling them within the web service
  • had no effect, still caused issues with the wannacry sample
• Check if its an issue within the docker container or something that would be an issue on my host machine as well
  • Its not an issue on the host machine, so that means something is missing within the docker container or how yara interacts with the ubuntu base image
• Checks yara versions
  • Yara and Yarac in the deploy container are both running version 3.11
• Adding libc-dev and file-dev as those pacakges are added in the official yara docker image
  • Does not seem to fix the issue

Failing tests on 32-bit platforms

On 32-bit platforms, the TestWriter test fails as both compiled rule binaries seem to differ:

(sid_armhf-dchroot)satta@abel:~/golang-github-hillu-go-yara$ go test -v
=== RUN   TestCompiler
--- PASS: TestCompiler (0.00s)
	compiler_test.go:15: expected error: syntax error, unexpected _IDENTIFIER_
=== RUN   TestPanic
--- PASS: TestPanic (0.00s)
	compiler_test.go:25: Everything ok, MustCompile panicked: syntax error, unexpected _IDENTIFIER_
=== RUN   TestWarnings
--- PASS: TestWarnings (0.00s)
	compiler_test.go:37: Recorded Errors=[]yara.CompilerMessage{yara.CompilerMessage{Filename:"", Line:1, Text:"syntax error, unexpected _IDENTIFIER_, expecting _CONDITION_"}}, Warnings=[]yara.CompilerMessage(nil)
=== RUN   TestCompilerFinalizer
Create compiler
Trigger GC
Trigger Gosched
Manually call destructure on copy
--- PASS: TestCompilerFinalizer (0.00s)
	crash_test.go:23: Did not crash due to yr_*_destroy() being called twice. Yay.
=== RUN   TestRulesFinalizer
Create rules
Trigger GC
Trigger Gosched
Manually call destructure on copy
--- PASS: TestRulesFinalizer (0.00s)
	crash_test.go:40: Did not crash due to yr_*_destroy() being called twice. Yay.
=== RUN   TestBooleanOperators
--- PASS: TestBooleanOperators (0.01s)
=== RUN   TestComparisonOperators
--- PASS: TestComparisonOperators (0.03s)
=== RUN   TestArithmeticOperators
--- PASS: TestArithmeticOperators (0.02s)
=== RUN   TestBitwiseOperators
--- PASS: TestBitwiseOperators (0.01s)
=== RUN   TestStrings
--- PASS: TestStrings (0.04s)
=== RUN   TestWildcardStrings
--- PASS: TestWildcardStrings (0.00s)
=== RUN   TestHexStrings
--- PASS: TestHexStrings (0.02s)
=== RUN   TestEntrypoint
--- PASS: TestEntrypoint (0.00s)
=== RUN   TestFilesize
--- PASS: TestFilesize (0.00s)
=== RUN   TestExternals
--- PASS: TestExternals (0.02s)
=== RUN   TestModules
--- PASS: TestModules (0.02s)
=== RUN   TestIntegerFunctions
--- PASS: TestIntegerFunctions (0.01s)
=== RUN   TestSimpleMatch
--- PASS: TestSimpleMatch (0.00s)
	rules_test.go:35: Matches: [{Rule:test Namespace:default Tags:[tag1] Meta:map[author:Hilko Bengen] Strings:[{Name:$a Offset:1 Data:[97 98 99]}]}]
=== RUN   TestSimpleFileMatch
--- PASS: TestSimpleFileMatch (0.00s)
	rules_test.go:50: Matches: [{Rule:test Namespace:default Tags:[tag1] Meta:map[author:Hilko Bengen] Strings:[{Name:$a Offset:1 Data:[97 98 99]}]}]
=== RUN   TestSimpleFileDescriptorMatch
--- PASS: TestSimpleFileDescriptorMatch (0.00s)
	rules_test.go:65: Matches: [{Rule:test Namespace:default Tags:[tag1] Meta:map[author:Hilko Bengen] Strings:[{Name:$a Offset:1 Data:[97 98 99]}]}]
=== RUN   TestEmpty
--- PASS: TestEmpty (0.00s)
	rules_test.go:71: Scan of null-byte slice did not crash. Yay.
=== RUN   TestLoad
--- PASS: TestLoad (0.00s)
=== RUN   TestReader
--- PASS: TestReader (0.01s)
	rules_test.go:116: Matches: [{Rule:test Namespace:default Tags:[tag1] Meta:map[author:Hilko Bengen] Strings:[{Name:$a Offset:1 Data:[97 98 99]}]}]
=== RUN   TestWriter
--- FAIL: TestWriter (0.25s)
	rules_test.go:142: buffers are not equal
=== RUN   TestReaderBZIP2
--- PASS: TestReaderBZIP2 (8.93s)
=== RUN   TestScanMemCgoPointer
--- PASS: TestScanMemCgoPointer (0.00s)
=== RUN   TestFileWalk
--- SKIP: TestFileWalk (0.00s)
	stress_test.go:16: Set TEST_WALK to enable scanning files from user's HOME with a dummy ruleset.
		(Setting -test.timeout may be a good idea for this.)
FAIL
exit status 1
FAIL	_/home/satta/golang-github-hillu-go-yara	9.440s

Dumping both buffers to files, it seems like 4 bytes are different:

(sid_armhf-dchroot)satta@abel:~$ xxd /tmp/f1 > f1.hex
(sid_armhf-dchroot)satta@abel:~$ xxd /tmp/f2 > f2.hex
(sid_armhf-dchroot)satta@abel:~$ diff f1.hex f2.hex 
5c5
< 00000040: 0000 c90d 320d b001 0000 e8b8 afb6 161d  ....2...........
---
> 00000040: 0000 c90d 320d b001 0000 00c9 0d32 161d  ....2........2..

I'm not enough of an expert on Yara to guess what the reason might be, but maybe it helps. libyara 3.6.0 was used here.

This seems to happen at least on armhf and i386, see https://tests.reproducible-builds.org/debian/rb-pkg/unstable/i386/golang-github-hillu-go-yara.html.

No way to get defined Variables

No way to get defined Variables
only can define variables by using DefineVariable

Compiler Warnings/Errors not collected (anymore)

Hi there,

It seems that a previously working behaviour, being the ability to pull warnings/errors from the compiler, no longer works.

Replication

OSX 10.11.6

$ go version
go version go1.8 darwin/amd64

$ yara -v
yara 3.5.0

$ cd $GOPATH/src/github.com/hillu/go-yara && git rev-parse HEAD
f0a137cfb146d381a6cca4ba9797251bfc5b06f1

To observe this behaviour, run the following code:

package main

import (
	"fmt"

	"github.com/hillu/go-yara"
)

func main() {
	// Build a Yara Compiler
	compiler, err := yara.NewCompiler()
	if err != nil {
		panic(err)
	}

	// Add an obviously broken rule
	compiler.AddString("rule foo { bar }", "default")

	// Debug the Compiler Errors/Warnings
	fmt.Println(compiler.Errors)
	fmt.Println(compiler.Warnings)
}

Output

$ go run main.go
[]
[]

Investigation

I have found that userData, when checked at https://github.com/hillu/go-yara/blob/master/compiler.go#L30, is appearing as nil. I am not too familiar with how this should work to offer a fix.

Errors compiling for yara 3.5

Hi,

When I try to build the most recent version of go-yara against yara 3.5 it fails with the following error:

go build -tags yara3.5
vendor/github.com/hillu/go-yara/config.go:17:39: could not determine kind of name for C.YR_CONFIG_MAX_STRINGS_PER_RULE

It seems that the config.go is not correctly excluded for yara 3.5. Building it against yara 3.7 works fine. Previous versions of go-yara also worked fine; building v1.0.8 against yara 3.5 does not generate errors.

Thank you!

C.YR_CONFIG_MAX_MATCH_DATA

Hi，
go install -tags no_pkg_config github.com/hillu/go-yara

error msg is：

../../github.com/hillu/go-yara/config_yara38.go:12:39: could not determine kind of name for C.YR_CONFIG_MAX_MATCH_DATA

Compiling go-yara with gccgo on AIX

Hi,

I tried to compile go-yara on AIX 7.2 POWER7 (ppc64) but I get the following error:

bash-5.0$ CGO_LDFLAGS="-L/usr/local/lib -lyara" CGO_CFLAGS="-I/usr/local/include" go.gcc build --gccgoflags="-Xlinker -bbigtoc" -o out -mod vendor -tags="no_pkg_config" aixtest/main.go

ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_add_string
ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_create
ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_define_boolean_variable
ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_define_float_variable
ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_define_integer_variable
ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_define_string_variable
...

Using -Xlinker -bnoquiet shows me the following information on top of the error:

(ld): setopt 64
(ld): halt 4
(ld): setopt r/o->w
(ld): setopt bigtoc
(ld): setfflag 4
(ld): savename $WORK/b001/exe/a.out
(ld): filelist 12 1
(ld): i /lib/crt0_64.o
(ld): i /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/crtcxa.o
(ld): i /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/crtdbase.o
(ld): i $WORK/b001/_pkg_.a
(ld): i $WORK/b001/_pkg1_.a
(ld): lib /usr/local/lib/libyara.a
(ld): lib /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/libgobegin.a
(ld): lib /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/libgo.a
(ld): lib /usr/lib/libm.a
(ld): lib /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/libgcc_s.a
(ld): i /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/libgcc.a
(ld): lib /usr/lib/libc.a
LIBRARY: Gemeinsam genutztes Objekt libgo.a[libgo.so.14]: Es wurden 44098 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libgcc_s.a[shr.o]: Es wurden 151 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[shr_64.o]: Es wurden 3148 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[posix_aio_64.o]: Es wurden 20 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[aio_64.o]: Es wurden 18 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[pse_64.o]: Es wurden 8 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[dl_64.o]: Es wurden 4 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[pty_64.o]: Es wurden 1 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[cthread_64.o]: Es wurden 25 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[uchar_64.o]: Es wurden 4 Symbole importiert.
FILELIST: Anzahl der zuvor eingefügten Dateien, die verarbeitet wurden: 12
(ld): resolve
RESOLVE: 1680 von 55377 Symbolen wurden beibehalten.
(ld): addgl /usr/lib/glinkl64.o
ADDGL: Es wurde Glink-Code für 131 Symbole hinzugefügt.
(ld): er full
ld: 0711-318 FEHLER: Es wurden nicht definierte Symbole gefunden.
        Die folgenden Symbole sind fehlerhaft:

libyara (3.11.0) was installed with ./bootstrap.sh && ./configure --disable-shared --disable-magic --disable-cuckoo --enable-dotnet && make && make install

It was installed to:

• /usr/local/lib
• /usr/local/include
• /usr/local/bin

ls /usr/local/lib /usr/local/include
/usr/local/include:
yara  yara.h

/usr/local/lib:
libyara.a  libyara.la  pkgconfig

/usr/local/bin/yara --version
3.11.0

ar -t /usr/local/lib/libyara.a
tests.o
pe.o
elf.o
math.o
time.o
pe_utils.o
hash.o
dotnet.o
grammar.o
ahocorasick.o
arena.o
atoms.o
bitmask.o
compiler.o
endian.o
exec.o
exefiles.o
filemap.o
lt1-hash.o
hex_grammar.o
hex_lexer.o
lexer.o
libyara.o
mem.o
modules.o
object.o
parser.o
proc.o
re.o
re_grammar.o
re_lexer.o
rules.o
scan.o
scanner.o
sizedstr.o
stack.o
stopwatch.o
strutils.o
stream.o
threading.o
none.o

There are no more libyaras installed:

find / -name "libyara.a"
/home/user/yara-3.11.0/libyara/.libs/libyara.a
/usr/local/lib/libyara.a
find / -name "yara.h"
/home/user/yara-3.11.0/libyara/include/yara.h
/usr/local/include/yara.h

I also tried using pkg-config but this results in same error.

go.gcc version
go version go1.12.2 gccgo (GCC) 9.1.0 aix/ppc64

git -C vendor/github.com/hillu/go-yara/ log | head -n 1
commit 510cc23f05d05782d7f600ee26154542127e6281

Do you know what I am doing wrong here?

Thanks.

Implement yr_scanner_scan_mem_blocks()-based io.Reader scanning interface?

As per the title, would it be possible to provide a scanning interface that accepts an io.Reader. The implementation could be based on a streaming interface by implementing the YR_MEMORY_BLOCK_ITERATOR* parameter for yr_scanner_scan_mem_block()?

With an io.Reader it'd be much easier to integrate go-yara in a clean and Go-like manner. Unless I missed something of course - happy to hear about alternatives (currently I believe there's either ScanMem(), which takes a singular []byte, ScanFile, which takes a filename, and ScanFileDescriptor, which takes a fd) :-)

Inconsistent type for flag argument in ScanProc

The flag argument for ScanProc function is int but ScanMem and ScanFile use the ScanFlags type. ScanProc should expect the ScanFlags type for keeping the API consistency.

cgo pointer rules break ScanMem [go 1.6]

When scanning a byte slice with ScanMem go panics with:

panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 7 [running]:
panic(0x5fbd00, 0xc82000e8b0)
    /usr/lib/go/src/runtime/panic.go:481 +0x3e6
github.com/hillu/go-yara._cgoCheckPointer0(0x5a9be0, 0xc820016fe4, 0x0, 0x0, 0x0, 0x54417e)
    ??:0 +0x4d
github.com/hillu/go-yara.addString(0xc82000e888, 0x11b3dd4, 0x0, 0xc820016fe4, 0xc800000030)
    /home/tom/src/github.com/hillu/go-yara/rules.go:117 +0xce
github.com/hillu/go-yara._cgoexpwrap_f3bc5924bb1c_addString(0xc82000e888, 0x11b3dd4, 0x0, 0xc820016fe4, 0xc800000030)
    ??:0 +0x47
github.com/hillu/go-yara._Cfunc_yr_rules_scan_mem(0x12360e0, 0xc820016fe4, 0x3b, 0x1, 0x58a900, 0xc82000e888, 0x1, 0x0)
    ??:0 +0x44
github.com/hillu/go-yara.(*Rules).ScanMem(0xc82002c068, 0xc820016fe4, 0x3b, 0x40, 0x1, 0x3b9aca00, 0x0, 0x0, 0x0, 0x0, ...)
    /home/tom/src/github.com/hillu/go-yara/rules.go:147 +0x23d

Using a bytes.Buffer in rules_test.go/TestReader seems to trigger the above error

diff --git a/rules_test.go b/rules_test.go
index ca59966..83b912d 100644
--- a/rules_test.go
+++ b/rules_test.go
@@ -109,7 +109,10 @@ func TestReader(t *testing.T) {
        if err != nil {
                t.Fatalf("ReadRules: %+v", err)
        }
-       m, err := r.ScanMem([]byte(" abc "), 0, 0)
+
+       buf := &bytes.Buffer{}
+       buf.Write([]byte(" abc "))
+       m, err := r.ScanMem(buf.Bytes(), 0, 0)
        if err != nil {
                t.Errorf("ScanMem: %s", err)
        }

Relative paths in include statements

Hi!

Can't seem to load files that has include statements like this:

include "subdir/SomeGreatYaraRules.yar"

I load these files with err := compiler.AddFile(f, "ns")