spyre-project / spyre

simple YARA-based IOC scanner

Spyre

...a simple, self-contained modular host-based IOC scanner

Spyre is a simple host-based IOC scanner built around the YARA pattern matching engine and other scan modules. The main goal of this project is easy operationalization of YARA rules and other indicators of compromise.

Users need to bring their own rule sets. The awesome-yara repository gives a good overview of free yara rule sets out there.

Spyre is intended to be used as an investigation tool by incident responders. It is not meant to evolve into any kind of endpoint protection service.

Getting Started

Using Spyre is easy:

1. Add YARA signatures. In its default configuration, Spyre will read YARA rules for file and process scanning from filescan.yar and procscan.yar, respectively. The following options exist for providing rules files to Spyre (and will be tried in this order):

   1. Add the rule files to a ZIP file and append that ZIP file to the binary.
   2. Add the rule files to a ZIP file whose base name is identical to the scanner binary's base name, i.e. if the Spyre binary is called spyre or spyre.exe, use spyre.zip.
   3. Put the rule files and the scanner binary into the same directory.

   ZIP file contents may be encrypted using the password infected (AV industry standard) to prevent antivirus software from scanning the ruleset, classifying it as malicious content and preventing the scan.

   YARA rule files may contain include statements.

2. Deploy, run the scanner

3. Collect report and evidence

Configuration

Run-time configuration is done via an optional file spyre.yaml.

If a ZIP file has been appended to the Spyre binary, configuration and other files such as YARA rules are only read from this ZIP file. Otherwise, they are read from the directory into which the binary has been placed.

See the example-configuration/ subdirectory for an example.

Global configuration

• hostname / command line switch --set-hostname: Explicitly set the hostname that will be used in the log file and in the report. This is usually not needed.

• max-file-size / command line switch --max-file-size: Maximum size for files to be scanned using expensive file scanning modules such as YARA. Default: 32MB

• proc-ignore-names / command line switch --proc-ignore: Names of processes that will not be scanned using process memory scanning modules.

• paths / command line switch --path: Paths to be scanned using file scanning modules. Default: / (Unix) or all fixed drives (Windows).

• report / comand line switch --report: Set one or more report targets. Default: spyre_${hostname}_${time}.log in the current working directory, using the plain format. A different output format can be specified by appending ,format=FORMAT.

  The following formats are currently supported:

  • plain, the default, a simple human-readable text format
  • tsjson, a JSON document that can be imported into Timesketch

  The hostname and time variables are only expanded in the target filename.

  Note: Configuration of report targets is likely to change in one of the next releases.

• high-priority / command line switch --high-priority: In its default configuration (with this setting disabled), Spyre instructs the OS scheduler to lower the priorities of CPU time and I/O operations, in order to avoid disruption of normal system operation.

• command line switch --loglevel=LEVEL: Set the log level. Valid: trace, debug, info, notice, warn, error, quiet.

Module-specific configuration

There are currently three areas for which scanning modules can be implemented: System-level checks, file scans, and process scans.

Listed below are the currently implemented modules and supported configuration parameters.

• system
  • eventobj (Windows)
    • iocs
  • registry (Windows)
    • iocs
  • winkernelobj (Windows)
    • iocs
  • findwindow (Windows)
    • iocs
• file
  • yara
    • rule-files
    • fail-on-warnings
• proc
  • yara
    • rule-files
    • fail-on-warnings

Please refer to the example configuration file example-spyre.yaml for hints on how to describe indicators of compromise for each module.

Notes about YARA rules

YARA is configured with default settings, plus the following explicit switches (cf. 3rdparty.mk):

• --disable-magic
• --disable-cuckoo
• --enable-dotnet
• --enable-macho
• --enable-dex

For file scans, the following variables are defined:

• filename,
• filepath,
• extension,
• filetype (not currently populated while scanning)

For process scans, the variables pid and executable are defined.

The spyre_collect_limit metavariable can be used to limit the number of writes collected from matching files or to inhibit collecting files altogether. This can be useful to limit the size of evidence packages and to avoid collecting sensitive information.

Building

Spyre can be built for 32bit and 64bit Linux and Windows targets.

Debian Buster (10.x) and later

On a Debian/buster system (or a chroot) in which the following packages have been installed:

• make
• gcc
• gcc-multilib
• gcc-mingw-w64
• autoconf
• automake
• libtool
• pkg-config
• wget
• patch
• sed
• golang-$VERSION-go, e.g. golang-1.8-go. The Makefile will automatically select the newest version unless GOROOT has been set.
• git-core
• ca-certificates
• zip

This describes the build environment that is exercised regularly via CI.

Fedora 30 and later

The same build has also been successfully tried on Fedora 30 with the following packages installed:

• make
• gcc
• mingw{32,64}-gcc
• mingw{32,64}-winpthreads-static
• autoconf
• automake
• libtool
• pkgconf-pkg-config
• wget
• patch
• sed
• golang
• git-core
• ca-certificates
• zip

Once everything has been installed, just type make. This should download archives for musl-libc, openssl, yara, build those and then build spyre.

The bare spyre binaries are created in _build/<triplet>/.

Running make release creates a ZIP file that contains those binaries for all supported architectures.

Generating binaries compatible with ancient Windows XP, Windows Server 2003

Compatibility with these systems was removed with Go 1.11, so a Go 1.10 toolchain is required. Since Go 1.10 does not support Go modules, third-party Go dependencies have to be vendored: Use a newer Go version do this (just run go vendor) and set GOROOT to point to the Go 1.10 toolchain before running make.

MacOSX

Currently, cross-compiling is not supported.

• GCC from Xcode
• Build-dependencies from Homebrew:
  • gnu-make
  • autoconf
  • automake
  • libtool
  • pkg-config
  • wget
  • gpatch
  • gnu-sed
  • gnu-tar
  • go
  • git
  • ca-certificates
  • zip

The system-supplied make is too old because Apple decided to be allergic to GPLv3. gmake from Homebrew works fine.

Coding

See HACKING.md

Copyright

Copyright 2018-2020 DCSO Deutsche Cyber-Sicherheitsorganisation GmbH

Copyright 2020-2021 Spyre Project Authors (see: AUTHORS.txt)

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

See the LICENSE file for the full license text.

spyre's Issues

Add log file scanner

The idea is to find log files and journald files on Linux, EVTX logs on Windows in their well-known locations and scan them for simple IOCs or YARA rules.

Implement pattern-based Event Object and Mutex scan

Make the build chain work on OSX

After installing gnu-make and openssl via homebrew the make build currently fails.

Log system information

…such as OS version, language, active networ adapters…

--report doesn't accept format option

I don't think that the --report flag works correctly

I think that, in order to create a TSJSON output file called 'log.json' it is supposed to be used like:

--report='log.json;format=tsjson'

but this isn't the case - this creates 2 log files called 'log.json' and 'format=tsjson'.

I believe the problem is that the simpleStringSlice in config.go is splitting the flag into 2 parts before it gets to target.go which tries to do the same thing - it is therefore impossible currently to set the output to be in the TSJSON format. My tiny patch is to change line 30 of target.go to use "@" as the delimiter for format, rather than ";".

Just as a side note, using ";" as a command-line delimiter adds an additional layer of complexity since it's usually a shell command delimiter so bash reads "spyre -r log.json;format=json" as "spyre -r log.json" and "format=json" - "@" is much easier to use. And "," would be an easier delimiter for multiple report files too.

Matching yara rule

Hi hillu,

it might be useful to get the information in the log which yara rule matched. Also it might be helpful to get the calling commandline at beginning of the log so you can reconstruct what you have done before (not sure about that)?

The shown hostname is not necasserarily the same host, which is analyzed. If I scan a mounted folder/partition an optional switch for giving the hostname might be useful.

Thx

Error when importing jsonl file in timesketch

Hi,

the current json output that is generated for timesketch cannot be imported.
The problem is that the Spyre output file starts with

[
{<JSON-DATA>}
,
...
]

Timesketch cannot handle this. You need to remove "[","]" and "," from the file. Also timesketch expects that the file has the file extension: .jsonl.

It would be great if this could be adjusted in Spyre.

Thx
Sven

Sigma scanning capability as feature request

Hi!

Are there any plans to implement plain sigma support for log scanning?

Would be a great feature to have the ability to scan log files with plain sigma rules alongside to the existing yara scanning! Depending on attacker / malware behavior Sigma is better suitable than yara in some cases.

As far as I know there is no (free) tool out there that has this capability.

Thx and greetings

Add module for network IOCs

The platform-specific equivalents of netstat can be used to find currently- or recently-active TCP or UDP sockets.

Allow setting command-line options from config file within appended zip

It would be very useful for some cases if the command-line config could be included in config.Fs as a simple file so that everything could be wrapped up into a single binary that doesn't require setting any custom flags to run.

Ideally, the config would be overwritten by any command-line config flags.

This would allow for someone doing IR to compile a single file that would have any custom settings and yara rules built into it so that all that needs to be done is for the file to be run on any suspected vulnerable host. Where the hosts are managed by other organisations, it may be difficult for the IR team to ensure that flags are correctly set.

Please tag 1.1.0 relase

I did not get to do that before losing access to the repository.
(Preferably merge #10 first, though.)

Cheers,
-Hilko

Introduce more than success/error state in module initialization

Currently, (spyre/modules.{System,File}Scanner).Init functions can return a single error value, indicating success or failure. There needs to be at least a third state:

• Success = Module initiialized – it should be used normally.
• Error = Module found configuration but could not parse it (user error) – spyre initialization should be aborted.
• Ignore = No configuration available or some API cannot be accessed – scans are run without this module

yara: Log matching strings

Move to yara 4.x

By moving spyre to the latest go-yara version we should update the supported yara version to 4.x

Please fix CI setup

I believe that the CI setup was broken due to invalidated credentials that I had granted towards CircleCI when my account was removed from the @DCSO organization (cf.). Please fix so tagged builds are released with a binary build. :-)

Add ignorelist for files

Implement YARA-based process memory scanning module.

Option to exclude vmtoolsd.exe Process from procscan

If you copy&paste procscan.yar and filescan.yar into a VMWare VM, then all procscan rules will match on vmtoolsd.exe process.

Skip large files before opening

The globel config.MaxFileSize (--max-file-size) parameter should not only be checked in the YARA file scanner module.

Support sending report to URL as POST

For widespread checks across a large (and poorly managed) IT estate, it would be useful if spyre could support https:// URLs in addition to file:// ones.

If all spyre did was send a POST to the URL given containing the body of the report then the IR team could manage collecting all the reports and processing them centrally without having to work on collecting the logs from each instance of spyre that ran (and without having to get IT management to correctly set up a file share that all hosts can access to deposit logs centrally).

Evaluate MXE as replacement for 3rdparty.mk

TIL that there has been an effort to provide a Make-based, stable cross-compiling environment going on since at least 2007: MXE. Perhaps we can use this for building the third-party dependencies.

More instructions about how to generate YARA and IOC.json?

Hi, everyone

 I have successfully compiled Spyre. However, When I run it on windows10, some errors occured:
 1) stat ioc.json, the system cannot find the file specified.
 2)yara init stat filescan.yar the system cannot find the file specified
 3) yara init state procscan.yar the system cannot find the file specified

 I think this may be caused by failing compiled yara.go.

 Could someone give me some advice about this?
 Thanks

Syntax of ioc.json for Registry

What's the right syntax for the file? I can't seem to get it to find keys.

{ "key": "\\HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows Defender\\Features\\MpPlatformKillbitsFromEngine", "name": "FeaturesMpPlatformKillbitsFromEngine", "description": "testing the key function" }

How to build for 32-bit program of linux on 64-bit linux using musl-gcc

Hi, I don't know how to build for 32-bit program of linux on 64-bit linux using musl-gcc, just like the spyre of i386-linux-musl, which I don't know how to generate on 64-bit linux. I have searched a lot,but don't find the solution.Could you help me out? Thanks in advance.

yara: Reject rules with warnings

When the YARA compiler complains that rules will be slowing down scanning, it is right, of course. We should simply reject such rules (and possibly make the behavior configurable at a later point, should the need arise.)

Good rules to test this with are the auto-generated rulesets from Intezer.

Add support for obfuscating IOCs

Instead of using plain archive/zip, we might use github.com/alexmullins/zip or github.com/yeka/zip. Next question: How to provide the password in a non-obvious way?

Exclude WinDefender from procscan

There are a tons of FP matches on the WinDefender svchost process. Testet on Win 7 32bit.

Implement scan for active windows with specific window classes

note: FindWindowA

Scan Modules

Hi, as I'm trying to use spyre, I successfully installed all packages. On a Kali Linux, I'm trying to launch the spyre running file. As I don't know much about yara scanning modules, I copy/pasted the filescan.yar and procscan.yar files from spyre/scanner.yara. Then, launching the running program, here's the error that pops up :
2021/10/25 14:26:13 Error initializing YARA-file module: syntax error, unexpected identifier
2021/10/25 14:26:13 Error initializing YARA-proc module: syntax error, unexpected identifier

Would you mind providing me with help concerning this error ? If it wouldnt bother you, maybe having an example file of these .yara files, and kind of a userguide to know how and where to put these said-scanning modules.
Thank you very much for your help and for providing such an interesting tool,

Log SHA256 sums of matched files.

Use Go modules

For better or worse, go dep fallen out of fashion, so we might as well get with the program.

(Of course this will not rid us of building the 3rd-party libraries.)

Report does not include files which could not be opened

At present the report file (either type) does not include a record of any files which were not scanned because the file could not be opened.

It would be useful to include these (at least because it might indicate that the scan was run with the incorrect privileges).

My hacky fix is to add a line to spyre.go after the current line 79 saying:
report.AddStringf(f, "Could not open file %s", path)
this isn't perfect as it's a different format from the other file warnings (like "Skipping large file") but I can't use AddFileInfo as this breaks (not 100% sure where, I suspect it's because the file 'f' isn't open-able so it can't be Stat'd by formatFileEntry, but I'm not sure).

Re-think handling of command line parameters, configuration file

There are a few problems regarding runtime configuration, e.g.:

1. configuration via the command line (and/or a params.txt file is clumsy as it is and will become worse as we add modules.
2. Thus far, spyre looks opportunistically for YARA files, either in the directory that holds the executable, or in an embedded executable. This is more complicated then necessary and error-prone for fragments of not-so-well-written YARA rule sets. The ad-hoc solution that was added for the registry and eventobj modules has similar problems.

I am currently leaning toward using a YAML-based configuration and mapping command line parameters directly to YAML document paths.

Every module could then have its default "seed" file(s) and a section in the configuration file where those files can be overridden.

Add proper support for yara `import` statement

The import statement when parsing YARA rules should be mapped to the filesystem abstraction that is used to read from the appended ZIP file if available. Currently, YARA will happily access the filesystem directly which is not what we want...

Note: (*yara.Compiler).SetIncludeCallback()

Separate the list of skipped large files from rule matches.

Maybe just inform about the list of skipped files at the end of the scan. It would significantly improve the readability of spyre.log.

Extend spyre to build on Fedora

spyre does not build on Fedora because make fails with

3rdparty.mk:14: *** (Currently) unsupported native triplet x86_64-redhat-linux.  Stop.

Add command line switch to set hostname

See comment in #1