target / strelka Goto Github PK

View Code? Open in Web Editor NEW

797.0 40.0 105.0 21.36 MB

Real-time, container-based file scanning at enterprise scale

License: Other

Dockerfile 1.32% Python 87.15% YARA 5.48% Go 5.51% Zeek 0.42% Shell 0.12%

security detection cfc target-cfc python3 golang yara

strelka's People

Contributors

Stargazers

Watchers

Forkers

zer0d0y kartikeyap aglerj cclauss killvxk crackercat deelmind jack51706 awesome-security kristinwaters steelzy jshlbrd rbramwell sasqwatch chemberger gregcopenhaver jpressnell zachsis davidjbianco tnvo pmelson johnjohnsp1 ufomorme seclab-int-dev-group thezedwards triplekill malvidin marcomafcorp lazerhawk nateicart bblenard quinn-yan mikekmiller tzf-omkey zate mambaproject blue-infosec ashishsecdev rhaist ravifatty kjruiz110 delbs27 forkkit bigbrobro scottpas raimundojimenez nasnaq billmcchesney1 weslambert sashka3076 chandanvatsa cybermung darthcyber ninoseki syllogy actorexpose aleclangford classicvalues sublime-security pombredanne 5l1v3r1 cawalch krautface ammasajan rroethof-anylinq cklinuxproject radizzle mdixey17 maryan21 tuduongquyet hartl3y94 gdesmar phycheyu colindean mars-suite chennqqi odoochain nasingfaund mesutozsoycom ryanohoro susohym aaronherman rondorondorondo morriscode derekt2 keiche gaojie-wang nighttardis cybermonk3y l3aalteshuva alexk307 ebolaware slooppe rwaight musadeola baixuanfeng prismacloudlabs utsav1256

strelka's Issues

Create ScanFooter Scanner

Is your feature request related to a problem? Please describe.
ScanHeader allows analysts to observe the first X characters in each file. This is useful for confirming file tastes, observing unknown file content, and analyzing miscellaneous content. Similarly, a scanner that reads and collects the last X characters in a file may be useful for collecting signatures or other known patterns.

Describe the solution you'd like
Implement ScanFooter that mirrors ScanHeader.

Describe alternatives you've considered
N/A

Additional context
N/A

Create BITS Scanner (FireEye BitsParser)

Is your feature request related to a problem? Please describe.
FireEye recently released a Windows BITS parser
which may be useful for passive data collection and storage of BITS jobs. While the BITS
database is unlikely be to transferred over the network (if you're using Strelka as a network
security tool), users may be able to submit this file from one or many hosts to perform
large scale BITS collection without the need for a separate endpoint querying tool.

Describe the solution you'd like
Create a scanner that will take in a BITS database and provide a list of
dictionaries, like the following:

{
    "JobType": "download",
    "JobPriority": "normal",
    "JobState": "suspended",
    "JobId": "b733e5e1-12ad-463e-a125-ade26cc1fab6",
    "JobName": "SpeechModelDownloadJob",
    "OwnerSID": "S-1-5-20",
    "Owner": "NT AUTHORITY\\NETWORK SERVICE",
    "CreationTime": "2021-01-25T11:52:05Z",
    "ModifiedTime": "2021-01-25T12:45:21Z"
}

Describe alternatives you've considered
N/A

Additional context
N/A

python-docx Python3.10 compatibility issue breaks docx parsing

Describe the bug
Utilizing the latest versions of Python3 and the python-docx module causes an error (ImportError: cannot import name 'Sequence' from 'collections') due to incompatibility with Python3.10's renaming of collections to collections.abc.

Steps to reproduce
Steps to reproduce the behavior:

Build Strelka using Python3.10 (default for docker deployment)
Process a docx file type though the fileshot process
See error ImportError: cannot import name 'Sequence' from 'collections'

Expected behavior
File processes successfully and outputs results in log file

Release

Release: 0.22.12.08

Additional context
Issue was raised on the official python-docx module repo in November, but has not received any interaction.

Add in 'root id' to 'tree' object

Is your feature request related to a problem? Please describe.
The current 'tree' object in Strelka displays a unique identifier for the current File (node) as well as (if applicable) the identifier for the parent File (parent). If the depth of a file is greater than 1 (i.e., if a file is insize a zip is inside a zip), then a user will have to walk from parent to node to get one level up, then walk from parent to node again to get the original object. While users can access the root node with request.id, it is not available in the 'tree' object.

Current working tree:

For depth: 0 
    "tree": {
      "node": "33523e49-a66c-404e-9aa9-da6afa7dd376",
    }

For depth: 1
    "tree": {
      "node": "a140807c-561e-41b0-b855-de806e87812b",
      "parent": "33523e49-a66c-404e-9aa9-da6afa7dd376",
    }

For depth: 2
    "tree": {
      "node": "34a377ac-3057-4b55-a333-af56e11d517e",
      "parent": "a140807c-561e-41b0-b855-de806e87812b",
    }

If you are examining the record at depth:2, you would need to use that tree.parent id to get to depth:1 and repeat to get to the root file.

Describe the solution you'd like
Add in a tree.root object that always maintains the root id.

For depth: 0 
    "tree": {
      "node": "33523e49-a66c-404e-9aa9-da6afa7dd376",
      "root": "33523e49-a66c-404e-9aa9-da6afa7dd376"
    }

For depth: 1
    "tree": {
      "node": "a140807c-561e-41b0-b855-de806e87812b",
      "parent": "33523e49-a66c-404e-9aa9-da6afa7dd376",
      "root": "33523e49-a66c-404e-9aa9-da6afa7dd376"
    }

For depth: 2
    "tree": {
      "node": "34a377ac-3057-4b55-a333-af56e11d517e",
      "parent": "a140807c-561e-41b0-b855-de806e87812b",
      "root": "33523e49-a66c-404e-9aa9-da6afa7dd376"
    }

Describe alternatives you've considered
Don't do this and add in documentation stating that you can obtain the root object with request.id.

Additional context
N/A

docker-compose fails to build due to dependency error

Describe the bug
Upon running docker-compose build for Strelka version 2021111, two errors are observed related to the mmbot package relating to dependencies: scikit-learn, Cython, and numpy. Strelka fails to complete the build process.

Steps to reproduce
Steps to reproduce the behavior:

Run docker-compose -f ./build/docker-compose.yml from the root directory.

Expected behavior
No errors, completed Strelka build.

Screenshots

  Traceback (most recent call last):
    File "/tmp/pip-build-f8e4hqbk/scikit-learn/setup.py", line 201, in check_package_status
      module = importlib.import_module(package)
    File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 994, in _gcd_import
    File "<frozen importlib._bootstrap>", line 971, in _find_and_load
    File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  ModuleNotFoundError: No module named 'numpy'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-f8e4hqbk/scikit-learn/setup.py", line 306, in <module>
      setup_package()
    File "/tmp/pip-build-f8e4hqbk/scikit-learn/setup.py", line 292, in setup_package
      check_package_status('numpy', min_deps.NUMPY_MIN_VERSION)
    File "/tmp/pip-build-f8e4hqbk/scikit-learn/setup.py", line 227, in check_package_status
      .format(package, req_str, instructions))
  ImportError: numpy is not installed.
  scikit-learn requires numpy >= 1.13.3.
  Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html
  
  
  ----------------------------------------
  Failed building wheel for scikit-learn
  Running setup.py clean for scikit-learn
  Running setup.py bdist_wheel for colorclass: started
  Running setup.py bdist_wheel for colorclass: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/d1/86/9d/16127127306a92d7fd30267890a5634026c045391979c4c317
  Running setup.py bdist_wheel for msoffcrypto-tool: started
  Running setup.py bdist_wheel for msoffcrypto-tool: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/f4/a6/40/413c706c56b1aee940dd6873b425dd0ab4d67bd4c0d11cf6a4
  Running setup.py bdist_wheel for olefile: started
  Running setup.py bdist_wheel for olefile: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/4b/f4/11/bc4166107c27f07fd7bba707ffcb439619197638a1ac986df3
Successfully built oletools colorclass msoffcrypto-tool olefile
Failed to build scikit-learn
Installing collected packages: cython, six, grpcio, protobuf, grpcio-tools, numpy, joblib, colorclass, easygui, pycparser, cffi, cryptography, olefile, msoffcrypto-tool, pcodedmp, pyparsing, oletools, pytz, python-dateutil, pandas, scipy, threadpoolctl, scikit-learn, mmbot
  Running setup.py install for scikit-learn: started
    Running setup.py install for scikit-learn: finished with status 'error'
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-f8e4hqbk/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-8zax0jyp-record/install-record.txt --single-version-externally-managed --compile:
    Partial import of sklearn during the build process.
    C compiler: x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC
    
    compile options: '-c'
    x86_64-linux-gnu-gcc: test_program.c
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/unixccompiler.py", line 51, in UnixCCompiler__compile
        extra_postargs, display = display)
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/ccompiler.py", line 90, in <lambda>
        m = lambda self, *args, **kw: func(self, *args, **kw)
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/ccompiler.py", line 174, in CCompiler_spawn
        (cmd, s, msg))
    distutils.errors.DistutilsExecError: Command "x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c test_program.c -o objects/test_program.o" failed with exit status 127
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-f8e4hqbk/scikit-learn/setup.py", line 306, in <module>
        setup_package()
      File "/tmp/pip-build-f8e4hqbk/scikit-learn/setup.py", line 302, in setup_package
        setup(**metadata)
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/core.py", line 135, in setup
        config = configuration()
      File "/tmp/pip-build-f8e4hqbk/scikit-learn/setup.py", line 188, in configuration
        config.add_subpackage('sklearn')
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/misc_util.py", line 1020, in add_subpackage
        caller_level = 2)
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/misc_util.py", line 989, in get_subpackage
        caller_level = caller_level + 1)
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/misc_util.py", line 926, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "sklearn/setup.py", line 83, in configuration
        cythonize_extensions(top_path, config)
      File "/tmp/pip-build-f8e4hqbk/scikit-learn/sklearn/_build_utils/__init__.py", line 45, in cythonize_extensions
        basic_check_build()
      File "/tmp/pip-build-f8e4hqbk/scikit-learn/sklearn/_build_utils/pre_build_helpers.py", line 106, in basic_check_build
        compile_test_program(code)
      File "/tmp/pip-build-f8e4hqbk/scikit-learn/sklearn/_build_utils/pre_build_helpers.py", line 67, in compile_test_program
        extra_postargs=extra_postargs)
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/ccompiler.py", line 90, in <lambda>
        m = lambda self, *args, **kw: func(self, *args, **kw)
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/ccompiler.py", line 361, in CCompiler_compile
        single_compile(o)
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/ccompiler.py", line 321, in single_compile
        self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/ccompiler.py", line 90, in <lambda>
        m = lambda self, *args, **kw: func(self, *args, **kw)
      File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/unixccompiler.py", line 54, in UnixCCompiler__compile
        raise CompileError(msg)
    distutils.errors.CompileError: Command "x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c test_program.c -o objects/test_program.o" failed with exit status 127
    
    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-f8e4hqbk/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-8zax0jyp-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-f8e4hqbk/scikit-learn/

Release

0.20.11.20

Additional context
N/A

JavaScript deobfuscation

Is your feature request related to a problem? Please describe.
There are two issues with ScanJavascript deobfuscation:
First, the scanner doesn't communicate if JS has been de-obfuscated
Second, if deobfuscation fails, then the entire scan fails

Describe the solution you'd like
Add a boolean field ("deobfuscated") that describes if deobfuscation was successful
If deobfuscation fails, then parse the obfuscated JS

Describe alternatives you've considered
N/A

Additional context
N/A

failed to connect to : context deadline exceeded

Describe the bug
Hey all I'm following the Quickstart steps in order to test this project but I'm running into failed to connect to : context deadline exceeded after starting the cluster and building/running the strelka-fileshot binary. Is there an extra step I'm missing? Looks like a great project and thanks in advance!

Steps to reproduce
Steps to reproduce the behavior:

1: Build and Start Strelka Cluster (Docker)

# Terminal 1
$ docker-compose -f build/docker-compose.yaml up

Build Strelka-Fileshot (File Submitter)

# Terminal 2
$ go build github.com/target/strelka/src/go/cmd/strelka-fileshot

Add File Paths / Patterns to be Scanned to a [fileshot.yaml] file

files:
    patterns:
      - '/tmp/*'

Error

./strelka-fileshot -c fileshot.yaml
2022/05/05 08:56:46 failed to connect to : context deadline exceeded

Expected behavior
Expect the scanner to scan and output results to ./strelka.log

Release

0.22.04.26 & 0.21.11.29

Additional context
Tested on macOS Monterey & Ubuntu 22.04
Tested with both fileshot binary and fileshot container.

ScanDocx extracted text missing newline characters

Describe the bug
ScanDocx produces an extracted text file that is missing new line characters.

Steps to reproduce
Steps to reproduce the behavior:

Scan file with ScanDocx
Review full output of extracted text (will require modification of ScanDocx)

Expected behavior
ScanDocx should produce an extracted text file that more closely matches the text in the document.

Screenshots
N/A

Release
N/A

Additional context

ScanElf failed with `uncaught_exception`

Describe the bug
ScanElf failed with uncaught_exception

Steps to reproduce
Steps to reproduce the behavior:

Build and run strelka-fileshot or strelka-oneshot on Ubuntu 20.04

./strelka-oneshot -f strelka-fileshot

Read response from server

{
  "file": {
    "depth": 0,
    "flavors": {
      "mime": [
        "application/x-executable"
      ],
      "yara": [
        "elf_file"
      ]
    },
    "scanners": [
      "ScanElf",
      "ScanEntropy",
      "ScanFooter",
      "ScanHash",
      "ScanHeader",
      "ScanYara"
    ],
    "size": 12777735,
    "tree": {
      "node": "7783472d-7d2a-4703-b714-b098f5365f18",
      "root": "7783472d-7d2a-4703-b714-b098f5365f18"
    }
  },
  "request": {
    "attributes": {
      "filename": "strelka-fileshot"
    },
    "client": "go-oneshot",
    "id": "7783472d-7d2a-4703-b714-b098f5365f18",
    "source": "ubuntu",
    "time": 1650964452
  },
  "scan": {
    "elf": {
      "elapsed": 0.583676,
      "flags": [
        "uncaught_exception"
      ]
    },
    "entropy": {
      "elapsed": 0.028771,
      "entropy": 6.690920985654504
    },
    "footer": {
      "elapsed": 5.7e-05,
      "footer": "me.buildVersion.str\u0000type.*\u0000runtime.textsectionmap\u0000"
    },
    "hash": {
      "elapsed": 0.177556,
      "md5": "40d131ae6f009d175c00aa12dec26895",
      "sha1": "26fb220f7450674ca63324299b28e9292b2ae73f",
      "sha256": "7cc948620272bdb96cb62ab1c94c4ae0dbcd1f261b9e35c53557773cb71b7d68",
      "ssdeep": "196608:hSsNJlt4pAgVLA7H2UINxrnK5YVKUKvKh:UQUpAgZiArrnKq4v2"
    },
    "header": {
      "elapsed": 7.7e-05,
      "header": "\u007fELF\u0002\u0001\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0002\u0000>\u0000\u0001\u0000\u0000\u0000�cF\u0000\u0000\u0000\u0000\u0000@\u0000\u0000\u0000\u0000\u0000\u0000\u0000�\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
    },
    "yara": {
      "elapsed": 0.035788,
      "matches": [
        "test"
      ]
    }
  }
}

Expected behavior
Elf scanner runs properly

Release

Release: [e.g. latest or 0.21.11.29]

scan_tnef - AttributeError / Data Exception

Describe the bug
While extracting files, an exception is thrown while trying to reference a 'data' key in the tnef_html object. No 'data' key exists.

Steps to reproduce
Run.

Expected behavior
N/A

Screenshots

backend_1      | Traceback (most recent call last):
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/strelka.py", line 148, in scan_wrapper
backend_1      |     self.scan(data, file, options, expire_at)
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/scanners/scan_tnef.py", line 57, in scan
backend_1      |     for c in strelka.chunk_string(tnef_html.data):
backend_1      | AttributeError: 'bytes' object has no attribute 'data'

Release

Release: 0.19.08.27

Additional context
N/A

PEP8 line length

Is your feature request related to a problem? Please describe.
While the project mostly adheres to PEP8, meeting the requirements for line lengths of 79 characters is problematic due to the necessity of nested code (especially in scanners where layers can be wrapped in multiple try/except and with statements). We should decide on a more appropriate line length and update the code to adhere to it.

Describe the solution you'd like
Decision regarding PEP8 line length requirements, an update to the documentation reflecting the decision, and an update to the code to adhere to the requirement.

Describe alternatives you've considered
Currently PEP8 line length requirements fall under "nice to have" -- we could make this the accepted standard for the project.

Additional context
N/A

It appears go-redis update changed the arguments to cd.cmdable.ping

Describe the bug
While running the docker-compose the manager fails to build with error.

# github.com/target/strelka/src/go/cmd/strelka-manager

./main.go:40:19: not enough arguments in call to cd.cmdable.Ping
	have ()
	want (context.Context)
./main.go:46:35: not enough arguments in call to cd.cmdable.ZRemRangeByScore
	have (string, string, string)
	want (context.Context, string, string, string)

Steps to reproduce
Steps to reproduce the behavior:

Run docker-compose -f build/docker-compose.yaml up -d
Scroll to bottom
See error fails the docker-compose build sequence

Expected behavior
Expect the docker-compose process to complete with built and running strelka containers.

Additional context
Go-redis pushed changes 4 days ago which changed the arguments to cmdable.Ping

Run pytest In Docker Build

Is your feature request related to a problem? Please describe.

Dependencies can get out of sync between different parts of the project, creating dependency problems that may be opaque in a dev environment.

Describe the solution you'd like

Run pytest in the docker build to assure that tests pass in the build, not just the dev environment.

Describe alternatives you've considered

Running the test process in a clean build environment.

Additional context

Build workflows simply build the docker containers, which makes the build process itself a good place to run pytest.

pytest runs in this workflow file separately from the Docker build

https://github.com/target/strelka/actions/runs/3679057853/workflow

scan_html AttributeError 'startswith'

Describe the bug
variable 'link' is not initialized in the scan_html scanner (line 37).

Steps to reproduce

Execution of default fileshot config on OSX.

Expected behavior
No errors, hopefully.

Screenshots

backend_1      | 2019-08-22 15:29:44 - [ERROR] root [strelka.scan_wrapper]: ScanHtml: exception while scanning uid aa331ccf-d87b-47b7-8343-8e7081da57bf (see traceback below)
backend_1      | Traceback (most recent call last):
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/strelka.py", line 148, in scan_wrapper
backend_1      |     self.scan(data, file, options, expire_at)
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/scanners/scan_html.py", line 37, in scan
backend_1      |     if link.startswith('data:') and ';base64,' in link:
backend_1      | AttributeError: 'NoneType' object has no attribute 'startswith'
backend_1      | 2019-08-22 15:32:17 - [ERROR] root [strelka.scan_wrapper]: ScanHtml: exception while scanning uid fa13e579-70d1-46d1-a2b4-f2127ee2196d (see traceback below)
backend_1      | Traceback (most recent call last):
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/strelka.py", line 148, in scan_wrapper
backend_1      |     self.scan(data, file, options, expire_at)
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/scanners/scan_html.py", line 37, in scan
backend_1      |     if link.startswith('data:') and ';base64,' in link:
backend_1      | AttributeError: 'NoneType' object has no attribute 'startswith'

Release

Release: 0.19.07.26

Additional context
N/A

Error when building docker container for strelka-frontend

Describe the bug
When using the docker-compose.yml file for a quickstart the build of the docker image for the strelka-frontend failes

Steps to reproduce
Steps to reproduce the behavior:

Clone the current strelka repository and use master or version 0.19.08.28
Run up strelka using the docker-compose file: docker-compose -f build/docker-compose.yaml up
See error

Step 4/8 : RUN cd /go/src/github.com/target/strelka/src/go/cmd/strelka-frontend/ &&     go get . &&     CGO_ENABLED=0 go build -o /tmp/strelka-frontend .
 ---> Running in 43f32b29e411
cannot find package "go.opentelemetry.io/otel/api/global" in any of:
        /usr/local/go/src/go.opentelemetry.io/otel/api/global (from $GOROOT)
        /go/src/go.opentelemetry.io/otel/api/global (from $GOPATH)
cannot find package "go.opentelemetry.io/otel/api/metric" in any of:
        /usr/local/go/src/go.opentelemetry.io/otel/api/metric (from $GOROOT)
        /go/src/go.opentelemetry.io/otel/api/metric (from $GOPATH)
cannot find package "go.opentelemetry.io/otel/api/trace" in any of:
        /usr/local/go/src/go.opentelemetry.io/otel/api/trace (from $GOROOT)
        /go/src/go.opentelemetry.io/otel/api/trace (from $GOPATH)
ERROR: Service 'frontend' failed to build: The command '/bin/sh -c cd /go/src/github.com/target/strelka/src/go/cmd/strelka-frontend/ &&     go get . &&     CGO_ENABLED=0 go build -o /tmp/strelka-frontend .' returned a non-zero code: 1

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Release

Release: github master branch
Release: 0.19.08.28

Additional context
It seems that this is a problem when resolving the golang package opentelemetry packages go.opentelemetry.io/otel/api/global.

Review Implementation of TLSH Hashing

Is your feature request related to a problem? Please describe.
TLSH (Trend Micro Locality Sensitive Hash) is a fuzzy matching library. Given a byte stream with a minimum length of 50 bytes TLSH generates a hash value which can be used for similarity comparisons. Similar objects will have similar hash values which allows for the detection of similar objects by comparing their hash values. Note that the byte stream should have a sufficient amount of complexity. For example, a byte stream of identical bytes will not generate a hash value.

Implement TLSH hash for additional comparison capabilities.

Describe the solution you'd like
Add TLSH to ScanHash.py

import hashlib
import ssdeep
import tlsh
from strelka import strelka


class ScanHash(strelka.Scanner):
    """Calculates file hash values."""
    def scan(self, data, file, options, expire_at):
        self.event['md5'] = hashlib.md5(data).hexdigest()
        self.event['sha1'] = hashlib.sha1(data).hexdigest()
        self.event['sha256'] = hashlib.sha256(data).hexdigest()
        self.event['ssdeep'] = ssdeep.hash(data)
        self.event['tlsh'] = tlsh.hash(data)

Describe alternatives you've considered
N/A

Additional context
N/A

[scan_docx] invalid distance too far back

Describe the bug
Upon submission of a specific document (specificity undefined), a zlib error is throw.

Offending line:

strelka/src/python/strelka/scanners/scan_docx.py

Line 23 in a7816bd

docx_doc = docx.Document(docx_io)

zlib.error: Error -3 while decompressing data: invalid distance too far back

Steps to reproduce
Steps to reproduce the behavior:

Submit document with known issue. Known document is malicious and no reference will be provided here.

Expected behavior
No error on file scan

Screenshots

backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/strelka.py", line 148, in scan_wrapper
backend_1      |     self.scan(data, file, options, expire_at)
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/scanners/scan_docx.py", line 23, in scan
backend_1      |     docx_doc = docx.Document(docx_io)
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/docx/api.py", line 25, in Document
backend_1      |     document_part = Package.open(docx).main_document_part
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/docx/opc/package.py", line 128, in open
backend_1      |     pkg_reader = PackageReader.from_file(pkg_file)
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/docx/opc/pkgreader.py", line 36, in from_file
backend_1      |     phys_reader, pkg_srels, content_types
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/docx/opc/pkgreader.py", line 69, in _load_serialized_parts
backend_1      |     for partname, blob, reltype, srels in part_walker:
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/docx/opc/pkgreader.py", line 110, in _walk_phys_parts
backend_1      |     for partname, blob, reltype, srels in next_walker:
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/docx/opc/pkgreader.py", line 105, in _walk_phys_parts
backend_1      |     blob = phys_reader.blob_for(partname)
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/docx/opc/phys_pkg.py", line 108, in blob_for
backend_1      |     return self._zipf.read(pack_uri.membername)
backend_1      |   File "/usr/lib/python3.6/zipfile.py", line 1338, in read
backend_1      |     return fp.read()
backend_1      |   File "/usr/lib/python3.6/zipfile.py", line 858, in read
backend_1      |     buf += self._read1(self.MAX_N)
backend_1      |   File "/usr/lib/python3.6/zipfile.py", line 948, in _read1
backend_1      |     data = self._decompressor.decompress(data, n)
backend_1      | zlib.error: Error -3 while decompressing data: invalid distance too far back

Release

Release: 0.19.08.27

Additional context
N/A

PyMuPDF function xref_length doesn't exist in version 1.18.0

Describe the bug
When trying to run a PDF through Strelka with the current "build/python/backend/requiments.txt", PDF always fail with

"pdf": { "elapsed": 0.000511, "flags": [ "pdf_load_error" ], "total": { "extracted": 0, "objects": 0 } },

This is occurring because the function "xref_length" does not exist in PyMuPDF version 1.18.0

Reviewing the change history for the requirements.txt file it looks like the initial deployment of scan_pdf.py included PyMuPDF version 1.18.10, which does steam to have function. It also appears that on Aug 3rd 2021 the version was bumped to 1.18.15 and then downgrade to 1.18.0 instead of 1.18.10.

Steps to reproduce
Steps to reproduce the behavior:

Build the backend using the current "build" folder
Attempt to submit a PDF.

Expected behavior
Expectation that PDF are properly parsed.

Screenshots
If applicable, add screenshots to help explain your problem.

Release

Release: [e.g. 1.19.05.16]

Additional context
Add any other context about the problem here.

Problem Building Frontend Container

Describe the bug
The frontend Docker container build fails due to a type error in the main.go file.

Steps to reproduce
Steps to reproduce the behavior:

Run docker-compose -f build/docker-compose.yaml up
See error

# github.com/target/strelka/src/go/cmd/strelka-frontend`
./main.go:146:25: cannot use &redis.Z literal (type *redis.Z) as type redis.Z in argument to s.coordinator.cli.cmdable.ZAdd

Expected behavior
Successful build of the Strelka frontend container.

Release

Commit ID: 669fd86

Additional context
Linux: SLES 15 4.12.14-197.15-default
Docker version: 19.03.1
Go version: 1.13.4 linux/amd64

This seems to be similar to #71.

Please let me know if you need any additional information.

[REQUEST] Single Binary for Ad-hoc Analysis

Is your feature request related to a problem? Please describe.
It would be great to have the ability to quickly analyze a file using a single slimmed-down binary, without having to connect to a remote backend. This would not necessarily require all components typically required for enterprise-level scanning, and would be excellent for demo/training/quick analysis without having to set up infrastructure. The expectation would be that it is used for ad-hoc analysis.

Describe alternatives you've considered
Running a local version of the backend/other components
Using docker-compose

Compilation Issues (grpcio dependency) on Apple M1 (ARM)

Describe the bug
While compiling the current version of Strelka on a Macbook M1 (using ARM), an error is thrown while compiling grpcio

#17 21.14   Downloading grpcio-1.27.2.tar.gz (16.9 MB)
#17 24.60     ERROR: Command errored out with exit status 1:
#17 24.60      command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cba9d7s2/grpcio/setup.py'"'"'; __file__='"'"'/tmp/pip-install-cba9d7s2/grpcio/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-cba9d7s2/grpcio/pip-egg-info
#17 24.60          cwd: /tmp/pip-install-cba9d7s2/grpcio/
#17 24.60     Complete output (13 lines):
#17 24.60     /tmp/pip-install-cba9d7s2/grpcio/src/python/grpcio/commands.py:102: SyntaxWarning: "is not" with a literal. Did you mean "!="?
#17 24.60       if exit_code is not 0:
#17 24.60     Traceback (most recent call last):
#17 24.60       File "<string>", line 1, in <module>
#17 24.60       File "/tmp/pip-install-cba9d7s2/grpcio/setup.py", line 191, in <module>
#17 24.60         if check_linker_need_libatomic():
#17 24.60       File "/tmp/pip-install-cba9d7s2/grpcio/setup.py", line 149, in check_linker_need_libatomic
#17 24.60         cc_test = subprocess.Popen(['cc', '-x', 'c++', '-std=c++11', '-'],
#17 24.60       File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
#17 24.60         self._execute_child(args, executable, preexec_fn, close_fds,
#17 24.60       File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
#17 24.60         raise child_exception_type(errno_num, err_msg, err_filename)
#17 24.60     FileNotFoundError: [Errno 2] No such file or directory: 'cc'
#17 24.60     ----------------------------------------
#17 24.67 ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c pip3 install --no-cache-dir -r /strelka/requirements.txt]: exit code: 1

Steps to reproduce
Steps to reproduce the behavior:

Run docker-compose -f build/docker-compose.yml build on an ARM host with docker and docker-compose installed.

Expected behavior
No compilation errors

Screenshots
See error above

Release

Release: 0.21.11.29

Additional context
N/A

Compilation Issues (pymupdf dependency) on Apple M1 (ARM)

Describe the bug
While compiling the current version of Strelka on a Macbook M1 (using ARM), an error is thrown while compiling pymupdf

#43 107.9     Running setup.py install for PyMuPDF: started
#43 108.1     Running setup.py install for PyMuPDF: finished with status 'error'
#43 108.1     ERROR: Command errored out with exit status 1:
#43 108.1      command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-exfn5fk_/pymupdf_48028f1220064d828bfdd4a85b07f456/setup.py'"'"'; __file__='"'"'/tmp/pip-install-exfn5fk_/pymupdf_48028f1220064d828bfdd4a85b07f456/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-iekrbeh6/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.9/PyMuPDF
#43 108.1          cwd: /tmp/pip-install-exfn5fk_/pymupdf_48028f1220064d828bfdd4a85b07f456/
#43 108.1     Complete output (20 lines):
#43 108.1     running install
#43 108.1     running build
#43 108.1     running build_py
#43 108.1     creating build
#43 108.1     creating build/lib.linux-aarch64-3.9
#43 108.1     creating build/lib.linux-aarch64-3.9/fitz
#43 108.1     copying fitz/__init__.py -> build/lib.linux-aarch64-3.9/fitz
#43 108.1     copying fitz/fitz.py -> build/lib.linux-aarch64-3.9/fitz
#43 108.1     copying fitz/utils.py -> build/lib.linux-aarch64-3.9/fitz
#43 108.1     copying fitz/__main__.py -> build/lib.linux-aarch64-3.9/fitz
#43 108.1     running build_ext
#43 108.1     building 'fitz._fitz' extension
#43 108.1     creating build/temp.linux-aarch64-3.9
#43 108.1     creating build/temp.linux-aarch64-3.9/fitz
#43 108.1     aarch64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/mupdf -I/usr/local/include/mupdf -I/usr/include/python3.9 -c fitz/fitz_wrap.c -o build/temp.linux-aarch64-3.9/fitz/fitz_wrap.o
#43 108.1     fitz/fitz_wrap.c:2733:10: fatal error: fitz.h: No such file or directory
#43 108.1      2733 | #include <fitz.h>
#43 108.1           |          ^~~~~~~~
#43 108.1     compilation terminated.
#43 108.1     error: command '/usr/bin/aarch64-linux-gnu-gcc' failed with exit code 1
#43 108.1     ----------------------------------------
#43 108.1 ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-exfn5fk_/pymupdf_48028f1220064d828bfdd4a85b07f456/setup.py'"'"'; __file__='"'"'/tmp/pip-install-exfn5fk_/pymupdf_48028f1220064d828bfdd4a85b07f456/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-iekrbeh6/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.9/PyMuPDF Check the logs for full command output.

Steps to reproduce
Steps to reproduce the behavior:

Run docker-compose -f build/docker-compose.yml build on an ARM host with docker and docker-compose installed.

Expected behavior
No compilation errors

Screenshots
See error above

Release

Release: 0.21.12.9

Additional context
N/A

Package version pinning / pip requirements file

Is your feature request related to a problem? Please describe.
Currently the project doesn't do any package version pinning nor use a pip requirements install file-- we should do that.

Describe the solution you'd like
Determine the appropriate versions for each package, combine them into a pip requirements file, update the README with the current package version.

Describe alternatives you've considered
N/A

Additional context
N/A

scan_tnef - Strip Exception

Describe the bug
While scanning [TNEF] (https://en.wikipedia.org/wiki/Transport_Neutral_Encapsulation_Format) files, an exception occurs when data is submitted an object string cannot be parsed by strip().

Expected behavior
N/A

Screenshots

backend_1 | 2019-06-11 17:55:06 - [ERROR] root [strelka.scan_wrapper]: ScanTnef: exception while scanning uid 4d30fbe0-93d2-4d93-ba70-ebbf8bd4e590 (see traceback below)
backend_1 | Traceback (most recent call last):
backend_1 | File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/strelka.py", line 148, in scan_wrapper
backend_1 | self.scan(data, file, options, expire_at)
backend_1 | File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/scanners/scan_tnef.py", line 19, in scan
backend_1 | object_data = tnef_object.data.strip(b'\0') or None
backend_1 | TypeError: strip arg must be None or str

Release

Release: 0.19.08.27

Additional context
N/A

scan_pkcs7 API mismatch

Describe the bug
I used the steps in the documentation to build the Dockerfile this morning (checked out 4a97567) and scanned a test file that has a good load of filth in it (link - WARNING actually contains malware, but you'd have to go out of your way to trigger it (i.e. not executable, doesn't have executable file extensions, etc) )

The file contains an x509 cert, some vt scan data, some archive formats, and some OS binaries.

Steps to reproduce
Steps to reproduce the behavior:

Build and run the Dockerfile according to the directions
Submit the linked zip archive
See the PKCS object error

2019-05-13 12:27:02 - [ERROR] root [objects.scan_wrapper]: ScanPkcs7: exception while scanning file with hash f0bcf8f9fde7c6b737ae48761c7298e98c2412576ef0d07474df3211e309adf2 and uid 238f900f-34cc-43a7-881d-bce61c528089 (see traceback below)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/strelka-0.99-py3.6.egg/server/objects.py", line 513, in scan_wrapper
    self.scan(file_object, options)
  File "/usr/local/lib/python3.6/dist-packages/strelka-0.99-py3.6.egg/server/scanners/scan_pkcs7.py", line 20, in scan
    pkcs7_certificates = pkcs7.get_certificates()
AttributeError: 'PKCS7' object has no attribute 'get_certificates'

Expected behavior
I expected that Strelka would be able to handle the certificate errors, or at least die gracefully.

Server and project version

OS: Bionic via Docker, run on Mac
Commit Hash: 4a97567

Additional context
Looks like the PKCS7 API changed, is all.

[Feature Request] Add scanner support for 7z archives

cc @rw-access

Is your feature request related to a problem? Please describe.
It seems that Strelka doesn't have the ability to scan 7z archives, in the same way that it can for zip archives. This leads to some uncertainly when scanning 7z archives, specifically the ability to list files inside archives.

Describe the solution you'd like
Support for 7z archives similar to current zip archive support (zip scanner: https://github.com/target/strelka/blob/master/src/python/strelka/scanners/scan_zip.py)

Describe alternatives you've considered
Cloning and refactoring the existing zip scanner, however my Python skills are not that advanced...

Thank you for your consideration!

Use go fmt

Looking at the Go source code and fixing some minor bugs I noticed that the official go fmt was not used to format the code. As every IDE applies go fmt automatically pull requests will be hard as the resulting git changes are basically every line of the original code.

I recommend either:

Running find . -iname '*.go' -exec gofmt -w {} \; from the project root once and push.
Let me prepare a PR for you with the changes.

Let me know what you think.

ScanUrl and JavaScript file issues

Describe the bug
The ScanUrl scanner regex doesn't seem to play well with JavaScript files -- it misinterprets JavaScript strings as URLs.

Steps to reproduce
Steps to reproduce the behavior:

Run large, complex JavaScript files through the system
Review url_metadata results

Expected behavior
Non-URL JavaScript shouldn't be interpreted as URLs. This may mean either excluding ScanUrl from running on files tasted as JavaScript (not a great solution), reducing the scope of the ScanUrl regex pattern (also not a great solution), or creating multiple regex patterns that are conditionally called based on file flavor (probably the best solution).

Screenshots
N/A

Server and project version

OS: Ubuntu Bionic
Commit Hash: 755bcef (most recent)

Additional context
N/A

[BUG] HTML/JavaScript recursion

Describe the bug
We've identified a bug in the HTML/JavaScript identification and extraction code. It's possible that libmagic will incorrectly identify a file as "text/html" while YARA will correctly identify a file as "javascript_file". When this happens, the ScanHtml scanner is applied to the JavaScript file and enters a recursive file extraction loop until the maximum depth is hit.

Steps to reproduce
Steps to reproduce the behavior:

Find an HTML file that contains embedded JavaScript that gets tasted as "text/html" by libmagic
Run the file through Strelka
Check for Python logs that describe "exceeded maximum depth" or scan results where the same HTML file is being repeatedly extracted

Expected behavior
JavaScript should not be tasted as HTML.

Screenshots
N/A

Server and project version

OS: Ubuntu Bionic
Commit Hash: N/A (first release)

Additional context
N/A

support multiple directories in dirstream

Is your feature request related to a problem? Please describe.
I have an existing process which has files extracted to multiple directories. I would like all of the directories to be monitored for new files and have them sent to the broker without running multiple instances of dirstream.

Describe the solution you'd like
Allow for multiple directories to be configured in the dirstream config. Ideally, each configured directory could have it's own unique configurations (meta_separator, delete_files, etc). Then monitor as each directory as configured and send files to the broker.

Describe alternatives you've considered
I've considered running multiple dirstream processes with different configurations, one for each directory to monitor.

Additional context
N/A

Build Failure

Describe the bug
When building the current iteration of Strelka, the following exception is thrown:

The command '/bin/sh -c apt-get -qq update &&     apt-get install --no-install-recommends -qq     automake     build-essential     curl     gcc     git     libtool     make     swig     python3-dev     python3-pip     python3-wheel     pkg-config     antiword     libarchive-dev     libfuzzy-dev     libmagic-dev     libssl-dev     libzbar0     python3-setuptools     redis-server     tesseract-ocr     unrar     upx     jq &&     cd /tmp/ &&     curl -OL https://exiftool.org/Image-ExifTool-12.30.tar.gz &&     tar -zxvf Image-ExifTool-12.30.tar.gz &&     cd Image-ExifTool-12.30/ &&     perl Makefile.PL &&     make &&     make install &&     cd /tmp/ &&     curl -OL https://github.com/fireeye/capa/releases/download/v$CAPA_VERSION/capa-linux &&     chmod +x /tmp/capa-linux &&     mkdir /.viv/ &&     chmod -R a+rw /.viv &&     cd /tmp/ &&     curl -OL https://s3.amazonaws.com/build-artifacts.floss.flare.fireeye.com/travis/linux/dist/floss &&     chmod +x /tmp/floss &&     cd /tmp/ &&     curl -OL https://github.com/VirusTotal/yara/archive/v$YARA_VERSION.tar.gz &&     tar -zxvf v$YARA_VERSION.tar.gz &&     cd yara-$YARA_VERSION/ &&     ./bootstrap.sh &&     ./configure --with-crypto --enable-dotnet --enable-magic &&     make && make install && make check &&     cd /tmp/ &&     curl -OL https://github.com/VirusTotal/yara-python/archive/v$YARA_PYTHON_VERSION.tar.gz &&     tar -zxvf v$YARA_PYTHON_VERSION.tar.gz &&     cd yara-python-$YARA_PYTHON_VERSION/ &&     python3 setup.py build --dynamic-linking &&     python3 setup.py install' returned a non-zero code: 2
Service 'backend' failed to build : Build failed

A dependency is causing issues in the build process.

Steps to reproduce

docker-compose -f build/docker-compose.yaml up

Expected behavior
Build without failure

Screenshots
N/A

Release

Release: 0.25.5.17 (Build on 11/28/2021)

Additional context
Add any other context about the problem here.

John The Ripper Inconsistency

Describe the bug
Per @cameron-dunn-sublime:

As each of these changes creates a new build and then gets tested, I've found that the encrypted zip password cracking functionality can behave inconsistently across builds. For example, today I found that the functionality wasn't working (same file, easy password in passwords file) and after rebuilding twice it began working again.

I first noticed this a while ago and figured it was due to grabbing the latest commit on the JohnTheRipper which could be unstable. I now reference a specific commit and still am seeing the inconsistent behavior. The build scripts in JTR seem pretty complex... I suspect JTR only because of the relative complexity but I have not definitively isolated the issue.

Release

Release: 0.21.11.29

Distribution and scanner unit testing

Is your feature request related to a problem? Please describe.
At release the project doesn't have unit tests for file distribution or scanners -- since these are some of the most critical parts of the project, we should have some test coverage for them.

Describe the solution you'd like
Unit tests for file distribution and scanners. These can be gradually introduced.

Describe alternatives you've considered
The current alternative and expectation is that contributors perform testing and and document the tests before submitting PRs, but given the density of the project, it can be difficult to test every critical component.

Additional context
N/A

Add VHD/VHDX Scanner

Is your feature request related to a problem? Please describe.

Threat actors are using VHD/VHDX images in a manner similar to how they use ISO9660 images for email attachment delivery. Strelka has no support for VHD/VHDX.

Describe the solution you'd like

A scanner, similar to ScanIso, that supports VHD/VHDX images, can extract files, and report on image metadata, including timestamp, file system, and volume labels.

Describe alternatives you've considered

N/A

Additional context

There is very little native Python support for VHD/VHDX parsing and extraction, however the Ubuntu package 7zip supports VHD/VHDX.

docker-compose compilation error

Following the QuickStart steps (https://target.github.io/strelka/#/?id=quickstart) and getting an error from docker-compose -f build/docker-compose.yaml up.

Seems like path isn't defined

$ docker-compose -f build/docker-compose.yaml up
Building frontend
Step 1/8 : FROM golang AS build
 ---> 315fc470b445
Step 2/8 : LABEL maintainer "Target Brands, Inc. [email protected]"
 ---> Using cache
 ---> d576b1ec7590
Step 3/8 : COPY ./src/go/ /go/src/github.com/target/strelka/src/go/
 ---> Using cache
 ---> 8111f515c425
Step 4/8 : RUN cd /go/src/github.com/target/strelka/src/go/cmd/strelka-frontend/ &&     go get . &&     CGO_ENABLED=0 go build -o /tmp/strelka-frontend .
 ---> Running in b3da8000ef30
# github.com/target/strelka/src/go/pkg/rpc
../../pkg/rpc/rpc.go:110:24: undefined: path
../../pkg/rpc/rpc.go:112:46: undefined: path
../../pkg/rpc/rpc.go:119:4: undefined: reader
ERROR: Service 'frontend' failed to build: The command '/bin/sh -c cd /go/src/github.com/target/strelka/src/go/cmd/strelka-frontend/ &&     go get . &&     CGO_ENABLED=0 go build -o /tmp/strelka-frontend .' returned a non-zero code: 2

strelka/src/go/pkg/rpc/rpc.go

Line 110 in 9685ce0

f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)

Docker-compose 1.29.2 error

Describe the bug
Using the latest docker-compose 1.29.2 on a fresh Ubuntu or Fedora latest install results in errors when building Strelka Server

Steps to reproduce
Steps to reproduce the behavior:

Run Install docker-compose on either linux distribution following these steps https://docs.docker.com/compose/install/ and the docker packages for the platform selected
installing the server of strelka following these steps https://target.github.io/strelka/#/?id=server-install
Then running docker-compose produces the below errors
docker-compose -f build/docker-compose.yaml up
Traceback (most recent call last):
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1277, in request
File "http/client.py", line 1323, in _send_request
File "http/client.py", line 1272, in endheaders
File "http/client.py", line 1032, in _send_output
File "http/client.py", line 972, in send
File "docker/transport/unixconn.py", line 43, in connect
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "requests/adapters.py", line 449, in send
File "urllib3/connectionpool.py", line 727, in urlopen
File "urllib3/util/retry.py", line 410, in increment
File "urllib3/packages/six.py", line 734, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1277, in request
File "http/client.py", line 1323, in _send_request
File "http/client.py", line 1272, in endheaders
File "http/client.py", line 1032, in _send_output
File "http/client.py", line 972, in send
File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "docker/api/client.py", line 214, in _retrieve_server_version
File "docker/api/daemon.py", line 181, in version
File "docker/utils/decorators.py", line 46, in inner
File "docker/api/client.py", line 237, in _get
File "requests/sessions.py", line 543, in get
File "requests/sessions.py", line 530, in request
File "requests/sessions.py", line 643, in send
File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "docker-compose", line 3, in
File "compose/cli/main.py", line 81, in main
File "compose/cli/main.py", line 200, in perform_command
File "compose/cli/command.py", line 70, in project_from_options
File "compose/cli/command.py", line 153, in get_project
File "compose/cli/docker_client.py", line 43, in get_client
File "compose/cli/docker_client.py", line 170, in docker_client
File "docker/api/client.py", line 197, in init
File "docker/api/client.py", line 222, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
[3647] Failed to execute script docker-compose

Expected behavior
I expect as per previous Strelka builds that it retrieves latest docker templates and associated packages and builds the server components of Strelka

Screenshots
CLI output above is the exact error

Release

1.29.2

Additional context
The way i worked around the problem for now is by removing this version of docker-compose and installing 1.28.6 following the documentation above and the server stack built successfully.

m2crypto support

Is your feature request related to a problem? Please describe.
Currently ScanX509 and ScanPkcs7 use an 'out-of-band' version of the pyopenssl package -- it should use a supported packaged.

Describe the solution you'd like
We should migrate this package to m2crypto (which now supports Python 3.X).

Describe alternatives you've considered
N/A.

Additional context
This will have an impact on Pkcs7 and X509 event fields.

Docker build error

Bug occurs when building using: docker-compose -f build/docker-compose.yaml --project-name strelka up

Steps to reproduce
Steps to reproduce the behavior:

Run docker-compose -f build/docker-compose.yaml --project-name strelka up
Get error:

docker-compose -f build/docker-compose.yaml --project-name strelka up
Building frontend                                                                                                                                                                                              
Step 1/8 : FROM golang AS build                                                                                                                                                                                
 ---> 7ced090ee82e                                                                                                                                                                                             
Step 2/8 : LABEL maintainer "Target Brands, Inc. [email protected]"                                                                                                                                
 ---> Using cache                                                                                                                                                                                              
 ---> 7b9dc4e2e493                                                                                                                                                                                             
Step 3/8 : COPY ./src/go/ /go/src/github.com/target/strelka/src/go/                                                                                                                                            
 ---> Using cache                                                                                                                                                                                              
 ---> 6b281c63b673                                                                                                                                                                                             
Step 4/8 : RUN cd /go/src/github.com/target/strelka/src/go/cmd/strelka-frontend/ &&     go get . &&     CGO_ENABLED=0 go build -o /tmp/strelka-frontend .                                                      
 ---> Running in d8840028999d                                                                                                                                                                                  
# github.com/target/strelka/src/go/cmd/strelka-frontend                                                                                                                                                        
./main.go:82:24: cannot use redis.Z literal (type redis.Z) as type *redis.Z in argument to s.coordinator.client.cmdable.ZAdd                                                                                   
ERROR: Service 'frontend' failed to build: The command '/bin/sh -c cd /go/src/github.com/target/strelka/src/go/cmd/strelka-frontend/ &&     go get . &&     CGO_ENABLED=0 go build -o /tmp/strelka-frontend .' returned a non-zero code: 2

Expected behavior
Expected clean build in docker environment

Release

Commit ID in use: 3b31cd7

Additional context
Host details:
Linux mybuild 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Client:
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        6247962
 Built:             Tue Feb 26 23:52:23 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       6247962
  Built:            Wed Feb 13 00:24:14 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Question - Yara scanning error?

Describe the bug
Hi All.
I'm not sure if I'm being a noob. I followed the install instructions and all is running and scanning fine. I however can't get Yara scanning working as all the files scanned produces the following:

 "yara": {
      "elapsed": 0.032276,
      "flags": [
        "compiling_error"
      ]
    }

I have downloaded the signature base from https://github.com/Neo23x0/signature-base and removed most yara rules to test.
Am I missing something obvious?

Steps to reproduce
Steps to reproduce the behavior:

Default install docker
Map yara rules from Git repo mentioned to /etc/yara/
Run Filestream to send data from Suricata
See error above

Expected behavior
Yara output from scanned files?

Release
Current latest Git clone

.deb/.rpm Package Support

Is your feature request related to a problem? Please describe.
Considering the number of dependencies this project has, it would be beneficial for the project to have stable releases maintained in .deb/.rpm packages. I believe this would also allow for wider adoption of the project being that it would add an ease-of-use / better user experience to get up and running with strelka.

Describe the solution you'd like
RPM support in EPEL, DEB Support in $ubuntu_package_repo_here

Describe alternatives you've considered
There is no alternative, there is only zuul. In all seriousness though, I think the alternative is what we are all doing now which is building the package + dependencies from source.

Additional context
This discussion came up on the project slack channel. It was recommended we place this as an issue for tracking if anyone in the community wants to try and tackle the feature request.

next release

Hi Team,

I've been looking at strelka again and noticed it has changed a lot since I looked at it last time. Before I download and start playing with it, are there any major releases coming up?

Great work! TIA

scan_plist Invalid Token Exception

Describe the bug
Upon running fileshot on OSX, receiving an error for scan_plist while scanning plist files.

Expected behavior
No exceptions, hopefully.

Screenshots

backend_1      | Traceback (most recent call last):
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/strelka.py", line 148, in scan_wrapper
backend_1      |     self.scan(data, file, options, expire_at)
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/scanners/scan_plist.py", line 13, in scan
backend_1      |     plist = plistlib.loads(data)
backend_1      |   File "/usr/lib/python3.6/plistlib.py", line 1024, in loads
backend_1      |     fp, fmt=fmt, use_builtin_types=use_builtin_types, dict_type=dict_type)
backend_1      |   File "/usr/lib/python3.6/plistlib.py", line 1015, in load
backend_1      |     return p.parse(fp)
backend_1      |   File "/usr/lib/python3.6/plistlib.py", line 325, in parse
backend_1      |     self.parser.ParseFile(fileobj)
backend_1      | xml.parsers.expat.ExpatError: not well-formed (invalid token): line 2, column 24

Release

Release: 0.19.07.26

Additional context
N/A

Add Python Supports for CAPA Scanner

Is your feature request related to a problem? Please describe.
The CAPA scanner did not originally support Python3 on release and required parsing of the results. CAPA has been updated with Python3 support.

Describe the solution you'd like
Port CAPA scanner to Python3.

Describe alternatives you've considered
N/A

Additional context
N/A

Go client

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
It'd be nice to see a working Go client. I'm not sure what the state of ZMQ support in Go is, but that should be the only technical roadblock.

Describe alternatives you've considered
N/A

Additional context
N/A

Explore pdfminer replacement

pdfminer is very slow -- we should explore other PDF parsing packages that provide similar capabilities.

docker image taste path mismatch

Describe the bug
When building a dev docker image, the docker container will create /etc/strelka/taste.yara instead of the /etc/strelka/**taste/**taste.yara path that is referenced in the strelka.yml file (and matches the project repo folder structure). This results in a "No such file or directory" error when submitting new files to the server for processing.

Steps to reproduce

git clone https://github.com/target/strelka /opt/strelka
cd /opt/strelka && docker build -t strelka-dev .
docker run --rm -v /tmp/sample_files:/tmp/sample_files strelka-dev strelka.py
run 'docker stats' to get $containerid
docker exec $containerid strelka_user_client.py --broker 127.0.0.1:5558 --path /tmp/sample_files/file0001
Error message will be present as an [ERROR] event on the screen where the container launched in step 3 is running.

Expected behavior
Files should be tasted and assigned to scanners properly. When bug is present, tasting fails, no worker is assigned by the broker, no scanning occurs.

Screenshots
N/A

Server and project version
OS: Xubuntu 18.04
Commit hash: N/A (current master as of Feb 25, 2019)

Additional context
N/A

Strelka unittest files repo

Is your feature request related to a problem? Please describe.
After writing a few more test files, I've realized it's much easier to host files in a repository rather than pulling them down in each test. I've created a repo (strelka_test_files) and want your opinion on moving forward with it.

Describe the solution you'd like
If the repository of test files is fine with you, I'd like to transfer/recreate it under the Target origanization just to keep everything in line with the project. I've refactored the PE scanner test to use this repository, and written a few more scanner tests that use it as well, which you can find on my org's page

Describe alternatives you've considered
Currently, the PE scanner test dynamically pulls a specific version of Putty down to perform tests on, but:

the added network call potentially makes the tests unacceptably slow depending on network speed - this will only compound with more tests regardless of network speeds
the Putty link is not guaranteed to be up forever (whereas I'm assuming if someone can get to the strelka GitHub repo they can get to a strelka_test repo)
it has been difficult to find test files hosted on the internet with permanent links and for all the needed test scenarios

Additional context
N/A

filestream staging directory created with 0755 permissions

Is your feature request related to a problem? Please describe.

The filestream staging directory allows r-x for group and public users.

strelka/src/go/cmd/strelka-filestream/main.go

Line 118 in 8b7e163

os.Mkdir(staging, 0755)

Describe the solution you'd like
Can this be set to 0600 without causing issues (least privilege)?

Invoke scanners based on yaraMetadata matches

Is your feature request related to a problem? Please describe.
Not related to a problem

Describe the solution you'd like
I'd like a method for invoking scanners if custom YARA rules are matched. For example, if we have a rule that matches PE files with specific version info, and we want to send those samples to a malware sandbox.

I think adding a yara entry to the scanners config next to flavors would be a good way to control which rules trigger this behavior.

Describe alternatives you've considered
Querying our logs for all records matching xx YARA rule in yy timeframe, but this does not scale well at all.

Additional context
N/A

Exception - ScanPe - PEFormatError - Invalid NT Headers signature.

Describe the bug
The PE parsing library we use expects there to be a 0x4D5A file header.
If the "DOS magic header" is missing then the library fails out. - @bblenard

Steps to reproduce
Configured fileshot to scan 'text/plain' flavored files and very the output in the strelka log shows ScanPE on a text file. -@bblenard

Expected behavior
An exception will be thrown on the backend and the scanner will not complete.

Screenshots

ScanPe: exception while scanning uid fc32252a-c773-4cb3-b286-6856172b3eea (see traceback below)
backend_1      | Traceback (most recent call last):
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/strelka.py", line 148, in scan_wrapper
backend_1      |     self.scan(data, file, options, expire_at)
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/strelka-0.0.0-py3.6.egg/strelka/scanners/scan_pe.py", line 214, in scan
backend_1      |     pe = pefile.PE(data=data)
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/pefile.py", line 1754, in _init_
backend_1      |     self._parse_(name, data, fast_load)
backend_1      |   File "/usr/local/lib/python3.6/dist-packages/pefile.py", line 1872, in _parse_
backend_1      |     raise PEFormatError('Invalid NT Headers signature. Probably a LE file')
backend_1      | pefile.PEFormatError: 'Invalid NT Headers signature. Probably a LE file'

Release

Release: 0.19.08.27

Additional context
This should be handled with an exception flag rather than a change to how data is scanned. No data modification should occur.

target / strelka Goto Github PK

strelka's People

Contributors

Stargazers

Watchers

Forkers

strelka's Issues

Recommend Projects

Recommend Topics

Recommend Org