Coder Social home page Coder Social logo

sandialabs / scot Goto Github PK

View Code? Open in Web Editor NEW
242.0 38.0 48.0 548.99 MB

Sandia Cyber Omni Tracker (SCOT)

Home Page: http://getscot.sandia.gov

License: Other

Perl 30.60% Shell 2.43% JavaScript 45.76% CSS 4.09% Dockerfile 0.18% Raku 2.74% SCSS 0.22% HTML 13.74% Less 0.26%
perl javascript cyber-security incident-response threat-intelligence applications snl-cyber-sec cybersecurity snl-applications scr-1627

scot's Introduction

ScotLogo

Sandia Cyber Omni Tracker

version Documentation Status license pipeline status

SCOT Online Demo

Give SCOT a try: https://scotdemo.com with the username: admin/ password: admin

For this demo, the app is reverted every hour on the hour.

SCOT Mailing List

Send e-mail to [email protected] with body of "subscribe scot-users"

Documentation and Install

Read our documentation at Read The Docs.

Also, for docker documentation, please read: https://github.com/sandialabs/scot/blob/scot-docker/docs/source/scotdocker.rst

UPGRADE NOTE

IMPORTANT: Read Issue #55 before upgrading from 3.5.5.

Install It!

** RPM Based Installer (CENTOS 7) **

  • clone scot repo or download the files

    • scot.perl.rpm.install.tar.gz
    • scot.rpm.install.tar.gz
  • extract both files:

    • tar xzvf scot*tar.gz
  • install scot perl first

    • cd scot-perl-install
    • ./install.sh
    • follow instructions presented at end of install
  • install scot second

    • cd scot-install
    • ./install.sh

Docker Method

The suggested method for installation of SCOT is using docker. For a walkthrough of installing SCOT via docker, please read: https://github.com/sandialabs/scot/blob/scot-docker/docs/source/scotdocker.rst

Once you have read the documentation, please run the restart-build-deploy.sh script in the root directory of the scot source code to begin the installation process.

Legacy Method

SCOT has an installer that will work with Ubuntu 16.04 and CentOS 7.3. While there should be nothing preventing SCOT to work on other Linux's, you will have to do a lot more work by hand to get SCOT installed. (Please post your successes for others)

To install on Ubuntu 16.04 or CentOS 7:

  • clone the SCOT repo:

    git clone https://github.com/sandialabs/scot.git

  • be sure to set http_proxy environment variables if necessary

  • run the installer as root:

    • $ sudo -E bash
    • cd SCOT
    • ./install.sh 2>&1 | tee ../scot.install.log

The install can take a while. The installer output is very verbose and saving it to a log is recommended to help with debugging any issues that might arise.

Overview

The Sandia Cyber Omni Tracker (SCOT) is a cyber security incident response management system and knowledge base. Designed by cyber security incident responders, SCOT provides a new approach to manage security alerts, analyze data for deeper patterns, coordinate team efforts, and capture team knowledge. SCOT integrates with existing security applications to provide a consistent, easy to use interface that enhances analyst effectiveness.

Scot-flow

Customer Need

Incident response (IR) teams utilize many systems to detect, collect and analyze cyber security event data. These systems, while solving pieces of the puzzle, often fail to give the analyst a holistic view of what is happening and their team’s response to those events. Many systems do not have the flexibility to work with the IR processes to research and document those activities. Research is not easily shared and searchable, so the team’s effectiveness decreases, especially when key personnel are on vacation or take other positions. Without a ready corpus of examples of past events, training new team members becomes a lengthy process. Each additional tool adds cognitive load to the analyst and the tool’s maintenance needs take the analyst away from the primary task of IR.

Our Approach

Focused on removing the friction between analysts and their tools, SCOT enables analysts to document and share their research and response efforts. As a software suite that integrates data from detectors, analysis, and other information sources, it provides real time updates of the team’s work to keep the team informed and coordinated. SCOT automatically identifies indicators to help the analyst discover and respond to advanced threats. Centralization of the data reduces the contextual shifts necessary to access each detection system. Fusing detection data with the accumulated team knowledge allows the team to quickly discover that a new alert might be part of a larger campaign. In addition, SCOT automates and simplifies common analyst tasks to increase analyst’s effectiveness by freeing them to concentrate on cyber security – not tool mastery.

Benefits

The number of alerts Sandia’s IR team has seen has nearly doubled in the past several years. SCOT enabled the team to keep up with this increase without adding additional team members. As a training tool, new team members started contributing in weeks, instead of months. In just over 4 years SCOT has amassed a database of over 700K indicators from analyst and alert input. These indicators help the team spot an adversary’s methods and tactics, as well as highlighting common targets within the enterprise. SCOT, processed over 1.6 million alerts since deployment, while maintaining 99.9% availability, and required minimal administration. SCOT is fully scalable to meet higher loads.

Competitive Advantage

Sandia's incident response team realized several advantages using SCOT over other solutions. SCOT's ease of use eliminated the steep learning curve of traditional SIEMS and captured team knowledge much more effectively. Designed for cyber security, SCOT allows the IR team to enter data easily, instead of struggling to conform to a ticketing system designed for other purposes. While workflow systems handle linear workflows easily, SCOT is purpose built for the looping nature of cyber security investigations. SCOT also solves the challenges of keeping wikis, spreadsheets and documents up-to-date and accessible to an IR team. While top-notch analysts may be able to keep everything in their brains, SCOT will capture their knowledge for when they go on vacation or to other employment.

Scot-venn

Contact US

News and Discussion: mailto:[email protected]

Vulnerabilities: mailto:[email protected]

Bugs/Feature Requests: Use our GitHub issue tracking

Collaboration: mailto:[email protected]

scot's People

Contributors

ahoying avatar angeorgieff avatar brymon68 avatar erikjlee avatar jecompton avatar kf4x avatar sandywater avatar toddbruner avatar umidevsec avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

scot's Issues

Need Log-off capability

Depending on browser configuration, logging off as one user and logging on as another can be daunting. The problem I think has to do with Single Sign-On capabilities.

Anyway, a "Sign off" button would be an awesome addition.

Several UI refresh bugs

Several different strange behaviours are described below.

On Windows, with IE 10, Chrome 45.0.2454, FF 40 -- When selecting alerts from the alert list, the detail pane at the bottom is always empty (for some reason nothing loads down there at all under windows). When I click the status button for an event, the button text doesn't change but it keeps accumulating small X's on its right side (not sure how else to describe that).

On Ubuntu, With Firefox 37--

The lower alert pane works here, unlike on Windows. However, certain kinds of changes do not cause a refresh of the UI--instead the whole page needs to be refreshed for the UI to update.

For example, when I click the grey button inside the "status" box for an event, it changes it from open to close logically (I can only verify this by refreshing the whole page), but the button itself doesn't show this change, and the status in the upper list doesn't change (until after refresh).

Or, also for an alert, when I delete an item in the lower detail pane using the trash can icon, I confirm the deletion, but nothing changes on the UI. When I refresh, that item disappears.

Another weird thing here is that even after I delete all the items in the lower pane, the corresponding event remains in the list of events, even after refreshing the page (maybe this part is by design?)

Edit: FYI the SCOT server is running on CentOS release 6.7, rather than Ubuntu. But I wouldn't expect that should impact the way the UI behaves...

Inclusion of Screenshots Would be Helpful

It would be extremely helpful to be able to attach screenshots, preferably via drag and drop or paste from clipboard to entries. This will allow more thorough documentation of incidents. For example, I may wish to include a screenshot of the alert or graph which initiated the event.

Please clarify the privileges needed for (remote) MongoDB access?

Hello, I deployed a remote MongoDB instance (maintained by a 3rd party) and I am running into this:

2016/02/22 11:40:31 [12155]        Mongo.pm: 776 mongodb://<MongoDBHostname:27021> Reading documents matching $VAR1 = {'collection' => 'users','match_ref' => {'username' => 'admin'},'all' => 1};
2016/02/22 11:40:31 [12155] DefaultHelpers.pm:   90 MongoDB::DatabaseError: not authorized on scotng-prod to execute command { count: "users", query: { username: "admin" } }

(It does access a local MongoDB instance OK.)

Can you please clarify the exact privileges needed on MongoDB to run SCOT (or point to any documentation that can clarify this?)

Thank you

build failed on AArch64, Fedora 33

[root@cn05 scot]# ./install.sh 2>&1 | tee ../scot.install.log
...
=========== installing APACHE =============

--- Installing Apache2

--
-- CENT/RH based system install of apache2

Last metadata expiration check: 2:51:35 ago on Thu 25 Mar 2021 11:28:19 AM CET.
Package httpd-2.4.46-9.fc33.aarch64 is already installed.
Package mod_ssl-1:2.4.46-9.fc33.aarch64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
-- adding firewalld command to allow web traffic
FirewallD is not running
FirewallD is not running
FirewallD is not running

-- configuring apache2

-- DEVDIR is /data/jw/scot
-- CSD is /data/jw/scot/install/src/apache2

  • looking for /data/jw/scot/../Scot-Internal-Modules/etc/scot-revproxy.conf
  • looking for /data/jw/scot/install/src/apache2/scot-revproxy-Fedora.conf
    FAILED to FIND revproxy config!
    Installation can not proceed until fixed!
    [root@cn05 scot]#

'firewalld' is yum installed.

LDAP: Error Found 'Test User' but this user has no groups...

This error occurs with OpenLDAP 2.4.39 server and SCOT 3.4.0 - Hindenberg. Ubuntu 14.04.2 LTS.

Error:Found 'Test User', but this user has no groups. Please check your 'User ID Attribute' and 'Membership Attr'

What's are correct parameters on the User Authentication page for LDAPS for OpenLDAP 2.4?

Does installing SCOT from source work? Installation on Ubuntu, CentOS, RedHat all seem to have issues.

I was wondering if anyone has had any success in installing SCOT from source recently (2016)?

I have followed the instructions, here: http://scot.readthedocs.org/en/latest/install.html , and done fresh installs on Ubuntu Server 14.04, CentOS 6.7, RedHat 6.7 using the specific Bash installers provided ... all with similar outcomes: Perl errors and the system not believing that the SCOT server is running (for example, when you try to test IMAP authentication, etc).

Specifically:

2016/02/17 21:18:44 [27804]         Scot.pm:  104 commandline was not set during initialization at /usr/share/perl5/Net/Server.pm line 103, <DATA> line 231.
2016/02/17 21:18:44 [27804]         Scot.pm:  105  at /usr/share/perl5/Net/Server.pm line 103, <DATA> line 231.
        Net::Server::commandline('Starman::Server=HASH(0x8f3a6d8)') called at /usr/share/perl5/Net/Server.pm line 92
        eval {...} called at /usr/share/perl5/Net/Server.pm line 92
        Net::Server::_initialize('Starman::Server=HASH(0x8f3a6d8)') called at /usr/share/perl5/Net/Server.pm line 50
        Net::Server::run('Starman::Server=HASH(0x8f3a6d8)', 'port', 'ARRAY(0x91e4008)', 'host', '*', 'proto', 'tcp', 'serialize', 'none', ...) called at /usr/share/perl5/Starman/Server.pm line 84
        Starman::Server::run('Starman::Server=HASH(0x8f3a6d8)', 'CODE(0x8f4b9a8)', 'HASH(0x8f4ba50)') called at /usr/share/perl5/Plack/Handler/Starman.pm line 18
        Plack::Handler::Starman::run('Plack::Handler::Starman=HASH(0x8f4bab0)', 'CODE(0x8f4b9a8)') called at /usr/share/perl5/Plack/Loader.pm line 84
        Plack::Loader::run('Plack::Loader=HASH(0x126def0)', 'Plack::Handler::Starman=HASH(0x8f4bab0)') called at /usr/share/perl5/Plack/Runner.pm line 277
        Plack::Runner::run('Plack::Runner=HASH(0x1019910)') called at /usr/bin/plackup line 10

I would appreciate any feedback or guidance.

Thank you

Promoting an HTML alert causes HTML entities to get munged

When an alert that contains HTML elements, like a forwarded email message, is promoted to an event, the html entities get encoded, which makes the resulting entry in the new event unreadable. The issue is caused by the make_data_row method in lib/Scot/Model/Alert.pm. It uses encode_entities on all data, which is what munges the HTML elements. It would be better to strip unwanted HTML tags using either HTML::Restrict or HTML::TagFilter and pass the filtered HTML through to the new event entry. Let me know if you'd like me to mock up and submit a patch for this.

Tags don't show when entered via alert_tool.pl

"Tags" do not show up in the SCOT UI when they are entered via alert_tool.pl

To reproduce:

  1. Create a tag manually in the SCOT UI, for example under "Intel":
https://scot/#/intel/1
  • enter "Manual" in Tags field, without quotes
  • Verify that you can see "Manual" in Tags selector on the various SCOT screens
  1. Create a tag programmatically by using alert_tool.pl
./alert_tool.pl
  • enter "Code" in Tags field, without quotes
  • Dumper will confirm POST data
  • You will not find/see "Code" tag anywhere in the SCOT UI

Install script ends abruptly

Brand new instance of Ubuntu 16.04 LTS, continue to get the following error:

Proxy is not required for internet access

root@Scot:/scot# ./install.sh 2>&1 | tee scot.install.log
==================== SCOT 3.5 Installer =======================
DEVDIR is /scot
Running as root: Yes

http_proxy  =
!!! http_proxy not set! if you are behind a proxy, install will
!!! likely fail.  If you are using "sudo" to install use the
!!! "-E" option to preserve your environment variables
root@Scot:/scot#

Unable to log in to SCOT - bad CSRF token

I resolved a number of issues to get the containers up and SCOT listening, but I am now stumped with getting authenticated. I get the error "Failed to log in due to bad CSRF token. Please reload the page and then log in. Error: Failed CSRF check.

The logs show:
POST "/auth"
Template "bad_csrf.html.ep" not found
403 Forbidden (0.001077s, 928.505/s)

I've used admin/admin (building the demo site). That didn't work. I changed the password and set the hash in Mongo, that didn't work either.

Here's the SCOT startup log:

Routing to a callback
304 Not Modified (0.007052s, 141.804/s)
POST "/scotaq/amq"
Template "not_found.development.html.ep" not found
Template "not_found.html.ep" not found
Rendering template "mojo/debug.html.ep"
GET "/scot/api/v2/whoami"
GET "/scot/api/v2/status"
GET "/scot/api/v2/handler"
Template "mojo/menubar.html.ep" not found
Routing to controller "Scot::Controller::Auth" and action "check"
401 Unauthorized (0.040213s, 24.868/s)
404 Not Found (0.061014s, 16.390/s)

Thinking this is a problem with the mojo CSRF plugin, and/or missing code in the headers but I'm not sure where to modify those.

Help would be appreciated!

RPM spec

Are you folks open to packaging this as an RPM? I've been tasked with installing this on our systems and we try to rely on RPMs in general. I'll be putting together a spec and would be happy to submit it back to you folks, though frankly it looks to be a daunting task given the number of dependencies.

Docker-Perl does not install

Installing scot from master branch, under Ubuntu 16.04, using latest docker-ce, perl did not build properly. Error during the 'cpanm' call was:
Configuring IO-AIO-4.6 ... ! Configure failed for IO-AIO-4.6. See /root/.cpanm/work/1536269103.6/build.log for details.

This is caused by not including libperl-dev in the docker image. Including the package in the Dockerfile-Perl will allow for IO-AIO to build properly and for the docker image to be created.

Ubuntu Installation Script

All the references to the mongod service and logs should be updated to mongodb. Also the script should start off by installing the package build-essentials.

Regards
Andy Dove

Mongo attempts to connect on port 27017 even if you specify a different port

OS: Ubuntu Server 14.04

If you modify /opt/sandia/webapps/scot3/scot.conf to utilize an alternative port for Mongo (let's say port 27021 for this example), even though SCOT acknowledges the alternative port, SCOT still tries to connect to the MongoDB using port 27017.

Edit: Here is the config snippet from scot.conf:

                    'database' => {
                                    'write_safety' => 1,
                                    'port' => 27021,
                                    'db_name' => 'scotng-prod',
                                    'host' => 'mongodb://<YourMongoDBHostname>',
                                    'user' => 'scot',
                                    'pass' => 'password'
                                  }

Here is the output from /var/log/scot.prod.log (I added ** for emphasis, since I can't add bolding inside code tags)


2016/02/19 16:13:13 [2462]        Mongo.pm:  676 Retrieving one document
2016/02/19 16:13:13 [2462]        Mongo.pm:   95 ---- Mongo client connection build ---
2016/02/19 16:13:13 [2462]        Mongo.pm:   96 ---- host:    mongodb://<YourMongoDBHostname>
2016/02/19 16:13:13 [2462]        Mongo.pm:   97 ---- port:    **27021**
2016/02/19 16:13:13 [2462]        Mongo.pm:   98 ---- name:    scotng-prod
2016/02/19 16:13:13 [2462]        Mongo.pm:   99 ---- user:    scot
2016/02/19 16:13:13 [2462]        Mongo.pm:  776 mongodb://<YourMongoDBHostname> Reading documents matching $VAR1 = {'collection' => 'users','all' => 1,'match_ref' => {'username' => 'admin'}};
2016/02/19 16:13:52 [2432] DefaultHelpers.pm:   90 MongoDB::SelectionError: No readable server available for matching read preference primary. MongoDB server status:
Topology type: Single; Member status:
  <YourMongoDBHostname>:**27017** (type: Unknown, error: MongoDB::NetworkError: Could not connect to '<YourMongoDBHostname>:**27017**': Connection timed out )
2016/02/19 16:13:52 [2432]  EPLRenderer.pm:   50 Template "exception.development.html.ep" not found
2016/02/19 16:13:52 [2432]  EPLRenderer.pm:   50 Template "exception.html.ep" not found
2016/02/19 16:13:52 [2432]  EPLRenderer.pm:   38 Rendering template "mojo/debug.html.ep"
2016/02/19 16:13:52 [2432]  EPLRenderer.pm:   38 Rendering template "mojo/menubar.html.ep"
2016/02/19 16:13:58 [2445] DefaultHelpers.pm:   90 MongoDB::SelectionError: No readable server available for matching read preference primary. MongoDB server status:
Topology type: Single; Member status:
  <YourMongoDBHostname>:**27017** (type: Unknown, error: MongoDB::NetworkError: Could not connect to '<YourMongoDBHostname>:**27017**': Connection timed out )

Email alert script alertbot.pl broken

In Docker, alertbot.pl doesn't work.

In troubleshooting I found several issues. First, Sys::RunAlone is missing from the required perl modules.

Once that is installed, lib/Scot/Bot/ForkAlerts.pm references Scot::Util::Phantom which does not exist, commenting that line out fixed that issue.

Then I got this error:

$ ./alertbot.pl
Attribute (env) is required at constructor Scot::Util::Imap::new (defined at ../lib/Scot/Util/Imap.pm line 426) line 40.
Scot::Util::Imap::new('Scot::Util::Imap', 'config', 'HASH(0x934f4c0)', 'log', 'Log::Log4perl::Logger=HASH(0xf0e8a8)', 'email_acct', undef, 'mongo', 'Scot::Util::Mongo=HASH(0x931d4b0)') called at ../lib/Scot/Env.pm line 334
Scot::Env::_build_imap('Scot::Env=HASH(0xf0ea10)') called at reader Scot::Env::imap (defined at ../lib/Scot/Env.pm line 319) line 7
Scot::Env::imap('Scot::Env=HASH(0xf0ea10)') called at ../lib/Scot/Bot/ForkAlerts.pm line 54
Scot::Bot::ForkAlerts::_get_imap('Scot::Bot::ForkAlerts=HASH(0x9343638)') called at reader Scot::Bot::ForkAlerts::imap (defined at ../lib/Scot/Bot/ForkAlerts.pm line 44) line 7
Scot::Bot::ForkAlerts::imap('Scot::Bot::ForkAlerts=HASH(0x9343638)') called at ../lib/Scot/Bot/ForkAlerts.pm line 61
Scot::Bot::ForkAlerts::run('Scot::Bot::ForkAlerts=HASH(0x9343638)', 'HASH(0x8521bd0)') called at ./alertbot.pl line 173

To fix that I updated Env.pm and changed the Scot::Util::Imap->new call to this:

my $imap    = Scot::Util::Imap->new(
    config      => $config->{$mode},
    log         => $self->log,
    email_acct  => $account,
    mongo       => $mongo,
    env         => $self,
);

After that the script started working fine and I can now get new alerts via email.

IMAP: Having issues authenticating with Outlook/Office365

I am testing IMAP against Outlook/Office365 using /opt/sandia/webapps/scot3/bin/alertbot.pl but not having success. I would appreciate any feedback or suggestions.

Result:

$ cd /opt/sandia/webapps/scot3/bin/
$ ./alertbot.pl
Attribute (imap_client) does not pass the type constraint because: Validation failed for 'Mail::IMAPClient' with value undef at accessor Scot::Util::Imap::imap_client (defined at ../lib/Scot/Util/Imap.pm line 53) line 21
        Scot::Util::Imap::imap_client('Scot::Util::Imap=HASH(0x9561f30)') called at ../lib/Scot/Util/Imap.pm line 111
        Scot::Util::Imap::get_messages_aref('Scot::Util::Imap=HASH(0x9561f30)', 'HASH(0x95309d0)') called at ../lib/Scot/Bot/ForkAlerts.pm line 67
        Scot::Bot::ForkAlerts::run('Scot::Bot::ForkAlerts=HASH(0x9540248)', 'HASH(0x95309d0)') called at alertbot.pl line 173

SCOT log (/var/log/scot.prod.log):

2016/02/25 08:55:50 [15302]         Imap.pm:   73 ++++ Building IMAP connection
2016/02/25 08:55:50 [15302]         Imap.pm:   74 ++++ OPTS  : $VAR1 = 'Server';
$VAR2 = 'outlook.office365.com';
$VAR3 = 'Port';
$VAR4 = 993;
$VAR5 = 'User';
$VAR6 = 'XXXXXXXXXXXXXXXX';
$VAR7 = 'Password';
$VAR8 = 'YYYYYYYYYYYYYYYY';
$VAR9 = 'Ssl';
$VAR10 = [
           'SSL_verify_mode',
           'SSL_VERIFY_NONE'
         ];
$VAR11 = 'Uid';
$VAR12 = 1;
$VAR13 = 'Ignoresizeerrors';
$VAR14 = 1;
$VAR15 = 'SSL_verify_mode';
$VAR16 = 'SSL_VERIFY_NONE';

Config (/opt/sandia/webapps/scot3/scot.conf):

'production' => {
                    'email_accounts' => {
                                          'XXXXXXXXXXXXXXXX' => 'YYYYYYYYYYYYYYYY',
                                          'scot-alerts-test' => 'password',
                                          'scot-alerts' => 'password'
                                        },
                    'version' => '3.3',
                    'imap' => {
                                'ignoresizeerrors' => 1,
                                'username' => 'XXXXXXXXXXXXXXXX',
                                'port' => 993,
                                'uid' => 1,
                                'hostname' => 'outlook.office365.com',
                                'ssl' => [
                                            'SSL_verify_mode',
                                            'SSL_VERIFY_NONE'
                                          ],
                              },

Issues with Scot::Env module

Anyone have any idea how to fix this issues with backup.pl and alert.pl? I installed SCOT 3.5.2 from the repository on Red Hat 7 with SE Linux.

Here are the errors:

[root@plrtir01 ~]# /opt/scot/bin/backup.pl
Can't locate Scot/Env.pm in @inc (you may need to install the Scot::Env module) (@inc contains: ../lib /opt/perl5/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux /opt/perl5/perls/perl-5.18.2/lib/site_perl/5.18.2 /opt/perl5/perls/perl-5.18.2/lib/5.18.2/x86_64-linux /opt/perl5/perls/perl-5.18.2/lib/5.18.2 .) at /opt/scot/bin/backup.pl line 9.
BEGIN failed--compilation aborted at /opt/scot/bin/backup.pl line 9.

[root@plrtir01 ~]# /opt/scot/bin/alert.pl
--- Starting Mail Ingester ---
Can't locate object method "log_config" via package "Scot::Env" at /opt/scot/lib/Scot/Env.pm line 113.
[root@plrtir01 ~]#

And the e-mail message:

X-Cron-Env: <XDG_SESSION_ID=1131>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: [email protected]
Date: Mon, 5 Jun 2017 11:50:02 -0300 (-03)

Can't locate object method "log_config" via package "Scot::Env" at /opt/scot/lib/Scot/Env.pm line 113.
--- Starting Mail Ingester ---

Thanks!

/var/log/mongo directory missing

Building scot from 'master' under Ubuntu 16.04 using latest docker-ce. Upon startup of the mongodb docker image the database does not respond, and will loop forever with:
mongodb | => Waiting for confirmation of MongoDB service startup....

This is caused by a missing directory: /var/log/mongodb

Ensuring the directory is created in Dockerfile-Mongodb solves this issue.

Multiple questions

Hello,

I have several questions, feel free to ask me if some of them are not clear. If some question are, in fine, a bug, just tell me and I will create a dedicated issue for that if you want.
For most of these question : either I didn't find a clear answer in the documentation (note that English is not my birth language :) ), or I did'nt find relative parameters in the Admin section of the demo website.

  1. Is it possible to connect a SIEM ? (which could be able to deliver json or xml datas)
  2. Does SCOT support 802.1X authentication ?
  3. Is SCOT able to send mail to users (if they are assigned to an event, incident, task... If something happens to an event they are assigned)
  4. How can we change the incident template (custom fields, custom list of values) ?
  5. [Bug?] On the demo site, if I "Make Task" on an entry, the new task does not appear in the "More > Task" list
  6. [Bug?] I can't create a new group on the demo website. And in? the Admin section, I don't see groups used in the permissions tab (in some events/incidents, for example : wg-scot-ir, wg-scot-researchers).
  7. How can we create personalized reports ?
  8. I am not sure to understand the difference between Alert and Intel section :/

Thank you very much for all informations you can give me.
Regards.

SCOT Alert API error. Can't POST JSON data

I'm trying to use the alert API because I'd like to forward Bro alert data to SCOT. It comes down to using curl, but I have been unsuccessful in getting it to complete against my local vm. So, I decided to try to go against the demo site and got the same error.

It seems there's maybe a mismatch in the version of the perl module Log::Log4perl::Logger? There's a missing history method.

Here's the full error (I've formatted the JSON to be easier to read):

$ curl -k -g -X POST -m60 -d '{
>   "subject": "bad things happened",
>   "sources": ["bro"],
>   "data": {
>     "msg": "bad things happened",
>     "note": "SCOT::JSON_Alert",
>     "dropped": "F",
>     "ts": "2016-01-17T10:04:12.465551Z",
>     "actions": [
>       "SCOT::ACTION_LOG",
>       "Notice::ACTION_LOG"
>     ],
>     "sub": "https://192.168.168.5/discover?q=Cfjwiwelskdjfie8123",
>     "peer_descr": "bro",
>     "suppress_for": 3600.0
>   }
> }' "https://admin:[email protected]/scot/alertgroup"
The application raised the following error:

  Can't locate object method "history" via package "Log::Log4perl::Logger" at inline template fd403ab55a4c875e35b42428816134c7 line 311.
306:             %= $kv->(Time => scalar localtime(time))
307:           </table>
308:         </div>
309:         <div class="tap">tap for more</div>
310:       </div>
311:       % if (@{app->log->history}) {
312:         <div id="log" class="box infobox spaced">
313:           <table>
314:             % for my $msg (@{app->log->history}) {
315:               <tr>
316:                 <td class="striped value wide">

and the StackTrace middleware couldn't catch its stack trace, possibly because your application overrides $SIG{__DIE__} by itself, preventing the middleware from working correctly. Remove the offending code or module that does it: known examples are CGI::Carp and Carp::Always.

mongodb errors on first-time login

I just installed scot, but I'm having problems trying to login for the first time. Initially I faced LDAP auth issues, and I disabled LDAP per #47. After restarting the server, I'm now getting MongoDB errors when trying to login. Trying to login via localhost/login redirects me to localhost/auth and yields the following error message
find_one error: MongoDB SelectionError: No readable server available for matching read preference primary. MongoDB server status: Topology type: Singe; Member status: 127.0.0.1:27017 (type Unknown, error: MongoDB::NetworkError: Could not connect to '127.0.0.1:27017': Connection Refused) at usr/local/share/perl/5.26.1/Meerkat/Collection.pm line 217

Do I need to manually start the MongoDB instance or is it something else? Any help would be greatly appreciated

Question about Scot.

Just out of curiosity are you able to import Screenshots or Documents and or Pictures in the SCOT Event or Task features. I think that would be a great feature

Redis & Mongo Entering FATAL State

I know that Docker is in Beta, but I'm having an issue with Redis and Mongo.

2016-01-11 04:01:01,800 INFO exited: redis (exit status 0; not expected)
2016-01-11 04:01:01,803 CRIT reaped unknown pid 112)
2016-01-11 04:01:01,857 INFO exited: mongo (exit status 100; not expected)
2016-01-11 04:01:03,861 INFO spawned: 'mongo' with pid 116
2016-01-11 04:01:03,862 INFO spawned: 'redis' with pid 117
2016-01-11 04:01:03,904 INFO exited: redis (exit status 0; not expected)
2016-01-11 04:01:03,907 CRIT reaped unknown pid 118)
2016-01-11 04:01:03,993 INFO exited: mongo (exit status 100; not expected)
2016-01-11 04:01:06,915 INFO spawned: 'redis' with pid 153
2016-01-11 04:01:06,923 INFO exited: redis (exit status 0; not expected)
2016-01-11 04:01:06,924 INFO gave up: redis entered FATAL state, too many start retries too quickly
2016-01-11 04:01:06,925 CRIT reaped unknown pid 154)
2016-01-11 04:01:07,197 INFO spawned: 'mongo' with pid 155
2016-01-11 04:01:07,297 INFO exited: mongo (exit status 100; not expected)
2016-01-11 04:01:07,379 INFO gave up: mongo entered FATAL state, too many start retries too quickly

I looked through the Dockerfile and I see that supervisor is installed, but it appears to be missing from the ubuntu_installer.sh file that the Dockerfile calls. I don't know a lot about Supervisor, but it is supposed to be a process manager, so maybe it could be the culprit?

Intel backend support

I don't know if this is reasonable, but it'd be really helpful if the intel portion could reach into a backend like CRITS via API. SCOT is a great start as an IR platform, but for larger uses, it's ideal to ingest a bunch of data into a better suited system like CRITS. I don't know how the current logic works, per se, but CRITS has a pretty extensive API. It also uses standard CybOX models for indicators.

Value to CRITS user: Allows integration with higher fidelity intel and integrate incident response team with threat intelligence team

How affects non-CRITS users: Doesn't affect at all. Existing simple intel function remains as-is.

scratchpad.html isn't working

With the docker install, scratchpad.html doesn't do anything. In troubleshooting I've found that the Apache configuration is reverse proxying /flair to http://127.0.0.1:8080, but nothing is listening on that port. I'm not sure what configuration is needed to resolve the issue.

Authentication failure after installation

Hello,
I'm getting an authentication failure message (mixed with HTML and CSS code - see the screenshot) on the most vanilla installation of SCOT on Ubuntu 16.04 64 bit (running on top of VMware). This is the full install, not the Docker container. I was getting the same message on some existing Ubuntu servers so I've decided to install a fresh Ubuntu, with nothing on it but SCOT. Yet, after the installation, I'm getting the same authentication error. The setup has everything set to default value, including the local authentication.

The scot.log shows several LDAP connectivity errors even though the authentication is configured as "Local":

2018/05/12 11:00:39   DEBUG [10651]  Mojolicious.pm:  131 POST "/auth"
2018/05/12 11:00:39   DEBUG [10651]       Routes.pm:  164 Routing to controller "Scot::Controller::Auth" and action "auth"
2018/05/12 11:00:39   DEBUG [10651]         Auth.pm:  155 Form based Login.  User = admin orig_url = /scot/api/v2/metric/response_avg_last_x_days?days=7&targetdate=2018-05-12&unit=day
2018/05/12 11:00:39   DEBUG [10651]         Auth.pm:  156 Auth Type = Local
2018/05/12 11:00:39   DEBUG [10651]         Auth.pm:  162 checking validity of username and password
2018/05/12 11:00:39   DEBUG [10651]         Auth.pm:  255 checking validity of username/password
2018/05/12 11:00:39   DEBUG [10651]         Auth.pm:  167 attempting to authenticate via ldap
2018/05/12 11:00:39   DEBUG [10651]         Auth.pm:  286 attempting authentication of admin by ldap
2018/05/12 11:00:39   DEBUG [10651]          Env.pm:  324 grabbing config item ldap
2018/05/12 11:00:39   DEBUG [10651]         Auth.pm:  328 seeing if ldap will authenticate
2018/05/12 11:00:39   DEBUG [10651]         Ldap.pm:  130 Connecting to LDAP server ldap.domain.tld
2018/05/12 11:00:39   ERROR [10651]         Ldap.pm:  137 Failed to connect to ldap.domain.tld
2018/05/12 11:00:39    WARN [10651]         Ldap.pm:  142 retrying ldap connection
2018/05/12 11:00:39   ERROR [10651]         Ldap.pm:  137 Failed to connect to ldap.domain.tld
2018/05/12 11:00:39    WARN [10651]         Ldap.pm:  142 retrying ldap connection
2018/05/12 11:00:39   ERROR [10651]         Ldap.pm:  137 Failed to connect to ldap.domain.tld
2018/05/12 11:00:39   ERROR [10651]         Ldap.pm:  146 Failed to Connect to LDAP Server: connection failed at /opt/scot/script/../lib/Scot/Util/Ldap.pm line 138.
2018/05/12 11:00:39   ERROR [10651] DefaultHelpers.pm:  101 Failed to Connect to LDAP server at /opt/scot/script/../lib/Scot/Util/Ldap.pm line 147.
2018/05/12 11:00:39   DEBUG [10651]  EPLRenderer.pm:   52 Template "exception.development.html.ep" not found
2018/05/12 11:00:39   DEBUG [10651]  EPLRenderer.pm:   52 Template "exception.html.ep" not found
2018/05/12 11:00:39   DEBUG [10651]  EPLRenderer.pm:   40 Rendering template "mojo/debug.html.ep"
2018/05/12 11:00:39   DEBUG [10651]  EPLRenderer.pm:   40 Rendering template "mojo/menubar.html.ep"
2018/05/12 11:00:39   DEBUG [10651]   Controller.pm:  209 500 Internal Server Error (0.608315s, 1.644/s)

Any suggestions on fixing this?

Regards,
Adrian

scot_auth_error

Alert.pl issues on Ubuntu 16.04 - SCOT 3.5.2

Anyone have any idea how to fix this issue with Alert.pl? I installed SCOT 3.5.2 from the repository on Ubuntu 16.04. When Alert.pl is run from any user (e.g. SCOT); I get this error which is right after "sender is approved" on messages which should be parsed and ingested. We are installing this new SCOT with "local" authorization at this time.

Here is the error:
Can't locate object method "parser_dir" via package "Scot::Env" at /opt/scot/lib/Scot/App/Mail.pm

Anyone have any idea as to how to fix this issue? Let me know if any additional details are needed.

Thanks,

Matt

Demo site down ?

Hello,

I try to access the demo website https://52.12.122.162/ but it does not response. Is the server still up ? Is there a new adress.

(I don't know if it is the right place to ask : feel free to redirect me if necessary).
Regards.

Blocklist Data

Where/How do you add data to the block list that is mentioned?

"SCOT Integrates into your blocklist to automatically tell you which domains are blocked without you having to look it up. If we had an IP Address, we’d see a small flag next to it indicating the country in which the IP Address is registered."

User Defined Forms Not Persisting Input Data

We have discovered a bug in our implementation of user defined forms that are available in Guides, Signatures, and Incidents. Some fields would not retain the changed information. Some would appear to retain the change but revert to old settings upon next page refresh.

The fix:
Upgrade SCOT to something later than 3.5.5. If you have existing Guides, Signatures, or Incidents, you will also need to run /opt/scot/bin/update_form_objects.pl . That script will update the database objects to work with the fixes to Guides, Signatures, and Incidents. As always, please back up your database before running update_form_objects.pl

How to set Email Injest Parser module for Arcsight

Hello,
The sample parser modules on the SCOT /opt/scot/lib/Scot/Parser/ does not have for ArcSight.

Can i adapt the parser for the splunk.pm.
Do i need to make a change on the "parse_message” function"
thanks

Issues with docker-compose.yml

I was getting the error "ERROR: Named volume "rfproxy_log_data:/var/log/scot:rw" is used in service "rfproxy" but no declaration was found in the volumes section.". I changed two lines in docker-compose.yml which possibly solved this (still troubleshooting):

rfproxy:
volumes:
- "./rfproxy_log_data:/var/log/scot/"

image: sandialabs/scot_perl

It seems the sandialabs/scot_rfproxy directory doesn't exist?

Also, the /etc/timezone files in the docker-compose.yml file for mongo, apache, rfproxy and scot were throwing the following error: ERROR: for apache Cannot start service apache: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting "/home/ec2-user/scot/etc/timezone" to rootfs at "/var/lib/docker/overlay2/6f815a39db2b8d8d95fc94c554bd55f82ae4fe55aba678c35baaa4f56df788ef/merged/etc/timezone" caused: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

So I commented them out.

# - "/etc/timezone:/etc/timezone:ro"
# - "/etc/localtime:/etc/localtime:ro"

Finally, the apache build had the following error:
ERROR: for apache Cannot create container for service apache: maximum retry count cannot be used with restart policy 'always'

Need to change that line to "on-failure":

deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3

Following Instructions but not able to do bash install_scot3

Here are the results when i try

root@ubuntu:~/scot# bash install_scot3
bash: install_scot3: No such file or directory
here is my LS

root@ubuntu:~/scot# ls
bin circle.yml Dockerfile etc lib license.txt public redhat_installer.sh t ubuntu_installer.sh
centos_installer.sh deploy docs install.sh LICENSE pkgs README.md script templates wercker.yml
HELP

alertbot.pl ignores command line flags

It seems that none of the flags described in the source code appear to be sent to any of the backend code. The result is that they are ignored when invoking the tool at the command line.

Re-write UI in React.JS using Flux methodology

The Current UI does DOM manipulations itself, which is hard to maintain and isn't always the most efficient. This suggestion is to re-write the using the Flux method of updating with React.JS.

DateTime::Cron::Simple missing from Docker install_perl_mods.sh

I pulled the docker file for sandialabs/scot yesterday to start setting up SCOT. When I checked the scot.prod.log file several scripts were dying with an error that DateTime::Cron::Simple was missing. I added it to install_perl_mods.sh, ran apt-get install make and then ran the install_perl_mods.sh script to fix it.

game exited with code 255

On clean install I see the following:
game | Can't locate object method "location" via package "Scot::Env" at /opt/scot/lib/Scot/Collection.pm line 40. game | --- Starting Game Tally --- game exited with code 255

It would appear a default "location" is not defined in the game.cfg.pl. Defining it in docker-configs/game/game.cfg.pl and re-issuing appears to resolve the issue although I need to test a clean install to be sure.

scot.cfg.pl and others define "location" as being for "future use". I simply copied one of these definitions over. Note there is an inconsistency in the defaults as most cfg.pl define location as "scot_demo" where scot.cfg.pl defines it as "demosite". Same applies to "site_identifier".

Multi-line (or other formatting) within an alert

Hi, I've successfully been able to add a custom alert parser, and I can see the multiple named columns that it creates in the GUI when such an alertgroup is selected.

With in a column of the alert, is there a way to make multi-line text look nice? When displayed, it appears to drop newlines, and also
is printed out literally.

activemq storage limits in Docker

Inside of docker, the default activemq configuration gives these error messages:

WARN | Store limit is 102400 mb, whilst the data directory: /opt/sandia/webapps/activemq/data/kahadb only has 5006 mb of usable space
ERROR | Temporary Store limit is 51200 mb, whilst the temporary data directory: /opt/sandia/webapps/activemq/data/localhost/tmp_storage only has 5006 mb of usable space

I updated the systemUsage in /opt/sandia/webapps/activemq/conf/scotamq.xml to the following to fix that:

      <systemUsage>
        <systemUsage>
            <memoryUsage>
                <memoryUsage limit="64 mb"/>
            </memoryUsage>
            <storeUsage>
                <storeUsage limit="4 gb"/>
            </storeUsage>
            <tempUsage>
                <tempUsage limit="1 gb"/>
            </tempUsage>
        </systemUsage>
    </systemUsage>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.