orlikoski / skadi Goto Github PK

Microscopic feature request to speed up usage - since Skadi can be used immediately after deployment by aiming CyLR at it, if the VM displayed it's ens33 IP address on the logon screen, a responder could more quickly begin firing data at it without logging in solely to to run ipconfig.

One issue that could interfere is Ubuntu's dynamic naming of the network interfaces... It may always be ens33, but it could change if someone were to alter the interface type assigned to the VM.

Feel free to ignore, just a minor suggestion on an overall fantastic tool.

Reference: https://askubuntu.com/a/217519

On a fresh install of CCF_VM 2.2, I am getting an error that there is no such file or directory for log2timeline.py when running the standard cdqr.py command to process a windows zip from CyLR. It looks like the new TimeSketch options aren't part of the documentation, so maybe this command or process has changed somehow but isn't documented?

cdqr@CCF_VM:~$ cdqr.py HOSTNAME.zip -p win --max_cpu -z --es_kb HOSTNAME
CDQR Version: 4.0.1
Traceback (most recent call last):
File "/usr/local/bin/cdqr.py", line 1585, in
main()
File "/usr/local/bin/cdqr.py", line 1450, in main
p_ver = plaso_version(log2timeline_location)
File "/usr/local/bin/cdqr.py", line 884, in plaso_version
myproc = subprocess.Popen([log2timeline_location,"--version"],stderr=subprocess.PIPE)
File "/usr/lib/python3.5/subprocess.py", line 947, in init
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'log2timeline.py'

In doing some testing with skadi_server-2019.3.ova I have user CyLR to capture data from my linux host (kubuntu 18.04). Ran CyLR -u skadi -p skadi -s to get the zip to the skadi server vm. Confirmed its there, it is (beast5.zip). When I run:
cdqr -p lin --max_cpu beast5.zip --es_kb beast5_sample
I get this error:
ERROR: "beast5.zip" cannot be found by the system.

Am I messing up the command? I looked at the python code and found where the error comes from, lines 1219-1232 (https://github.com/orlikoski/CDQR/blob/master/src/cdqr.py). I have tried adding the full directory, "/home/skadi/beast5.zip", same error.

All I want to do is use cdqr to process the zip so I can add into kibana. Thanks.

Imported the CCF-VM into virtualbox and attempted to start on company managed laptop. The FW rules prevented access to the running server so had to use Virtualbox Host-Only Ethernet Adapter under Virtualbox network settings. When the VM boots up it displayed the error "FAILED: failed to start raise network interfaces". logged into the VM and the "ifconfig" command revealed no eth0 interface and no assigned IPV4 address.

Resolution:
Ubuntu uses Predictable Network Interface Naming
ifconfig -a to view the available interfaces
sudo vi /etc/network/interfaces
add the following lines:
auto enp0s17 (this was the Ethernet interface name from the ifconfig -a command)
iface enp0s17 inet dhcp

Restart resulted in interface enp0s17 being assigned a private IP address accessible from the Host system.

I did not have an issue deploying the previous release of Skadi, but this version will not deploy with OVF. Also, I tried creating a VM and using the portable version to install as an ISO and it runs unbearably slow even with flash storage, 16CPU and 64GB of RAM.

I have the previous version running without flaws and deployed with no issue. Can you please resolve these issues in your OVF templates.

sudo ./CyLR -zp test

./CyLR: error while loading shared libraries: libcurl-gnutls.so.4: cannot open shared object file: no such file or directory

./CyLR: /lib64/libcurl-gnutls.so.4: no version information available (required by ./CyLR)

Look at maybe using https://github.com/yeti-platform/yeti

i want to add users to the timesketch GUI

I noticed that Skadi has the proper dependencies for these features (Celery, Neo4j, etc), but the features are not enabled in the OVF's /etc/timesketch.conf.

Might be a good reason for it, but otherwise, could these be enabled by default in the OVF?

think about using Watchtower to keep the docker image up to date

All work is done on the build script and supporting elements. Time to make the official builds

https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/

Make some sort of alias or script that enables the use of the CDQR docker but uses the same cdqr commands as older versions of Skadi

The final solution should include options for Windows, Linux, and MacOS

Apologies if this would have been easier by commenting on the closed issue.

Same issue around trying to clone master resulting in the script failing. I've done some digging and it looks like the signedbuildskadi.sh that is inside the .tgz still has "master" for $install_build.

Hello,

I wanted to configure a static ip address for the ELK stack.
or the entire Skadi VM. how should I proceed?

Thanks and Regards
Tej Gandhi

At the tail end of the Ubuntu build script, it fails on the step where it attempts to randomly create the credentials. This is a brand new Ubuntu 16.04 system built only for this purpose.

Using random username and passwords for OS Account, TimeSketch, Nginx proxy, and Grafana
==> Creating skadi_HXYs user
BAD PASSWORD: it is too simplistic/systematic
BAD PASSWORD: it is too simplistic/systematic
BAD PASSWORD: it is too simplistic/systematic
chpasswd: (user skadi_HXYs) pam_chauthtok() failed, error:
Have exhausted maximum number of retries for service
chpasswd: (line 1, user skadi_HXYs) password not changed

The Secure Networking Add-On pack no longer works with v2019.1.

Update the Secure Networking pack to support the Nginx Docker container. Probably going to involve https://hub.docker.com/r/jrcs/letsencrypt-nginx-proxy-companion/

ElasticSearch can sometimes fail to load when it tries to use the IPv6 version of localhost.

IPv6 is disabled by default in the build process https://github.com/orlikoski/Skadi/blob/master/scripts/signedbuildskadi.sh#L267-L271 .

The fix is to comment out all IPv6 lines in the /etc/hosts

Example:

# The following lines are desirable for IPv6 capable hosts
# ::1     localhost ip6-localhost ip6-loopback
# ff02::1 ip6-allnodes
# ff02::2 ip6-allrouters

When analysing a new timeline or existing timeline in TimeSketch, TimeSketch shows no data and a pop up "Internal Server Error"

I have installed docker and tested but receiving the following error when installing Skadi = wget -O /tmp/buildskadi.sh https://raw.githubusercontent.com/orlikoski/skadi/master/scripts/buildskadi.sh
The following ports are open: 80,5432,9200.

Building dependency tree
Reading state information... Done
vm.max_map_count = 262144
vm.max_map_count=262144
Renaming Host to skadi
Enabling UFW firewall to only allow OpenSSH and Ngninx Full
Skipping adding existing rule
Skipping adding existing rule (v6)
Skipping adding existing rule
Skipping adding existing rule (v6)
Skipping adding existing rule
Skipping adding existing rule (v6)
Firewall is active and enabled on system startup
Downloading Skadi
Cloning into '/opt/Skadi'...
fatal: Remote branch master not found in upstream origin

- "ES_JAVA_OPTS=-Xms${HEAP_SIZE} -Xmx${HEAP_SIZE}"

and then put HEAP_SIZE=1g in /opt/Skadi/Docker/.env

Based on issue log2timeline/plaso#2398

Create a script/process that accepts data from automation tools to perform functions on the server.

Ensure it is secure and cannot be misused by bad actors.
Ensure it can be used by the cloud version.

Such as (but not limited to):

• Process data
• System for user to initial Start/Stop/Restart services/components on-demand
• Data management (move things around on disk or in the DB like purge timesketches)

I'm attempting to take data that was collected from cdqr and output it to csv reports. I am currently using the ova file from the Skadi site as my server and cdqr is version 5.0. Below is the multiple ways I tried to create these reports and the error output.

command: cdqr demo.zip - z --max_cpu
Output Error: "demo.zip" cannot be found by the system. Please verify filename and path are correct.

command: in: demo.zip -z --max_cpu
Output Error: "demo.zip" cannot be found by the system. Please verify filename and path are correct.

Need to change rc.local to use /opt/cerebro-0.6.6 vs 0.6.5

Hi, I tried using docker-compose to get everything up and running.

I keep on running into this error:
docker_yeti-beat_1 exited with code 1

Detailed logs below:

yeti-beat_1       | celery beat v4.2.1 (windowlicker) is starting.
yeti-beat_1       | Traceback (most recent call last):
yeti-beat_1       |   File "/usr/local/bin/celery", line 10, in <module>
yeti-beat_1       |     sys.exit(main())
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 16, in main
yeti-beat_1       |     _main()
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 322, in main
yeti-beat_1       |     cmd.execute_from_commandline(argv)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 496, in execute_from_commandline
yeti-beat_1       |     super(CeleryCommand, self).execute_from_commandline(argv)))
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 275, in execute_from_commandline
yeti-beat_1       |     return self.handle_argv(self.prog_name, argv[1:])
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 488, in handle_argv
yeti-beat_1       |     return self.execute(command, argv)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 420, in execute
yeti-beat_1       |     ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 279, in run_from_argv
yeti-beat_1       |     sys.argv if argv is None else argv, command)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 363, in handle_argv
yeti-beat_1       |     return self(*args, **options)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 238, in __call__
yeti-beat_1       |     ret = self.run(*args, **kwargs)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/beat.py", line 109, in run
yeti-beat_1       |     return beat().run()
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 81, in run
yeti-beat_1       |     self.start_scheduler()
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 100, in start_scheduler
yeti-beat_1       |     print(self.banner(service))
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 122, in banner
yeti-beat_1       |     c.reset(self.startup_info(service))),
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 132, in startup_info
yeti-beat_1       |     scheduler = service.get_scheduler(lazy=True)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 618, in get_scheduler
yeti-beat_1       |     lazy=lazy,
yeti-beat_1       |   File "/opt/yeti/core/scheduling.py", line 61, in __init__
yeti-beat_1       |     self.load_entries()
yeti-beat_1       |   File "/opt/yeti/core/scheduling.py", line 79, in load_entries
yeti-beat_1       |     self.loaded_entries = get_plugins()
yeti-beat_1       |   File "/opt/yeti/core/yeti_plugins.py", line 21, in get_plugins
yeti-beat_1       |     entry = obj.get_or_create(name=obj.default_values['name'])
yeti-beat_1       |   File "/opt/yeti/core/database.py", line 90, in get_or_create
yeti-beat_1       |     r = cls.objects(**select_dict).modify(upsert=True, **update_dict)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 648, in modify
yeti-beat_1       |     raise OperationError(u'Update failed (%s)' % err)
yeti-beat_1       | mongoengine.errors.OperationError: Update failed (BSON field 'no_cursor_timeout' is an unknown field.)
mongodb           | 2019-07-09T13:14:44.533+0000 I  NETWORK  [conn59] end connection 172.25.0.14:54382 (17 connections now open)
mongodb           | 2019-07-09T13:14:44.534+0000 I  NETWORK  [conn60] end connection 172.25.0.14:54384 (16 connections now open)
docker_yeti-beat_1 exited with code 1

it looks as if the README and the userguide does not yet reflect the change from "kopf" to "cerebro", right?
so cluster management should be done with

http://<CCF-VM IP Address or localhost>:9000

BTW: netstat shows still a process listening on port 9200 on localhost

It would be useful to have something like this:
http://christophe.vandeplas.com/2014/06/mactime-magic-with-elk.html

cdqr --max_cpu --nohash in:/tmp/cfreds_2015_data_leakage_pc.E01 out:/tmp/results-mormanual

Fails with a "pyewf_handle_read_buffer" error, but running the same command from inside the docker container works. I narrowed it down to this part of the translated docker command:

docker run --network host -v /tmp/cfreds_2015_data_leakage_pc.E01:/tmp/cfreds_2015_data_leakage_pc.E01 -v /tmp/output:/tmp/output --add-host=elasticsearch:127.0.0.1 --add-host=postgres:127.0.0.1 -v /opt/Skadi/Docker/timesketch/timesketch_default.conf:/etc/timesketch.conf aorlikoski/cdqr:5.1.0 -y /tmp/cfreds_2015_data_leakage_pc.E01 /tmp/output

If that is changed to /tmp:/tmp then the command runs.

I haven't looked at the helper file, but the change should be reasonably simple, right?

Store all config files and such inside Docker / Kubernetes objects instead of placing them on the host OS

Update the build script to deploy Skadi as 100% containers