orlikoski / skadi Goto Github PK
View Code? Open in Web Editor NEWCollect, Process, and Hunt with host based data from MacOS, Windows, and Linux
Home Page: https://www.skadivm.com
License: GNU General Public License v3.0
Collect, Process, and Hunt with host based data from MacOS, Windows, and Linux
Home Page: https://www.skadivm.com
License: GNU General Public License v3.0
On a fresh install of CCF_VM 2.2, I am getting an error that there is no such file or directory for log2timeline.py when running the standard cdqr.py command to process a windows zip from CyLR. It looks like the new TimeSketch options aren't part of the documentation, so maybe this command or process has changed somehow but isn't documented?
cdqr@CCF_VM:~$ cdqr.py HOSTNAME.zip -p win --max_cpu -z --es_kb HOSTNAME
CDQR Version: 4.0.1
Traceback (most recent call last):
File "/usr/local/bin/cdqr.py", line 1585, in
main()
File "/usr/local/bin/cdqr.py", line 1450, in main
p_ver = plaso_version(log2timeline_location)
File "/usr/local/bin/cdqr.py", line 884, in plaso_version
myproc = subprocess.Popen([log2timeline_location,"--version"],stderr=subprocess.PIPE)
File "/usr/lib/python3.5/subprocess.py", line 947, in init
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'log2timeline.py'
The Secure Networking Add-On pack no longer works with v2019.1.
Update the Secure Networking pack to support the Nginx Docker container. Probably going to involve https://hub.docker.com/r/jrcs/letsencrypt-nginx-proxy-companion/
At the tail end of the Ubuntu build script, it fails on the step where it attempts to randomly create the credentials. This is a brand new Ubuntu 16.04 system built only for this purpose.
Using random username and passwords for OS Account, TimeSketch, Nginx proxy, and Grafana
==> Creating skadi_HXYs user
BAD PASSWORD: it is too simplistic/systematic
BAD PASSWORD: it is too simplistic/systematic
BAD PASSWORD: it is too simplistic/systematic
chpasswd: (user skadi_HXYs) pam_chauthtok() failed, error:
Have exhausted maximum number of retries for service
chpasswd: (line 1, user skadi_HXYs) password not changed
think about using Watchtower to keep the docker image up to date
Store all config files and such inside Docker / Kubernetes objects instead of placing them on the host OS
Update the build script to deploy Skadi as 100% containers
Based on issue log2timeline/plaso#2398
I did not have an issue deploying the previous release of Skadi, but this version will not deploy with OVF. Also, I tried creating a VM and using the portable version to install as an ISO and it runs unbearably slow even with flash storage, 16CPU and 64GB of RAM.
I have the previous version running without flaws and deployed with no issue. Can you please resolve these issues in your OVF templates.
I have installed docker and tested but receiving the following error when installing Skadi = wget -O /tmp/buildskadi.sh https://raw.githubusercontent.com/orlikoski/skadi/master/scripts/buildskadi.sh
The following ports are open: 80,5432,9200.
Building dependency tree
Reading state information... Done
vm.max_map_count = 262144
vm.max_map_count=262144
Renaming Host to skadi
Enabling UFW firewall to only allow OpenSSH and Ngninx Full
Skipping adding existing rule
Skipping adding existing rule (v6)
Skipping adding existing rule
Skipping adding existing rule (v6)
Skipping adding existing rule
Skipping adding existing rule (v6)
Firewall is active and enabled on system startup
Downloading Skadi
Cloning into '/opt/Skadi'...
fatal: Remote branch master not found in upstream origin
All work is done on the build script and supporting elements. Time to make the official builds
Look at maybe using https://github.com/yeti-platform/yeti
i want to add users to the timesketch GUI
ElasticSearch can sometimes fail to load when it tries to use the IPv6 version of localhost
.
IPv6 is disabled by default in the build process https://github.com/orlikoski/Skadi/blob/master/scripts/signedbuildskadi.sh#L267-L271 .
The fix is to comment out all IPv6 lines in the /etc/hosts
Example:
# The following lines are desirable for IPv6 capable hosts
# ::1 localhost ip6-localhost ip6-loopback
# ff02::1 ip6-allnodes
# ff02::2 ip6-allrouters
Microscopic feature request to speed up usage - since Skadi can be used immediately after deployment by aiming CyLR at it, if the VM displayed it's ens33
IP address on the logon screen, a responder could more quickly begin firing data at it without logging in solely to to run ipconfig
.
One issue that could interfere is Ubuntu's dynamic naming of the network interfaces... It may always be ens33
, but it could change if someone were to alter the interface type assigned to the VM.
Feel free to ignore, just a minor suggestion on an overall fantastic tool.
Reference: https://askubuntu.com/a/217519
Make some sort of alias or script that enables the use of the CDQR docker but uses the same cdqr commands as older versions of Skadi
The final solution should include options for Windows, Linux, and MacOS
It would be useful to have something like this:
http://christophe.vandeplas.com/2014/06/mactime-magic-with-elk.html
it looks as if the README and the userguide does not yet reflect the change from "kopf" to "cerebro", right?
so cluster management should be done with
http://<CCF-VM IP Address or localhost>:9000
BTW: netstat shows still a process listening on port 9200 on localhost
When analysing a new timeline or existing timeline in TimeSketch, TimeSketch shows no data and a pop up "Internal Server Error"
Imported the CCF-VM into virtualbox and attempted to start on company managed laptop. The FW rules prevented access to the running server so had to use Virtualbox Host-Only Ethernet Adapter under Virtualbox network settings. When the VM boots up it displayed the error "FAILED: failed to start raise network interfaces". logged into the VM and the "ifconfig" command revealed no eth0 interface and no assigned IPV4 address.
Resolution:
Ubuntu uses Predictable Network Interface Naming
ifconfig -a to view the available interfaces
sudo vi /etc/network/interfaces
add the following lines:
auto enp0s17 (this was the Ethernet interface name from the ifconfig -a command)
iface enp0s17 inet dhcp
Restart resulted in interface enp0s17 being assigned a private IP address accessible from the Host system.
In doing some testing with skadi_server-2019.3.ova I have user CyLR to capture data from my linux host (kubuntu 18.04). Ran CyLR -u skadi -p skadi -s to get the zip to the skadi server vm. Confirmed its there, it is (beast5.zip). When I run:
cdqr -p lin --max_cpu beast5.zip --es_kb beast5_sample
I get this error:
ERROR: "beast5.zip" cannot be found by the system.
Am I messing up the command? I looked at the python code and found where the error comes from, lines 1219-1232 (https://github.com/orlikoski/CDQR/blob/master/src/cdqr.py). I have tried adding the full directory, "/home/skadi/beast5.zip", same error.
All I want to do is use cdqr to process the zip so I can add into kibana. Thanks.
sudo ./CyLR -zp test
./CyLR: error while loading shared libraries: libcurl-gnutls.so.4: cannot open shared object file: no such file or directory
./CyLR: /lib64/libcurl-gnutls.so.4: no version information available (required by ./CyLR)
Need to change rc.local to use /opt/cerebro-0.6.6 vs 0.6.5
Hello,
I wanted to configure a static ip address for the ELK stack.
or the entire Skadi VM. how should I proceed?
Thanks and Regards
Tej Gandhi
I'm attempting to take data that was collected from cdqr and output it to csv reports. I am currently using the ova file from the Skadi site as my server and cdqr is version 5.0. Below is the multiple ways I tried to create these reports and the error output.
command: cdqr demo.zip - z --max_cpu
Output Error: "demo.zip" cannot be found by the system. Please verify filename and path are correct.
command: in: demo.zip -z --max_cpu
Output Error: "demo.zip" cannot be found by the system. Please verify filename and path are correct.
Hi, I tried using docker-compose to get everything up and running.
I keep on running into this error:
docker_yeti-beat_1 exited with code 1
Detailed logs below:
yeti-beat_1 | celery beat v4.2.1 (windowlicker) is starting.
yeti-beat_1 | Traceback (most recent call last):
yeti-beat_1 | File "/usr/local/bin/celery", line 10, in <module>
yeti-beat_1 | sys.exit(main())
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 16, in main
yeti-beat_1 | _main()
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 322, in main
yeti-beat_1 | cmd.execute_from_commandline(argv)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 496, in execute_from_commandline
yeti-beat_1 | super(CeleryCommand, self).execute_from_commandline(argv)))
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 275, in execute_from_commandline
yeti-beat_1 | return self.handle_argv(self.prog_name, argv[1:])
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 488, in handle_argv
yeti-beat_1 | return self.execute(command, argv)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 420, in execute
yeti-beat_1 | ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 279, in run_from_argv
yeti-beat_1 | sys.argv if argv is None else argv, command)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 363, in handle_argv
yeti-beat_1 | return self(*args, **options)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 238, in __call__
yeti-beat_1 | ret = self.run(*args, **kwargs)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/beat.py", line 109, in run
yeti-beat_1 | return beat().run()
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 81, in run
yeti-beat_1 | self.start_scheduler()
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 100, in start_scheduler
yeti-beat_1 | print(self.banner(service))
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 122, in banner
yeti-beat_1 | c.reset(self.startup_info(service))),
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 132, in startup_info
yeti-beat_1 | scheduler = service.get_scheduler(lazy=True)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 618, in get_scheduler
yeti-beat_1 | lazy=lazy,
yeti-beat_1 | File "/opt/yeti/core/scheduling.py", line 61, in __init__
yeti-beat_1 | self.load_entries()
yeti-beat_1 | File "/opt/yeti/core/scheduling.py", line 79, in load_entries
yeti-beat_1 | self.loaded_entries = get_plugins()
yeti-beat_1 | File "/opt/yeti/core/yeti_plugins.py", line 21, in get_plugins
yeti-beat_1 | entry = obj.get_or_create(name=obj.default_values['name'])
yeti-beat_1 | File "/opt/yeti/core/database.py", line 90, in get_or_create
yeti-beat_1 | r = cls.objects(**select_dict).modify(upsert=True, **update_dict)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 648, in modify
yeti-beat_1 | raise OperationError(u'Update failed (%s)' % err)
yeti-beat_1 | mongoengine.errors.OperationError: Update failed (BSON field 'no_cursor_timeout' is an unknown field.)
mongodb | 2019-07-09T13:14:44.533+0000 I NETWORK [conn59] end connection 172.25.0.14:54382 (17 connections now open)
mongodb | 2019-07-09T13:14:44.534+0000 I NETWORK [conn60] end connection 172.25.0.14:54384 (16 connections now open)
docker_yeti-beat_1 exited with code 1
- "ES_JAVA_OPTS=-Xms${HEAP_SIZE} -Xmx${HEAP_SIZE}"
and then put HEAP_SIZE=1g
in /opt/Skadi/Docker/.env
cdqr --max_cpu --nohash in:/tmp/cfreds_2015_data_leakage_pc.E01 out:/tmp/results-mormanual
Fails with a "pyewf_handle_read_buffer" error, but running the same command from inside the docker container works. I narrowed it down to this part of the translated docker command:
docker run --network host -v /tmp/cfreds_2015_data_leakage_pc.E01:/tmp/cfreds_2015_data_leakage_pc.E01 -v /tmp/output:/tmp/output --add-host=elasticsearch:127.0.0.1 --add-host=postgres:127.0.0.1 -v /opt/Skadi/Docker/timesketch/timesketch_default.conf:/etc/timesketch.conf aorlikoski/cdqr:5.1.0 -y /tmp/cfreds_2015_data_leakage_pc.E01 /tmp/output
If that is changed to /tmp:/tmp then the command runs.
I haven't looked at the helper file, but the change should be reasonably simple, right?
I noticed that Skadi has the proper dependencies for these features (Celery, Neo4j, etc), but the features are not enabled in the OVF's /etc/timesketch.conf
.
Might be a good reason for it, but otherwise, could these be enabled by default in the OVF?
Create a script/process that accepts data from automation tools to perform functions on the server.
Ensure it is secure and cannot be misused by bad actors.
Ensure it can be used by the cloud version.
Such as (but not limited to):
Apologies if this would have been easier by commenting on the closed issue.
Same issue around trying to clone master resulting in the script failing. I've done some digging and it looks like the signedbuildskadi.sh that is inside the .tgz still has "master" for $install_build.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.