log2timeline / plaso Goto Github PK
View Code? Open in Web Editor NEWSuper timeline all the things
Home Page: https://plaso.readthedocs.io
License: Apache License 2.0
Super timeline all the things
Home Page: https://plaso.readthedocs.io
License: Apache License 2.0
2014-11-02 11:10:45,240 [ERROR] (MainProcess) PID:5791 <worker> 'NoneType' object has no attribute 'data'
Traceback (most recent call last):
File "plaso/engine/worker.py", line 181, in _ParseFileEntryWithParser
parser_object.Parse(self._parser_context, file_entry)
File "plaso/parsers/winreg.py", line 318, in Parse
codepage=parser_context.codepage)
File "plaso/parsers/winreg_plugins/interface.py", line 198, in Process
registry_type=registry_type, codepage=codepage, **kwargs)
File "plaso/parsers/winreg_plugins/mrulistex.py", line 372, in GetEntries
parser_context, key, registry_type=registry_type, codepage=codepage)
File "plaso/parsers/winreg_plugins/mrulistex.py", line 91, in _ParseMRUListExKey
for index, entry_number in self._ParseMRUListExValue(key):
File "plaso/parsers/winreg_plugins/mrulistex.py", line 72, in _ParseMRUListExValue
mru_list = self._MRULISTEX_STRUCT.parse(mru_list_value.data)
AttributeError: 'NoneType' object has no attribute 'data'
Error is:
$ PYTHONPATH=. python2.7 plaso/lib/output.py
Traceback (most recent call last):
File "plaso/lib/output.py", line 39, in
import pytz
File "/Library/Python/2.7/site-packages/pytz/init.py", line 29, in
from pkg_resources import resource_stream
File "build/bdist.macosx-10.9-intel/egg/pkg_resources.py", line 72, in
File "/Users/dmwhite/code/plaso/plaso/lib/parser.py", line 33, in
File "/Users/dmwhite/code/plaso/plaso/lib/event.py", line 25, in
from plaso.lib import timelib
File "/Users/dmwhite/code/plaso/plaso/lib/timelib.py", line 51, in
class Timestamp(object):
File "/Users/dmwhite/code/plaso/plaso/lib/timelib.py", line 142, in Timestamp
def CopyToIsoFormat(cls, timestamp, timezone=pytz.utc, raise_error=False):
AttributeError: 'module' object has no attribute 'utc'
Which appears to be due to a circular dependency somewhere.
Workaround is stopping pytz from loading resource_stream from pkg_resources. In init.py for pytz, comment out these lines:
try:
from pkg_resources import resource_stream
except ImportError:
resource_stream = None
2014-11-08 13:53:35,906 ERROR PID:22370 257L
Traceback (most recent call last):
File "plaso/engine/worker.py", line 126, in _ParseFileEntryWithParser
parser_object.Parse(self._parser_context, file_entry)
File "plaso/parsers/winreg.py", line 318, in Parse
codepage=parser_context.codepage)
File "plaso/parsers/winreg_plugins/interface.py", line 198, in Process
registry_type=registry_type, codepage=codepage, **kwargs)
File "plaso/parsers/winreg_plugins/msie_zones.py", line 236, in GetEntries
value_string = self.CONTROL_VALUES_PERMISSIONS[value.data]
I have looked at the output of psort.py on a image of mine for shell bag activity.
The activity shows up, but there is not any folder information in the output file. If I use ShellBagExplorer I do have folder info in some of the active shell bags so it is not a case of no data being available.
== details
I'm calling log2timeline / psort as:
/log2timeline.py -d --logfile plaso-debug.log --workers 4 --offset 411648 cu01c1.plasodb /mnt/imageCU01/ewf1
psort -z EST5EDT -w $dir.plaso.converted cu01c1.plasodb
I'm searching through the *.converted file.
Plaso "loops" on p2
PYTHONPATH=~/Projects/plaso/ python ~/Projects/plaso/plaso/frontend/log2timeline.py plaso.db SIFT\ Workstation\ 3.0\ Core\ Drive.vmdk
The following partitions were found:
Identifier Offset (in bytes) Size (in bytes)
p1 1048576 (0x00100000) 534723428352
p2 534725525504 (0x7c80200000) 2144337920
Please specify the identifier of the partition that should be processed:
Note that you can abort with Ctrl^C.
Some enhancements to build_depencies:
2014-06-12 05:18:09,447 [ERROR] (Worker_3 ) PID:6970 <worker> expected 3506438144, found 29665
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/plaso/engine/worker.py", line 188, in ParseFile
for event_object in parsing_object.Parse(file_entry):
File "/usr/lib/python2.7/dist-packages/plaso/parsers/mac_keychain.py", line 238, in Parse
for object_record in self._ReadEntryApplication(file_object):
File "/usr/lib/python2.7/dist-packages/plaso/parsers/mac_keychain.py", line 439, in _ReadEntryApplication
file_object, record.record_header, offset)
File "/usr/lib/python2.7/dist-packages/plaso/parsers/mac_keychain.py", line 350, in _ReadEntryHeader
comments = self.TEXT.parse_stream(file_object)
File "/usr/local/lib/python2.7/dist-packages/construct-2.5.1-py2.7.egg/construct/core.py", line 197, in parse_stream
return self._parse(stream, Container())
File "/usr/local/lib/python2.7/dist-packages/construct-2.5.1-py2.7.egg/construct/core.py", line 287, in _parse
return self._decode(self.subcon._parse(stream, context), context)
File "/usr/local/lib/python2.7/dist-packages/construct-2.5.1-py2.7.egg/construct/core.py", line 287, in _parse
return self._decode(self.subcon._parse(stream, context), context)
File "/usr/local/lib/python2.7/dist-packages/construct-2.5.1-py2.7.egg/construct/core.py", line 723, in _parse
subobj = sc._parse(stream, context)
File "/usr/local/lib/python2.7/dist-packages/construct-2.5.1-py2.7.egg/construct/core.py", line 397, in _parse
return _read_stream(stream, self.lengthfunc(context))
File "/usr/local/lib/python2.7/dist-packages/construct-2.5.1-py2.7.egg/construct/core.py", line 304, in _read_stream
raise FieldError("expected %d, found %d" % (length, len(data)))
FieldError: expected 3506438144, found 29665
When running:
PYTHONPATH=~/Projects/plaso/ python ~/Projects/plaso/plaso/frontend/log2timeline.py plaso.db SIFT\ Workstation\ 3.0\ Core\ Drive.vmdk
This seems to loop on:
/home/sansforensics/.cpan/sources/authors/id/M/MA/lib/lib/core/common.py/zones.py/core/...
Also see: https://code.google.com/p/plaso/issues/detail?id=96
Renewed pickling issues on Windows for pyparsing, surfaces in log2timeline_test.py. What has changed since the last fix that resurfaces this issue.
Offending code in PyparsingMultiLineTextParser:
"""Fill the buffer."""
if len(self._buffer) > self._buffer_size:
return
self._buffer += filehandle.read(self._buffer_size)
# If a parser specifically indicates specific encoding we need
# to handle the buffer as it is an unicode string.
# If it fails we fail back to the original raw string.
if self.encoding:
try:
buffer_decoded = self._buffer.decode(self.encoding)
self._buffer = buffer_decoded
except UnicodeDecodeError:
pass
The issue:
Traceback (most recent call last):
File "setup.py", line 114, in <module>
packages=find_packages('.'),
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/dist-packages/setuptools/command/sdist.py", line 108, in run
self.make_distribution()
File "/usr/lib/python2.7/distutils/command/sdist.py", line 456, in make_distribution
self.make_release_tree(base_dir, self.filelist.files)
File "/usr/lib/python2.7/dist-packages/setuptools/command/sdist.py", line 200, in make_release_tree
_sdist.make_release_tree(self, base_dir, files)
File "/usr/lib/python2.7/distutils/command/sdist.py", line 438, in make_release_tree
dest = os.path.join(base_dir, file)
File "/usr/lib/python2.7/posixpath.py", line 80, in join
path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 11: ordinal not in range(128)
log2timeline.py --help
Traceback (most recent call last):
File "plaso/frontend/log2timeline.py", line 436, in <module>
if not Main():
File "plaso/frontend/log2timeline.py", line 374, in Main
options = arg_parser.parse_args()
File "/usr/lib64/python2.7/argparse.py", line 1688, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/usr/lib64/python2.7/argparse.py", line 1720, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/usr/lib64/python2.7/argparse.py", line 1926, in _parse_known_args
start_index = consume_optional(start_index)
File "/usr/lib64/python2.7/argparse.py", line 1866, in consume_optional
take_action(action, args, option_string)
File "/usr/lib64/python2.7/argparse.py", line 1794, in take_action
action(self, namespace, argument_values, option_string)
File "/usr/lib64/python2.7/argparse.py", line 994, in __call__
parser.print_help()
File "/usr/lib64/python2.7/argparse.py", line 2313, in print_help
self._print_message(self.format_help(), file)
File "/usr/lib64/python2.7/argparse.py", line 2287, in format_help
return formatter.format_help()
File "/usr/lib64/python2.7/argparse.py", line 279, in format_help
help = self._root_section.format_help()
File "/usr/lib64/python2.7/argparse.py", line 210, in format_help
item_help = join([func(*args) for func, args in self.items])
File "/usr/lib64/python2.7/argparse.py", line 288, in _join_parts
if part and part is not SUPPRESS])
when comparing the json dicts make sure they are sorted
When running l2t twice on the the same store and input data it yields this exception:
2014-10-28 14:20:22,826 INFO PID:19888 Processing is done, waiting for storage to complete.
/home/user/Projects/plaso/plaso/lib/storage.py:804: UserWarning: Duplicate name: 'information.dump'
self._zipfile.writestr(stream_name, stream_data)
Traceback (most recent call last):
File "plaso/engine/worker.py", line 151, in _ParseFileEntryWithParser
parser_object.Parse(self._parser_context, file_entry)
File "plaso/parsers/plist.py", line 143, in Parse
parser_context, plist_name=plist_name, top_level=top_level_object)
File "plaso/parsers/plist_plugins/interface.py", line 174, in Process
self.GetEntries(parser_context, top_level=top_level, match=match)
File "plaso/parsers/plist_plugins/airport.py", line 50, in GetEntries
u'/RememberedNetworks', u'item', wifi['LastConnected'], description)
KeyError: 'LastConnected'
RE: Git head 20141124
I initiated log2timeline.py yesterday against a partition. I just checked its progress and it seems to be in an infinite loop. I invoked log2timeline via:
log2timeline.py -d --logfile plaso-debug.log --workers 4 --offset 411648 cu01c1.plasodb /mnt/imageCU01/ewf1
For the last 12+ hours I'm getting this in the debug output every 10 seconds:
2014-11-26 15:26:48,732 WARNING PID:11761 Unable to connect to RPC socket to: Worker_0 at http://localhost:11784
2014-11-26 15:26:48,733 ERROR PID:11761 Process Worker_0 [11784] is not functioning when it should be. Terminating it and removing from list.
2014-11-26 15:26:48,734 WARNING PID:11761 Process Worker_0 [11784] is still alive.
2014-11-26 15:26:48,736 WARNING PID:11761 Unable to connect to RPC socket to: Worker_1 at http://localhost:11786
2014-11-26 15:26:48,737 ERROR PID:11761 Process Worker_1 [11786] is not functioning when it should be. Terminating it and removing from list.
2014-11-26 15:26:48,738 WARNING PID:11761 Process Worker_1 [11786] is still alive.
2014-11-26 15:26:48,740 WARNING PID:11761 Unable to connect to RPC socket to: Worker_2 at http://localhost:11788
2014-11-26 15:26:48,740 ERROR PID:11761 Process Worker_2 [11788] is not functioning when it should be. Terminating it and removing from list.
2014-11-26 15:26:48,742 WARNING PID:11761 Process Worker_2 [11788] is still alive.
2014-11-26 15:26:48,743 WARNING PID:11761 Unable to connect to RPC socket to: Worker_3 at http://localhost:11790
2014-11-26 15:26:48,744 ERROR PID:11761 Process Worker_3 [11790] is not functioning when it should be. Terminating it and removing from list.
2014-11-26 15:26:48,745 WARNING PID:11761 Process Worker_3 [11790] is still alive.
I went back and looked at the first occurrence and I don't see any relevant logs just before it started but I don't really know what I'm looking for.
I have the full debug.log, so let me know what I can provide.
Likely an unrelated issue but before the infinite loop started I was getting this "ERROR" by the 10,000s of thousands, but it was more or less continuous for the whole time the image was processing:
2014-11-25 20:32:21,290 ERROR PID:11788 'ascii' codec can't decode byte 0xf9 in position 1: ordinal not in range(128)
Here it is with some context:
2014-11-25 20:32:21,290 DEBUG PID:11788 Trying to parse: f_000022 with parser: skydrive_log_error
2014-11-25 20:32:21,290 WARNING PID:11788 [skydrive_log_error] Unable to process file: type: OS, location: /mnt/imageCU01/ewf1
type: RAW
type: TSK_PARTITION, location: /p2, part index: 3, start offset: 0x0c900000
type: VSHADOW, store index: 0
type: TSK, inode: 114343, location: /Users/bbenisrael/AppData/Local/Google/Chrome/User Data/Default/Media Cache/f_000022
with error: 'ascii' codec can't decode byte 0xf9 in position 1: ordinal not in range(128).
2014-11-25 20:32:21,290 DEBUG PID:11788 The path specification that caused the error: type: OS, location: /mnt/imageCU01/ewf1
type: RAW
type: TSK_PARTITION, location: /p2, part index: 3, start offset: 0x0c900000
type: VSHADOW, store index: 0
type: TSK, inode: 114343, location: /Users/bbenisrael/AppData/Local/Google/Chrome/User Data/Default/Media Cache/f_000022
2014-11-25 20:32:21,290 ERROR PID:11788 'ascii' codec can't decode byte 0xf9 in position 1: ordinal not in range(128)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/plaso/engine/worker.py", line 123, in _ParseFileEntryWithParser
parser_object.Parse(self._parser_context, file_entry)
File "/usr/lib/python2.7/site-packages/plaso/parsers/text_parser.py", line 1032, in Parse
self._text_reader.ReadLines(file_object)
File "/usr/lib/python2.7/site-packages/plaso/parsers/text_parser.py", line 970, in ReadLines
self.lines = u''.join([self.lines, line])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf9 in position 1: ordinal not in range(128)
If that is unrelated and worthy of its own issue, let me know.
I've started down the path of debugging precisely why, but first off - thanks for adding the TLN outputs. Unfortunately, it isn't quite right - specifically, when outputting filesystem timestamps, there's no indication of what the timestamp is (MACB, SI/FN, etc.).
This appears to be because the FILE (and similar) types' "short" output lacks this metadata, and the TLN format uses these short outputs. This unfortunately means the output isn't useful, I'm still outputting l2tcsv and using 0.65 to convert that to TLN. I'm trying to get free time to work on a patch, but after a few months decided to just submit a placeholder ticket at least.
In the tests I see the following output, let's silence this:
Key: \Software\Microsoft\Windows\CurrentVersion\Explorer\UserAssist\{FA99DFC7-6AC2-453A-A5E2-5E2AFF4507BD} not found
Key: \Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Explorer\UserAssist\{FA99DFC7-6AC2-453A-A5E2-5E2AFF4507BD} not found
Using the latest update from the git repo (as of 11/20/2014 6:14pm UTC):
user@server:/mnt/cases/test_evidence$ log2timeline.py -o 2048 test_image_20141119.dump KINGSTON\ SV300S37A120G.E01
Traceback (most recent call last):
File "/usr/local/bin/log2timeline.py", line 5, in <module>
pkg_resources.run_script('plaso==1.1.1-20141119', 'log2timeline.py')
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1401, in run_script
exec(script_code, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/plaso-1.1.1_20141119-py2.7.egg/EGG-INFO/scripts/log2timeline.py", line 428, in <module>
File "/usr/local/lib/python2.7/dist-packages/plaso-1.1.1_20141119-py2.7.egg/EGG-INFO/scripts/log2timeline.py", line 418, in Main
File "build/bdist.linux-x86_64/egg/plaso/frontend/frontend.py", line 1603, in ProcessSource
File "build/bdist.linux-x86_64/egg/plaso/frontend/frontend.py", line 681, in ProcessSource
File "build/bdist.linux-x86_64/egg/plaso/frontend/frontend.py", line 765, in ScanSource
File "build/bdist.linux-x86_64/egg/dfvfs/helpers/source_scanner.py", line 407, in Scan
File "build/bdist.linux-x86_64/egg/dfvfs/helpers/source_scanner.py", line 274, in _ScanNode
File "build/bdist.linux-x86_64/egg/dfvfs/helpers/source_scanner.py", line 514, in ScanForVolumeSystem
File "build/bdist.linux-x86_64/egg/dfvfs/analyzer/analyzer.py", line 211, in GetVolumeSystemTypeIndicators
File "build/bdist.linux-x86_64/egg/dfvfs/analyzer/analyzer.py", line 102, in _GetTypeIndicators
File "build/bdist.linux-x86_64/egg/dfvfs/resolver/resolver.py", line 106, in OpenFileObject
File "build/bdist.linux-x86_64/egg/dfvfs/resolver/ewf_resolver_helper.py", line 45, in OpenFileObject
File "build/bdist.linux-x86_64/egg/dfvfs/file_io/file_object_io.py", line 85, in open
File "build/bdist.linux-x86_64/egg/dfvfs/file_io/ewf_file_io.py", line 69, in _OpenFileObject
IOError: pyewf_handle_open_file_objects: unable to open file. libewf_segment_table_append_segment_by_segment_file: invalid segment table. libewf_handle_open_file_io_pool: unable to append segment: 1 to segment table.
If I mount the E01 ahead of time with ewfmount, and then run log2timeline against that mount point, the program runs fine. This particular test image has four E0# segments.
I've not been able to get in and debug precisely why, but memory consumption on processing Windows 7 machines is rather high. On a 20-core, 20GB machine Plaso has 11 of 17 workers that are consuming more than 500MB apiece, two peaking out at 1.2GB and totaling 14GB. It almost appears that workers are taking a page from Perl's book and being memory packrats - once they move on from parsing a given file they still hold on to a large amount of memory.
For the particular image I'm processing right now, I've had to kick up to the 20x20 resources, 10x10 was too small even for --workers=5. Sorry I don't have more concrete info right now, but hopefully in the future consumption will drop a bit to make more reasonable machines useful.
In context of: 8bec638
The test coverage of preg needs to be improved.
I am getting tons of these from a 32-bit Windows 7 dd image. This is the current git build on Ubuntu 14.04.1 64-bit libyal build.
2014-10-24 12:34:45,760 WARNING PID:26694 [winiis] Unable to process file: type: OS, location: /home/analyst/Case/disk0.dd
type: RAW
type: TSK_PARTITION, location: /p3, part index: 5, start offset: 0x31800000
type: TSK, inode: 72719, location: /Users/user/AppData/Local/Microsoft/BingBar/Apps/Mail_15642ee020d2449d86382022aa6f2548/7.1.391/css/mail.css
with error: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128).
2014-10-24 12:34:45,760 ERROR PID:26694 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/plaso/engine/worker.py", line 163, in _ParseFileEntryWithParser
parser_object.Parse(self._parser_context, file_entry)
File "build/bdist.linux-x86_64/egg/plaso/parsers/text_parser.py", line 780, in Parse
if not self.VerifyStructure(parser_context, line):
File "build/bdist.linux-x86_64/egg/plaso/parsers/iis.py", line 161, in VerifyStructure
if u'#Software: Microsoft Internet Information Services' in line:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)
NOTE: Originally reported to code.google.com:
https://code.google.com/p/plaso/issues/detail?id=100
Reported Oct 7, 2014
Currently plaso does not create an ES index template.
When you use feature such as faceting on "analyzed" fields, the result is not quite what you expect (grouping is not done based on the whole field).
That is why usually a raw field (not analysed) is created for every field (ex: username.raw). That is what logstash does to allow faceting as expected by the user, this news explain pretty well the issue and solution they choosed : http://www.elasticsearch.org/blog/logstash-1-3-1-released/.
An easy fix is to import the logstash index template (https://github.com/elasticsearch/logstash/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json) as the default one in ES, then when data from plaso is indexed by ES, you get you raw fields and you can easilly facet on them.
In order to do that :
curl -XPUT localhost:9200/_template/plaso -d '
{
"template" : "",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true},
"dynamic_templates" : [ {
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@Version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}'
Traceback (most recent call last):
File "plaso/engine/worker.py", line 151, in _ParseFileEntryWithParser
parser_object.Parse(self._parser_context, file_entry)
File "plaso/parsers/cups_ipp.py", line 253, in Parse
eventdata.EventTimestamp.CREATION_TIME, data_dict)
File "plaso/parsers/cups_ipp.py", line 94, in __init__
self.job_name = self._ListToString(data_dict.get('job_name', None))
File "plaso/parsers/cups_ipp.py", line 123, in _ListToString
return u', '.join(values)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 37: ordinal not in range(128)
The current dpkg files contains:
Architecture: any
Since plaso is pure Python this should be:
Architecture: all
I ran log2timeline (version: 1.1.1_20141103 ) to create a dumpfile after which I used psort to extract the events.
When I look at the output of the winevt module, I see a number of event_id's that are incorrect. I obtained the same output from the following:
import pyevt
a=pyevt.open('SysEvent.Evt')
a1=1.get_record(0)
a1.event_identifier
-- this returns 1073748859, while it should be 7036 (i.e. 1073748859 - 2^30 + 1)
-- (checked the eventlog/eventid with a Windows program, Event Log Explorer)
The eventlog came from a Windows XP machine. Most of the event_id's (about 99%) seem to be incorrect (larger than 65536), but some of the id's look alright (below 65536).
Haven't tested this yet with eventlogs from Vista/7 though.
Thanks for reading,
chrome_cache_parser: add version 2.1 format support
https://code.google.com/p/plaso/issues/detail?id=79
Also look at newer formats
2014-10-28 16:47:38,395 [ERROR] (Worker_4 ) PID:18984 <worker> 60
Traceback (most recent call last):
File "plaso/engine/worker.py", line 151, in _ParseFileEntryWithParser
parser_object.Parse(self._parser_context, file_entry)
File "plaso/parsers/bsm.py", line 569, in Parse
is_bsm = self.VerifyFile(parser_context, file_object)
File "plaso/parsers/bsm.py", line 744, in VerifyFile
if self.BSM_TYPE_LIST[token_id][0] != 'BSM_TOKEN_TEXT':
KeyError: 60
FAIL: testRealEvents (__main__.WindowsServicesTest)
Test the plugin against real events from the parser.
----------------------------------------------------------------------
Traceback (most recent call last):
File "plaso/analysis/windows_services_test.py", line 127, in testRealEvents
self.assertEquals(len(report.text), 136830)
AssertionError: 140625 != 136830
PYTHONPATH=/Projects/plaso/ /Projects/plaso/plaso/frontend/log2timeline.py", line 426, in Main/Projects/plaso/plaso/frontend/log2timeline.py plaso.db activity.sqlite/Projects/plaso/plaso/frontend/log2timeline.py", line 436, in
Traceback (most recent call last):
File "
if not Main():
File "
front_end.ProcessSource(options)
File "/Projects/plaso/plaso/frontend/frontend.py", line 1620, in ProcessSource/Projects/plaso/plaso/frontend/frontend.py", line 623, in ProcessSource
super(ExtractionFrontend, self).ProcessSource(options)
File "
self.ScanSource(options)
File "~/Projects/plaso/plaso/frontend/frontend.py", line 699, in ScanSource
self._scan_context, scan_path_spec=scan_path_spec)
File "/usr/lib/python2.7/dist-packages/dfvfs/helpers/source_scanner.py", line 361, in Scan
return self._ScanNode(scan_context, scan_node)
File "/usr/lib/python2.7/dist-packages/dfvfs/helpers/source_scanner.py", line 228, in _ScanNode
if os_file_entry.IsDirectory():
AttributeError: 'NoneType' object has no attribute 'IsDirectory'
Now:
The following partitions were found:
Identifier Offset (in bytes) Size (in bytes)
p1 1048576 (0x00100000) 1572864000
Want:
The following partitions were found:
Identifier Offset (in bytes) Size (in bytes)
p1 1048576 (0x00100000) 100 MiB/96 MB (1572864000)
[and yes the actual values in the example are not correct]
I did not test this with the v1.1.0 release. With the latest 2014-11-09 git code:
preg.py is referring to the vss resident hives as both 1-7 and 2-8. I assume it should be always 1-7.
Example:
When running preg.py on my test image I get:
The following Volume Shadow Snapshots (VSS) were found:
Identifier VSS store identifier Creation Time
vss1 2a7505fb-069a-11e4-9c17-e8e0b74a6d5f 2014-07-08T13:42:02.895597+00:00
vss2 0b1e5cff-0dad-11e4-9c7c-e8e0b74a6d5f 2014-07-17T15:41:44.447528+00:00
vss3 88be1a87-13f6-11e4-90b3-e8e0b74a6d5f 2014-07-25T16:26:10.064554+00:00
vss4 c646617b-1c9a-11e4-a28e-e8e0b74a6d5f 2014-08-05T14:14:51.020955+00:00
vss5 28118317-2474-11e4-9c6e-e8e0b74a6d5f 2014-08-15T16:30:29.807312+00:00
vss6 8596bf7c-2de4-11e4-a2af-e8e0b74a6d5f 2014-08-27T14:28:48.108970+00:00
vss7 8b514f86-3366-11e4-a2be-e8e0b74a6d5f 2014-09-03T14:54:22.763851+00:00
If I request all 7 be processed in the output I get:
cu01c1-registry-vss1..7-USBstor.txt: Hive File : /Windows/System32/config/SYSTEM
cu01c1-registry-vss1..7-USBstor.txt: Hive File : /Windows/System32/config/SYSTEM:VSS Store 2
cu01c1-registry-vss1..7-USBstor.txt: Hive File : /Windows/System32/config/SYSTEM:VSS Store 3
cu01c1-registry-vss1..7-USBstor.txt: Hive File : /Windows/System32/config/SYSTEM:VSS Store 4
cu01c1-registry-vss1..7-USBstor.txt: Hive File : /Windows/System32/config/SYSTEM:VSS Store 5
cu01c1-registry-vss1..7-USBstor.txt: Hive File : /Windows/System32/config/SYSTEM:VSS Store 6
cu01c1-registry-vss1..7-USBstor.txt: Hive File : /Windows/System32/config/SYSTEM:VSS Store 7
cu01c1-registry-vss1..7-USBstor.txt: Hive File : /Windows/System32/config/SYSTEM:VSS Store 8
Attempting to run update_dependencies.py, but encountering the following error:
root@myBOX:/opt/tools/plaso/utils# python update_dependencies.py
ERROR:root:Linux variant: Ubuntu 14.04 not supported.
Traceback (most recent call last):
File "update_dependencies.py", line 627, in
if not Main():
File "update_dependencies.py", line 260, in Main
u'{0:s}/3rd%20party/{1:s}'.format(google_drive_url, sub_directory))
UnboundLocalError: local variable 'sub_directory' referenced before assignment
root@myBOX:/opt/tools/plaso/utils#
analyst@myBOX:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty
I dont see why this isnt working. I am on python 2.7.6
I am trying to install log2timeline. I have a version from aptitude, but it doesnt work due to dependency issues. I cant check dependencies for a manual compilation due to errors above but the manual build also doesnt work.
Thanks for your help!
I'm running plaso using the development branch from ppa:kristinn-l/plaso-dev on Ubuntu 14.10 against a 2tb EWF image. The target partition is 1.8tb with about 100 GB of allocated data. My system is a quad-core i5, 4gb ram, 6gb swap.
Running the command:
$ log2timeline.py plaso/plaso.dump image.E01
and selecting the 1.8tb partition and parsing 3 shadow volumes resulted in an eventual system freeze. Swap was growing toward max capacity, and I think it maxed and locked the system as plaso was out of memory. Restarting the system, the zipped dump file was corrupted, so it did not complete writing.
I have started plaso a second time, limiting the parsers to win_gen preset subtracting some parsers to try to lighten the load:
$ log2timeline.py -p --parsers "win_gen,-win_,-symantec_" plaso/plaso.dump image.E01
Swap is again growing toward max though the dump file, predicatably, is growing slower.
As title: build script fails as sqlite3.h is not available (clean Fedora 20, installed the build-essentials as noted here: https://sites.google.com/a/kiddaland.net/plaso/developer/building-the-tool/linux#TOC-Fedora-Core-20---Manual-build).
If I install libsqlite3x-devel beforehand, it succeeds (sudo yum install libsqlite3x-devel).
I thought Kristinn had a fix for this in a CL a week or two ago but I just tested git head 20141124 and the problem continues.
This command is producing no useful output:
preg.py -o $offset -i *.E01 --no_vss -p bagmru
(The offset and E01 file are valid, Changing bagmru to USBstor works.)
It does generate a traceback:
Traceback (most recent call last):
File "/usr/bin/preg.py", line 2087, in
if not Main():
File "/usr/bin/preg.py", line 2075, in Main
front_end.RunModeRegistryPlugin(options, options.plugin_names)
File "/usr/bin/preg.py", line 718, in RunModeRegistryPlugin
options, plugin_names=plugin_names)
File "/usr/bin/preg.py", line 633, in GetHivesAndCollectors
searchers = self._GetSearchersForImage(self.GetSourcePathSpec().parent)
File "/usr/bin/preg.py", line 568, in _GetSearchersForImage
for store_index in vss_stores:
TypeError: 'NoneType' object is not iterable
Manager tries to call GetPluginNames on parsers that don't have plugins:
2014-11-21 19:28:50,805 DEBUG PID:29458 Preprocessing done.
Traceback (most recent call last):
File "plaso/frontend/log2timeline.py", line 428, in
if not Main():
File "plaso/frontend/log2timeline.py", line 418, in Main
front_end.ProcessSource(options)
File "/Users/dmwhite/code/plaso/plaso/frontend/frontend.py", line 1633, in ProcessSource
self._ProcessSourceMultiProcessMode(options)
File "/Users/dmwhite/code/plaso/plaso/frontend/frontend.py", line 1130, in _ProcessSourceMultiProcessMode
parser_filter_string=parser_filter_string):
File "/Users/dmwhite/code/plaso/plaso/parsers/manager.py", line 145, in GetParsers
includes, excludes = cls.GetFilterListsFromString(parser_filter_string)
File "/Users/dmwhite/code/plaso/plaso/parsers/manager.py", line 85, in GetFilterListsFromString
active_list.extend(parser_class.GetPluginNames())
AttributeError: type object 'AslParser' has no attribute 'GetPluginNames'
Parsing a disk image now produces quite a lot of "N/A" parsers:
...
Parser counter information:
Counter: total = 1912439
Counter: N/A = 1012356
...
Looking at that:
psort.py -q sample.dump "SELECT parser,store_number,store_index" |head
parser,store_number,store_index
-,1,126
-,1,128
-,1,127
-,1,129
-,2,119
filestat,1,139
filestat,1,130
That reveals that "parser" is not set if this is coming from a plugin. Looking at for instance the winregistry I see all plugins call:
parser_context.ProduceEvent(event_object, plugin_name=self.NAME)
That is they do not call out which parser produce the results, only which plugin. This results in parser_name being set to None and thus not saved in the event object.
This leads to high level view of the data set being lost, we need to add parser name as well, eg. one would like to know all events created by the winregistry parser, how many registry related events are there? And then to dig into more specifics by looking at what registry plugins parsed them, eg:
Counter: winreg_mountpoints2 = 16
Counter: winreg_run_software = 12
Counter: winreg_boot_execute = 12
Counter: winreg_mrulistex_string_and_shell_item = 12
Change the following error to be tell what file entry caused the error.
2014-11-08 18:58:41,242 [WARNING] (Worker_3 ) PID:18333 <timelib> Unable to create timestamp from 0000-00-00 00:00:00.000000 with error: year is out of range
This requires:
Will investigate, but here's the crash:
[INFO] Extracting: dfvfs-20141028.tar.gz
Traceback (most recent call last):
File "./utils/build_dependencies.py", line 1760, in
if not Main():
File "./utils/build_dependencies.py", line 1749, in Main
if not dependency_builder.Build(project_name, project_type):
File "./utils/build_dependencies.py", line 1650, in Build
return self._BuildDependency(download_helper, project_name)
File "./utils/build_dependencies.py", line 1463, in _BuildDependency
download_helper, source_filename, project_name, project_version):
File "./utils/build_dependencies.py", line 1572, in _BuildPythonModule
source_filename, project_name, project_version):
File "./utils/build_dependencies.py", line 744, in Build
source_directory = self.Extract(source_filename)
File "./utils/build_dependencies.py", line 534, in Extract
elif not filename.startswith(directory_name):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 25: ordinal not in range(128)
Triggered on Travis CI https://travis-ci.org/log2timeline/plaso/builds/41174472:
Unable to run test: plaso.frontend.preg_test [./plaso/frontend/preg_test.py] due to error: 'module' object has no attribute 'magics_class'
log2timeline.py --help | less
Traceback (most recent call last):
File "/usr/bin/log2timeline.py", line 428, in <module>
if not Main():
File "/usr/bin/log2timeline.py", line 366, in Main
options = arg_parser.parse_args()
File "/usr/lib/python2.7/argparse.py", line 1688, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/usr/lib/python2.7/argparse.py", line 1720, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/usr/lib/python2.7/argparse.py", line 1926, in _parse_known_args
start_index = consume_optional(start_index)
File "/usr/lib/python2.7/argparse.py", line 1866, in consume_optional
take_action(action, args, option_string)
File "/usr/lib/python2.7/argparse.py", line 1794, in take_action
action(self, namespace, argument_values, option_string)
File "/usr/lib/python2.7/argparse.py", line 994, in __call__
parser.print_help()
File "/usr/lib/python2.7/argparse.py", line 2313, in print_help
self._print_message(self.format_help(), file)
File "/usr/lib/python2.7/argparse.py", line 2327, in _print_message
file.write(message)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 8848: ordinal not in range(128)
At the moment plaso is not processing .gz log files. Make sure it does.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.