mandiant / flare-ida

IDA Pro utilities from FLARE team

FLARE Team Reversing Repository

This repository contains a collection of IDA Pro scripts and plugins used by the FireEye Labs Advanced Reverse Engineering (FLARE) team.

plugins

To install, copy the contents of the plugins directory in this repository to your %PROGRAMFILES%\IDA\plugins folder.

python

The python directory here can be copied to your %PROGRAMFILES%\IDA\python folder, or you can modify your PYTHONPATH environment variable to include the directory.

Provided Plugins

Shellcode Hashes

The shellcode_hashes_search_plugin.py IDA plugin implements the hash searching described here: https://www.mandiant.com/blog/precalculated-string-hashes-reverse-engineering-shellcode/.

The shellcode_hashes directory contains the script used to create the database for the shellcode_hash_search.py script, along with a provided database.

Struct Typer

The struct_typer_plugin.py plugin implements the struct typing described here: https://www.mandiant.com/blog/applying-function-types-structure-fields-ida/

StackStrings

The stackstrings_plugin.py implements the recovery of manually constructed strings described here: http://www.fireeye.com/blog/threat-research/2014/08/flare-ida-pro-script-series-automatic-recovery-of-constructed-strings-in-malware.html

MSDN Annotations

This script for IDA Pro adds MSDN information from a XML file to the database. Information about this plugin can be found at: https://www.fireeye.com/blog/threat-research/2014/09/flare-ida-pro-script-series-msdn-annotations-ida-pro-for-malware-analysis.html

The following functionality is included:

• Backup the original database
• Retrieve all imported functions
• Import function descriptions
• Import argument descriptions
• Create custom enumerations for identified constants including descriptions
• Rename constants to their readable values

MSDN Annotations Usage

TL;DR: In IDA run annotate_IDB_MSDN.py.

All files (IDAPython scripts, XML parser, MSDN information XML file, etc.) should be located in the same directory accessible by IDA Pro. In IDA use File - Script file... (ALT + F7) to open annotate_IDB_MSDN.py. The form will allow you to change the settings and annotate the IDB file after you click OK.

After executing the script once, View - Recent scripts (ALT + F9) can be used as well.

ApplyCalleeType

This plugin allows you to specify or choose a function type for indirect calls as described here: https://www.fireeye.com/blog/threat-research/2015/04/flare_ida_pro_script.html

idb2pat

This script allows you to easily generate function patterns from an existing IDB database that can then be turned into FLIRT signatures to help identify similar functions in new files. More information is available at: https://www.fireeye.com/blog/threat-research/2015/01/flare_ida_pro_script.html

argtracker

This utility can help you identify static arguments to functions used within a program. This is most commonly used to extract arguments to string decoder functions. Example usage is available in

• examples/argtracker_example1.py
• examples/argtracker_example2.py

A blog post with further information is available at:

https://www.fireeye.com/blog/threat-research/2015/11/flare_ida_pro_script.html

objc2_analyzer

This script creates cross-references between selector references and their implementations as defined in the Objective-C runtime related sections of the target Mach-O executable. It also patches selector reference pointers to instead point to their implementation function. This makes analysis of Objective-C code easier by enabling smooth transitions between an implementation and the locations where its selector is referenced throughout the code. Helpful Objective-C code comments are added to each call to objc_msgSend variants to clearly indicate which method is being called on which class.

ironstrings

ironstrings.py is an IDAPython script that uses code emulation to recover constructed strings (stackstrings) from malware. Please see the details in the script's README.

Code Grafter

code_grafter.py is an IDAPython script that grafts code to an IDA database to implement various imported functions and increase the likelihood of being able to execute an unpacker or decoder entirely under Bochs (or any other emulation tools that don't implement special handling for these functions). This prevents faults when emulated execution reaches functions such as VirtualAlloc or lstrlenA.

flare-ida's Issues

MSDN data doesn't have constant values

I cannot use your script to generate the xml file (MSDN data) successfully because "Windows Software
Development Kit (SDK) for Windows 7 and .NET Framework 3.5 Service Pack 1" was unavailable in Microsoft website years ago. So I straightly used the xml file you provided in

https://github.com/mr-tz/flare-ida/blob/master/MSDN_data/msdn_data.zip

But no constant in this xml file has value. So when I execute the script in IDA, no API constant will rename to macro.

This is a part of the xml file:

<constants>
	<constant>
		<name>BFT_NTDS_DATABASE</name>
		<description>The file is an NTDS database file. This file should be copied to the file identified as BFT_NTDS_DATABASE when the data is restored.</description>
	</constant>
	<constant>
		<name>BFT_LOG</name>
		<description>The file is a log file. All log files are copied to the directory identified as BFT_LOG_DIR when the data is restored.</description>
	</constant>
	<constant>
		<name>BFT_PATCH_FILE</name>
		<description>The file is a patch file. All patch files are copied to the directory identified as BFT_CHECKPOINT_DIR when the data is restored.</description>
	</constant>
</constants>

As you can see, all constants only have <name> and <description>. I can't see <value> in all <constant>.

Parse_decl2 too many arguments error

This line fails because the current parse_decl2 has one less argument now:

https://github.com/fireeye/flare-ida/blob/master/python/flare/apply_callee_type.py#L67

Taking out "blah" still throws an error, and I'm not sure why.

issue when run python script annotate_IDB_MSDN

i got the issue when run python script annotate_IDB_MSDN, please help me

Traceback (most recent call last):
  File "C:/Program Files/IDAPro6.6/python/flare/annotate_IDB_MSDN.py", line 117, in on_ok_button
    IDB_MSDN_Annotator.main(config)
  File "C:/Program Files/IDAPro6.6/python/flare\IDB_MSDN_Annotator\__init__.py", line 523, in main
    functions_map = parse_xml_data_files(msdn_data_dir)
  File "C:/Program Files/IDAPro6.6/python/flare\IDB_MSDN_Annotator\__init__.py", line 486, in parse_xml_data_files
    additional_functions = xml_parser.parse(xml_file)
  File "C:/Program Files/IDAPro6.6/python/flare\IDB_MSDN_Annotator\xml_parser.py", line 283, in parse
    parser.parse(xmlfile)
  File "C:\Program Files\IDAPro6.6\lib\xml\sax\expatreader.py", line 107, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "C:\Program Files\IDAPro6.6\lib\xml\sax\xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "C:\Program Files\IDAPro6.6\lib\xml\sax\expatreader.py", line 211, in feed
    self._err_handler.fatalError(exc)
  File "C:\Program Files\IDAPro6.6\lib\xml\sax\handler.py", line 38, in fatalError
    raise exception
xml.sax._exceptions.SAXParseException: C:\Program Files\IDAPro6.6\python\flare\annotate_IDB_MSDN.py:1:2: not well-formed (invalid token)

Thank you very much!

Issue parsing files with MSDN_crawler

I seem to have hit an issue with the parsing of the files, tried this on 6.6 on x64, and 6.5 on x86.

C:\Users\luser\Desktop\IDA stuff\flare-ida\MSDN_crawler [master]> python .\msdn_
crawler.py 'C:\\sdk_help\\' 'C:\\Program Files\\IDA 6.5\\tilib.exe' 'C:\\Program
 Files\\IDA 6.5\\til\\pc'
MSDN crawler based on zynamics msdn-crawler - Copyright 2010
Traceback (most recent call last):
  File ".\msdn_crawler.py", line 413, in <module>
    main()
  File ".\msdn_crawler.py", line 398, in main
    (file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
  File ".\msdn_crawler.py", line 371, in parse_files
    result = parse_file(os.path.join(root, file), const_enum)
  File ".\msdn_crawler.py", line 276, in parse_file
    return parse_new_style(file, content, const_enum)
  File ".\msdn_crawler.py", line 183, in parse_new_style
    parsed_html.find_all(width='60%')]
TypeError: 'NoneType' object is not callable

(This happens after a few minutes of processing). Upon running the same command -v, I get:

Lots of this "Could not retrieve function description...", which I figure it's okay since not all files will be relevant to the script.

[...] 
DEBUG:__main__:Error: Could not retrieve function description from file C:\\sdk_
help\\abff2e90-4c42-4c07-816f-efde05343e03.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abff323b-e6c6-45e0-93bd-eeb68bca80e0.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abff3c41-301f-4273-9103-8e6197ba41fe.htm
Traceback (most recent call last):
  File "c:\python27\lib\logging\__init__.py", line 842, in emit
    msg = self.format(record)
  File "c:\python27\lib\logging\__init__.py", line 719, in format
    return fmt.format(record)
  File "c:\python27\lib\logging\__init__.py", line 464, in format
    record.message = record.getMessage()
  File "c:\python27\lib\logging\__init__.py", line 328, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Logged from file msdn_crawler.py, line 118
DEBUG:__main__:Error: Could not retrieve function description from file C:\\sdk_
help\\abff3c41-301f-4273-9103-8e6197ba41fe.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abffd0fe-d047-4670-a728-eea8253f3f2d.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_activate.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_getautohidebar.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_getstate.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_gettaskbarpos.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_new.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_querypos.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_remove.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_setautohidebar.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_setpos.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_setstate.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_windowposchanged.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abnormaltermination.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abn_fullscreenapp.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abn_poschanged.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abn_statechange.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abn_windowarrange.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abort.htm
DEBUG:__main__:Error: Could not retrieve function description from file C:\\sdk_
help\\abort.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abortall.htm
DEBUG:__main__:Error: Could not retrieve function description from file C:\\sdk_
help\\abortall.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abortdoc.htm
Traceback (most recent call last):
  File ".\msdn_crawler.py", line 413, in <module>
    main()
  File ".\msdn_crawler.py", line 398, in main
    (file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
  File ".\msdn_crawler.py", line 371, in parse_files
    result = parse_file(os.path.join(root, file), const_enum)
  File ".\msdn_crawler.py", line 276, in parse_file
    return parse_new_style(file, content, const_enum)
  File ".\msdn_crawler.py", line 183, in parse_new_style
    parsed_html.find_all(width='60%')]
TypeError: 'NoneType' object is not callable

Just for some 4 eyeing:
I am running windows version of IDA (tried both 64 and 32 bit hosts).
Python is always 32 bit.
I pip-installed "beautifulsoup" (not beautifulsoup4)
I decompressed all the HxS help files to a flat directory, i.e. all htm files in the same directory.

ps. as a sidenote I get tilib.exe errors on different files, on clean installs, on pretty much every version from 6.1 to 6.6 except for 6.5, (tilib.exe version matching that of the IDA distribution) anyone else experiencing this?

print_tinfo expected 8 arguments, got 7

Hi FLARE Team,

ApplyCalleeType: Exception caught: print_tinfo expected 8 arguments, got 7
Traceback (most recent call last):
  File "C:\Tools\IDA\python\flare\apply_callee_type.py", line 267, in run
    typename = idaapi.print_tinfo('', '', 0, 0, idaapi.PRTYPE_1LINE, tinfo, '', '')
  File "C:\Tools\IDA\python\idaapi.py", line 36101, in print_tinfo
    return _idaapi.print_tinfo(*args)
TypeError: print_tinfo expected 8 arguments, got 7

After looking at idaapi.py

def print_tinfo(_args):
  """
    print_tinfo(result, prefix, indent, cmtindent, flags, tif, name,
        cmt) -> bool
    """
  return _idaapi.print_tinfo(_args)

Thanks.

can't find idb2pat.py in your repository

Hi, I read your blog post and can't find the script.
Where it is ?
Thanks

Python 3 support

As you probably already know IDA 7.4 ships with Python 3.x support.
Default may still be Python 2.x, but as of this year Python 2 is deprecated and many plugins are being migrated to 3.x at the moment.

Are you considering supporting Python 3 support?

Broken loading of IDA libs in IDA 7.0 (Found in StructTyper)

Hello,

Setup:

• Clean Install of IDA 7.0 with Flare-IDA install per README.md
• WIndows 7 x64

When running the StructTyper plugin, I receive the following error:

Failed while executing plugin_t.run():
Traceback (most recent call last):
  File "C:/Program Files/IDA 7.0/plugins/struct_typer_plugin.py", line 47, in run
    idaapi.require('flare.struct_typer')
  File "C:\Program Files\IDA 7.0\python\ida_idaapi.py", line 153, in require
    m = importlib.import_module(modulename, package)
  File "C:\python27-x64\Lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
  File "C:\Program Files\IDA 7.0\python\flare\struct_typer.py", line 48, in <module>
    g_dll = ctypes.windll[idaname + ".wll"]
  File "C:\python27-x64\Lib\ctypes\__init__.py", line 437, in __getitem__
    return getattr(self, name)
  File "C:\python27-x64\Lib\ctypes\__init__.py", line 432, in __getattr__
    dll = self._dlltype(name)
  File "C:\python27-x64\Lib\ctypes\__init__.py", line 362, in __init__
    self._handle = _dlopen(self._name, mode)
WindowsError: (126, 'The specified module could not be found')

The key error seems to be in:

g_dll = ctypes.windll[idaname + ".wll"]

Version 7.0 looks to have dropped the .will files (or I missed something on the installation). I tried changing it to .DLL and now get the following:

    Failed while executing plugin_t.run():
    Traceback (most recent call last):
      File "C:/Program Files/IDA 7.0/plugins/struct_typer_plugin.py", line 47, in run
        idaapi.require('flare.struct_typer')
      File "C:\Program Files\IDA 7.0\python\ida_idaapi.py", line 153, in require
        m = importlib.import_module(modulename, package)
      File "C:\python27-x64\Lib\importlib\__init__.py", line 37, in import_module
        __import__(name)
      File "C:\Program Files\IDA 7.0\python\flare\struct_typer.py", line 70, in <module>
        print_type_to_one_line = g_dll.print_type_to_one_line
      File "C:\python27-x64\Lib\ctypes\__init__.py", line 375, in __getattr__
        func = self.__getitem__(name)
      File "C:\python27-x64\Lib\ctypes\__init__.py", line 380, in __getitem__
        func = self._FuncPtr((name_or_ordinal, self))
    AttributeError: function 'print_type_to_one_line' not found

IDA 7.1: NameError: global name 'flare' is not defined

I put plugins in the "IDA 7.1/plugin/" directory, and python files in "IDA 7.1/python" dir; installed dependency vivisect
i'm getting this now:

Failed while executing plugin_t.run():
Traceback (most recent call last):
  File "C:/Program Files/IDA 7.1/plugins/stackstrings_plugin.py", line 57, in run
    flare.stackstrings.main()
NameError: global name 'flare' is not defined

Any help, much obliged.

OverflowError with 64-bit IDBs

Attempting to use shellcode hash search for 64-bit IDBs may elicit the following error (hand-typed, HOPEFULLY free of transcription errors):

Traceback (most recent call last):
  File "C:/Program Files/IDA 7.1/python/flare/shellcode_hash_search.py", line 297, in lookForOpArgs
    hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
  File "C:/Program Files/IDA 7.1/python/flare/shellcode_hash_search.py", line 184, in getSymbolByTypeHash
    cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER

Commonly, 64-bit shellcode uses 32-bit hash algorithms with 32-bit registers, so one workaround might be to check that the upper 32 bits are null or all ones (sign-extended negative quantity) and truncate (respecting sign extension) before querying (or raise an exception if the upper 32 bits contain numeric information of any significance. It might be better to promote the db schema and existing hashes to 64-bit quantities and make this work seamlessly for 32- and 64-bit IDBs if possible.

idb2pat has issues with functions < 32 in length

Looks like the zrange implementation doesn't consider the case when (func.endEA - func.startEA) < 32. For REALLY small functions, this seems to return a really long iterator.

Try it zrange(32, 8). Script gets stuck at line 265.

issue when execute MSDN Crawler

i got the issue of MSDN Crawler, anyone has this issue when execute MSDN Crawler?

C:\Program Files\IDA 6.4>python msdn_crawler.py "C:\\Program Files\\Microsoft SDKs\\Windows\\v7.0\\Help\\1033" "C:\\Program Files\\IDA 6.4\\tilib.exe" "C:\\Program Files\\IDA 6.4\\til\\pc"
MSDN crawler based on zynamics msdn-crawler - Copyright 2010
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\bc31.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\bc31.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\bc5dos.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\bc5dos.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\bc5w16.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\bc5w16.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\bcb4win.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\bcb4win.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\bcb5win.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\bcb5win.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\geos.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\geos.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\gnuwin.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\gnuwin.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\ms16dos.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\ms16dos.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\ms16win.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\ms16win.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mscor.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mscor.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk_2000.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk_2000.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk_nt.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk_nt.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk_vista.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk_vista.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk_win7.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk_win7.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk_ws03.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk_ws03.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\nlm.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\nlm.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\ntapi.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\ntapi.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\ntddk.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\ntddk.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\ntddk64.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\ntddk64.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\vc10.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\vc10.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\vc10_64.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\vc10_64.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\vc6win.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\vc6win.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\vc8amd64.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\vc8amd64.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\vc9.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\vc9.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\w16dos.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\w16dos.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\w16os2.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\w16os2.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\w32dos.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\w32dos.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\w32os2.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\w32os2.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\wdm.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\wdm.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\wnet.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\wnet.til']' returned non-zero exit status 1
Traceback (most recent call last):
  File "msdn_crawler.py", line 413, in <module>
    main()
  File "msdn_crawler.py", line 398, in main
    (file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
  File "msdn_crawler.py", line 357, in parse_files
    const_enum = extract_til_constant_info.main(tilib_exe, til_dir)
  File "C:\Program Files\IDA 6.4\extract_til_constant_info.py", line 95, in main
    for enum_name, enum in enums.iteritems():
UnboundLocalError: local variable 'enums' referenced before assignment

C:\Program Files\IDA 6.4>

Import error with PySide inside IDA Pro

Hi,

I have a weird error inside IDA-Pro. I've installed PySide with pip, as recommended in the official documentation.

The following command runs fine in a terminal:

C:\>python
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from PySide import QtGui

But inside IDA Pro (Windows > Output Window), it fails:

Python>from PySide import QtGui
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: DLL load failed: La proc?dure sp?cifi?e est introuvable.

Though, I'm able to import PySide (still in IDA Pro):

Python>import PySide

I've created following environment variables:

PATH:
  [REMOVED];C:\Python27;C:\Python27\scripts;C:\Python27\Lib\site-packages\PySide

PYTHONPATH:
  C:\Python27\Lib;C:\Python27\DLLs;C:\Python27\Lib\lib-tk;c:\Python27\Lib\flare;c:\Python27\Lib\vivisect;C:\Python27\Lib\site-packages\PySide

Any idea what I'm missing?

Many thanks in advance for your help.

0 functions applied in IDA from .sig file

Pretty sure it's not me doing something wrong(and creating all those issues I mean). Works on a simple VC++ Hello World project as expected and explained in the articles(One, Two):

On the other hand, in a big UE4 project 0 functions get applied in IDA from .sig file after generating the .sig file with sigmake from idb2pat:

I generate the .sig file using command sigmake -lrsub_ "S05_TestingGrounds-Win64-Shipping - No Xdigit errors(deleted lines with errors).pat" "S05_TestingGrounds-Win64-Shipping - No Xdigit errors(deleted lines with errors).sig". -lrsub_ parameter is to exclude functions that have sub_ in them.

After generating the .sig file and then trying to apply the .sig file, that's what I get: 0 functions applied.

Here's the link with the project, pat and sig file, so you can try to generate and apply this sig file onto executable yourself:
https://www.dropbox.com/h?preview=TestingGrounds_DebugSymbols.zip

This is a Unreal Engine 4.26 C++ project created from FPS template which I packaged in UE4 with debug symbols.

The executable, pat and .sig files are in \WindowsNoEditor\S05_TestingGrounds\Binaries\Win64\ folder. The original pat file with xdigit problem is called S05_TestingGrounds-Win64-Shipping - Original.pat. The pat file with problematic lines deleted which cause xdigit problem is called S05_TestingGrounds-Win64-Shipping - No Xdigit errors(deleted lines with errors).pat.

Problem for 0 functions applied is somewhere between lines 30000 and 35000 in the .pat file because if I delete all lines after line 30000, it generates valid .sig file and then applies this sig file appropriately on the executable:

After I delete lines 30000 - 35000 in the pat file and then delete all the lines after 50000, it generates valid .sig files and applies the sig file appropriately on the executable:

As you can see on the screenshots, no meaningful logs is generated after applying new FLIRT signature in IDA, just Plan FLIRT signature: Unnamed sample library in the case of success and failure to apply any function signatures.

shellcode_hash_search.py has some logic errors

Manually executing shellcode_hash_search on IDA7.5 and Python3 shows some logic problems:

In promptForSearchTypes()

if using_ida7api:
            if idaapi.ASKBTN_YES == idaapi.ask_yn(idaapi.ASKBTN_YES, str('Search for DWORD array of hashes?')):
                self.params.searchDwordArray = True
            if idaapi.ASKBTN_YES == idaapi.ask_yn(idaapi.ASKBTN_YES, str('Search for DWORD array of hashes?')):
                self.params.searchDwordArray = True
        else:
            if idc.AskYN(1, str('Search for push argument hash value?')) == 1:
                self.params.searchPushArgs = True
            if idc.AskYN(1, str('Search for DWORD array of hashes?')) == 1:
                self.params.searchDwordArray = True

I presume the first call should ask for push arguments and set searchPushArgs accordingly.

def promptForHashTypes()

hashTypes = self.dbstore.getAllHashTypes()
if len(self.params.hashTypes) == 0:
raise RuntimeError('No hashes selected')
# we used to prompt y/n for each one. too obnoxious, just force all hashes
self.params.hashTypes = hashTypes

This code will always raise an exception as self.params.hashTypes is always 0 in this case

Will do a fork and fix the code and set a pull request if you'd like

Problems with MSDN_Crawler (MSDN Annotation) - FEB/12/2016

A couple of weeks ago I’ve download the flare-ida and, among the scripts, I’ve tested the msdn_crawler. However, I have faced several problems with the parsing process. Additionally, I have tested the MSDN Annotation (MSDN Crawler) in two distinct environments. Few details follow:

1. The tested environment were WinXP SP2 and Win7x64.
2. IDA 6.9 installed (with tilib.exe or tilib64.exe depending of the system)
3. Python version 2.7.9
4. The original script from flare-ida package (from https://github.com/fireeye/flare-ida) did not worked for me. Instead of, I downloaded the modified script from http://blog.depressedmarvin.com/upload/2015/02/10/msdn_crawler.py (Chinese version) and it worked partially (it failed in processing 197 files).
5. I extracted all help files to Z:\Host_Drive_C\extracted_msdn. Question: Is there some special method (I used 7-zip) to accomplish it? I don' t think so because the Chinese script partialy works with the same extraction.
6. I installed the BeautifulSoup package (version 3, because version 4 doesn’t work) by running the following commands: pip install beautifulsoup
7. I executed the following command:
   C:\flare-ida-master>python -v MSDN_crawler\msdn_crawler.py "Z:\Host_Drive_C\extracted_msdn" "c:\Program Files (x86)\IDA 6.9\tilib64.exe" "C:\Program Files (x86)\IDA 6.9\til\pc"
8. The output was:

installing zipimport hook
import zipimport  builtin
 installed zipimport hook
 C:\Python27\lib\site.pyc matches C:\Python27\lib\site.py
import site  precompiled from C:\Python27\lib\site.pyc
 C:\Python27\lib\os.pyc matches C:\Python27\lib\os.py
import os  precompiled from C:\Python27\lib\os.pyc
import errno  builtin
import nt  builtin
 C:\Python27\lib\ntpath.pyc matches C:\Python27\lib\ntpath.py
import ntpath  precompiled from C:\Python27\lib\ntpath.pyc
 C:\Python27\lib\stat.pyc matches C:\Python27\lib\stat.py
import stat  precompiled from C:\Python27\lib\stat.pyc
 C:\Python27\lib\genericpath.pyc matches C:\Python27\lib\genericpath.py
import genericpath  precompiled from C:\Python27\lib\genericpath.pyc
 C:\Python27\lib\warnings.pyc matches C:\Python27\lib\warnings.py
import warnings  precompiled from C:\Python27\lib\warnings.pyc
 C:\Python27\lib\linecache.pyc matches C:\Python27\lib\linecache.py
import linecache  precompiled from C:\Python27\lib\linecache.pyc
 C:\Python27\lib\types.pyc matches C:\Python27\lib\types.py
import types  precompiled from C:\Python27\lib\types.pyc
 C:\Python27\lib\UserDict.pyc matches C:\Python27\lib\UserDict.py
import UserDict  precompiled from C:\Python27\lib\UserDict.pyc
 C:\Python27\lib\_abcoll.pyc matches C:\Python27\lib\_abcoll.py
import _abcoll  precompiled from C:\Python27\lib\_abcoll.pyc
 C:\Python27\lib\abc.pyc matches C:\Python27\lib\abc.py
import abc  precompiled from C:\Python27\lib\abc.pyc
 C:\Python27\lib\_weakrefset.pyc matches C:\Python27\lib\_weakrefset.py
import _weakrefset  precompiled from C:\Python27\lib\_weakrefset.pyc
import _weakref  builtin
 C:\Python27\lib\copy_reg.pyc matches C:\Python27\lib\copy_reg.py
import copy_reg  precompiled from C:\Python27\lib\copy_reg.pyc
 C:\Python27\lib\traceback.pyc matches C:\Python27\lib\traceback.py
import traceback  precompiled from C:\Python27\lib\traceback.pyc
 C:\Python27\lib\sysconfig.pyc matches C:\Python27\lib\sysconfig.py
import sysconfig  precompiled from C:\Python27\lib\sysconfig.pyc
 C:\Python27\lib\re.pyc matches C:\Python27\lib\re.py
import re  precompiled from C:\Python27\lib\re.pyc
 C:\Python27\lib\sre_compile.pyc matches C:\Python27\lib\sre_compile.py
import sre_compile  precompiled from C:\Python27\lib\sre_compile.pyc
import _sre  builtin
 C:\Python27\lib\sre_parse.pyc matches C:\Python27\lib\sre_parse.py
import sre_parse  precompiled from C:\Python27\lib\sre_parse.pyc
 C:\Python27\lib\sre_constants.pyc matches C:\Python27\lib\sre_constants.py
import sre_constants  precompiled from C:\Python27\lib\sre_constants.pyc
 C:\Python27\lib\locale.pyc matches C:\Python27\lib\locale.py
import locale  precompiled from C:\Python27\lib\locale.pyc
import encodings  directory C:\Python27\lib\encodings
 C:\Python27\lib\encodings\__init__.pyc matches C:\Python27\lib\encodings\__init__.py
import encodings  precompiled from C:\Python27\lib\encodings\__init__.pyc
 C:\Python27\lib\codecs.pyc matches C:\Python27\lib\codecs.py
import codecs  precompiled from C:\Python27\lib\codecs.pyc
import _codecs  builtin
 C:\Python27\lib\encodings\aliases.pyc matches C:\Python27\lib\encodings\aliases.py
import encodings.aliases  precompiled from C:\Python27\lib\encodings\aliases.pyc
import operator  builtin
 C:\Python27\lib\functools.pyc matches C:\Python27\lib\functools.py
import functools  precompiled from C:\Python27\lib\functools.pyc
import _functools  builtin
import _locale  builtin
 C:\Python27\lib\encodings\cp1252.pyc matches C:\Python27\lib\encodings\cp1252.py
import encodings.cp1252  precompiled from C:\Python27\lib\encodings\cp1252.pyc
Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
 C:\Python27\lib\csv.pyc matches C:\Python27\lib\csv.py
import csv  precompiled from C:\Python27\lib\csv.pyc
import _csv  builtin
import cStringIO  builtin
import logging  directory C:\Python27\lib\logging
 C:\Python27\lib\logging\__init__.pyc matches C:\Python27\lib\logging\__init__.py
import logging  precompiled from C:\Python27\lib\logging\__init__.pyc
import time  builtin
 C:\Python27\lib\weakref.pyc matches C:\Python27\lib\weakref.py
import weakref  precompiled from C:\Python27\lib\weakref.pyc
import thread  builtin
 C:\Python27\lib\threading.pyc matches C:\Python27\lib\threading.py
import threading  precompiled from C:\Python27\lib\threading.pyc
 C:\Python27\lib\collections.pyc matches C:\Python27\lib\collections.py
import collections  precompiled from C:\Python27\lib\collections.pyc
import _collections  builtin
 C:\Python27\lib\keyword.pyc matches C:\Python27\lib\keyword.py
import keyword  precompiled from C:\Python27\lib\keyword.pyc
 C:\Python27\lib\heapq.pyc matches C:\Python27\lib\heapq.py
import heapq  precompiled from C:\Python27\lib\heapq.pyc
import itertools  builtin
import _heapq  builtin
 C:\Python27\lib\atexit.pyc matches C:\Python27\lib\atexit.py
import atexit  precompiled from C:\Python27\lib\atexit.pyc
 C:\Python27\lib\pprint.pyc matches C:\Python27\lib\pprint.py
import pprint  precompiled from C:\Python27\lib\pprint.pyc
 C:\Python27\lib\site-packages\BeautifulSoup.pyc matches C:\Python27\lib\site-packages\BeautifulSoup.py
import BeautifulSoup  precompiled from C:\Python27\lib\site-packages\BeautifulSoup.pyc
 C:\Python27\lib\__future__.pyc matches C:\Python27\lib\__future__.py
import __future__  precompiled from C:\Python27\lib\__future__.pyc
 C:\Python27\lib\sgmllib.pyc matches C:\Python27\lib\sgmllib.py
import sgmllib  precompiled from C:\Python27\lib\sgmllib.pyc
 C:\Python27\lib\markupbase.pyc matches C:\Python27\lib\markupbase.py
import markupbase  precompiled from C:\Python27\lib\markupbase.pyc
 C:\Python27\lib\htmlentitydefs.pyc matches C:\Python27\lib\htmlentitydefs.py
import htmlentitydefs  precompiled from C:\Python27\lib\htmlentitydefs.pyc
 C:\flare-ida-master\MSDN_crawler\extract_til_constant_info.pyc matches C:\flare-ida-master\MSDN_crawler\extract_t
il_constant_info.py
import extract_til_constant_info  precompiled from C:\flare-ida-master\MSDN_crawler\extract_til_constant_info.pyc
 C:\Python27\lib\subprocess.pyc matches C:\Python27\lib\subprocess.py
import subprocess  precompiled from C:\Python27\lib\subprocess.pyc
import gc  builtin
import msvcrt  builtin
import _subprocess  builtin
MSDN crawler based on zynamics msdn-crawler - Copyright 2010
 C:\Python27\lib\encodings\ascii.pyc matches C:\Python27\lib\encodings\ascii.py
import encodings.ascii  precompiled from C:\Python27\lib\encodings\ascii.pyc
Traceback (most recent call last):
  File "MSDN_crawler\msdn_crawler.py", line 413, in <module>
    main()
  File "MSDN_crawler\msdn_crawler.py", line 398, in main
    (file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
  File "MSDN_crawler\msdn_crawler.py", line 371, in parse_files
    result = parse_file(os.path.join(root, file), const_enum)
  File "MSDN_crawler\msdn_crawler.py", line 276, in parse_file
    return parse_new_style(file, content, const_enum)
  File "MSDN_crawler\msdn_crawler.py", line 183, in parse_new_style
    parsed_html.find_all(width='60%')]
TypeError: 'NoneType' object is not callable

1. By using the Chinese version, the output was (during the first lines):

…..

'ascii' codec can't encode character u'\xa0' in position 174: ordinal not in range(128) - ERROR: virtualallocex.htm

And the last three lines were:

Parsed 341767 files
Extracted information about 34214 functions
ERROR processing 197 files

1. I renamed the msdn_data_nn.xml file to msdn_data.xml and copied it to IDA 6.8 directory. After having configured the plugin (executing the annotate_IDB_MSDN.py script) and having tested it with a malware, the following message came up at IDA Pro output section (probably, because of 197 errors):

MSDN Annotator SUMMARY
----------------------------------
Functions not found
-------------------
  1            LCMapStringW
  2            SetHandleCount
  3            LCMapStringA
  4            MultiByteToWideChar
  5            WideCharToMultiByte
  6            InternetOpenUrlA
  7            VirtualAlloc

I’ve already seen someone reporting the same problem, but I wasn’t able to find the appropriate explanation.

Please, could you help me?

Thank you.

AB.

idb2pat sigmake FATAL: Bad xdigit: error

I get the FATAL: Bad xdigit: error error when trying to use sigmake on a file with long lines, such as this one:
https://slexy.org/view/s2UyxIhZQR

Here's the .pat file with quite a few problem lines like that if you want to try it yourself: https://www.dropbox.com/s/ym8mfm0k037gbd4/S05_TestingGrounds-Win64-Shipping.pat?dl=0

It's a pat file generated from a compiled executable with Unreal Engine 4 for Win64 with .pdb supplied. And UE4 uses MSBuild, iirc, if that helps in any way.

I use updated idb2pat from #105 but I get the exact same issue on non-updated idb2pat on IDA 7.2 with all the default settings.

I think the problem is here: 48894C2408B808160100E8D11A2A02482BE0C78424D086000002000000488B84 FF ABF9 60247. Fourth member with numbers - 60247. Usually there are 4 digits in most lines but on the lines with that exact error it's always more than 4 digits for the fourth member. I don't know how to fix that, I think I'll go try a version from #98 with fixes, they might have fixed exactly this?

Removing all the lines with more than 4 digits on the fourth member does let sigmake tool finish generating .sig file.

No table with addresses is getting printed in ironstring, and so many "DEBUG:root..." in the output

so I'm using ironstring to print the stack string of a sample, but the problem is the string shown in the summary don't have an address related to them, so i can't find them where in the binary they are easily, and there is no table being printed like shown in the readme, just a load of "DEBUG:root..." stuff and no table, what's the problem?

checked the flare emu python files as well and the default verbose is set to 0 so not sure what these DEBUG:root stuff is all about

here's the end of output :

...
DEBUG:root:target 0040B94C hit
DEBUG:root:we missed our target! bailing out of this function..
DEBUG:root:unmapped 0045D000 to 0045EFFF
DEBUG:root:mapping 00002000 bytes @0045D000
DEBUG:root:running on 0x43C3EE
DEBUG:root:getting paths to 0040E539, 1 of 1 targets
DEBUG:root:run #1, 1 targets remaining: 0040E539 (2 paths)
DEBUG:root:emulating path #1 of 2 from 0043C3EE to 0040E539 via basic blocks: [0, 1, 2, 3]
DEBUG:root:0040E515 is outside of block #0 (0043C3EE -> 0043C3F3), forcing PC to 0040E515
DEBUG:root:0040E538 is outside of block #1 (0040E515 -> 0040E51E), forcing PC to 0040E520
DEBUG:root:extracting stackstrings at checkpoint: 0x40E522, stacksize: 0x4
DEBUG:root:extracting global stackstrings at checkpoint: 0x40E522
DEBUG:root:extracting stackstrings at checkpoint: 0x40E52A, stacksize: 0xC
DEBUG:root:extracting global stackstrings at checkpoint: 0x40E52A
DEBUG:root:0040E538 is outside of block #2 (0040E520 -> 0040E537), forcing PC to 0040E538
DEBUG:root:target 0040E539 hit
DEBUG:root:we missed our target! bailing out of this function..
DEBUG:root:emulating path #2 of 2 from 0043C3EE to 0040E539 via basic blocks: [0, 1, 3]
DEBUG:root:0040E515 is outside of block #0 (0043C3EE -> 0043C3F3), forcing PC to 0040E515
DEBUG:root:0040E538 is outside of block #1 (0040E515 -> 0040E51E), forcing PC to 0040E538
DEBUG:root:target 0040E539 hit
DEBUG:root:we missed our target! bailing out of this function..
DEBUG:root:unmapped 0045D000 to 0045EFFF
DEBUG:root:mapping 00002000 bytes @0045D000
DEBUG:root:running on 0x43C3F8
DEBUG:root:getting paths to 0043C400, 1 of 2 targets
DEBUG:root:getting paths to 0043C406, 2 of 2 targets
DEBUG:root:run #1, 2 targets remaining: 0043C406 (1 paths)
DEBUG:root:emulating path #1 of 1 from 0043C3F8 to 0043C406 via basic blocks: [0, 1]
DEBUG:root:target 0043C400 found on the way to 0043C406
DEBUG:root:0043C406 is outside of block #0 (0043C3F8 -> 0043C400), forcing PC to 0043C406
DEBUG:root:target 0043C406 hit
DEBUG:root:run #2, 2 targets remaining: 0043C400 (1 paths)
DEBUG:root:emulating path #1 of 1 from 0043C3F8 to 0043C400 via basic blocks: [0]
DEBUG:root:target 0043C400 hit
DEBUG:root:unmapped 0045D000 to 0045EFFF
DEBUG:root:mapping 00002000 bytes @0045D000

ironstrings summary
------------------------
Ran successfully on 1296/1298 functions
Found 15 stackstrings
Commented 15 stackstrings
Encountered 2 errors
 - Error analyzing function 0x410442: Invalid memory read (UC_ERR_READ_UNMAPPED)
 - Error analyzing function 0x412C3E: Invalid memory read (UC_ERR_READ_UNMAPPED)

Recovered 10 unique stackstrings
------------------------
strings..
...
Finished ironstrings stackstring deobfuscation after 99.51 seconds

shellcode_hash_search crash with 64 bit IDA (7.3)

The shellcode_hash_search plugin crashes during execution due to an apparently 64 bit integer size problem with the underlying SQLite DB.

---------------------------------------------------------------------------------------------
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40) [MSC v.1500 64 bit (AMD64)] 
IDAPython 64-bit v7.3.0 final (serial 0) (c) The IDAPython Team <[email protected]>
---------------------------------------------------------------------------------------------
shellcode_hash: Starting up
shellcode_hash: Processing current segment only: 0x00000000 - 0x00000207
shellcode_hash: 0x000000a3: ror13AddHash32Dll:0xe553a458 kernel32.dll!VirtualAlloc
shellcode_hash: 0x000000bd: ror13AddHash32Dll:0x0726774c kernel32.dll!LoadLibraryA
shellcode_hash: 0x0000012f: ror13AddHash32Dll:0xc99cc96a dnsapi.dll!DnsQuery_A
shellcode_hash: 0x00000198: ror13AddHash32Dll:0x56a2b5f0 kernel32.dll!ExitProcess
shellcode_hash: 0x000001a4: ror13AddHash32Dll:0xe035f044 kernel32.dll!Sleep
shellcode_hash: 0x000001e4: ror13AddHash32Dll:0xcc8e00f4 kernel32.dll!lstrlenA
shellcode_hash: Exception: Python int too large to convert to SQLite INTEGER
Traceback (most recent call last):
  File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 342, in lookForOpArgs
    hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
  File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 192, in getSymbolByTypeHash
    cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER
shellcode_hash: 0x000000bc: ror13AddHash32Dll:0x0726774c kernel32.dll!LoadLibraryA
shellcode_hash: Exception: Python int too large to convert to SQLite INTEGER
Traceback (most recent call last):
  File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 342, in lookForOpArgs
    hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
  File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 192, in getSymbolByTypeHash
    cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER
shellcode_hash: 0x00000197: ror13AddHash32Dll:0x56a2b5f0 kernel32.dll!ExitProcess
shellcode_hash: Exception: Python int too large to convert to SQLite INTEGER
Traceback (most recent call last):
  File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 342, in lookForOpArgs
    hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
  File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 192, in getSymbolByTypeHash
    cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER
shellcode_hash: Exception: Python int too large to convert to SQLite INTEGER
Traceback (most recent call last):
  File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 342, in lookForOpArgs
    hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
  File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 192, in getSymbolByTypeHash
    cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER
shellcode_hash: Done

Apparently the problem arises due to the storing of the hashes as INT in the SQLite not as string?

Please ref: https://www.google.com/search?q=64+bit+"Python+int+too+large+to+convert+to+SQLite+INTEGER"

MSDN_Crawler: many tilib.exe errors

as reported by @wzr in #2.

In the meantime I have included a try: except: block around

   if file.endswith('htm'):
        file_counter += 1
        try:
            result = parse_file(os.path.join(root, file), const_enum)
        except:
            error_files.append(file)
        if result:
            results.append(result)
    print 'ERROR processing %d files' % len(error_files)

Which resulted in:

ERROR processing 21828 files
Parsed 329993 files
Extracted information about 15263 functions

Does this correspond with your numbers?

Ignore unidientified functions sub_xxxx

Feature to ignore all not-yet-identified functions, named sub_xxxx
in idb2pat

idb2pat fix bugs

Hi William, I am TQN, the old author of idb2sig plugin.
I need to update my plugin, so I used your script to test the result and compared the resulting .pat files with IDA flair tool.
After some work, I found some bugs in the old plugin code.
So I have fix those bugs in your code. But it still have a small bug:
find_ref_loc will return BADADDR for the following sequence:

48 8D 05 29 06 00 00         lea     rax, `vtable for'loader_failure_t+10h

with:

• dref_from = 0xC40
• ref = 0x619

But in byte pattern is 0x0629 (29 06) because of +10h
Hope you have a time and fix it.
Thanks and best regards.
My code:
https://github.com/HongThatCong/idb2sig

MSDN_MACROS not defined in apply_callee_type?

see https://github.com/fireeye/flare-ida/blob/4fdc6aeaf997576410afdd3f06e6020898e2a09d/python/flare/apply_callee_type.py#L218

stackstrings-plugin excepion

Failed while executing plugin_t.run():
Traceback (most recent call last):
File "C:/Program Files/IDA 7.0/plugins/stackstrings_plugin.py", line 67, in run
flare.stackstrings.main()
File "C:\Python27\Lib\flare\stackstrings.py", line 276, in main
vw = jayutils.loadWorkspace(filePath)
File "C:\Python27\Lib\flare\jayutils.py", line 341, in loadWorkspace
vw.loadFromFile(filename)
File "C:\Python27\Lib\vivisect_init_.py", line 2265, in loadFromFile
fname = mod.parseFile(self, filename, baseaddr=baseaddr)
File "C:\Python27\Lib\vivisect\parsers\blob.py", line 45, in parseFile
raise Exception('Blob loader requires arch option (-O viv.parsers.blob.arch="")')
Exception: Blob loader requires arch option (-O viv.parsers.blob.arch="

idb2pat - OverflowError on 64 bit applications.

The script works great on x86 applications but fails on x64 applications.

ERROR:idb2pat:make_func_sigs:Python int too large to convert to C long
Traceback (most recent call last):
File "X:/Software/IDA 6.8/python/flare/idb2pat.py", line 370, in make_func_sigs
sigs.append(make_func_sig(config, f))
File "X:/Software/IDA 6.8/python/flare/idb2pat.py", line 219, in make_func_sig
ref_loc = find_ref_loc(config, ea, ref)
File "X:/Software/IDA 6.8/python/flare/idb2pat.py", line 142, in find_ref_loc
for i in xrange(ea, get_item_end(ea) - config.pointer_size + 1):
OverflowError: Python int too large to convert to C long
ERROR:idb2pat:make_func_sigs:Failed to create signature for function at [...]

msdn_crawler.py not working on IDA for Linux (6.95)

Hello,

I have tried generating the the XML database file using the msdn_crawler.py on a Linux machine (Ubuntu 16.04 x64) and it seems to have failed with the following errors (for both tilib and tilib64)

python ./MSDN_crawler/msdn_crawler.py "/home/<username>/extracted1033/" "/opt/bin/ida-6.95/tilib" "/opt/bin/ida-6.95/til/pc"
MSDN crawler based on zynamics msdn-crawler - Copyright 2010
WARNING:til_extractor:Error calling tilib.exe with /opt/bin/ida-6.95/til/pc/nlm.til -- Command '['/opt/bin/ida-6.95/tilib', '-l', '/opt/bin/ida-6.95/til/pc/nlm.til']' returned non-zero exit status 126
...SNIP...
WARNING:til_extractor:Error calling tilib.exe with /opt/bin/ida-6.95/til/pc/ntapi.til -- Command '['/opt/bin/ida-6.95/tilib', '-l', '/opt/bin/ida-6.95/til/pc/ntapi.til']' returned non-zero exit status 126
Traceback (most recent call last):
  File "./MSDN_crawler/msdn_crawler.py", line 413, in <module>
    main()
  File "./MSDN_crawler/msdn_crawler.py", line 398, in main
    (file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
  File "./MSDN_crawler/msdn_crawler.py", line 357, in parse_files
    const_enum = extract_til_constant_info.main(tilib_exe, til_dir)
  File "/opt/bin/flare-ida-master/MSDN_crawler/extract_til_constant_info.py", line 95, in main
    for enum_name, enum in enums.iteritems():
UnboundLocalError: local variable 'enums' referenced before assignment

I have also tried with the Windows version of IDA and failed like others have pointed out in other issue threads.

MSDN crawler based on zynamics msdn-crawler - Copyright 2010
Traceback (most recent call last):
  File "MSDN_crawler\msdn_crawler.py", line 413, in <module>
    main()
  File "MSDN_crawler\msdn_crawler.py", line 398, in main
    (file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
  File "MSDN_crawler\msdn_crawler.py", line 371, in parse_files
    result = parse_file(os.path.join(root, file), const_enum)
  File "MSDN_crawler\msdn_crawler.py", line 276, in parse_file
    return parse_new_style(file, content, const_enum)
  File "MSDN_crawler\msdn_crawler.py", line 183, in parse_new_style
    parsed_html.find_all(width='60%')]
TypeError: 'NoneType' object is not callable 

I tried obtaining the modified version of the crawler and already generated database but it's no longer there.

Installing on Mac OSX

Any instructions on how to install on OSX? I tried following the general directions for the windows installation, but cant import either the plugins or vivisect, nor run any of them.
This is on ida 7.0
Thanks!

objc2_xrefs_helper erroneous selector replacement

Note to self for later...

python/flare/objc2_xrefs_helper.py can replace selectors erroneously.

Example: 4fe4b9560e99e33dabca553e2eeee510 (OSX XAgent), virtual addresses 0x100014F61 and 0x100014FA5

ID for each is an NSThread object held in r12 and later r13, and SEL is start, but objc2_xrefs_helper.py replaces this with __Keylogger_start (-[Keylogger start]). Here is the first case:

100014F58 call    cs:_objc_msgSend_ptr ; [NSThread initWithTarget:selector:object]
100014F58                         ; Creates NSThread targeting [BootXLoader postThread]
100014F5E mov     r12, rax
100014F61 mov     rsi, qword ptr cs:__Keylogger_start_ ; Keylogger - (void)start
100014F68 mov     rdi, r12
100014F6B call    cs:_objc_msgSend_ptr ; [r12 start]

Perhaps objc2_xrefs_helper.py doesn't realize this selector is only unique within this binary but also pertains to objects whose interfaces are not implemented in this binary?

annotate_IDB_MSDN.py not working w IDA 6.95

I tried running the plugin w the db file from the url https://github.com/mr-tz/flare-ida/blob/master/MSDN_data/msdn_data.zip

the plugin threw this error:
Traceback (most recent call last):
File "C:/bin/flare/annotate_IDB_MSDN.py", line 118, in on_ok_button
IDB_MSDN_Annotator.main(config)
File "C:/bin/flare\IDB_MSDN_Annotator_init_.py", line 521, in main
functions_map = parse_xml_data_files(msdn_data_dir)
File "C:/bin/flare\IDB_MSDN_Annotator_init_.py", line 484, in parse_xml_data_files
additional_functions = xml_parser.parse(xml_file)
File "C:/bin/flare\IDB_MSDN_Annotator\xml_parser.py", line 283, in parse
parser.parse(xmlfile)
File "C:\bin\Python27\Lib\xml\sax\expatreader.py", line 110, in parse
xmlreader.IncrementalParser.parse(self, source)
File "C:\bin\Python27\Lib\xml\sax\xmlreader.py", line 123, in parse
self.feed(buffer)
File "C:\bin\Python27\Lib\xml\sax\expatreader.py", line 217, in feed
self._err_handler.fatalError(exc)
File "C:\bin\Python27\Lib\xml\sax\handler.py", line 38, in fatalError
raise exception
xml.sax._exceptions.SAXParseException: C:/bin/flare\annotate_IDB_MSDN.py:1:2: not well-formed (invalid token)

sc_hashes.db: add process name database + filename database

Sometimes malware uses for example ror-13 to check process and file names. I suggest to eiter include popular process/file names to existing database with decoy library names "processes" "filesystem" or make a new database located at the same folder.

idb2pat.py issue on IDA 7.5

If you run idb2pat.py on IDA version 7.5, you'll get this error:

 error: Traceback (most recent call last):
  File "<string>", line 478, in <module>
  File "<string>", line 455, in main
  File "<string>", line 416, in get_pat_file
NameError: global name 'askfile_c' is not defined

Which is caused by newer version of IDA and renaming of the askfile_c member to ask_file in the ida_kernwin type, as explained in this porting guide by hex-rays.

There are many more errors like this one for many members in different types.

One possible solution I see is to have 2 versions of the file now: idb2pat.py and idb2pat_74plus.py for appropriate IDA version.

After updating the file and changing names for all the members(which changed names), I was able to succesfully generate .sig file for a simple test Hello World VC++ solution for Windows from a version with .pdb loaded for the same executable without the .pdb loaded:

#include <iostream>

void PrintHelloWorld()
{
    std::cout << "Hello World!\n";
}

int main()
{
    PrintHelloWorld();
}

shellcode hashes operand size issue

For

mov     r8d, 95902B19h

IDAPython returns (MSB is 1)

Python>get_operand_value(here(), 1)
0xffffffff95902b19

This values is then queried in the database and not found resulting in no annotation.

For mov r8d, 8FE060Ch IDAPython returns 0x8fe060c (MSB is 0) and everything works as expected.

PR suggestion for this incoming.

Callee Plugin Missing Some Arguments?

I noticed this using the Callee plugin today:

The push 1Ch isn't identified as the third argument to the function. Is it perhaps an issue with the lea between the instructions?

I applied the VirtualQuery signature by selecting the call, pressing Alt-J, and using the global function VirtualQuery.

shellcode-hashes - create enum of resolved values

Create an enum & apply it for all identified hash values.

MSDN_Crawler issue

I stumble on this error, but in .py file, I don't find any kind of that string appear in source code, can you review it for me, please?

Simple way to fix ImportError (PyQt5 and PySide conflict)?

Above image probably illustrates error better than I can explain.

another IDA 7.0 issue - MSDN_crawler

debugging output from Visual Studio (paths shorten):

MSDN_crawler.error.txt

The script was first invoked from the commadline as:
python.exe msdn_crawler.py C:\TEMP\msdn D:\devapps\IDA\util\idasdk70\bin\tilib70\win\tilib64.exe D:\devapps\IDA\IDA.7.0\til\pc
where C:\TEMP\msdn is expanded MSDN docs from Visual Studio 2017

2 questions:

• Do you think an IDA 6.8 MSDN_crawler-database will work in IDA 7.0?
• Will the script work with current MSDN html files?

objc2_analyzer.py cannot work for IDA 7.5

There are some changes in IDA 7.*. For example:

1. "ADRP Rx, label@PAGE; ADD Rx, Rx, label@PAGEOFF" is replaced by "ADRL Rx, label"

2. without idaapi.auto_wait(), python script cannot get Xref.

Please update objc2_analyzer.py to support IDA 7.*. Thank you.

'itertools.count' object has no attribute 'next'

Python 3.9.4. x64 - IDA 7.5 SP3

Pat generated from idb2pat has some issues with sigmake

I'm running into a few errors when I run sigmake on the generated pat file.
The most common one of which is "FATAL: Name is too long.", but also once in a while, "FATAL: Bad xdigit:"

My current workaround for both errors is just deleting the specified lines from the pat file.
(idb2pat was ran on IDA 7)

idb2pat in ida6.8 with "NoneType" error

I was using an earlier commit(Jan. 2016) of idb2pat.py in IDA 6.8. However I came across an error as following:

"idb2pat.py:'NoneType' object has no attribute 'rfind'
Traceback(most recent call last)
File D IDA 6.8\python idaapi py line 601, in IDAPython ExecScript execfile (script, g)
File D: /IDA 6.8/plugins/idb2pat py line 478, in <module> 
main()
File D: /DA 6.8/plugins/idb2pat py, line 449, in main 
update config(c)
File D: /Da 6.8/plugins/idb2pat py, line 426, in update config 
name, extension =oS path.splitext(get input file path())
File "C: \Python2/(Lib\ntpath py line 190, in spliter return genericpath. splitext(p, sep, altsep, extsep)
File C:\Python2/Lib\genericpath py", line 91, in splitext
seplndex= p.find(sep)
AttributeError: None Type object has no attribute find"

I put idb2pat under ./plugin. Is it correct?

Annotate API calls via wrapper/thunk functions

I tried to run the MSDN Annotations plugin but it turns out that it does not work well with IDA 7.1 Pro.
It creates a Segment .msdn with content but it does not change the Symbols as expected.

idb2pat RuntimeError: zrange only iterates from smaller to bigger numbers only

RuntimeError: zrange only iterates from smaller to bigger numbers only: 9531258, 9531255
ERROR:idb2pat:make_func_sigs:Failed to create signature for function at 0x916f20L (golang_org_x_net_html_atom_match)
INFO:idb2pat:make_func_sigs:[ 10372 / 10380 ] golang_org_x_net_html_atom_Lookup 0x916f90L
ERROR:idb2pat:make_func_sigs:zrange only iterates from smaller to bigger numbers only: 9531361, 9531358
Traceback (most recent call last):
  File "D:/IDAPro6.8/python/idb2pat.py", line 392, in make_func_sigs
    sigs.append(make_func_sig(config, f))
  File "D:/IDAPro6.8/python/idb2pat.py", line 240, in make_func_sig
    ref_loc = find_ref_loc(config, ea, ref)
  File "D:/IDAPro6.8/python/idb2pat.py", line 163, in find_ref_loc
    for i in zrange(ea, get_item_end(ea) - config.pointer_size):
  File "D:/IDAPro6.8/python/idb2pat.py", line 37, in zrange
    raise RuntimeError("zrange only iterates from smaller to bigger numbers only: {:d}, {:d}".format(start, end))

idb2pat.py: no definition of "loc" variable lines ~260-267

I think "loc" got lost in the merge - in the previous version, "loc" was defined as "loc = 32" prior to the loop at line 260 which contains this line:

crc_data[loc - 32] = get_byte(func.startEA + loc)

Since the loc definition was removed, the module is throwing "NameError: global name 'loc' is not defined"

Thanks for the great scripts!
Erika

Possible problem with 64 bit code (find_ref_loc fucntion)?

I'd also like to create flirt libraries for Go code (64 bit) - however, in first tests, only around 30% matched. I'm not sure where the problem exactly is, but it could be here:

    ...
    if isCode(getFlags(ea)):
        for i in zrange(ea, max(ea, 1 + get_item_end(ea) - config.pointer_size)):
            if get_long(i) == ref:
                return i

That code should find outgoing data or code references so they can be excluded from CRC32 checksums and signature and assumes these to be 32- or 64-bit values depending or architecture, not really sure about it... but it seems to fail with 64-bit instructions like

48 8B 05 F6 FF A3 00                    mov     rax, cs:main_something

as this is a 32-bit relative offset (0xa3fff6) in 64 bit code. Actually I think all references in 64 bit code is 32 bit relative... In these cases, nothing is put into variable_bytes and the offset is added to crc32, which it probably should not - if I understand the code correctly.

Another small thing I find odd is

    # this will be either " :%04d %s" or " :%08d %s"
    public_format = " :%%0%dX %%s" % (config.pointer_size)

Shouldn't this be config.pointer_size*2 as each character only covers one nibble, so each byte requires 2 characters? Or is this a specific feature of Ida that these values only cover half the bitsize?

idb2pat, is it working for MIPS ?

Hello, I'm trying to use your script on a MIPSL binary and I'm getting this error.

ERROR:Python int too large to convert to C long

StackStrings, error when using with binary/shellcode files

Error
I get the error below when trying to use StackStrings on a shellcode blob, however it does work (excellently!) when I use it on normal PE files.

Error:

Failed while executing plugin_t.run():
Traceback (most recent call last):
  File "C:/Program Files/IDA 7.1/plugins/stackstrings_plugin.py", line 60, in run
    flare.stackstrings.main()
  File "C:/Program Files/IDA 7.1/plugins\flare\stackstrings.py", line 250, in main
    vw = jayutils.loadWorkspace(filePath)
  File "C:/Program Files/IDA 7.1/plugins\flare\jayutils.py", line 247, in loadWorkspace
    vw.loadFromFile(filename)
  File "C:\python27-x64\lib\site-packages\vivisect\__init__.py", line 2140, in loadFromFile
    fname = mod.parseFile(self, filename)
  File "C:\python27-x64\lib\site-packages\vivisect\parsers\blob.py", line 44, in parseFile
    raise Exception('Blob loader *requires* arch option (-O viv.parsers.blob.arch="<archname>")')
Exception: Blob loader *requires* arch option (-O viv.parsers.blob.arch="<archname>")

Cause
I believe the error is because in blob.py parseFile() tries to work out the architecture based on magic bytes which obviously aren't there, since the architecture is needed for emulation it throws an exception.

Solution
In jaysutils.py when loading a Vivisect workspace if the IDA loader is a Binary one then either ask the user the architecture or base it on info from the IDA API, code in https://reverseengineering.stackexchange.com/a/11398 gives you that. Then explicitly tell Vivisect the architecture.

Python scripts: Version check IDA prior to using idaapii.require

Via Minh Triet Pham Tran

Hello, I think for the flare-ida scripts, you should have a check for the
version of IDA (>=6.5 or <=6.4) because idaapi.require is introduced from
IDA >6.4 as mentioned in this article:
http://www.hexblog.com/?p=749
The scripts including idaapi.require:
https://github.com/fireeye/flare-ida/blob/master/python/flare/annotate_IDB_MSDN.py
https://github.com/fireeye/flare-ida/blob/master/plugins/shellcode_hashes_search_plugin.py
https://github.com/fireeye/flare-ida/blob/master/plugins/stackstrings_plugin.py
https://github.com/fireeye/flare-ida/blob/master/plugins/apply_callee_type_plugin.py

MSDN Annotations standalone

Which files do I need in order to let "MSDN Annotations" run standalone without the other plugins?

_TOKEN_INFORMATION_CLASS: failed to add constant

When I run the script I get the following error:

_TOKEN_INFORMATION_CLASS: failed to add constant TokenUser=1 (0x1)
_TOKEN_INFORMATION_CLASS: failed to add constant TokenGroups=2 (0x2)
_TOKEN_INFORMATION_CLASS: failed to add constant TokenPrivileges=3 (0x3)
_TOKEN_INFORMATION_CLASS: failed to add constant TokenOwner=4 (0x4)