checkpointsw / karta

Karta - source code assisted fast binary matching plugin for IDA

 /$$   /$$                       /$$              
| $$  /$$/                      | $$              
| $$ /$$/   /$$$$$$   /$$$$$$  /$$$$$$    /$$$$$$ 
| $$$$$/   |____  $$ /$$__  $$|_  $$_/   |____  $$
| $$  $$    /$$$$$$$| $$  \__/  | $$      /$$$$$$$
| $$\  $$  /$$__  $$| $$        | $$ /$$ /$$__  $$
| $$ \  $$|  $$$$$$$| $$        |  $$$$/|  $$$$$$$
|__/  \__/ \_______/|__/         \___/   \_______/

Purpose

"Karta" (Russian for "Map") is an IDA Python plugin that identifies and matches open-sourced libraries in a given binary. The plugin uses a unique technique that enables it to support huge binaries (>200,000 functions), with almost no impact on the overall performance.

The matching algorithm is location-driven. This means that it's main focus is to locate the different compiled files, and match each of the file's functions based on their original order within the file. This way, the matching depends on K (number of functions in the open source) instead of N (size of the binary), gaining a significant performance boost as usually N >> K.

We believe that there are 3 main use cases for this IDA plugin:

1. Identifying a list of used open sources (and their versions) when searching for a useful 1-Day
2. Matching the symbols of supported open sources to help reverse engineer a malware
3. Matching the symbols of supported open sources to help reverse engineer a binary / firmware when searching for 0-Days in proprietary code

Read The Docs

https://karta.readthedocs.io/

Installation (Python 3 & IDA >= 7.4)

For the latest versions, using Python 3, simply git clone the repository and run the setup.py install script. Python 3 is supported since versions v2.0.0 and above.

Installation (Python 2 & IDA < 7.4)

As of the release of IDA 7.4, Karta is only actively developed for IDA 7.4 or newer, and Python 3. Python 2 and older IDA versions are still supported using the release version v1.2.0, which is most probably going to be the last supported version due to python 2.X end of life.

Identifier

Karta's identifier is a smaller plugin that identifies the existence, and fingerprints the versions, of the existing (supported) open source libraries within the binary. No more need to reverse engineer the same open-source library again-and-again, simply run the identifier plugin and get a detailed list of the used open sources. Karta currently supports more than 10 open source libraries, including:

• OpenSSL
• Libpng
• Libjpeg
• NetSNMP
• zlib
• Etc.

Matcher

After identifying the used open sources, one can compile a .JSON configuration file for a specific library (libpng version 1.2.29 for instance). Once compiled, Karta will automatically attempt to match the functions (symbols) of the open source in the loaded binary. In addition, in case your open source used external functions (memcpy, fread, or zlib_inflate), Karta will also attempt to match those external functions as well.

Folder Structure

• src: source directory for the plugin
• configs: pre-supplied *.JSON configuration files (hoping the community will contribute more)
• compilations: compilation tips for generating the configuration files, and lessons from past open sources
• docs: sphinx documentation directory

Additional Reading

• https://research.checkpoint.com/karta-matching-open-sources-in-binaries/
• https://research.checkpoint.com/thumbs-up-using-machine-learning-to-improve-idas-analysis

Credits

This project was developed by me (see contact details below) with help and support from my research group at Check Point (Check Point Research).

Contact (Updated)

This repository was developed and maintained by me, Eyal Itkin, during my years at Check Point Research. Sadly, with my departure of the research group, I will no longer be able to maintain this repository. This is mainly because of the long list of requirements for running all of the regression tests, and the IDA Pro versions that are involved in the process.

Please accept my sincere apology.

@EyalItkin

karta's Issues

[13/08/2022 09:07:54] - Karta - WARNING: Found multiple instances of "libpng" - multiple instances are not supported right now [13/08/2022 09:08:02] - Karta - WARNING: Found multiple instances of "zlib" - multiple instances are not supported right now

[13/08/2022 09:07:54] - Karta - WARNING: Found multiple instances of "libpng" - multiple instances are not supported right now
[13/08/2022 09:08:02] - Karta - WARNING: Found multiple instances of "zlib" - multiple instances are not supported right now

Support Python 3.X

The code was developed in Python 2.7.X, and probably won't work out of the box for Python 3.X users. Should update the code so it will support all Python versions.

A problem on adding support for a new open source

when I use karta_analyze_src.py to add new open source , karta told me that there are not anchor function, how should I do next?

Supporting multiple versions in the same binary

In the current state, when the identifier for library X finds more than one match, it refuses to match the library because the two versions will collide and mess up the file map logic.

Should attempt to identify the cluster for each version, and zoom-in on each cluster independently, to support this edge case. Turns out that some programs (PDF Viewers) use multiple versions of zlib (for instance).

Create a community collection of configuration and identifiers

I think it is a good idea to have a community-driven collection of Identifiers and Configurations for Karta. When such a collection get bigger, it will increase the chances that the configuration you are looking for was already created by a community member.

I think that such a solution can be achieved in several ways:

1. Encourage users to push their configs and identifier scripts to Karta/configs and Karta/src/libs
2. Create a new repository to store configs and libs, and use it as a submodule from inside Karta
3. Create an external repository (similar to FLIRTDB) to store the configs and identifiers.

thumbs_up_ELF crashing on ARM binary

When I run thumbs_up_ELF on a ARM 32 bit binary I get the following exception:

C:\Users\pedro.peck\Desktop\Karta\src\thumbs_up\thumbs_up_ELF.py: Expected 2D array, got 1D array instead:
array=[].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
Traceback (most recent call last):
File "C:\Program Files\IDA 7.2\python\ida_idaapi.py", line 572, in IDAPython_ExecScript
execfile(script, g)
File "C:/Users/pedro.peck/Desktop/Karta/src/thumbs_up/thumbs_up_ELF.py", line 186, in
main()
File "C:/Users/pedro.peck/Desktop/Karta/src/thumbs_up/thumbs_up_ELF.py", line 178, in main
result = analysisStart(analyzer, code_segments, data_segments)
File "C:/Users/pedro.peck/Desktop/Karta/src/thumbs_up/thumbs_up_ELF.py", line 43, in analysisStart
if not gatherIntel(analyzer, scs, sds):
File "C:/Users/pedro.peck/Desktop/Karta/src/thumbs_up\analyzer_utils.py", line 20, in gatherIntel
if not analyzer.func_classifier.calibrateFunctionClassifier(scs):
File "C:/Users/pedro.peck/Desktop/Karta/src/thumbs_up\utils\function.py", line 217, in calibrateFunctionClassifier
clf.fit(X_train, Y_train)
File "C:\Python27\lib\site-packages\sklearn\ensemble\forest.py", line 250, in fit
X = check_array(X, accept_sparse="csc", dtype=DTYPE)
File "C:\Python27\lib\site-packages\sklearn\utils\validation.py", line 552, in check_array
"if it contains a single sample.".format(array))
ValueError: Expected 2D array, got 1D array instead:
array=[].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

Let me know if anything is unclear! :)

installation problem (IDA 7.4 Python 3.8 @ Windows 10)

Following the README Python3 instructions led to the following error at installation:

Traceback (most recent call last):
  File "C:\Users\David\Desktop\Karta-master\setup.py", line 9, in <module>
    setup(name='Karta',
  File "C:\Python\Python38\lib\site-packages\setuptools\__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "C:\Python\Python38\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "C:\Python\Python38\lib\distutils\dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "C:\Python\Python38\lib\distutils\dist.py", line 985, in run_command
    cmd_obj.run()
  File "C:\Python\Python38\lib\site-packages\setuptools\command\install.py", line 67, in run
    self.do_egg_install()
  File "C:\Python\Python38\lib\site-packages\setuptools\command\install.py", line 117, in do_egg_install
    cmd.run()
  File "C:\Python\Python38\lib\site-packages\setuptools\command\easy_install.py", line 418, in run
    self.easy_install(spec, not self.no_deps)
  File "C:\Python\Python38\lib\site-packages\setuptools\command\easy_install.py", line 660, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "C:\Python\Python38\lib\site-packages\setuptools\command\easy_install.py", line 707, in install_item
    self.process_distribution(spec, dist, deps)
  File "C:\Python\Python38\lib\site-packages\setuptools\command\easy_install.py", line 751, in process_distribution
    distros = WorkingSet([]).resolve(
  File "C:\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 780, in resolve
    dist = best[req.key] = env.best_match(
  File "C:\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 1065, in best_match
    return self.obtain(req, installer)
  File "C:\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 1077, in obtain
    return installer(requirement)
  File "C:\Python\Python38\lib\site-packages\setuptools\command\easy_install.py", line 679, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "C:\Python\Python38\lib\site-packages\setuptools\command\easy_install.py", line 705, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "C:\Python\Python38\lib\site-packages\setuptools\command\easy_install.py", line 890, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "C:\Python\Python38\lib\site-packages\setuptools\command\easy_install.py", line 1158, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "C:\Python\Python38\lib\site-packages\setuptools\command\easy_install.py", line 1144, in run_setup
    run_setup(setup_script, args)
  File "C:\Python\Python38\lib\site-packages\setuptools\sandbox.py", line 253, in run_setup
    raise
  File "C:\Python\Python38\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Python\Python38\lib\site-packages\setuptools\sandbox.py", line 195, in setup_context
    yield
  File "C:\Python\Python38\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Python\Python38\lib\site-packages\setuptools\sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "C:\Python\Python38\lib\site-packages\setuptools\sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "C:\Python\Python38\lib\site-packages\setuptools\_vendor\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Python\Python38\lib\site-packages\setuptools\sandbox.py", line 154, in save_modules
    yield saved
  File "C:\Python\Python38\lib\site-packages\setuptools\sandbox.py", line 195, in setup_context
    yield
  File "C:\Python\Python38\lib\site-packages\setuptools\sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "C:\Python\Python38\lib\site-packages\setuptools\sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "C:\Users\David\AppData\Local\Temp\easy_install-qven3kou\scikit-learn-0.22.1\setup.py", line 303, in <module>
  File "C:\Users\David\AppData\Local\Temp\easy_install-qven3kou\scikit-learn-0.22.1\setup.py", line 299, in setup_package
  File "C:\Python\Python38\lib\site-packages\numpy\distutils\core.py", line 137, in setup
    config = configuration()
  File "C:\Users\David\AppData\Local\Temp\easy_install-qven3kou\scikit-learn-0.22.1\setup.py", line 182, in configuration
  File "C:\Python\Python38\lib\site-packages\numpy\distutils\misc_util.py", line 1033, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "C:\Python\Python38\lib\site-packages\numpy\distutils\misc_util.py", line 999, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "C:\Python\Python38\lib\site-packages\numpy\distutils\misc_util.py", line 941, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "sklearn\setup.py", line 86, in configuration
  File "C:\Users\David\AppData\Local\Temp\easy_install-qven3kou\scikit-learn-0.22.1\sklearn\_build_utils\__init__.py", line 75, in cythonize_extensions
  File "C:\Python\Python38\lib\site-packages\Cython\Build\Dependencies.py", line 1072, in cythonize
    pool = multiprocessing.Pool(
  File "C:\Python\Python38\lib\multiprocessing\context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Python\Python38\lib\multiprocessing\pool.py", line 212, in __init__
    self._repopulate_pool()
  File "C:\Python\Python38\lib\multiprocessing\pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "C:\Python\Python38\lib\multiprocessing\pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "C:\Python\Python38\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Python\Python38\lib\multiprocessing\context.py", line 326, in _Popen
    return Popen(process_obj)
  File "C:\Python\Python38\lib\multiprocessing\popen_spawn_win32.py", line 70, in __init__
    with open(wfd, 'wb', closefd=True) as to_child:
  File "C:\Python\Python38\lib\site-packages\setuptools\sandbox.py", line 417, in _open
    if mode not in ('r', 'rt', 'rb', 'rU', 'U') and not self._ok(path):
  File "C:\Python\Python38\lib\site-packages\setuptools\sandbox.py", line 428, in _ok
    realpath = os.path.normcase(os.path.realpath(path))
  File "C:\Python\Python38\lib\ntpath.py", line 626, in realpath
    path = normpath(path)
  File "C:\Python\Python38\lib\ntpath.py", line 452, in normpath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not int

Better handle faulty dependencies

analyze_src_file.py is running inside a dissasembler, in batch mode, and the only way for it to realize inside which program it is being executed is catching ImportErrors. However, Karta currently assumes that it will find the disassembler, otherwise an ugly exception will be thrown at the user.

Since we lately has some issues when importing sark and other dependencies within ida_api.py, try to check in advance if these requirements are installed correctly, so to be able to warn the user in advance in a clear message. In addition, improve the exception that is thrown when no disassembler could be found.

Installation dependency error

When installing Karta (tag v2.0.0) using py -3 setup.py install

I face the following problem:

Installed c:\users\user\appdata\local\programs\python\python38\lib\site-packages\karta-2.0.0-py3.8.egg
Processing dependencies for Karta==2.0.0
Searching for sark==2.0
Reading https://pypi.org/simple/sark/
No local packages or working download links found for sark==2.0
error: Could not find suitable distribution for Requirement.parse('sark==2.0')

This was solved by editing setup.py file, removing the version from sark dependency

file_layer.py : variable "matched_bin_index" should be initialized before try/except block

In file "file_layer.py" the variable called "matched_bin_index" should be initialized before try/except block, because in case of error the code execution goes via "except:" branch and user gets "variable was not initialized" error.

Thumbs Up on Broadcom Wi-Fi RAM State Snapshots

Hi,

I've been using the Thumbs Up script a lot these days and most of the time it worked perfectly :) I definitely owe you a thank you fax.

Now I tried running the script on a Broadcom Wi-Fi state snapshot and couldn't get it working at all. I think Broadcom Wi-Fi is an ARM Cortex R. The ROM is mapped to 0x0, however, for that specific chip I didn't extract it yet. The RAM holds a major part of the chip's code and is loaded by the operating system driver. For example, it redefines if that chip is running in access point or in station mode. Whenever a crash happens, recent operating systems generate a crash log that contains a full copy of the RAM region. The RAM starts at 0x170000.

To get some of the functions in it recognized at all, I ran idc.plan_and_wait(0x170000,0x300000). When I correctly mark the whole region as RWX RAM section, Thumbs Up doesn't start at all. Thus, I divided it in ROM (despite not being ROM) from 0x170000-0x1fffff and RAM for the remaining part. Then, I get the following output:

[10/06/2020 11:02:26] - Thumbs Up Logger - INFO: Segments, as marked by the disassembler:
[10/06/2020 11:02:26] - Thumbs Up Logger - INFO: Code Segment: 0x170000 - 0x200000
[10/06/2020 11:02:26] - Thumbs Up Logger - INFO: Data Segment: 0x200000 - 0x300000
[10/06/2020 11:02:26] - Thumbs Up Logger - INFO: Processor: ARM, 32bit
[10/06/2020 11:02:26] - Thumbs Up Logger - INFO: Supported ARM code types:
[10/06/2020 11:02:26] - Thumbs Up Logger - INFO: 0 - ARM
[10/06/2020 11:02:26] - Thumbs Up Logger - INFO: 1 - THUMB
[10/06/2020 11:02:26] - Thumbs Up Logger - INFO: Starting the analysis
[10/06/2020 11:02:26] - Thumbs Up Logger - INFO: Phase #1
[10/06/2020 11:02:26] - Thumbs Up Logger - INFO: Tell IDA to analyze all of the code segments
[10/06/2020 11:02:26] - Thumbs Up Logger - INFO: Re-Analyzing code segment: 0x170000 - 0x200000
[10/06/2020 11:02:36] - Thumbs Up Logger - INFO: Phase #2
[10/06/2020 11:02:36] - Thumbs Up Logger - INFO: Observe all code patterns from the initial analysis
[10/06/2020 11:02:36] - Thumbs Up Logger - INFO: There are 7 scoped functions for code type 0
[10/06/2020 11:02:37] - Thumbs Up Logger - INFO: Calibration: Function Prologue Accuracy: 40.00%
[10/06/2020 11:02:37] - Thumbs Up Logger - INFO: Calibration: Function Epilogue Accuracy: 50.00%
[10/06/2020 11:02:37] - Thumbs Up Logger - INFO: Calibration: Function Prologue/Epilogue Accuracy: 40.00%
[10/06/2020 11:02:37] - Thumbs Up Logger - INFO: Testing: Function Prologue Accuracy: 40.00%
[10/06/2020 11:02:37] - Thumbs Up Logger - ERROR: Function Prologue Accuracy is too low, can't continue: 40.00% < 75.00%
[10/06/2020 11:02:37] - Thumbs Up Logger - ERROR: Failed during intelligence gathering, exiting
[10/06/2020 11:02:37] - Thumbs Up Logger - ERROR: Encountered an error during the analysis

Is there any chance to force the script running nonetheless? Do you think the results were still meaningful or is the whole setup of code in RAM mixed with data (yes, lol...) just not what Thumbs Up was made for?

What you can see here is how it looks like after loading it into IDA, with the marker being set to the border between what I defined as "ROM" (actually also being RWX) and RAM.

If you want to test it on your own, I also added the binary :) It's from a BCM4377.
SoC_RAM.bin.zip

Does not work on IDA 7.0 (mac)

1. Can't load karta_identifier in IDA 7.0 on Mac

Steps to reproduce

1. Install Homebrew Python 2.7

2. Run pip install -r requirements.txt

3. Use sys.path.append to add PYTHONPATH in karta_identifier.py:

   import sys
   sys.path.append('/usr/local/homebrew/lib/python2.7/site-packages/')
   

Load it in IDA and it reports error:

/tmp/Karta-master/src/karta_identifier.py: 'NoneType' object has no attribute 'logHandler'
Traceback (most recent call last):
  File "/Applications/IDA Pro 7.0/ida64.app/Contents/MacOS/python/ida_idaapi.py", line 553, in IDAPython_ExecScript
    execfile(script, g)
  File "/tmp/Karta-master/src/karta_identifier.py", line 180, in <module>
    pluginMain()
  File "/tmp/Karta-master/src/karta_identifier.py", line 164, in pluginMain
    initUtils(logger, createDisassemblerHandler(logger))
  File "/tmp/Karta-master/src/config/utils.py", line 129, in initUtils
    global_logger.linkHandler(disas_layer.logHandler())
AttributeError: 'NoneType' object has no attribute 'logHandler'

2. karta_manual_identifier.py does not work at all

From strings xxx I can tell that target is linked against OpenSSL 1.0.1h

...
OpenSSL 1.0.1h 5 Jun 2014
...

Now if I try to identify the library:

%> python karta_manual_identifier.py /tmp/target.i64
[+] Starting the Script
[+] Starting the input loop
    [+] Enter the details for the current manually identified library:
    [+] Library Name (case sensitive): OpenSSL
    [+] Library Version (case sensitive): 1.0.1h
    [+] Do you want to identify another library version? <Y/N>:
[+] Storing the data to the knowledge configuration file
[+] Finished Successfully

Karta shows nothing. Since there is a config/OpenSSL_1.0.1j.json available, I tried to use 1.0.1j as well, nothing changed.

Can you help?

Enabling user to manually define anchors

Yannay requested that he could manually define an anchor function, which seems legit.
By using a .json file that will store this information, we could probably solve more feature enhancements using the same file.

Needs to finalize the wanted solution before I develop this feature.

problem running karta scripts in IDA 7.4

I am trying to run Karta scripts in IDA but I get the following error:

Karta - INFO: Chose the IDA handler�[0m
Karta - ERROR: Failed to create disassembler handler "IDA": DLL load failed while importing sip: The specified module could not be found.� Karta - ERROR: Traceback (most recent call last):
File "C:/Users/User/Karta/src\disassembler\factory.py", line 55, in createDisassemblerHandler
return verifier.disas()
File "C:/Users/User/Karta/src\disassembler\IDA\ida_verifier_api.py", line 42, in disas
from .ida_api import IDA
File "C:/Users/User/Karta/src\disassembler\IDA\ida_api.py", line 9, in
import sark
File "C:\Python38\lib\site-packages\sark-7.4.0-py3.8.egg\sark_init_.py", line 14, in
from . import (core,
File "C:\Python38\lib\site-packages\sark-7.4.0-py3.8.egg\sark\code_init_.py", line 10, in
from . import line
File "C:\Python38\lib\site-packages\sark-7.4.0-py3.8.egg\sark\code\line.py", line 7, in
from ..ui import updates_ui
File "C:\Python38\lib\site-packages\sark-7.4.0-py3.8.egg\sark\ui.py", line 6, in
from .qt import MenuManager
File "C:\Python38\lib\site-packages\sark-7.4.0-py3.8.egg\sark\qt.py", line 4, in
from PyQt5 import QtCore, QtWidgets, QtGui
ImportError: DLL load failed while importing sip: The specified module could not be found.
�[0m

I installed every repository required but still cant run it. Will be glad to get some help.

'year' variable referenced before assignment in libjpeg.py,line:34

Add blog link to readme

Can you add your blog link to readme.md?

Issue regarding setup.py

I'm not quite well-versed in python, and I require some assistance in building this plugin.

I've went ahead and installed all required dependencies through pip, yet I am still getting errors in the library files themselves. If anybody could be of assistance, it would be greatly appreciated.

File:
error.txt

IOError: [Errno 13] Permission denied even though all file permissions are enabled

I was attempting to run an IDA plugin called Karta through Cygwin's CLI and I kept on getting the error message:

Traceback (most recent call last):
File "karta_analyze_src.py", line 310, in
main(sys.argv[1:])
File "karta_analyze_src.py", line 303, in main
analyzeLibrary(constructConfigPath(library_name, library_version), bin_dirs, archive_paths, prompter)
File "karta_analyze_src.py", line 106, in analyzeLibrary
archive_files = list(locateFiles(bin_dir, filter(lambda x: x.endswith("." + bin_suffix), getArchiveFiles(compiled_ar)) if not ignore_archive else None, bin_suffix))
File "C:\Users\wyin\Documents\Karta-master\src\ar_parser.py", line 10, in getArchiveFiles
ar_fd = open(ar_path, 'rb')
IOError: [Errno 13] Permission denied: 'C:/Users/wyin/Documents/cryptopp820'

I looked at my file permissions for C:/Users/wyin/Documents/cryptopp820 and i saw that I had every permission already enabled.

What I can I do to resolved this error?

it seems like its trying to open the directory in python and that causes the error

karta_analyze_src.py on windows

Is it possible "execute IDAPython scripts outside IDA"? I am trying to run the karta_analyze_src.py via following command:
C:\Python27\Lib\site-packages\Karta-1.0.0-py2.7.egg\src>ida.exe -S"karta_analyze_src.py -D " -t
.. no matter what args I provide to the script, the result is always the same:
karta_analyze_src.py: error: too few arguments

Store and share information between matched libraries

The matcher today attempts to match every library independently of the other libraries that should be / were already matched. This means that we lose useful information that can improve our matching logic:

1. Libpng usually locates zlib (external) functions, zlib could use it during its matching (or vice versa)
2. Known library borders from previous matchings could create an informative upper / lower bound on our file map

TODO: Should think if could somehow merge the solution for this issue with the solution for manually defining anchors.

KeyError when adding support for Libpcap 1.8.1

I'm currently in the process of adding support for libpcap 1.8.1 for Karta. I've created a new JSON configuration file using karta_analyze_src.py and added my libpcap identification script to init. When I run the matcher program in Karta using IDA Pro 7.3, Karta is able to match certain anchor functions in lipcap to those in my program, but when it begins to link the binary functions to their respective tentative files, it throws this exception:
C:\Users\wyin\Documents\Karta-master\src\karta_matcher.py: 4734272L Traceback (most recent call last): File "C:\Program Files\IDA Pro 7.3\python\ida_idaapi.py", line 572, in IDAPython_ExecScript execfile(script, g) File "C:/Users/wyin/Documents/Karta-master/src/karta_matcher.py", line 193, in <module> pluginMain() File "C:/Users/wyin/Documents/Karta-master/src/karta_matcher.py", line 183, in pluginMain matchLibraries() File "C:/Users/wyin/Documents/Karta-master/src/karta_matcher.py", line 144, in matchLibraries matchLibrary(lib_name, actual_version) File "C:/Users/wyin/Documents/Karta-master/src/karta_matcher.py", line 95, in matchLibrary startMatch(cur_config_path, lib_name) File "C:/Users/wyin/Documents/Karta-master/src/karta_matcher.py", line 60, in startMatch matching_engine.locateFileBoundaries() File "C:/Users/wyin/Documents/Karta-master/src\core\matching_engine.py", line 586, in locateFileBoundaries local_bins_ctx = map(lambda ea: self.bin_functions_ctx[ea], all_bin_functions[local_bin_start_index:local_bin_end_index + 1]) File "C:/Users/wyin/Documents/Karta-master/src\core\matching_engine.py", line 586, in <lambda> local_bins_ctx = map(lambda ea: self.bin_functions_ctx[ea], all_bin_functions[local_bin_start_index:local_bin_end_index + 1]) KeyError: 4734272L
The libpcap identification script is the exact same as the identification script for zlib except that the key string has been changed to fit libpcap. Do you know what exactly this error message means and what I could do to fix this?

Support function tables

Function tables can often be found in the global variables in many projects: C++ / firmware. Should add better support of finding such tables, and using them for the matching logic.

This support is crucial for C++ programs.

unable to recognize zlib and openssl in so file

Dear developer, when I first saw this project, I was attracted by your ideas and it's prospects, but when I used it to test a so, it contains obvious zlib and openssl features, it Failed,.I'm so confused .Looking forward to hearing from you.

Karta Identifier - libnh.so:

Identified Open Sources:

(none)

Identified Closed Sources:

(none)

Missing Open Sources:

libpng: Was not found
zlib: Was not found
OpenSSL: Was not found
OpenSSH: Was not found
net-snmp: Was not found
gSOAP: Was not found
libxml2: Was not found
libtiff: Was not found
mDNSResponder: Was not found
MAC-Telnet: Was not found
libjpeg-turbo: Was not found
libjpeg: Was not found
icu: Was not found
libvpx: Was not found

Running ThumbsUp on raw firmware binaries

Hi,

I encountered the a few issues when running the Thumbs Up script with the following configuration:

• Up-to-date Ubuntu 19.10
• Python 3.7.5
• IDA Pro 7.4

The requirement sark==2.0 could not be installed, so I replaced it in the install script and just took the most recent one from GitHub, which was 7.8. This might already be the source of my subsequent errors ;)

/media/sf_seemoo/software/Karta/src/thumbs_up/thumbs_up_firmware.py: unpack requires a buffer of 8 bytes
Traceback (most recent call last):
  File "/opt/idapro-7.4/python/3/ida_idaapi.py", line 593, in IDAPython_ExecScript
    exec(code, g)
  File "/media/sf_seemoo/software/Karta/src/thumbs_up/thumbs_up_firmware.py", line 226, in <module>
    main()
  File "/media/sf_seemoo/software/Karta/src/thumbs_up/thumbs_up_firmware.py", line 210, in main
    analyzer.linkFunctionClassifier()
  File "/media/sf_seemoo/software/Karta/src/thumbs_up/analyzers/arm.py", line 48, in linkFunctionClassifier
    self.func_classifier = FunctionClassifier(self, function_feature_size, function_inner_offset, classifiers_start_offsets, classifiers_end_offsets, classifiers_mixed_offsets, classifier_type_offsets)
  File "/media/sf_seemoo/software/Karta/src/thumbs_up/utils/function.py", line 68, in __init__
    numpy.random.seed(seed=struct.unpack("L", ida_nalt.retrieve_input_file_md5()[:4])[0])
struct.error: unpack requires a buffer of 8 bytes

Fixed this by replacing line 68 with numpy.random.seed(1337) and it worked.

Console output in IDA continues as follows:

[27/05/2020 08:14:08] - Thumbs Up Logger - INFO: Phase #4
[27/05/2020 08:14:08] - Thumbs Up Logger - INFO: Observe all code patterns from the improved analysis
[27/05/2020 08:14:08] - Thumbs Up Logger - INFO: There are 8913 scoped functions for code type 1
[27/05/2020 08:14:09] - Thumbs Up Logger - INFO: Calibration: Function Prologue Accuracy: 91.47%
[27/05/2020 08:14:11] - Thumbs Up Logger - INFO: Calibration: Function Epilogue Accuracy: 96.50%
[27/05/2020 08:14:12] - Thumbs Up Logger - INFO: Calibration: Function Prologue/Epilogue Accuracy: 97.00%
[27/05/2020 08:14:14] - Thumbs Up Logger - INFO: Testing: Function Prologue Accuracy: 91.72%
[27/05/2020 08:14:15] - Thumbs Up Logger - INFO: Testing: Function Epilogue Accuracy: 97.44%
[27/05/2020 08:14:16] - Thumbs Up Logger - INFO: Testing: Function Prologue/Epilogue Accuracy: 97.25%
[27/05/2020 08:14:22] - Thumbs Up Logger - INFO: Start marking functions, even without xrefs

Got the following error displayed in IDA:

/media/sf_seemoo/software/Karta/src/thumbs_up/thumbs_up_firmware.py: 0
Traceback (most recent call last):
  File "/opt/idapro-7.4/python/3/ida_idaapi.py", line 593, in IDAPython_ExecScript
    exec(code, g)
  File "/media/sf_seemoo/software/Karta/src/thumbs_up/thumbs_up_firmware.py", line 226, in <module>
    main()
  File "/media/sf_seemoo/software/Karta/src/thumbs_up/thumbs_up_firmware.py", line 218, in main
    result = analysisStart(analyzer, code_segments, data_segments)
  File "/media/sf_seemoo/software/Karta/src/thumbs_up/thumbs_up_firmware.py", line 122, in analysisStart
    functionScan(analyzer, scs)
  File "/media/sf_seemoo/software/Karta/src/thumbs_up/analyzer_utils.py", line 172, in functionScan
    if analyzer.func_classifier.predictFunctionStart(line.start_ea, guess_code_type):
  File "/media/sf_seemoo/software/Karta/src/thumbs_up/utils/function.py", line 368, in predictFunctionStart
    return self._start_classifiers[code_type].predict([sample])
KeyError: 0

IDA still continues automatic analysis afterward. Not sure if it worked or didn't. The results are definitely better than after just running a linear analysis on the ROM :)

the problem of karta_identifier.py

Karta - INFO: Chose the IDA handler�[0m
�[91m[11/08/2022 12:57:36] - Karta - ERROR: Failed to create disassembler handler "IDA": DLL load failed while importing sip: 找不到指定的模块。�[0m
�[91m[11/08/2022 12:57:36] - Karta - ERROR: Traceback (most recent call last):
File "D:\ChromeCoreDownloads/Karta-2.1.0/src\disassembler\factory.py", line 55, in createDisassemblerHandler
return verifier.disas()
File "D:\ChromeCoreDownloads/Karta-2.1.0/src\disassembler\IDA\ida_verifier_api.py", line 42, in disas
from .ida_api import IDA
File "D:\ChromeCoreDownloads/Karta-2.1.0/src\disassembler\IDA\ida_api.py", line 9, in
import sark
File "D:\Python3.9.13\lib\site-packages\sark-7.4.1-py3.9.egg\sark_init_.py", line 14, in
from . import (core,
File "D:\Python3.9.13\lib\site-packages\sark-7.4.1-py3.9.egg\sark\code_init_.py", line 10, in
from . import line
File "D:\Python3.9.13\lib\site-packages\sark-7.4.1-py3.9.egg\sark\code\line.py", line 7, in
from ..ui import updates_ui
File "D:\Python3.9.13\lib\site-packages\sark-7.4.1-py3.9.egg\sark\ui.py", line 6, in
from .qt import MenuManager
File "D:\Python3.9.13\lib\site-packages\sark-7.4.1-py3.9.egg\sark\qt.py", line 4, in
from PyQt5 import QtCore, QtWidgets, QtGui
ImportError: DLL load failed while importing sip: 找不到指定的模块。

ValueError when adding support for Libgcrypt 1.8.4

So right now I'm trying to add support for identification of the library libgcrypt. This time, I created a new JSON configuration file from the non-inlined and compiled version of libgcrypt. When I run karta_matcher.py on a test case for this library, it is capable of identifying libgcrypt. However, when it reaches the point of Linking the binary functions to their respective tentative files, Karta throws this exception:

C:\Users\wyin\Documents\Karta\src\karta_matcher.py: None is not in list
Traceback (most recent call last):
  File "C:\Program Files\IDA Pro 7.3\python\ida_idaapi.py", line 572, in IDAPython_ExecScript
    execfile(script, g)
  File "C:/Users/wyin/Documents/Karta/src/karta_matcher.py", line 193, in <module>
    pluginMain()
  File "C:/Users/wyin/Documents/Karta/src/karta_matcher.py", line 183, in pluginMain
    matchLibraries()
  File "C:/Users/wyin/Documents/Karta/src/karta_matcher.py", line 144, in matchLibraries
    matchLibrary(lib_name, actual_version)
  File "C:/Users/wyin/Documents/Karta/src/karta_matcher.py", line 95, in matchLibrary
    startMatch(cur_config_path, lib_name)
  File "C:/Users/wyin/Documents/Karta/src/karta_matcher.py", line 60, in startMatch
    matching_engine.locateFileBoundaries()
  File "C:/Users/wyin/Documents/Karta/src\core\matching_engine.py", line 587, in locateFileBoundaries
    file_match = file_class(file_name, src_start_index, src_end_index, local_bins_ctx, local_bin_start_index, local_bin_end_index, src_end_index - src_start_index + 1, self)
  File "C:/Users/wyin/Documents/Karta/src\file_layer.py", line 29, in __init__
    super(FileMatcher, self).__init__(name, src_index_start, src_index_end, fuzzy_bin_functions_ctx, bin_limit_lower, bin_limit_upper, src_scope, engine)
  File "C:/Users/wyin/Documents/Karta/src\core\file_layer.py", line 142, in __init__
    bin_range = range(self._bin_functions_ctx.index(self._lower_match_ctx), self._bin_functions_ctx.index(self._upper_match_ctx) + 1)
ValueError: None is not in list

Again, the identification script I'm using for libgcrypt is just a slightly edited version of the identification script for zlib. I'm running Karta on IDA Pro 7.3
Do you know what could be causing this error and what I can do to fix it?

A problem in isAnchor function.

I find a problem in src->config->anchor->isAnchor() when I look at your code.

this function can only run the first branch, because case can only equal to 1.

Integrate knowledge from IDA's flirt signatures

On windows binaries we could match several libC external functions using the FLIRT signatures.

• Upside: improves the accuracy of our match
• Downside: when a library uses ss2 optimized calls (memcpy, memset) it will cause FPs

Should think if / how we could integrate this feature without damaging the matching results.

x86/x64 Karta/Thumbs Up support

Please????

karta_identifier.py createDisassemblerHandler returns None

Can not run karta_identifier.py from IDA, I get following error:

Support python-idb

Is it possible to use famous python-idb library?
To alleviate the need to run IDA Pro itself, perfect for automation purposes.

The latest sklearn doesn't support Python2.7

Scikit-learn 0.20 was the last version to support Python2.7，so I think we'd better to use this version in setup.py like this

install_requires=['elementals', 'sark', 'pydocstyle', 'flake8', 'click', 'scikit-learn==0.20.0'],

Partial support for unicode values

Our canonical representation is stored in a *.json file, using python's json library. Reportedly, in some edge cases there are function names (that is really weird) or strings that include special unicode chars that makes it impossible to load back the *.json content.

Couldn't re-create it, and waiting for more information on this bug.