octopus-platform / joern-tools Goto Github PK

I would like to use joern-tools, so currently I am trying to learn how to use it but I found when the joern-tools/examples/subtreeEmbed.sh run , there are some missing python
files.

./subtreeEmbed.sh: line 1: ./lookup.py: No such file or directory
./subtreeEmbed.sh: line 1: ./getAst.py: No such file or directory
./subtreeEmbed.sh: line 1: ./astlabel.py: No such file or directory
./subtreeEmbed.sh: line 1: ./ast2features.py: No such file or directory
./subtreeEmbed.sh: line 1: ./demux.py: No such file or directory
./subtreeEmbed.sh: line 3: sally: command not found

I found similar names to the missing files at /joern-tools/build/scripts-2.7 folder. Is the files in this folder are the same missing files?
can you please let me know where I can find these files? especially getAst.py file.
Thank you!

When finding the nearest neighbor in the VLC tutorial, I execute the following example:

joern-list-funcs -p VLCEyeTVPluginInitialize | awk -F "\t" '{print $2}' | joern-knn

An ImportError arises stating that there is no module named "sklearn.datasets":

    "from sklearn.datasets import load_svnmlight_file"

I thought the only dependency for joern-tools was pygraphviz.

After downloading scikit-learn v0.15.1 to provide the module "sklearn.datasets", I'm presented with the error in _svmlight_format.c line 2505, "ValueError: Feature indices in SVMlight/LibSVM data file should be sorted and unique.

Which version of scikit is being used for sklearn.datasets?

When finding the nearest neighbor in the VLC tutorial, I execute the following example:

joern-list-funcs -p VLCEyeTVPluginInitialize | awk -F "\t" '{print $2}' | joern-knn

I'm presented with the error in _svmlight_format.c line 2505, "ValueError: Feature indices in SVMlight/LibSVM data file should be sorted and unique.

I've identified that when the SVM file is created by joern-apiembedder, the key-value pairs are in descending order by key; therefore, when the keys are parsed, the current index will be less than the previous index. Thus, the error is triggered.

For example, a line from the SVM file is:

3382 1:1.00000 0:1.00000 #3382

The current key, 0, is less than the previous key, 1, so the exceptions is thrown.

I get this error regardless of the source code repository I'm analyzing.

Thoughts on how to correct this, so that I can properly use joern-embedder and joern-knn together?

It would be nice if joern-plot-proggraph could also plot call graphs to get a quick overview of how functions are linked to one another by calls.

when execute joern-lookup , it turns out :

Traceback (most recent call last):
File "/usr/local/bin/joern-lookup", line 4, in
import('pkg_resources').run_script('joerntools==0.1', 'joern-lookup')
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 719, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 1504, in run_script
exec(code, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/joerntools-0.1-py2.7.egg/EGG-INFO/scripts/joern-lookup", line 3, in
from joerntools.shelltool.LookupTool import LookupTool
File "/usr/local/lib/python2.7/dist-packages/joerntools-0.1-py2.7.egg/joerntools/shelltool/LookupTool.py", line 4, in
from TraversalTool import TraversalTool
File "/usr/local/lib/python2.7/dist-packages/joerntools-0.1-py2.7.egg/joerntools/shelltool/TraversalTool.py", line 3, in
from joerntools.shelltool.JoernTool import JoernTool
File "/usr/local/lib/python2.7/dist-packages/joerntools-0.1-py2.7.egg/joerntools/shelltool/JoernTool.py", line 3, in
from joerntools.DBInterface import DBInterface
File "/usr/local/lib/python2.7/dist-packages/joerntools-0.1-py2.7.egg/joerntools/DBInterface.py", line 2, in
from joern.all import JoernSteps
File "/usr/local/lib/python2.7/dist-packages/joern-0.1-py2.7.egg/joern/all.py", line 2, in
from py2neo.ext.gremlin import Gremlin
ImportError: No module named gremlin

how to solve this problems?

E.g works fine:

echo "fun1().fun2()" | joern-lookup -g

echo "fun1()
.fun2()" | joern-lookup -g

Fails on unexpected query

Here is the time cost of GlobalAPIEmbedding operation in chucky-ng.
By changing it's embedder, I got two different result about the time cost .

This means the current PythonAPIEmbedder.py or SimplifiedAPIEmbedder.py still have the scalability problem when processing large code base. And SallyBasedEmbedder is the best choice by far when we encounter large code bases.