evilsocket / ergo Goto Github PK

tried running

1. ergo help

and

2. ergo example

and the following error occurs:
Traceback (most recent call last): File "/home/hg/anaconda3/bin/ergo", line 22, in <module> from ergo.actions.create import action_create File "/home/hg/anaconda3/lib/python3.7/site-packages/ergo/actions/create.py", line 6, in <module> from ergo.project import Project File "/home/hg/anaconda3/lib/python3.7/site-packages/ergo/project.py", line 13, in <module> from keras.utils.training_utils import multi_gpu_model ModuleNotFoundError: No module named 'keras.utils.training_utils'

any clues?

You need to add pillow to requirements in case it is not installed to prevent this error

Hello,
Seems like there's a parsing issue in serialize_classification_report (utils.py)
When (like in my case) the accuracy has no value for precision and recall, it leads to crash when trying to serialize to json. The text file is properly displayed.

I don't think counting spaces is a good idea, however it could be a temporary fix.

Apologies to bother again. Using the provided dataset from your blog, but getting the below error-stack. What am I doing wrong?

(ergoproject) remnux@remnux:~$ ergo train malware --dataset dataset.csv
[2019-05-30 09:48:29,475] (INFO) loading project /home/remnux/malware ...
[2019-05-30 09:48:29,476] (INFO) building model for training ...
[2019-05-30 09:48:29,496] (WARNING) From /home/remnux/ergoproject/lib/python3.4/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[2019-05-30 09:48:29,507] (WARNING) From /home/remnux/ergoproject/lib/python3.4/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use rate instead of keep_prob. Rate should be set to rate = 1 - keep_prob.
[2019-05-30 09:48:29,550] (INFO) preparing data from dataset.csv ...
[2019-05-30 09:48:45,318] (INFO) data shape: (199970, 487)
[2019-05-30 09:48:45,985] (CRITICAL) 'str' object has no attribute 'shape'
[2019-05-30 09:48:45,985] (ERROR)

Traceback (most recent call last):
File "/home/remnux/ergoproject/bin/ergo", line 81, in main
ACTIONS[action].cb(argc - 2, sys.argv[2:])
File "/home/remnux/ergoproject/lib/python3.4/site-packages/ergo/actions/train.py", line 62, in action_train
prj.prepare(args.dataset, args.test, args.validation, not args.no_shuffle)
File "/home/remnux/ergoproject/lib/python3.4/site-packages/ergo/project.py", line 143, in prepare
return self.dataset.source(data, p_test, p_val, shuffle)
File "/home/remnux/ergoproject/lib/python3.4/site-packages/ergo/dataset.py", line 113, in source
log.info("detected non scalar input: %s", x.shape)
AttributeError: 'str' object has no attribute 'shape'

When i type ergo no error and no output is return, the exit code remain 0, i have try to put a python debugger line inside the ergo script installed under bin, but the debugger doesn't start.

I use Python 3.7.3 installed with pyenv, i think the problem is on pyenv side but i have posted this issue here because is the first time i have this problem, other packages are not effected i can execute other packages binary regularly.

Getting the below error when I try to run the encode command to classify my malicious and clean files:

:~$ ergo encode malware/ data/ --output dataset.csv
[2019-05-29 03:41:35,405] (INFO) loading project /home/remnux/malware ...
[2019-05-29 03:41:35,410] (INFO) building model for training ...
[2019-05-29 03:41:35,449] (WARNING) From /usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[2019-05-29 03:41:35,465] (WARNING) From /usr/local/lib/python3.4/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use rate instead of keep_prob. Rate should be set to rate = 1 - keep_prob.
[2019-05-29 03:41:35,509] (INFO) using auto labeling
[2019-05-29 03:41:35,510] (INFO) enumerating data/clean ...
[2019-05-29 03:41:35,510] (INFO) enumerating data/malicious ...
[2019-05-29 03:41:35,510] (INFO) labeling 0 files ...
[2019-05-29 03:41:35,510] (INFO) starting 8 workers for encoding
[2019-05-29 03:41:35,522] (INFO) encoding 0 inputs to dataset.csv ...

Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap
self.run()
File "/usr/lib/python3.4/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.4/dist-packages/ergo/actions/encode.py", line 86, in appender
on_progress(done, total)
File "/usr/local/lib/python3.4/dist-packages/ergo/actions/encode.py", line 70, in on_progress
perc = done / total
ZeroDivisionError: division by zero

Is there a way to turn off logging? It takes a considerable amount of disk space.

Hello,
I'm not sure it's an issue, probably just a question.
I'm trying to convert my model to tflite format to be used on clients (still the one from pe-av)

I've converted to Tensoflow using ergo to-tf
Then, I'm using tflite_convert (with the input/output arrays gotten from a script found here and there, extracted from the pb file) but I have an error:

I'm wondering, is that TF .pb format something compatible with tflite ?
How do I know for sure that I'm using the correct input/output arrays, is there a way to extract that from ergo model ?