facebook / fai-pep Goto Github PK

Hello all,
It seems that the Wiki of this project needs an update.
For example, "~/caffe2/pytorch" in https://github.com/facebook/FAI-PEP/wiki/Run-FAI-PEP-for-the-first-time has already been invalid because caffee2 is part of pytorch now.

If I use pytorch/caffe2 instead, the build.sh for [email protected] cannot generate caffe2_benchmark binary file as expected for the latest pytorch repo.

Thank you!

Allowing user input to control paths used in file system operations could enable an attacker to access or modify otherwise protected system resources

Hi there,

I just new to here. So I typed by following the tutorial:

benchmarking/run_bench.py -b specifications/models/caffe2/shufflenet/shufflenet.json --platforms android

After a long time compiling, all of compile and link tasks are finished in the build_android folder in my pytorch repo. But it throws an error:

cmake unknown rule to install xxxx

It looks the caffe2_benchmark executable has been generated by failed to copied to the install folder, so I manually copied to the folder, namely:
/home/new/.aibench/git/exec/caffe2/android/2019/4/5/fefa6d305ea3e820afe64cec015d2f6746d9ca88
Then I modified repo_driver.py to avoid compiling again and run function _runBenchmarkSuites
But failed :

In file included from ../third_party/zstd/lib/common/pool.h:20:0,
from ../third_party/zstd/lib/common/pool.c:14:
../third_party/zstd/lib/common/zstd_internal.h:382:37: error: unknown type name ‘ZSTD_dictMode_e’; did you mean ‘FSE_decode_t’?
ZSTD_dictMode_e dictMode,
^~~~~~~~~~~~~~~
FSE_decode_t

Questions:

1. Any suggestions on how to run the tutorial correctly?
2. How to avoid the long time compiling for each time running

benchmarking/run_bench.py -b specifications/models/caffe2/shufflenet/shufflenet.json --platforms android

thanks!

Sorry for being creating an issue in this repo. Are you guys working on Django ?

In top level README one of the bullets about performance metrics says,

"energy/power : the energy per inference and average power of running the the ML model on a phone without battery"

I assume the "the the" is a typing error that should be "the".

But what about "without battery" ? If inference engine hardware in mobile phones could truly run without a battery that would be very low power indeed !! Is this a typo or if not, what is meant by this ?

-- jS

The above syntax can be changed to the following.

kw_extra['x_is_date'] = kw_extra.get('x_is_date', False)
kw_extra['x_axis_format'] = kw_extra.get('x_axis_format', "%d %b %Y")
kw_extra['color_category'] = kw_extra.get('color_category', "category20")
kw_extra['tag_script_js'] = kw_extra.get('tag_script_js', True)
kw_extra['chart_attr'] = kw_extra.get('chart_attr', {})

I have tried to run shufflenet and modified inputs to gpu_0/data getting following issue
INFO 12:24:56 subprocess_with_logger.py: 24: Running: /tmp/FAI-PEP/libraries/python/imagenet_test_map.py --image-dir /tmp/imagenet/val --label-file /tmp/FAI-PEP/libraries/python/labels.txt --output-image-file /tmp/tmpLwDpGk/caffe2/host/images.txt --output-label-file /tmp/tmpLwDpGk/caffe2/host/labels.txt --shuffle
INFO 12:24:56 subprocess_with_logger.py: 24: Running: awk (NR>050000/1000)&&(NR<=050000/1000+50000/1000) {print > "/tmp/tmpLwDpGk/caffe2/host/inputs/labels_0.txt"} /tmp/tmpLwDpGk/caffe2/host/labels.txt
INFO 12:24:56 subprocess_with_logger.py: 24: Running: /tmp/config/exec/caffe2/host/incremental/2019/5/23/90182a7332997fb0edf666abc4b554b83a1670d1/convert_image_to_tensor --input_image_file /tmp/tmpLwDpGk/caffe2/host/inputs/labels_0.txt --output_tensor /tmp/tmpLwDpGk/caffe2/host/images_tensor.pb --batch_size 1 --scale 256,-1 --crop 224,224 --preprocess normalize,mean,std --report_time json|Caffe2Observer
INFO 12:24:57 hdb.py: 27: push /tmp/tmpLwDpGk/caffe2/host/images_tensor.pb to /tmp/tmpJ2TNu5/6ea951fe0a41/images_tensor.pb
INFO 12:24:57 hdb.py: 27: push /tmp/config/model_cache/caffe2/shufflenet/model.pb to /tmp/tmpJ2TNu5/6ea951fe0a41/model.pb
INFO 12:24:57 hdb.py: 27: push /tmp/config/model_cache/caffe2/shufflenet/model_init.pb to /tmp/tmpJ2TNu5/6ea951fe0a41/model_init.pb
{u'softmax': u'/tmp/tmpJ2TNu5/6ea951fe0a41/output/softmax.txt'}
INFO 12:24:57 subprocess_with_logger.py: 24: Running: /tmp/tmpJ2TNu5/6ea951fe0a41/caffe2_benchmark --net /tmp/tmpJ2TNu5/6ea951fe0a41/model.pb --init_net /tmp/tmpJ2TNu5/6ea951fe0a41/model_init.pb --warmup 0 --iter 50 --input gpu_0/data --input_file /tmp/tmpJ2TNu5/6ea951fe0a41/images_tensor.pb --input_type float --output gpu_0/softmax --text_output true --output_folder /tmp/tmpJ2TNu5/6ea951fe0a41/output

/tmp/tmpLwDpGk/caffe2

/tmp/tmpLwDpGk/caffe2/output
INFO 12:25:00 hdb.py: 38: pull /tmp/tmpJ2TNu5/6ea951fe0a41/output/softmax.txt to /tmp/tmpLwDpGk/caffe2/output/softmax.txt
INFO 12:25:00 hdb.py: 40: directory /tmp/tmpJ2TNu5/6ea951fe0a41/output
INFO 12:25:00 hdb.py: 46: filenames /tmp/tmpJ2TNu5/6ea951fe0a41/output/
INFO 12:25:00 benchmark_driver.py: 64: Exception caught when running benchmark
INFO 12:25:00 benchmark_driver.py: 65: [Errno 2] No such file or directory: u'/tmp/tmpJ2TNu5/6ea951fe0a41/output/softmax.txt'
ERROR 12:25:00 benchmark_driver.py: 69: Traceback (most recent call last):

As gpu_0 is prefixed in every node in shufflenet checkpoint.

We run multiple tests from a single config, and introduced an error in the second test in the group. This failure was not noticed, because each test reuses the same temp directory so report.json from the previous run is reused. That means that even though the run fails, we report data for the previous run.

bazel build command in specifications/frameworks/tflite/android/build.sh requires a lot of dependencies, like appropriate version of bazel, Android SDK and NDK. It's burdensome.

I found that the resulted binary is provided in here which is form https://www.tensorflow.org/lite/performance/measurement.

I commented out

#   --config=android_arm \
#   --cxxopt='--std=c++11' \
#   tensorflow/lite/tools/benchmark:benchmark_model

these lines and saved the downloaded binary in {tensorflow_dir}/bazel-bin/tensorflow/lite/tools/benchmark/benchmark_model

Then,

python ${FAI_PEP_DIR}/benchmarking/run_bench.py -b "${BENCHMARK_FILE}" --config_dir "${CONFIG_DIR}"

executed without problems.

Below is my configuration.

{
  \"--commit\": \"master\",
  \"--exec_dir\": \"${CONFIG_DIR}/exec\",
  \"--framework\": \"tflite\",
  \"--local_reporter\": \"${CONFIG_DIR}/reporter\",
  \"--model_cache\": \"${CONFIG_DIR}/model_cache\",
  \"--platforms\": \"android\",
  \"--remote_repository\": \"origin\",
  \"--repo\": \"git\",
  \"--repo_dir\": \"${REPO_DIR}\",
  \"--tmp_model_dir\": \"${CONFIG_DIR}/tmp_model_dir\",
  \"--root_model_dir\": \"${CONFIG_DIR}/root_model_dir\",
  \"--screen_reporter\": null
}

When trying to run the experiment with docker for TFLite example, I encountered the following error:

+ python /tmp/FAI-PEP/benchmarking/run_bench.py -b /tmp/FAI-PEP/specifications/models/tflite/mobilenet_v2/mobilenet_v2_0.35_96.json --config_dir /tmp/config
usage: run_bench.py [-h] [--app_id APP_ID] [-b BENCHMARK_FILE] [--lab]
                    [--logger_level {info,warning,error}] [--remote]
                    --root_model_dir ROOT_MODEL_DIR [--token TOKEN]
                    [-c CUSTOM_BINARY] [--pre_built_binary PRE_BUILT_BINARY]
                    [--user_string USER_STRING]
run_bench.py: error: argument --root_model_dir is require

Does anyone know how to resolve this? Did I missed anything before running the example?

I tried benchmarking/run_bench.py -b specifications/models/tflite/mobilenet_v2/mobilenet_v2_1.4_224.json but don't know what should I enter at the prompt "Please enter the directory the framework repo resides" - should it be the local PyTorch or TF Lite repo directory? An example of this?

It would be great if you could add a general tutorial which allows practitioners to benchmark all sorts of models. The idea behind FAI-PEP is really good but all tutorials are geared towards image-classification models.

Hello, I'm trying to get benchmarks from my phone running MobilenetV2.

I was reading this document and got one question.

Just passing "--platform=android" is enough to run benchmark binaries? How does this tell where's my phone to the executed binary file? I thought I should set up some "adb" kind of things, but I couldn't find any mention of "adb". Now I am wondering if FAI-PEP has some android simulators that runs binary..

Could you tell me where should I refer to if I want to run FAI-PEP on the phone I connect?

Can you show an example on how to use the system with iOS?
Does the system compiles the .ipa that get sent to the iOS device or do we have to provide it?

When I first run the command,

python ${FAI_PEP_DIR}/benchmarking/run_bench.py -b "${BENCHMARK_FILE}" --config_dir "${CONFIG_DIR}"

I encounter this message.
"UnboundLocalError: local variable 'abs_name' referenced before assignment"

However, when I ran the same command again, it disappears. I doubt .md5 autogeneration somehow broke at the first time.