About the tensorflow benchmark in half-percision about deep-learning-benchmark

[u39kun]

Hi @foreverYoungGitHub
I borrowed that piece of code from https://github.com/tensorflow/benchmarks/blob/master/scripts/tf_cnn_benchmarks/convnet_builder.py#L112-L118
Looks like this came from tensorflow/benchmarks#63

You may want to ask the TensorFlow community on that.

[foreverYoungGitHub]

Oh, Thanks for that!

Actually, I just did the time benchmark for that. And the filter/kernel is actually computed as float16, which is much faster.

But when I run your code and the my own time benchmark code. I found that the speed of your code is much faster than my. Except the warm-up part in your code, what causes this different?

I attach my code as following.

import tensorflow as tf
import time
from datetime import datetime
import math
import argparse
import sys

import numpy as np

slim = tf.contrib.slim

import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"   # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"]="0"

def time_tensorflow_run_placeholder(session, target, feed_dict, info_string):
  num_steps_burn_in = 10
  total_duration = 0.0
  total_duration_squared = 0.0

  for i in range(FLAGS.num_batches + num_steps_burn_in):
    start_time = time.time()
    _ = session.run(target,feed_dict=feed_dict)
    duration = time.time() - start_time
    if i >= num_steps_burn_in:
      if not i % 10:
        print('%s: step %d, duration = %.3f' % (datetime.now(), i - num_steps_burn_in, duration))
      total_duration += duration
      total_duration_squared += duration * duration

  mn = total_duration / FLAGS.num_batches
  vr = total_duration_squared / FLAGS.num_batches - mn * mn
  sd = math.sqrt(vr)
  print('%s: %s across %d steps, %.3f +/- %.3f sec / batch' % (datetime.now(), info_string, FLAGS.num_batches, mn, sd))

def run_benchmark():
  graph_filename = FLAGS.graph_dir + "-{DATA_FORMAT}-{PRECISION}/frozen_graph.pb".format(DATA_FORMAT=FLAGS.data_format, PRECISION=FLAGS.precision)
  # Create a graph def object to read the graph
  with tf.gfile.GFile(graph_filename, "rb") as f:
    graph_def = tf.GraphDef()
    graph_def.ParseFromString(f.read())

  with tf.Graph().as_default() as graph:
    with tf.device('/'+FLAGS.mode+':0'):
      if FLAGS.data_format == 'NCHW':
        inputs = np.random.random((FLAGS.batch_size, 3, FLAGS.input_width, FLAGS.input_height))
      elif FLAGS.data_format == 'NHWC':
        inputs = np.random.random((FLAGS.batch_size, FLAGS.input_width, FLAGS.input_height, 3))
      if precision == 'fp16':
        inputs = inputs.astype(np.float16)
      tf.import_graph_def(graph_def)
      config = tf.ConfigProto()
      config.gpu_options.allocator_type = 'BFC'
      sess = tf.Session(config=config)
      # We define the input and output node we will feed in
      input_node = graph.get_tensor_by_name('import/input:0')
      output_node = graph.get_tensor_by_name('import/predictions/Reshape_1:0')
      time_tensorflow_run_placeholder(sess, output_node, {input_node: inputs}, "Forward")


def main(_):
  run_benchmark()

[u39kun]

@foreverYoungGitHub Off the top of my head, things that can definitely affect performance is the data format. NCHW is much faster than NHWC. Also, I noticed with TensorFlow is that it takes many warmup runs to get to the optimal speed. That's why in my code the number of warmup runs is set to 20... This was to specifically accommodate TensorFlow (PyTorch, for example, would "warmup" in just one run.)