Run AWS DeepLearning Benchmarks agains TS containers about serve

[mycpuorg]

A report from the quick run of benchmarks on select models.

TorchServeBenchmarkingResults.pdf

What's Useful:

• Gauge the Inference Times, Client Round Trip Times.
• Stability of Server over sustained Client requests, internal tests show this sustains for much longer.
• Worst Case Latency expected.

What's missing are:

• Needs running on GPU instances
• More models to be covered
• Separation of Inference Time and Overhead from TorchServe

[chauhang]

@mycpuorg For Throughput Benchmarking you can use: https://github.com/pytorch/pytorch/blob/master/torch/utils/throughput_benchmark.py#L62

Please ensure to have some warmup threads prior to collecting the actual stats.

[harshbafna]

Closing this as we now have an apache-bench based benchmarking script which is used for benchmarking TorchServe against different PyTorch models.