In trtton, the difference in inference speed is not large depending on the number of GPUs. about server

[SunnyGhj]

Make sure CPU and network bandwidth are sufficient

[SunnyGhj]

Try adding intra_op_thread_count, inter_op_thread_count in your config like the following configuration,

name: “yolov8_16batch_dynamic”
platform: “tensorrt_plan”
max_batch_size: 0
parameters { key: "execution_mode" value: { string_value: "1" } }
parameters { key: "intra_op_thread_count" value: { string_value: "4" } }
parameters { key: "inter_op_thread_count" value: { string_value: "4" } }
input: [
{
name: “input”
data_type: TYPE_FP32
format: FORMAT_NONE
dims: [ -1, 3, 640, 640 ]
}
]
output: [
{
name: “boxes”,
data_type: TYPE_FP32
dims: [ -1, 8400, 4 ]
},
{
name: “scores”,
data_type: TYPE_FP32
dims: [ -1, 8400, 1 ]

},
{
name: "classes",
data_type: TYPE_FP32
dims: [ -1, 8400, 1 ]
}
]
instance_group: [
{
kind: KIND_GPU
count: 8
gpus: [1]
},
{
kind: KIND_GPU
count: 8
gpus: [2]
},
{
kind: KIND_GPU
count: 8
gpus: [3]
}
]

[SunnyGhj]

Make sure --backend-config=tensorrt,execution-policy=BLOCKING is in your docker run startup parameters

[yoo-wonjun]

I encountered the same thing trying to run it symmetrically.

W0104 05:44:23.226400 1 server.cc:238] failed to enable peer access for some device pairs
[libprotobuf ERROR /tmp/tritonbuild/tritonserver/build/_deps/repo-third-party-build/grpc-repo/src/grpc/third_party/protobuf/src/google/protobuf/text_format.cc:335] Error parsing text-format inference.ModelConfig: 1:7: Interpreting non ascii codepoint 226.
[libprotobuf ERROR /tmp/tritonbuild/tritonserver/build/_deps/repo-third-party-build/grpc-repo/src/grpc/third_party/protobuf/src/google/protobuf/text_format.cc:335] Error parsing text-format inference.ModelConfig: 1:7: Expected string, got: �
E0104 05:44:23.228000 1 model_repository_manager.cc:1309] Poll failed for model directory 'yolov8_16batch_dynamic': failed to read text proto from /models/yolov8_16batch_dynamic/config.pbtxt