How does share memory speed up inference? about server

[Tabrizian]

Is shared memory in terms of NVIDIA triton difference from CUDA shared memory?

Yes. CUDA Shared memory is a Triton terminology for transferring CUDA tensors between client and server without having to pass them over the network.

But what it the reason for the increased performance?

The reason for performance improvement is that you don't have to transfer the tensor over the network. The benefit would be more significant with larger tensors.

[NikeNano]

Thank you for the answer @Tabrizian!

perf_analyzer -m defect-classifier -u triton:8500 -i gRPC --concurrency-range=1

Concurrency: 1, throughput: 184.18 infer/sec, latency 5425 usec

vs

perf_analyzer -m defect-classifier -u triton:8500 -i gRPC --shared-memory=cuda --concurrency-range=1

Concurrency: 1, throughput: 555.442 infer/sec, latency 1797 usec

It seems to make a huge difference in our use case. However we never seem to be able to get the same throughput for our own Python client. Is there any best practices in terms of client implementation in Python or C++ to achieve similar results as pref_analyzer. Does pref_analyzer throughput also include transfering data back to CPU when cuda share memory is used?

[Tabrizian]

Does pref_analyzer times also include transfering data back to CPU when cuda share memory is used?

@matthewkotila / @tgerdesnv do know whether Perf Analyzer includes the time to copy data to CUDA shared memory?

@NikeNano For Python clients, did you also use CUDA shared memory?

[matthewkotila]

@NikeNano: Does pref_analyzer throughput also include transfering data back to CPU when cuda share memory is used?

I'm not sure if I understand. The calculation for throughput simply counts how many inferences (request-response sets) were completed during a period of time, and divides by the period of time. Everything that has to happen in order for the inference to complete (including CUDA shared memory, CPU transfers, etc) is inherently included in that measurement.

[NikeNano]

@NikeNano For Python clients, did you also use CUDA shared memory?

Yes, we are trying to reimplement it in c++ as well but our feeling now is that we somehow are bottlenecked and are very far from the pref_analyzer performance.

[NikeNano]

@NikeNano: Does pref_analyzer throughput also include transfering data back to CPU when cuda share memory is used?

I'm not sure if I understand. The calculation for throughput simply counts how many inferences (request-response sets) were completed during a period of time, and divides by the period of time. Everything that has to happen in order for the inference to complete (including CUDA shared memory, CPU transfers, etc) is inherently included in that measurement.

Questions for clarification when using pref_analyzer with shared cuda memory(based upon the python example):

• Do we allocate cuda memory once, one call to cudashm.create_shared_memory_region or reallocate for each request?
• Do we move data from CPU to the GPU region for each request cudashm.set_shared_memory_region(shm_ip0_handle, [data]) for each request?
• Do we move the data back from GPU to CPU for each request cudashm.get_contents_as_numpy?

Based upon your previous answer @matthewkotila , I understand that the answer is Yes.

Thanks for the help.