Large growing native memory usage by the Java app which uses tdlib about td

[levlam]

1. All native memory used by a client is freed when the client is closed. After that the memory can be reused by other clients subject to implementation of the system memory allocator.
2. Do you send requests from a limited amount of Java threads, or often sending requests from short-living threads?
3. You can only close inactive and uneeded instances.

[andreymerchenko]

Hi @levlam , thanks for the quick response!

1. Attaching a graph of the total memory usage when 100 TDLib clients are used:
   
   1-2. Application was in idle mode with just 100 clients listening for updates. We can see steady growth of used memory (~18MiB/h)
   2-3. Application was working without any clients open. We see constant memory usage.
   3-4. Once the same 100 clients are loaded again, we see a jump in memory consumption ~200MiB and then constant growth (~18MiB/h).
   4-5. Stop and loading of clients after 1-2mins adds 58MiB to memory usage.
   5-6. Stop and loading of clients after 1-2mins adds 36MiB to memory usage.

2. Client requests are sent from different types of threads: for example, in case of web requests these are short-living threads, and there are cases when Client.send requests are sent from the threads in the thread pool (such threads can be reused). Does it make any difference whether Client.send requests are performed from short-living threads or from pooled threads?

[levlam]

1. Memory growth should be slower and slower over time. You can see that on the graph, but this also could be caused by the night time.
   180kb of new data per hour per client isn't a lot for more workloads.
   10MB of data per new instance is also fine if the whole chat list isn't loaded. Otherwise, ot should be less than 1MB.
   Memory growth on client reopening is likely caused by memory fragmentation within native allocator. You need to hook the allocator itself or ask him for statistics to see real memory usage. It should decrease slightly right after close/reopen.

2. Sending requests from short-living threads may affect memory usage, because some thread-local data can be allocated by TDLib for each thread.

[andreymerchenko]

1. We see at point in time 1(on the graph) that those 100 clients are already loaded and occupy some memory, then memory consumption is growing till point in time 2 and at that time those 100 clients are occupying also some memory, lets name that memory as X. I am under impression that when we close those 100 clients, most of those X should be free and when after 1 hour we start the same 100 clients again then should reuse that X memory that is free at that moment of time and that will take some time for them to occupy the same X of already allocated memory for application, however they start occupying X+400MiB right after they are started. If we close them and open they start occupying X+460 if we close and open them - X+500.
   Is that expected behavior?
   As eventually that leads to usage of 100% memory on the device without any possibility to free some memory without restarting the app.
2. Is there any cross-client shared cache that is not being cleaned up when we close those 100 clients?

[levlam]

1. This is correct, but memory must be allocated in continuous segments and very often native allocators can't reuse previously freed memory. In this case they are forced to allocate new memory from OS, but some memory is reused. That's why you see only +400 MiB instead of +1GB. To correctly check used native memory you need to ask the allocator. This can't be measured externally. And yes, memory fragmentation can lead to ever growing memory usage.
2. There is, but it is small and of bounded size. Some objects are instantiated only once per app lifetime and can be shared between instances.