The server crash after some minutes of gaming session about palworld-server-docker

[MSpreckels]

I have also seen the same situation going on when using not enough ram / too little ram. I've read on xgamingserver that 16 gigs of RAM are recommended but it'll work with less.

I'm operating my server with 20 gigs and am restarting it every day due to lag build up. I suspect that there is, as @thijsvanloef said, a memory leak.

[thijsvanloef]

I think this might be a memory leak of the Palworld Dedicated server, especially because there is only 6GB of ram available for the server:

2024-01-24 09:56:27 [2024.01.24-08.56.25:421][ 0]LogInit: - Physical RAM available (not considering process quota): 6 GB (5930 MB, 6073144 KB, 6218899456 bytes)

[softshelled]

I also met the same problem
Shutdown handler: initalize.

• Existing per-process limit (soft=65535, hard=65535) is enough for us (need only 65535)
  Increasing per-process limit of core file size to infinity.
  dlopen failed trying to load:
  steamclient.so
  with error:
  steamclient.so: cannot open shared object file: No such file or directory
  [S_API] SteamAPI_Init(): Loaded '/home/steam/.steam/sdk64/steamclient.so' OK. (First tried local 'steamclient.so')
  CAppInfoCacheReadFromDiskThread took 11 milliseconds to initialize
  dlmopen steamservice.so failed: steamservice.so: cannot open shared object file: No such file or directory
  Setting breakpad minidump AppID = 2394010
  [S_API FAIL] Tried to access Steam interface SteamUser021 before SteamAPI_Init succeeded.
  [S_API FAIL] Tried to access Steam interface SteamFriends017 before SteamAPI_Init succeeded.
  [S_API FAIL] Tried to access Steam interface STEAMAPPS_INTERFACE_VERSION008 before SteamAPI_Init succeeded.
  [S_API FAIL] Tried to access Steam interface SteamNetworkingUtils004 before SteamAPI_Init succeeded.
  LowLevelFatalError [File:D:/works/repos/Pal_SVN/Pal-UE-App/Source/Pal/PalSaveGameManager.cpp] [Line: 359]
  Failed save to backup.
  Signal 11 caught.
  Malloc Size=262146 LargeMemoryPoolOffset=262162
  CommonUnixCrashHandler: Signal=11
  Malloc Size=131160 LargeMemoryPoolOffset=393352
  Malloc Size=131160 LargeMemoryPoolOffset=524536
  LogPakFile: Display: Found Pak file ../../../Engine/Programs/CrashReportClient/Content/Paks/CrashReportClient.pak attempting to mount.
  LogPakFile: Display: Mounting pak file ../../../Engine/Programs/CrashReportClient/Content/Paks/CrashReportClient.pak.
  LogPakFile: Display: Mounted Pak file '../../../Engine/Programs/CrashReportClient/Content/Paks/CrashReportClient.pak', mount point: '../../../Engine/'
  LogICUInternationalization: ICU TimeZone Detection - Raw Offset: +8:00, Platform Override: ''
  LogInit: Build: ++UE5+Release-5.1-CL-0
  LogInit: Engine Version: 5.1.1-0+++UE5+Release-5.1
  LogInit: Compatible Engine Version: 5.1.0-0+++UE5+Release-5.1
  LogInit: Net CL: 0
  LogInit: OS: Ubuntu 22.04.3 LTS (5.15.0-91-generic), CPU: Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz, GPU: UnknownVendor PCI-id: 1013-00b8 (1af4-1100)
  LogInit: Compiled (64-bit): Dec 31 2023 20:12:22
  LogInit: Compiled with Clang: 13.0.1 (https://github.com/llvm/llvm-project 75e33f71c2dae584b13a7d1186ae0a038ba98838)
  LogInit: Build Configuration: Shipping
  LogInit: Branch Name: ++UE5+Release-5.1
  LogInit: Command Line: -Abslog="/home/steam/Steam/steamapps/common/PalServer/Pal/Saved/Logs/Pal-CRC.log" -Unattended -ImplicitSend "/home/steam/Steam/steamapps/common/PalServer/Pal/Saved/Crashes/crashinfo-Pal-pid-1433-A4C4C38C7BE34157B1C7CAB5D13DF9A7/" -unattended
  LogInit: Base Directory: /home/steam/Steam/steamapps/common/PalServer/Engine/Binaries/Linux/
  LogInit: Allocator: Mimalloc
  LogInit: Installed Engine Build: 1
  LogInit: Presizing for max 100000 objects, including 0 objects not considered by GC, pre-allocating 0 bytes for permanent pool.
  LogInit: Object subsystem initialized
  [2024.01.24-11.11.46:297][ 0]LogConfig: Applying CVar settings from Section [ConsoleVariables] File [Engine]
  [2024.01.24-11.11.46:297][ 0]LogInit: Unix hardware info:
  [2024.01.24-11.11.46:297][ 0]LogInit: - we are the first instance of this executable
  [2024.01.24-11.11.46:297][ 0]LogInit: - this process' id (pid) is 1524, parent process' id (ppid) is 1433
  [2024.01.24-11.11.46:297][ 0]LogInit: - we are not running under debugger
  [2024.01.24-11.11.46:297][ 0]LogInit: - machine network name is 'iZbp10y7ycczxqa3rvg5gqZ'
  [2024.01.24-11.11.46:297][ 0]LogInit: - user name is 'steam' (steam)
  [2024.01.24-11.11.46:297][ 0]LogInit: - we're logged in remotely
  [2024.01.24-11.11.46:297][ 0]LogInit: - we're running with rendering
  [2024.01.24-11.11.46:297][ 0]LogInit: - CPU: GenuineIntel 'Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz' (signature: 0x50657)
  [2024.01.24-11.11.46:297][ 0]LogInit: - Number of physical cores available for the process: 2
  [2024.01.24-11.11.46:297][ 0]LogInit: - Number of logical cores available for the process: 4
  [2024.01.24-11.11.46:297][ 0]LogInit: - Cache line size: 64
  [2024.01.24-11.11.46:297][ 0]LogInit: - GPU Brand Info: UnknownVendor PCI-id: 1013-00b8 (1af4-1100)
  [2024.01.24-11.11.46:297][ 0]LogInit: - Memory allocator used: Mimalloc
  [2024.01.24-11.11.46:297][ 0]LogInit: - This binary is optimized with LTO: no, PGO: no, instrumented for PGO data collection: no
  [2024.01.24-11.11.46:297][ 0]LogInit: - This is an internal build.
  [2024.01.24-11.11.46:297][ 0]LogCore: Skipped benchmarking clocks because the engine is running in a standalone program mode - CLOCK_MONOTONIC will be used.
  [2024.01.24-11.11.46:297][ 0]LogInit: Unix-specific commandline switches:
  [2024.01.24-11.11.46:297][ 0]LogInit: -ansimalloc - use malloc()/free() from libc (useful for tools like valgrind and electric fence)
  [2024.01.24-11.11.46:297][ 0]LogInit: -jemalloc - use jemalloc for all memory allocation
  [2024.01.24-11.11.46:297][ 0]LogInit: -binnedmalloc - use binned malloc for all memory allocation
  [2024.01.24-11.11.46:297][ 0]LogInit: -filemapcachesize=NUMBER - set the size for case-sensitive file mapping cache
  [2024.01.24-11.11.46:297][ 0]LogInit: -useksm - uses kernel same-page mapping (KSM) for mapped memory (OFF)
  [2024.01.24-11.11.46:297][ 0]LogInit: -ksmmergeall - marks all mmap'd memory pages suitable for KSM (OFF)
  [2024.01.24-11.11.46:297][ 0]LogInit: -preloadmodulesymbols - Loads the main module symbols file into memory (OFF)
  [2024.01.24-11.11.46:297][ 0]LogInit: -sigdfl=SIGNAL - Allows a specific signal to be set to its default handler rather then ignoring the signal
  [2024.01.24-11.11.46:297][ 0]LogInit: -crashhandlerstacksize - Allows setting crash handler stack sizes (204800)
  [2024.01.24-11.11.46:297][ 0]LogInit: -noexclusivelockonwrite - disables marking files created by the engine as exclusive locked while the engine has them opened
  [2024.01.24-11.11.46:297][ 0]LogInit: -httpproxy=ADDRESS:PORT - redirects HTTP requests to a proxy (only supported if compiled with libcurl)
  [2024.01.24-11.11.46:297][ 0]LogInit: -reuseconn - allow libcurl to reuse HTTP connections (only matters if compiled with libcurl)
  [2024.01.24-11.11.46:297][ 0]LogInit: -virtmemkb=NUMBER - sets process virtual memory (address space) limit (overrides VirtualMemoryLimitInKB value from .ini)
  [2024.01.24-11.11.46:297][ 0]LogInit: - Physical RAM available (not considering process quota): 31 GB (31188 MB, 31936860 KB, 32703344640 bytes)
  [2024.01.24-11.11.46:297][ 0]LogInit: - VirtualMemoryAllocator pools will grow at scale 1.4
  [2024.01.24-11.11.46:297][ 0]LogInit: - MemoryRangeDecommit() will be a no-op (re-run with -vmapoolevict to change)
  [2024.01.24-11.11.46:297][ 0]LogInit: - PageSize 4096
  [2024.01.24-11.11.46:297][ 0]LogInit: - BinnedPageSize 65536
  [2024.01.24-11.11.46:300][ 0]LogUObjectArray: 419 objects as part of root set at end of initial load.
  [2024.01.24-11.11.46:300][ 0]LogUObjectAllocator: 89064 out of 0 bytes used by permanent object pool.
  [2024.01.24-11.11.46:300][ 0]LogUObjectArray: CloseDisregardForGC: 0/0 objects in disregard for GC pool
  [2024.01.24-11.11.46:300][ 0]LogPaths: Warning: No paths for game localization data were specifed in the game configuration.
  [2024.01.24-11.11.46:300][ 0]LogInit: Using OS detected language (en-US).
  [2024.01.24-11.11.46:300][ 0]LogInit: Using OS detected locale (en-US).
  [2024.01.24-11.11.46:300][ 0]LogInit: Warning: No paths for engine localization data were specifed in the engine configuration.
  [2024.01.24-11.11.46:301][ 0]LogTextLocalizationManager: No localization for 'en-US' exists, so 'en' will be used for the language.
  [2024.01.24-11.11.46:301][ 0]LogTextLocalizationManager: No localization for 'en-US' exists, so 'en' will be used for the locale.
  [2024.01.24-11.11.46:301][ 0]LogInit: Using OS detected language (en-US).
  [2024.01.24-11.11.46:301][ 0]LogInit: Using OS detected locale (en-US).
  [2024.01.24-11.11.46:301][ 0]LogTextLocalizationManager: No localization for 'en-US' exists, so 'en' will be used for the language.
  [2024.01.24-11.11.46:301][ 0]LogTextLocalizationManager: No localization for 'en-US' exists, so 'en' will be used for the locale.
  [2024.01.24-11.11.46:301][ 0]LogPackageLocalizationCache: Processed 2 localized package path(s) for 1 prioritized culture(s) in 0.000368 seconds
  [2024.01.24-11.11.46:301][ 0]CrashReportCoreLog: CrashReportClientVersion=1.0
  [2024.01.24-11.11.46:301][ 0]CrashReportCoreLog: CrashReportReceiver disabled
  [2024.01.24-11.11.46:301][ 0]CrashReportCoreLog: DataRouterUrl: https://o1291919.ingest.sentry.io/api/6513339/unreal/4a1a3921f51f4975b4cf8dd19022cb20/
  [2024.01.24-11.11.46:302][ 0]CrashReportCoreLog: Initial state = Unknown UploadState value
  [2024.01.24-11.11.46:302][ 0]CrashReportCoreLog: Initial state = Unknown UploadState value
  [2024.01.24-11.11.46:302][ 0]LogCrashDebugHelper: DepotName: //UE5/Release-5.1
  [2024.01.24-11.11.46:302][ 0]LogCrashDebugHelper: BuiltFromCL: 0
  [2024.01.24-11.11.46:302][ 0]LogCrashDebugHelper: EngineVersion: 5.1.1-0+++UE5+Release-5.1
  [2024.01.24-11.11.46:302][ 0]LogCrashDebugHelper: BuildVersion: ++UE5+Release-5.1-CL-0
  [2024.01.24-11.11.47:272][ 0]CrashReportCoreLog: Got 3 pending files to upload from 'crashinfo-Pal-pid-1433-A4C4C38C7BE34157B1C7CAB5D13DF9A7'
  [2024.01.24-11.11.47:272][ 0]CrashReportCoreLog: State change from Ready to SendingFiles
  [2024.01.24-11.11.47:272][ 0]CrashReportCoreLog: CompressAndSendData have 3 pending files
  [2024.01.24-11.11.47:272][ 0]CrashReportCoreLog: CompressAndSendData compressing 9946 bytes ('/home/steam/Steam/steamapps/common/PalServer/Pal/Saved/Crashes/crashinfo-Pal-pid-1433-A4C4C38C7BE34157B1C7CAB5D13DF9A7/CrashContext.runtime-xml')
  [2024.01.24-11.11.47:272][ 0]CrashReportCoreLog: CompressAndSendData compressing 2572 bytes ('/home/steam/Steam/steamapps/common/PalServer/Pal/Saved/Crashes/crashinfo-Pal-pid-1433-A4C4C38C7BE34157B1C7CAB5D13DF9A7/Diagnostics.txt')
  [2024.01.24-11.11.47:272][ 0]CrashReportCoreLog: CompressAndSendData compressing 152 bytes ('/home/steam/Steam/steamapps/common/PalServer/Pal/Saved/Crashes/crashinfo-Pal-pid-1433-A4C4C38C7BE34157B1C7CAB5D13DF9A7/CrashReportClient.ini')
  [2024.01.24-11.11.47:280][ 0]LogInit: Using libcurl 7.83.1
  [2024.01.24-11.11.47:280][ 0]LogInit: - built for Linux
  [2024.01.24-11.11.47:280][ 0]LogInit: - supports SSL with OpenSSL/1.1.1n
  [2024.01.24-11.11.47:280][ 0]LogInit: - supports HTTP deflate (compression) using libz 1.2.12
  [2024.01.24-11.11.47:280][ 0]LogInit: - other features:
  [2024.01.24-11.11.47:280][ 0]LogInit: CURL_VERSION_SSL
  [2024.01.24-11.11.47:280][ 0]LogInit: CURL_VERSION_LIBZ
  [2024.01.24-11.11.47:280][ 0]LogInit: CURL_VERSION_IPV6
  [2024.01.24-11.11.47:280][ 0]LogInit: CURL_VERSION_ASYNCHDNS
  [2024.01.24-11.11.47:280][ 0]LogInit: CURL_VERSION_LARGEFILE
  [2024.01.24-11.11.47:281][ 0]LogInit: CurlRequestOptions (configurable via config and command line):
  [2024.01.24-11.11.47:281][ 0]LogInit: - bVerifyPeer = false - Libcurl will NOT verify peer certificate
  [2024.01.24-11.11.47:281][ 0]LogInit: - bUseHttpProxy = false - Libcurl will NOT use HTTP proxy
  [2024.01.24-11.11.47:281][ 0]LogInit: - bDontReuseConnections = false - Libcurl will reuse connections
  [2024.01.24-11.11.47:281][ 0]LogInit: - MaxHostConnections = 16 - Libcurl will limit the number of connections to a host
  [2024.01.24-11.11.47:281][ 0]LogInit: - LocalHostAddr = Default
  [2024.01.24-11.11.47:281][ 0]LogInit: - BufferSize = 65536
  [2024.01.24-11.11.47:281][ 0]CrashReportCoreLog: Sending HTTP request: https://o1291919.ingest.sentry.io/api/6513339/unreal/4a1a3921f51f4975b4cf8dd19022cb20/?AppID=CrashReporter&AppVersion=5.1.1-0%2B%2B%2BUE5%2BRelease-5.1&AppEnvironment=Release&UploadType=crashreports&UserID=323e332042e543d9857f8b1da9178588-000003e8%7C%7C323e332042e543d9857f8b1da9178588
  [2024.01.24-11.11.58:892][ 0]LogHttp: Warning: 0xacaa37d5f00: invalid HTTP response code received. URL: https://o1291919.ingest.sentry.io/api/6513339/unreal/4a1a3921f51f4975b4cf8dd19022cb20/?AppID=CrashReporter&AppVersion=5.1.1-0%2B%2B%2BUE5%2BRelease-5.1&AppEnvironment=Release&UploadType=crashreports&UserID=323e332042e543d9857f8b1da9178588-000003e8%7C%7C323e332042e543d9857f8b1da9178588, HTTP code: 0, content length: 0, actual payload size: 0
  [2024.01.24-11.11.58:892][ 0]LogHttp: Warning: 0xacaa37d5f00: request failed, libcurl error: 35 (SSL connect error)
  [2024.01.24-11.11.58:892][ 0]LogHttp: Warning: 0xacaa37d5f00: libcurl info message cache 0 ( Trying 34.120.195.249:443...)
  [2024.01.24-11.11.58:892][ 0]LogHttp: Warning: 0xacaa37d5f00: libcurl info message cache 1 (Connected to o1291919.ingest.sentry.io (34.120.195.249) port 443 (#0))
  [2024.01.24-11.11.58:892][ 0]LogHttp: Warning: 0xacaa37d5f00: libcurl info message cache 2 (ALPN: offers http/1.1)
  [2024.01.24-11.11.58:892][ 0]LogHttp: Warning: 0xacaa37d5f00: libcurl info message cache 3 (TLSv1.3 (OUT), TLS handshake, Client hello (1):)
  [2024.01.24-11.11.58:892][ 0]LogHttp: Warning: 0xacaa37d5f00: libcurl info message cache 4 (OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to o1291919.ingest.sentry.io:443 )
  [2024.01.24-11.11.58:892][ 0]LogHttp: Warning: 0xacaa37d5f00: libcurl info message cache 5 (Closing connection 0)
  [2024.01.24-11.11.58:892][ 0]CrashReportCoreLog: OnProcessRequestComplete(), State=SendingFiles bSucceeded=0
  [2024.01.24-11.11.58:892][ 0]CrashReportCoreLog: Warning: File upload failed to data router
  [2024.01.24-11.11.58:892][ 0]CrashReportCoreLog: State change from SendingFiles to Cancelled
  [2024.01.24-11.11.59:326][ 0]CrashReportCoreLog: Final state (Receiver) = Cancelled
  [2024.01.24-11.11.59:326][ 0]CrashReportCoreLog: Final state (Receiver) = Unknown UploadState value
  [2024.01.24-11.11.59:326][ 0]LogCore: Engine exit requested (reason: CrashReportClientApp RequestExit)
  [2024.01.24-11.11.59:326][ 0]LogExit: Preparing to exit.
  [2024.01.24-11.11.59:327][ 0]LogExit: Object subsystem successfully closed.
  [2024.01.24-11.11.59:327][ 0]LogModuleManager: Shutting down and abandoning module HTTP (12)
  [2024.01.24-11.11.59:343][ 0]LogModuleManager: Shutting down and abandoning module SSL (11)
  [2024.01.24-11.11.59:343][ 0]LogModuleManager: Shutting down and abandoning module CrashDebugHelper (8)
  [2024.01.24-11.11.59:343][ 0]LogModuleManager: Shutting down and abandoning module CoreUObject (6)
  [2024.01.24-11.11.59:343][ 0]LogModuleManager: Shutting down and abandoning module PakFile (4)
  [2024.01.24-11.11.59:343][ 0]LogModuleManager: Shutting down and abandoning module RSA (3)
  [2024.01.24-11.11.59:343][ 0]LogExit: Exiting.
  Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
  Segmentation fault (core dumped)

[TombSte]

I think this might be a memory leak of the Palworld Dedicated server, especially because there is only 6GB of ram available for the server:

2024-01-24 09:56:27 [2024.01.24-08.56.25:421][ 0]LogInit: - Physical RAM available (not considering process quota): 6 GB (5930 MB, 6073144 KB, 6218899456 bytes)

So we have to wait a fix of palword developers, right?

[thijsvanloef]

@TombSte Sadly yes