Simulation dies above 4000 ticks when running with 2600 clients, 40 relays, and 200 servers.
When it dies, memory usage is around 30GB and single threaded cpu usage is at 100%. Well over 50% of the cpu usage is system calls.
|t=4389.568|s=0|w=1|shadow|105.221.37.25|1178.proxy.shd| WARNING: vsocket_setsockopt: setsockopt not implemented
|t=4389.570|s=0|w=1|shadow|209.1.219.198|2499.proxy.shd| WARNING: vsocket_setsockopt: setsockopt not implemented
|t=4389.573|s=0|w=1|module|105.221.37.25|1178.proxy.shd| [tor-warn] log_unsafe_socks_warning() Your application (using socks5 to port 8080) is giving Tor only an IP address. Applications that do DNS resolves themselves may leak information. Consider using Socks4A (e.g. via privoxy or socat) instead. For more information, please see https://wiki.torproject.org/TheOnionRouter/TorFAQ#SOCKSAndDNS.
|t=4389.575|s=0|w=1|module|209.1.219.198|2499.proxy.shd| [tor-warn] log_unsafe_socks_warning() Your application (using socks5 to port 8080) is giving Tor only an IP address. Applications that do DNS resolves themselves may leak information. Consider using Socks4A (e.g. via privoxy or socat) instead. For more information, please see https://wiki.torproject.org/TheOnionRouter/TorFAQ#SOCKSAndDNS.
|t=4389.599|s=0|w=1|shadow|113.24.24.95|5.exit.shd| WARNING: vsocket_setsockopt: setsockopt not implemented
|t=4389.609|s=0|w=1|shadow|113.24.24.95|5.exit.shd| WARNING: vsocket_setsockopt: setsockopt not implemented
|t=4389.645|s=0|w=1|shadow|89.179.146.148|15.exit.shd| WARNING: vsocket_setsockopt: setsockopt not implemented
|t=4389.662|s=0|w=1|module|25.185.119.55|829.proxy.shd| MESSAGE: [fg-download-complete] got first bytes in 10.625 seconds and 327680 of 327680 bytes in 21.906 seconds (download 41 of 0)
|t=4389.687|s=0|w=1|shadow|225.57.90.11|303.proxy.shd| WARNING: vsocket_s
#0 0x00007ffff713f295 in mq_timedreceive () from /lib/x86_64-linux-gnu/librt.so.1
#1 0x0000000000424034 in pipecloud_localize_reads (pipecloud=0x652200)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/util/pipecloud.c:205
#2 0x00000000004240b8 in pipecloud_write (pipecloud=0x652200, dest=<value optimized out>, data=0x5a39cde80 "", data_size=211)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/util/pipecloud.c:187
#3 0x000000000040e456 in dvn_packet_route (dest_type=8 '\b', dest_layer=1 '\001', dest_major=0,
frametype=<value optimized out>, frame=<value optimized out>)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/routing.c:88
#4 0x000000000040bde5 in dlog_channel_write (channel=<value optimized out>, data=<value optimized out>,
length=<value optimized out>) at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/log.c:238
#5 0x000000000040c286 in dlogf_main (level=<value optimized out>, context=<value optimized out>, fmt=<value optimized out>,
vargs=<value optimized out>) at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/log.c:368
#6 0x0000000000412ae1 in snricall_log (va=<value optimized out>)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/node/snricall.c:152
#7 0x0000000000413260 in snricall (call_code=<value optimized out>)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/node/snricall.c:366
#8 0x00007ffff62f6123 in service_filegetter_log (sfg=0x7ffff667a518, level=SFG_NOTICE, format=<value optimized out>)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/plug-ins/filetransfer/shd-service-filegetter.c:51
#9 0x00007ffff62f6e62 in service_filegetter_report (sfg=0x7ffff667a518, sockd=30046)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/plug-ins/filetransfer/shd-service-filegetter.c:57
#10 service_filegetter_activate (sfg=0x7ffff667a518, sockd=30046)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/plug-ins/filetransfer/shd-service-filegetter.c:438
#11 0x00007ffff62f2584 in _plugin_socket_readable (sockd=30046)
at /raid/home/cwacek/shadow/shadow-scallion-v1.0.0/src/scallion.c:364
#12 0x0000000000411f99 in context_execute_socket (provider=0x6c58e0, sockd=30046, can_read=1 '\001', can_write=1 '\001',
do_read_first=1 '\001') at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/node/context.c:131
#13 0x00000000004136ec in vepoll_execute_notification (provider=<value optimized out>, vep=0x5d0ea7f90)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/vevent/vepoll.c:180
#14 0x0000000000418719 in vci_exec_event (vci_mgr=0x658b00, vci_event=0x549fa5080)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/vnetwork/vci.c:1016
#15 0x000000000041052c in sim_worker_heartbeat (worker=0x6567f0, num_event_worker_executed=0x7fffffffd410)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/sim_worker.c:395
#16 0x000000000040d21b in dvn_worker_main (process_id=1, total_workers=1, pipecloud=0x652200)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/process.c:119
#17 0x000000000040dd86 in dvn_create_slave (daemon=0, num_processes=1, slave_listen_port=6201, socketset=0x653b20)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/process.c:529
#18 0x000000000040df10 in dvn_create_instance (config=0x7fffffffd510)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/process.c:625
#19 0x000000000040e04f in dvn_main (config=0x7fffffffd510)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/process.c:665
#20 0x000000000040ce41 in main (argc=<value optimized out>, argv=<value optimized out>)
at /raid/home/cwacek/shadow/shadow-shadow-v1.0.0/src/core/shd-main.c:267
Stepping through from that point in the code, it appears to continuously loop on lines 185 - 190 in the function below. I'm not sure if that's proper behavior.
174 size_t pipecloud_write(pipecloud_tp pipecloud, unsigned int dest, char * data, size_t data_size) {
175 struct timespec ts;
176 int rv;
177
178 assert(dest < pipecloud->num_pipes);
179
180 ts.tv_sec = PIPECLOUD_TIMEOUT_SEC;
181 ts.tv_nsec = PIPECLOUD_TIMEOUT_NSEC;
182
183 rv = mq_timedsend(pipecloud->mqs[dest], data, data_size, 0, &ts);
184
185 while(rv != 0) {
186 /* first, try to pull in any waiting writes that we might have to avoid deadlocks */
187 pipecloud_localize_reads(pipecloud);
188
189 /* then, try to send again. */
190 rv = mq_timedsend(pipecloud->mqs[dest], data, data_size, 0, &ts);
191 }
192
193 return data_size;
194 }