worker keep crashing about f-stack

[whl739]

I tested it on ubuntu 14.04, it worked fine. And what's your platform?
Could you gdb the coredump file and find out where it crashed on?

[pkking]

My env is also ubuntu 14.04.05:

# cat /etc/os-release 
NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.5 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

gdb output:

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: master process /usr/local/ng'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000005ee6d8 in eth_virtio_dev_init ()
(gdb) bt
#0  0x00000000005ee6d8 in eth_virtio_dev_init ()
#1  0x00000000005c47c4 in rte_eth_dev_init ()
#2  0x00000000005d5c54 in pci_probe_all_drivers ()
#3  0x00000000005d6138 in rte_eal_pci_probe ()
#4  0x00000000005cac16 in rte_eal_init ()
#5  0x0000000000511205 in ff_dpdk_init ()
#6  0x000000000051616d in ff_init ()
#7  0x000000000044d827 in ff_mod_init (conf=0x9b497c "/usr/local/nginx_fstack/conf/f-stack.conf", proc_id=proc_id@entry=0, proc_type=<optimized out>, 
    proc_type@entry=0) at src/event/modules/ngx_ff_module.c:150
#8  0x000000000044a899 in ngx_worker_process_init (cycle=cycle@entry=0x9b1fb0, worker=0) at src/os/unix/ngx_process_cycle.c:1118
#9  0x000000000044aa2f in ngx_worker_process_cycle (cycle=0x9b1fb0, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:949
#10 0x0000000000449894 in ngx_spawn_process (cycle=cycle@entry=0x9b1fb0, proc=0x44aa14 <ngx_worker_process_cycle>, data=0x0, name=0x5f6a4f "worker process", 
    respawn=respawn@entry=1) at src/os/unix/ngx_process.c:198
#11 0x000000000044b62c in ngx_reap_children (cycle=0x9b1fb0) at src/os/unix/ngx_process_cycle.c:785
#12 ngx_master_process_cycle (cycle=0x9b1fb0) at src/os/unix/ngx_process_cycle.c:338
#13 0x000000000042b0ed in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:381

[whl739]

It seemed that it crashed on virto initialization. Does the example/helloworld work fine?
Try ./start.sh.

[pkking]

The process not exit when i lynx 192.167.167.14
Respone is 200 OK, but i can't see any output in my screen

# ./start.sh 
./example/helloworld --conf config.ini --proc-type=primary --proc-id=0 
[dpdk]: lcore_mask=1
[dpdk]: channel=4
[dpdk]: nb_ports=1
[dpdk]: promiscuous=1
[dpdk]: numa_on=0
[dpdk]: tso=0
[dpdk]: vlan_strip=1
[port0]: addr=192.167.167.14
[port0]: netmask=255.255.255.0
[port0]: broadcast=192.167.167.255
[port0]: gateway=192.167.167.254
[freebsd.boot]: hz=100
[freebsd.boot]: fd_reserve=1024
[freebsd.boot]: kern.ipc.maxsockets=262144
[freebsd.boot]: net.inet.tcp.syncache.hashsize=4096
[freebsd.boot]: net.inet.tcp.syncache.bucketlimit=100
[freebsd.boot]: net.inet.tcp.tcbhashsize=65536
[freebsd.sysctl]: kern.ipc.somaxconn=32768
[freebsd.sysctl]: kern.ipc.maxsockbuf=16777216
[freebsd.sysctl]: net.inet.tcp.fast_finwait2_recycle=1
[freebsd.sysctl]: net.inet.tcp.sendspace=16384
[freebsd.sysctl]: net.inet.tcp.recvspace=8192
[freebsd.sysctl]: net.inet.tcp.nolocaltimewait=1
[freebsd.sysctl]: net.inet.tcp.cc.algorithm=htcp
[freebsd.sysctl]: net.inet.tcp.sendbuf_max=16777216
[freebsd.sysctl]: net.inet.tcp.recvbuf_max=16777216
[freebsd.sysctl]: net.inet.tcp.sendbuf_auto=1
[freebsd.sysctl]: net.inet.tcp.recvbuf_auto=1
[freebsd.sysctl]: net.inet.tcp.sendbuf_inc=16384
[freebsd.sysctl]: net.inet.tcp.recvbuf_inc=524288
[freebsd.sysctl]: net.inet.tcp.inflight.enable=0
[freebsd.sysctl]: net.inet.tcp.sack=1
[freebsd.sysctl]: net.inet.tcp.blackhole=1
[freebsd.sysctl]: net.inet.tcp.msl=2000
[freebsd.sysctl]: net.inet.tcp.delayed_ack=0
[freebsd.sysctl]: net.inet.udp.blackhole=1
[freebsd.sysctl]: net.inet.ip.redirect=0
EAL: Detected 8 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: PCI device 0000:00:03.0 on NUMA socket 0
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL: PCI device 0000:00:06.0 on NUMA socket 0
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
create mbuf pool on socket 0
create ring:arp_ring_0_0 success, 2047 ring entries are now free!
Port 0 MAC: fa 16 3e 8c 61 31
TSO is disabled
set port 0 to promiscuous mode ok

Checking link statusdone
Port 0 Link Up - speed 10000 Mbps - full-duplex
link_elf_lookup_symbol: missing symbol hash table
link_elf_lookup_symbol: missing symbol hash table
netisr_init: forcing maxthreads from 1 to 0
Timecounters tick every 10.000 msec
Timecounter "ff_clock" frequency 100 Hz quality 1
f-stack-0: Ethernet address: fa:16:3e:8c:61:31
sockfd:1024

[pkking]

Well, links 192.167.167.14 shows the welcome html page.

[whl739]

The example works fine, there maybe something wrong with nginx_fstack.
Did you test it on a virtual machine? Because i saw virtio pmd.
I tested on a physical machine but not a virtual machine.

[pkking]

Yes, on an openstack vm.

[whl739]

There's really a bug of nginx_fstack on virtio VMs.
When the secondary starts up, virtio_hw->vtpci_ops is not initialized successfully.
I'll try to fix it.

[whl739]

Besides, kni couldn't be enabled in this version, because kni is only processed by primary and worker ff_primary won't execute ff_run().

[whl739]

http://dpdk.org/dev/patchwork/patch/20686/
This shows that there's an issuse on virtio with multi-process.
But the odd thing is, example/helloworld works fine.